Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

NavBench: Probing Multimodal Large Language Models for Embodied Navigation

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read NavBench probes whether multimodal LLMs can actually navigate, scoring comprehension and execution in one benchmark.

desk verdict A useful diagnostic benchmark for MLLM navigation, with a real temporal-reasoning finding, but the headline correlation and difficulty weights are under-supported as written. read the letter →

arxiv 2506.01031 v1 pith:A6LWBT7L submitted 2025-06-01 cs.CV

Yanyuan Qiao , Haodong Hong , Wenqi Lyu , Dong An , Siqi Zhang , Yutong Xie , Xinyu Wang , Qi Wu This is my paper · ORCID
classification cs.CV
keywords multimodallargelanguagemodelsembodiednavigationzero-shotevaluationvision-and-languagetemporalprogressestimationcomprehensionbenchmarkdesignroboticdeployment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces NavBench, a zero-shot benchmark that asks whether multimodal large language models can understand navigation behavior and act on it, rather than merely answering static visual questions. It builds 3,200 comprehension questions at three levels—global instruction alignment, temporal progress estimation, and local observation-action reasoning—plus 432 step-by-step execution episodes in 72 Matterport3D scenes stratified by difficulty. The central claim is that these two measurements are linked: models that understand navigation instructions better also execute them better, and temporal progress estimation is a persistent bottleneck across current models. A reader should care because the benchmark offers a diagnostic lens that standard success-rate metrics miss, and a deployment pipeline that transfers viewpoint decisions to a real robot.

What carries the argument

The central object is NavBench itself, a two-part benchmark: three multiple-choice comprehension tasks (global instruction alignment, temporal progress estimation, and local observation-action reasoning) built from instruction-trajectory pairs rendered in the Matterport3D simulator, plus 432 step-by-step execution episodes stratified into easy, medium, and hard by a composite score of spatial, cognitive, and execution complexity. The machinery that carries the argument is the paired design: the same underlying instruction-trajectory data is reused to generate both comprehension questions and execution episodes, so a model's ability to understand a navigation behavior can be directly compared with its ability to act on it, and the difficulty stratification lets the benchmark separate skills that appear only in easy or medium cases.

What would settle it

If models given the same comprehension questions with the panoramic views removed, leaving only instruction text and answer options, scored near the multimodal models' accuracy, the benchmark would be measuring language priors rather than embodied visual understanding; this text-only probe is a concrete way to falsify the claim.

Watch

Extended reading notes

Core claim

NavBench establishes that the embodied navigation ability of current multimodal large language models can be decomposed into comprehension and execution, and that on this decomposition, comprehension and execution track each other across models. The paper finds that GPT-4o leads on both axes, that lightweight open-source models such as Qwen2.5-VL-7B can reliably complete easy episodes, and that temporal progress estimation—knowing how far along a multi-step instruction a partial trajectory has reached—is the weakest skill for nearly every model. The benchmark also shows that adding map-based topological context improves GPT-4o's success rate, most strongly on medium-difficulty episodes, and that a real-world pilot with the top proprietary and open-source models mirrors the simulation ranking.

Load-bearing premise

The load-bearing premise is that correct answers on the three multiple-choice comprehension tasks reflect genuine embodied navigation understanding rather than surface regularities in the text or images; if models solve them by language priors or visual matching, the benchmark's conclusions about comprehension and execution no longer follow.

Editorial extensions

If this is right

  • Models that score higher on comprehension tend to score higher on execution, so comprehension scores could serve as a proxy signal for embodied ability in zero-shot navigation settings.
  • Temporal progress estimation is a consistent weakness, and error analysis links it to 'failure to stop' mistakes, so improving progress tracking should improve overall navigation execution.
  • Providing map-based topological context improves decision accuracy, with the largest gain on medium-difficulty episodes, indicating that structured spatial context helps where spatial ambiguity is highest.
  • Lightweight open-source models such as Qwen2.5-VL-7B can complete easy navigation episodes, suggesting resource-efficient models are viable for simple real-world deployments.
  • The real-world pilot, with 60% and 40% success for the top proprietary and open-source models, suggests NavBench's simulation results transfer to physical robot settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One inference the paper leaves implicit is that the comprehension tasks, especially temporal progress estimation, could be used as a cheap screening test for embodied reasoning before running full execution episodes.
  • If the comprehension-execution correlation reflects a causal link, then fine-tuning models on sub-instruction alignment or progress-tracking data could improve their downstream navigation success, a testable extension the paper does not run.
  • The benchmark's diagnostic value depends on the multiple-choice questions resisting surface solutions; a natural extension would be to counterbalance distractors for length, lexical overlap, and visual similarity, or to measure human agreement on item validity, to confirm the tasks probe genuine spatial understanding.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. NavBench is a benchmark for zero-shot evaluation of multimodal large language models (MLLMs) on embodied navigation, with two components: a navigation-comprehension set of 3,200 multiple-choice QA pairs (1,200 global instruction alignment, 1,000 temporal progress estimation, and 1,000 local observation-action reasoning) and a navigation-execution set of 432 episodes across 72 Matterport3D scenes stratified into easy, medium, and hard difficulty. The paper evaluates GPT-4o, GPT-4o-mini, and six open-source MLLMs, reports human performance on a small 'VLN-Bench (tiny)' subset, and additionally describes a real-world deployment pipeline with a 10-case pilot study. The headline findings are that GPT-4o performs best overall, Qwen2.5-VL-7B is the best open-source model, comprehension and execution abilities are closely linked, temporal progress estimation is a persistent weakness, and providing map-based context improves execution accuracy, especially at medium difficulty.

Significance. If the benchmark's construct validity is established, NavBench would be a useful diagnostic tool for evaluating MLLMs in embodied navigation, going beyond end-to-end success rates by separating comprehension from execution and by stratifying difficulty along multiple dimensions. The paper's strengths include the substantial scale of the comprehension set, the combination of simulation-based execution with a real-world pilot, the inclusion of a human reference subset, and the evaluation of both proprietary and open-source models. The diagnostic conclusions, however, depend on statistical and methodological details that are currently missing, so the significance of the findings is conditional on those details being supplied.

major comments (4)
  1. [§5.2, Table 1] The claim that comprehension and execution abilities are 'closely linked' (Abstract and Section 5.2) is supported only by a visual rank-order trend across eight models, with no significance test or confidence intervals. A Spearman rank correlation computed from Table 1 is approximately 0.62, which is not significant at conventional levels for n = 8, and there are visible inversions (e.g., InternVL2.5-8B has higher comprehension than InternVL2.5-2B but lower execution; LLaVA-OneVision-7B has lower comprehension than Qwen2.5-VL-3B but higher execution). Because this correlation is also the only indirect evidence offered for the construct validity of the comprehension tasks, the headline diagnostic conclusion should be re-supported with a permutation or partial-correlation check, per-model repeated trials, and explicit confidence intervals.
  2. [§3.2.2, Eqs. (2)-(4)] The difficulty stratification is load-bearing for the execution analysis, but the weights α, β, γ in Equations (2)-(4) are described only as 'empirically set' and their numerical values are never reported. The normalization in Equation (5) and the easy/medium/hard thresholds are therefore not reproducible. The human evaluation in the same section is described in one paragraph without the number of annotators, inter-annotator agreement (e.g., Krippendorff's alpha or Cohen's kappa), or a comparison between human ratings and the automatic scores; without these, the reported difficulty-level results in Section 5.2 cannot be checked.
  3. [§3.2.1] The Progress Estimation task is filtered using a 'curated list of valid instruction-path pairs,' but the curation rule is not specified, so it is unclear whether this step removes ambiguous items in a way that changes the task's difficulty or introduces selection bias. The Local Observation-Action distractors are said to be 'carefully sampled from nearby observations or visually similar actions,' but no protocol or example of the distractor distribution is given. This matters because the three comprehension tasks have different chance levels (Table 1: Global 19.33, Progress 25.4, Local 29.34), so raw accuracy gaps—specifically the claimed 'persistent weakness' in temporal progress estimation—could reflect candidate-set size or task difficulty rather than a specifically temporal deficit.
  4. [§5.2, §3.2.1] The benchmark's diagnostic interpretation assumes that multiple-choice performance measures embodied navigation understanding rather than surface regularities. The paper offers no controlled test of this assumption: no text-only baselines, no counterbalanced distractor-difficulty checks, and no per-item human-model agreement analysis. Adding such probes, or at least reporting per-item error bars and a baseline that sees the instruction text without the panoramic views, would substantially strengthen the claim that the comprehension-execution correlation supports an embodied-cognition interpretation.
minor comments (6)
  1. [Table 1] The model name 'LLaV A-OneVision-7B' contains an unintended space, and the block labeled 'VLN-Bench (tiny) Performance' mixes human performance with model results on the tiny subset; this layout is confusing and should be relabeled.
  2. [Figures 6 and 7] The accuracy differences in Figures 6 and 7 are presented without error bars or significance tests, so statements such as 'most models handle the basic condition well' are not quantitatively supported.
  3. [§4] The waypoint predictor is described as 'RES-50' (probably ResNet-50), but no architecture details, training data, or accuracy of the predictor are reported; the real-world pilot reports success rates for only 10 cases per model, so it should be described as a feasibility demonstration rather than a validated deployment result.
  4. [General] The paper does not state where the benchmark data and evaluation code will be released; a data/code availability statement is important for a benchmark paper of this kind.
  5. [§5.1] The human reference subset is collected from 'volunteer students from our research institute,' which may not represent a general population; at minimum, the recruitment and instruction procedures should be described, and the response counting should be clarified for the 72 execution episodes.
  6. [§3.2.1] The label 'cognitively grounded tasks' cites the spatial semantic hierarchy (Kuipers) but does not connect the three tasks to specific cognitive constructs or validate them against behavioral data beyond a small human accuracy number.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: NavBench is an evaluation benchmark whose correctness labels come from external navigation annotations and simulator rendering, not from the models being tested.

full rationale

I walked the claimed derivation chain: the comprehension QA pairs are constructed from instruction–trajectory annotations in R2R, RxR, GEL-R2R, and FGR2R, with visual observations rendered from the Matterport3D simulator; the correct answers are fixed by the original trajectory/instruction ground truth, not by any MLLM output. The execution episodes are also defined by simulator states and measured with SR/SPL, and the difficulty stratification weights affect only the binning of episodes into easy/medium/hard, not the correctness of any benchmark item. The abstract's claim that comprehension and execution are closely related is an empirical rank-order observation from Table 1, not a quantity fitted from model scores and then reported as a prediction; its weakness as a statistical inference is a robustness/validity concern, not circularity. Self-citations appear as dataset sources (R2R and FGR2R involve the authors) and as prior-work context, but those datasets are externally published, independently created resources whose annotations determine the benchmark answers; no load-bearing step reduces to the present paper's own claims or to the evaluated models. No uniqueness theorem, ansatz-via-citation, or known-result renaming occurs. Hence the paper is self-contained as an evaluation study, with no circular step to report.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

Everything the central claim rests on that the reader did not pay for upstream: the complexity weights are undisclosed free parameters; the validity of MCQ as a probe for embodied understanding and the reliability of human difficulty ratings are domain assumptions; the curated filtering of Progress Estimation examples is an ad hoc choice that could bias results.

free parameters (2)
  • Complexity weights alpha, beta, gamma = Not reported
    The spatial, cognitive, and execution complexity scores (Equations 2-4) rely on coefficients alpha_i, beta_i, gamma_i that determine difficulty categorization. Values are not disclosed, so the easy/medium/hard split is not reproducible.
  • Difficulty thresholds = 1-3 easy, 4-6 medium, 7-9 hard
    The mapping from normalized scores (range 1-9) to three difficulty levels is a hand-chosen quantization that drives all difficulty-stratified conclusions.
assumptions (4)
  • domain assumption Multiple-choice comprehension questions are valid probes of embodied navigation understanding.
    The paper equates MCQ accuracy on global/progress/local tasks with navigation comprehension, without behavioral validation tying these scores to agent performance beyond a coarse correlation across 8 models.
  • domain assumption Matterport3D simulator observations are representative of real indoor environments for zero-shot MLLM evaluation.
    Used as the visual substrate for both comprehension and execution; no analysis of domain gap to the real robot except a 10-episode pilot.
  • domain assumption Human difficulty ratings are reliable ground truth.
    Human evaluation is mentioned but no number of annotators, guidelines, or inter-annotator agreement is reported, even though it is used to validate the automatic difficulty scoring.
  • ad hoc to paper Curated list of valid instruction-path pairs does not bias Progress Estimation data.
    Section 3.2.1 filters examples using a curated list without criteria; this post-hoc exclusion could remove hard or ambiguous cases and skew model scores.

how reviews work

0 comments
Cite this review

Pith. "Pith review of NavBench: Probing Multimodal Large Language Models for Embodied Navigation." pith.science (2026). https://pith.science/paper/A6LWBT7L

@misc{pith2026250601031,
  author       = {Pith},
  title        = {Pith review of: NavBench: Probing Multimodal Large Language Models for Embodied Navigation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/A6LWBT7L}},
  note         = {Machine review of arXiv:2506.01031}
}
read the original abstract

Multimodal Large Language Models (MLLMs) have demonstrated strong generalization in vision-language tasks, yet their ability to understand and act within embodied environments remains underexplored. We present NavBench, a benchmark to evaluate the embodied navigation capabilities of MLLMs under zero-shot settings. NavBench consists of two components: (1) navigation comprehension, assessed through three cognitively grounded tasks including global instruction alignment, temporal progress estimation, and local observation-action reasoning, covering 3,200 question-answer pairs; and (2) step-by-step execution in 432 episodes across 72 indoor scenes, stratified by spatial, cognitive, and execution complexity. To support real-world deployment, we introduce a pipeline that converts MLLMs' outputs into robotic actions. We evaluate both proprietary and open-source models, finding that GPT-4o performs well across tasks, while lighter open-source models succeed in simpler cases. Results also show that models with higher comprehension scores tend to achieve better execution performance. Providing map-based context improves decision accuracy, especially in medium-difficulty scenarios. However, most models struggle with temporal understanding, particularly in estimating progress during navigation, which may pose a key challenge.

Figures

Figures reproduced from arXiv: 2506.01031 by the authors.

Figure 1
Figure 1. NavBench evaluates MLLMs across three comprehension tasks and a step-by-step execution [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Illustration of the Navigation Comprehension task. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. NavBench construction pipeline and statistics. (a) QA generation for comprehension tasks at global, progress, and local levels. (b) Execution pipeline combining automatic difficulty scor￾ing and human ratings. (c) Benchmark statistics, including comprehension (comp.) task distribution, QA counts, and execution statistics (e.g., instruction length, steps, distance). Specifically, at each step, the model receives the … view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Radar chart of average complexity scores across cognitive, spatial, and execution dimensions for different difficulty levels. Difficulty Categorization Based on the final scores, each case is categorized into one of three levels, as illustrated in [PITH_FULL_IMAGE:fig…
Figure 5
Figure 5. Figure 5: Overview of the real-world embodied navigation pipeline. To demonstrate the real-world feasibility of MLLM￾guided embodied navigation, we implement a modular pipeline that complements our benchmark evaluation, as illustrated in [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Model Performance under Different Instruction Perturbations [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 8
Figure 8. Figure 8: Distribution of navigation error types identified from manual analysis. more challenging reasoning tasks. In particular, Progress Estimation remains a consistent weakness across models: aside from GPT-4o (42.90%), all others perform poorly, highlighting current MLLMs’ …

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. NavVerse: Benchmarking Indoor-to-Outdoor Embodied Navigation in Continuous Robot Simulation

    cs.RO 2026-07 conditional novelty 7.0 of 10

    A new physics-enabled benchmark with 10,000 indoor, outdoor, and indoor-to-outdoor navigation episodes shows current zero-shot agents fail most when crossing the indoor-outdoor boundary, especially on the new PlaceNav task.

  2. MapTab: A Diagnostic Benchmark for Long-Horizon Multi-Criteria Multimodal Reasoning on Heterogeneous Topological Graphs

    cs.LG 2026-02 unverdicted novelty 6.0 of 10

    MapTab is a new multimodal benchmark with 328 images and nearly 200k queries that shows current MLLMs have substantial difficulty with multi-criteria route planning when visual and tabular information must be combined.

Reference graph

Works this paper leans on

65 extracted references · 36 canonical work pages · cited by 2 Pith papers

  1. [1]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems, volume 36, 2023

    2023

  2. [2]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024

    arXiv 2024

  3. [3]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024

    arXiv 2024

  4. [4]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913, 2017

    2017

  5. [5]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Rongrong Ji, and Xing Sun. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. ArXiv, abs/24...

    arXiv 2024

  6. [6]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023

    arXiv

  7. [7]

    Spatialbot: Precise spatial understanding with vision language models

    Wenxiao Cai et al. Spatialbot: Precise spatial understanding with vision language models. In IEEE international conference on robotics and automation, 2025

    work page 2025

  8. [8]

    Thinking in Space: How Multimodal Large Language Models See, Remember and Recall Spaces

    Jihan Yang, Shusheng Yang, Anjali Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in Space: How Multimodal Large Language Models See, Remember and Recall Spaces. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025

    work page 2025

Show all 65 references
  1. [9]

    Reid, Stephen Gould, and Anton van den Hengel

    Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian D. Reid, Stephen Gould, and Anton van den Hengel. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In CVPR, pages 3674–3683, 2018. 10

    2018

  2. [10]

    Object goal navigation using goal-oriented semantic exploration

    Devendra Singh Chaplot, Dhiraj Prakashchand Gandhi, Abhinav Gupta, and Russ R Salakhutdinov. Object goal navigation using goal-oriented semantic exploration. Advances in Neural Information Processing Systems, 33:4247–4258, 2020

    2020

  3. [11]

    The spatial semantic hierarchy

    Benjamin Kuipers. The spatial semantic hierarchy. Artificial intelligence, 119(1-2):191–233, 2000

    2000

  4. [12]

    Intern vl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Zhong Muyan, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Intern vl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. IEEE/CVF Co...

    2023

  5. [13]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian...

    2022 arXiv

  6. [14]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. ArXiv, abs/2308.12966, 2023

    2023 arXiv

  7. [15]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26286–26296, 2023

    2023

  8. [16]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709, 2019

    2019

  9. [17]

    Ok-vqa: A visual question answering benchmark requiring external knowledge

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195–3204, 2019

    2019

  10. [18]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019

    2019

  11. [19]

    A survey on multimodal large language models

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint arXiv:2306.13549, 2023

    2023 arXiv

  12. [20]

    Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, pages 216–233. Springer, 2025

    2025

  13. [21]

    Mm-vet: Evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023

    2023 arXiv

  14. [22]

    Scanreason: Empowering 3d visual grounding with reasoning capabilities

    Chenming Zhu, Tai Wang, Wenwei Zhang, Kai Chen, and Xihui Liu. Scanreason: Empowering 3d visual grounding with reasoning capabilities. In European Conference on Computer Vision, 2024

    2024

  15. [23]

    REVERIE: remote embodied visual referring expression in real indoor environments

    Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang, William Yang Wang, Chunhua Shen, and Anton van den Hengel. REVERIE: remote embodied visual referring expression in real indoor environments. In CVPR, pages 9979–9988, 2020

    2020

  16. [24]

    Navigating the nuances: A fine-grained evaluation of vision-language navigation

    Zehao Wang, Minye Wu, Yixin Cao, Yubo Ma, Meiqi Chen, and Tinne Tuytelaars. Navigating the nuances: A fine-grained evaluation of vision-language navigation. ArXiv, abs/2409.17313, 2024

    2024 arXiv

  17. [25]

    Target-driven visual navigation in indoor scenes using deep reinforcement learning

    Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In IEEE international conference on robotics and automation, pages 3357–3364. IEEE, 2017

    2017

  18. [26]

    Instance-specific image goal navigation: Training embodied agents to find object instances.arXiv preprint arXiv:2211.15876, 2022

    Jacob Krantz, Stefan Lee, Jitendra Malik, Dhruv Batra, and Devendra Singh Chaplot. Instance-specific image goal navigation: Training embodied agents to find object instances.arXiv preprint arXiv:2211.15876, 2022

    2022 arXiv

  19. [27]

    Habitat: A platform for embodied ai research

    Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai research. In Proceedings of the IEEE/CVF international conference on computer vision, ...

    2019

  20. [28]

    History aware multimodal transformer for vision-and-language navigation

    Shizhe Chen, Pierre-Louis Guhur, Cordelia Schmid, and Ivan Laptev. History aware multimodal transformer for vision-and-language navigation. In Advances in Neural Information Processing Systems, 2021

    2021

  21. [29]

    Vision-and-language navigation today and tomorrow: A survey in the era of foundation models

    Yue Zhang et al. Vision-and-language navigation today and tomorrow: A survey in the era of foundation models. Transactions on Machine Learning Research, 2024

    2024

  22. [30]

    Room-across-room: Mul- tilingual vision-and-language navigation with dense spatiotemporal grounding

    Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. Room-across-room: Mul- tilingual vision-and-language navigation with dense spatiotemporal grounding. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 4392–4412, 2020

    2020

  23. [31]

    Vision-and-dialog navigation

    Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. Vision-and-dialog navigation. In CoRL, pages 394–406, 2019

    2019

  24. [32]

    Find what you want: Learning demand-conditioned object attribute space for demand-driven navigation

    Hongcheng Wang, Andy Guan Hong Chen, Xiaoqi Li, Mingdong Wu, and Hao Dong. Find what you want: Learning demand-conditioned object attribute space for demand-driven navigation. In Advances in Neural Information Processing Systems, 2023

    2023

  25. [33]

    Object-and-action aware model for visual language navigation

    Yuankai Qi, Zizheng Pan, Shengping Zhang, Anton van den Hengel, and Qi Wu. Object-and-action aware model for visual language navigation. In European Conference on Computer Vision, pages 303–317. Springer, 2020

    2020

  26. [34]

    Language and visual entity relationship graph for agent navigation

    Yicong Hong, Cristian Rodriguez, Yuankai Qi, Qi Wu, and Stephen Gould. Language and visual entity relationship graph for agent navigation. Advances in Neural Information Processing Systems, 33:7685– 7696, 2020

    2020

  27. [35]

    Neighbor-view enhanced model for vision and language navigation

    Dong An, Yuankai Qi, Yan Huang, Qi Wu, Liang Wang, and Tieniu Tan. Neighbor-view enhanced model for vision and language navigation. In Proceedings of the 29th ACM International Conference on Multimedia, pages 5101–5109, 2021

    2021

  28. [36]

    Towards learning a generic agent for vision-and-language navigation via pre-training

    Weituo Hao, Chunyuan Li, Xiujun Li, Lawrence Carin, and Jianfeng Gao. Towards learning a generic agent for vision-and-language navigation via pre-training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13137–13146, 2020

    2020

  29. [37]

    Airbert: In- domain pretraining for vision-and-language navigation

    Pierre-Louis Guhur, Makarand Tapaswi, Shizhe Chen, Ivan Laptev, and Cordelia Schmid. Airbert: In- domain pretraining for vision-and-language navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1634–1643, 2021

    2021

  30. [38]

    Improving vision-and-language navigation with image-text pairs from the web

    Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter Anderson, Devi Parikh, and Dhruv Batra. Improving vision-and-language navigation with image-text pairs from the web. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part ...

    2020

  31. [39]

    History aware multimodal transformer for vision-and-language navigation

    Shizhe Chen, Pierre-Louis Guhur, Cordelia Schmid, and Ivan Laptev. History aware multimodal transformer for vision-and-language navigation. Advances in neural information processing systems, 34:5834–5847, 2021

    2021

  32. [40]

    Hop: History-and-order aware pre-training for vision-and-language navigation

    Yanyuan Qiao, Yuankai Qi, Yicong Hong, Zheng Yu, Peng Wang, and Qi Wu. Hop: History-and-order aware pre-training for vision-and-language navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15418–15427, 2022

    2022

  33. [41]

    Hop+: History-enhanced and order-aware pre-training for vision-and-language navigation

    Yanyuan Qiao, Yuankai Qi, Yicong Hong, Zheng Yu, Peng Wang, and Qi Wu. Hop+: History-enhanced and order-aware pre-training for vision-and-language navigation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(7):8524–8537, 2023

    2023

  34. [42]

    Bevbert: Mul- timodal map pre-training for language-guided navigation

    Dong An, Yuankai Qi, Yangguang Li, Yan Huang, Liang Wang, Tieniu Tan, and Jing Shao. Bevbert: Mul- timodal map pre-training for language-guided navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2737–2748, 2023

    2023

  35. [43]

    Bird’s-eye-view scene graph for vision-language navigation

    Rui Liu, Xiaohan Wang, Wenguan Wang, and Yi Yang. Bird’s-eye-view scene graph for vision-language navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10968– 10980, 2023

    2023

  36. [44]

    Gridmm: Grid memory map for vision-and-language navigation

    Zihan Wang, Xiangyang Li, Jiahao Yang, Yeqi Liu, and Shuqiang Jiang. Gridmm: Grid memory map for vision-and-language navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15625–15636, 2023

    2023

  37. [45]

    Scaling data generation in vision-and-language navigation

    Zun Wang, Jialu Li, Yicong Hong, Yi Wang, Qi Wu, Mohit Bansal, Stephen Gould, Hao Tan, and Yu Qiao. Scaling data generation in vision-and-language navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12009–12020, 2023. 12

    2023

  38. [46]

    A new path: Scaling vision-and-language navigation with synthetic instructions and imitation learning

    Aishwarya Kamath, Peter Anderson, Su Wang, Jing Yu Koh, Alexander Ku, Austin Waters, Yinfei Yang, Jason Baldridge, and Zarana Parekh. A new path: Scaling vision-and-language navigation with synthetic instructions and imitation learning. In Proceedings of the IEEE/CVF Conferenc...

    2023

  39. [47]

    Navid: Video-based vlm plans the next step for vision-and-language navigation

    Jiazhao Zhang, Kunyu Wang, Rongtao Xu, Gengze Zhou, Yicong Hong, Xiaomeng Fang, Qi Wu, Zhizheng Zhang, and He Wang. Navid: Video-based vlm plans the next step for vision-and-language navigation. arXiv preprint arXiv:2402.15852, 2024

    2024 arXiv

  40. [48]

    Esc: Exploration with soft commonsense constraints for zero-shot object navigation

    Kaiwen Zhou, Kaizhi Zheng, Connor Pryor, Yilin Shen, Hongxia Jin, Lise Getoor, and Xin Eric Wang. Esc: Exploration with soft commonsense constraints for zero-shot object navigation. In International Conference on Machine Learning, pages 42829–42842. PMLR, 2023

    2023

  41. [49]

    Cows on pasture: Baselines and benchmarks for language-driven zero-shot object navigation

    Samir Yitzhak Gadre, Mitchell Wortsman, Gabriel Ilharco, Ludwig Schmidt, and Shuran Song. Cows on pasture: Baselines and benchmarks for language-driven zero-shot object navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2317...

    2023

  42. [50]

    Vlfm: Vision- language frontier maps for zero-shot semantic navigation

    Naoki Yokoyama, Sehoon Ha, Dhruv Batra, Jiuguang Wang, and Bernadette Bucher. Vlfm: Vision- language frontier maps for zero-shot semantic navigation. In IEEE international conference on robotics and automation, pages 42–48. IEEE, 2024

    2024

  43. [51]

    Instructnav: Zero-shot system for generic instruction navigation in unexplored environment

    Yuxing Long, Wenzhe Cai, Hongcheng Wang, Guanqi Zhan, and Hao Dong. Instructnav: Zero-shot system for generic instruction navigation in unexplored environment. arXiv preprint arXiv:2406.04882, 2024

    2024 arXiv

  44. [52]

    Navgpt: Explicit reasoning in vision-and-language navigation with large language models

    Gengze Zhou, Yicong Hong, and Qi Wu. Navgpt: Explicit reasoning in vision-and-language navigation with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 7641–7649, 2024

    2024

  45. [53]

    Open- nav: Exploring zero-shot vision-and-language navigation in continuous environment with open-source llms

    Yanyuan Qiao, Wenqi Lyu, Hui Wang, Zixu Wang, Zerui Li, Yuan Zhang, Mingkui Tan, and Qi Wu. Open- nav: Exploring zero-shot vision-and-language navigation in continuous environment with open-source llms. arXiv preprint arXiv:2409.18794, 2024

    2024 arXiv

  46. [54]

    Mapgpt: Map- guided prompting with adaptive path planning for vision-and-language navigation

    Jiaqi Chen, Bingqian Lin, Ran Xu, Zhenhua Chai, Xiaodan Liang, and Kwan-Yee Wong. Mapgpt: Map- guided prompting with adaptive path planning for vision-and-language navigation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1:...

    2024

  47. [55]

    Chang, Angela Dai, Thomas A

    Angel X. Chang, Angela Dai, Thomas A. Funkhouser, Maciej Halber, Matthias Nießner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from RGB-D data in indoor environments. In 3DV, pages 667–676, 2017

    2017

  48. [56]

    Grounded entity-landmark adaptive pre-training for vision-and-language navigation

    Yibo Cui, Liang Xie, Yakun Zhang, Meishan Zhang, Ye Yan, and Erwei Yin. Grounded entity-landmark adaptive pre-training for vision-and-language navigation. IEEE/CVF International Conference on Com- puter Vision, pages 12009–12019, 2023

    2023

  49. [57]

    Sub-instruction aware vision-and-language navigation

    Yicong Hong, Cristian Rodriguez, Qi Wu, and Stephen Gould. Sub-instruction aware vision-and-language navigation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 3360–3376, 2020

    2020

  50. [58]

    Are large vision language models good game players? In Proceedings of the International Conference on Learning Representations, 2025

    Xinyu Wang, Bohan Zhuang, and Qi Wu. Are large vision language models good game players? In Proceedings of the International Conference on Learning Representations, 2025

    2025

  51. [59]

    Benchmarking complex instruction-following with multiple constraints composition

    Bosi Wen, Pei Ke, Xiaotao Gu, Lindong Wu, Hao Huang, Jinfeng Zhou, Wenchuang Li, Binxin Hu, Wendy Gao, Jiaxin Xu, Yiming Liu, Jie Tang, Hongning Wang, and Minlie Huang. Benchmarking complex instruction-following with multiple constraints composition. ArXiv, abs/2407.03978, 2024

    2024 arXiv

  52. [60]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuehui Wang, Qingyun Li, Yiming Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang, Tan Jiang, Bo Wang, Conghui He, Botian Shi, Xingcheng Zhang, Han Lv, Y...

    2024 arXiv

  53. [61]

    Qwen2.5-vl technical report

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Han...

    2025 arXiv

  54. [62]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  55. [63]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  56. [64]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems...

    2023

  57. [65]

    Lmdeploy: A toolkit for compressing, deploying, and serving llm

    LMDeploy Contributors. Lmdeploy: A toolkit for compressing, deploying, and serving llm. https: //github.com/InternLM/lmdeploy, 2023. 14

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.