Pith. sign in

REVIEW 3 major objections 5 minor 77 references

NavComposer: Composing Language Instructions for Navigation Trajectories through Action-Scene-Object Modularization

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read NavComposer claims that decomposing a navigation trajectory into actions, scenes, and objects before writing the instruction produces higher-quality language than end-to-end generators, with no training on the target dataset.

desk verdict The paper's headline superiority claim is undermined by a circular semantic-consistency evaluation, but the modular generation framework and the annotation-free evaluation system are real contributions worth a careful revision. read the letter →

arxiv 2507.10894 v1 pith:44AM6IZN submitted 2025-07-15 cs.AI cs.CLcs.CV

Zongtao He , Liuyi Wang , Lu Chen , Chengju Liu , Qijun Chen This is my paper
classification cs.AIcs.CLcs.CV
keywords language-guidednavigationvision-and-languageinstructiongenerationmodularframeworksemanticentitydecompositionannotation-freeevaluationegocentricvideocaptioningVLN-CE
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

NavComposer claims that the best way to write navigation instructions for a trajectory is to split the problem: first extract three kinds of semantic entities from the egocentric video—actions, scenes, and objects—then let a large language model stitch those entities into fluent instructions. The paper argues this modular decomposition produces instructions that are more accurate, richer, and more varied than those from end-to-end trained generators or from video-language models that see the whole video at once, and that it does so without training on the target dataset. To make the claim testable without expert annotations, the paper also introduces NavInstrCritic, a three-part evaluation covering contrastive matching, semantic consistency, and linguistic diversity. If the claim holds, data-hungry vision-and-language navigation could be trained at scale from automatically generated instructions, including from real robot and internet videos.

What carries the argument

The central object is the semantic-entity tuple of actions, scenes, and objects extracted per trajectory segment, together with the synthesis protocol that converts that tuple into language. Action classification uses frame-pair motion analysis with temporal correction rules; scene recognition and object detection can be unimodal or multimodal-LLM based; and the synthesis module downsamples repeated actions to key frames, applies a double random synonym replacement for diversity, and prompts an LLM to phrase the temporal entity sequence. The evaluation is carried by NavInstrCritic, whose contrastive matcher maps trajectory and instruction into one space, whose LLM grader scores action, scene, and object consistency, and whose diversity metrics measure vocabulary and compression properties.

What would settle it

Run a human rating study in which annotators judge NavComposer and baseline instructions for correctness, completeness, and naturalness without being shown NavComposer's entity lists; if humans do not rank NavComposer first, the reported superiority is an artifact of its own evaluator. A complementary check is to train a navigation agent on instructions from each generator and compare success rate and SPL on unseen environments.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that explicit entity-based decomposition is the load-bearing ingredient for instruction quality. On 872 continuous-navigation trajectories, the best NavComposer variant reaches a MAP of 0.685, a mean semantic consistency of 5.112, and a compression-ratio diversity score of 4.478, beating the end-to-end baseline, two classical speaker-model baselines, and three video-language models on nearly every metric without ever being trained on the target dataset. Ablations removing actions, scenes, or objects each lower the mean semantic consistency, and the object entity is the largest contributor to contrastive matching and diversity. The framework is also applied to ten trajectory sources spanning simulators, handheld cameras, driving footage, and a wheeled robot, generating 232,569 instructions in total.

Load-bearing premise

The semantic-consistency evaluation assumes that the actions, scenes, and objects that NavComposer itself extracts are the correct and complete checklist for judging any instruction for a trajectory, so a baseline that describes the same path through different landmarks or wording can be unfairly marked as wrong.

Editorial extensions

If this is right

  • Large-scale instruction corpora can be generated for any trajectory video, including domains where no expert annotations exist.
  • The modular design means future improvements in action classification, scene recognition, object detection, or language synthesis can be swapped in without retraining the whole pipeline.
  • Synonym replacement and prompt control allow instruction style, length, and vocabulary to be shaped to match a downstream agent or user preference.
  • Generated instructions are roughly twice as long as classical speaker outputs (58 vs 27 words on average), carrying more landmarks and scene context for an agent to ground on.
  • The same generator can cover simulated, handheld-camera, driving, and robot-camera trajectories after only light preprocessing.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the reported semantic-consistency scores partly measure agreement with NavComposer's own entity extraction, because baselines without explicit extraction are scored against the unified entity lists from NavComposer's best variant; a human reference or a fixed third-party entity extractor would make the comparison less self-referential.
  • Beyond the paper: the contrastive matcher is trained on the continuous-navigation benchmark, so cross-source MAP scores reflect domain mismatch as much as instruction quality; a matcher trained on broader video-text data would be needed to separate the two.
  • Beyond the paper: a direct downstream test—training a navigation policy on NavComposer-generated instructions and measuring success rate and SPL on unseen environments—would show whether the instruction quality improvements translate into agent performance, which the paper does not run.
  • Beyond the paper: because the entity lists are explicit, the same pipeline could be pointed backwards to debug why an instruction fails: the action, scene, or object that caused the error is identifiable by construction.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes NavComposer, a modular pipeline that generates natural-language navigation instructions from egocentric video by first extracting action, scene, and object entities and then prompting an LLM to compose them into an instruction. It also introduces NavInstrCritic, an annotation-free evaluation system covering contrastive matching, semantic consistency (ASC/SSC/OSC/MSC), and linguistic diversity. Experiments on VLN-CE and ten additional datasets compare NavComposer with an end-to-end CoCa baseline, classical speaker models, and video LLMs, reporting superiority on most metrics and demonstrating adaptation to diverse data sources.

Significance. If the claims were supported, the paper would make a practical contribution: modular zero-shot instruction generation for continuous navigation, an annotation-free evaluation suite, and a broad adaptability study across 10 datasets. The modular design is flexible, the module comparison is systematic, and the diversity metrics are standard. However, the headline comparison is currently undermined by the semantic-consistency protocol, which evaluates all methods against entities produced by NavComposer itself. The core superiority result therefore needs to be re-established with an unbiased reference before the contribution can be assessed.

major comments (3)
  1. [IV-A and Algorithm 1] The semantic-consistency evaluation is circular. Section IV-A states that for methods without explicit entity extraction, the evaluator uses 'unified results from our best-performing variant,' i.e., NavComposer's A, S, O lists. Algorithm 1 (lines 27-31) constructs the instruction I from exactly those same lists via the LLM. Consequently ASC/SSC/OSC/MSC in Table VI measure whether a baseline happens to reproduce NavComposer's entity choices, not whether the instruction is faithful to the trajectory. A baseline using different but valid landmarks or scene granularity is penalized. The MSC gap in Table VI (5.112 vs 2.768-3.049) is therefore an artifact of the protocol, and the claim in Section IV-C1 that NavComposer 'outperforms all other methods across nearly all metrics' is unsupported by this metric.
  2. [IV-B] The module selection procedure inflates the comparison. The best-performing variant is chosen on the same NavInstrCritic metrics and the same validation set that is later used for the headline comparison in Table VI. Since the semantic-consistency metrics are biased toward NavComposer's own entities, selecting the variant that maximizes these metrics guarantees an advantage over methods that do not generate from those entity lists. A separate validation split or an unbiased semantic metric is needed before the comparison can be interpreted.
  3. [III-B2 and Table VI] The semantic-consistency scores are not shown to correlate with any external notion of instruction quality. The paper's ablation in rows #8-#10 shows that removing one entity type can raise the scores for the other two (e.g., removing scenes raises ASC and OSC), which is consistent with score redistribution rather than absolute quality. Without validation against human judgments, downstream navigation success, or an independent entity annotation set, the MSC scores cannot carry the weight of the paper's central claim.
minor comments (5)
  1. [II-B] There is a typo: 'the genralization of instructions' should be 'the generalization of instructions.'
  2. [Figure 6] The generated examples contain typographical artifacts such as 'Ve e rat', 'V eerleft', and 'Cease at desk', which conflict with the claimed instruction quality; please clean up or regenerate these examples.
  3. [IV-A] The phrase 'unified results from our best-performing variant' needs a precise definition: which variant was used, how its entities were extracted for each baseline, and whether those entities were available to the baselines.
  4. [Algorithm 1] The sampling logic for 'move forward' actions is not self-explanatory: line 8 gates on Z ≤ 1/2 and then splits into 1/6, 1/3, and 1/3; please clarify the intended probabilities in the text.
  5. [IV-D] The paper honestly notes that the contrastive matcher is trained on VLN-CE and may suffer domain mismatch, but this also means the MAP scores in Table VII should be interpreted as measurements of matcher domain transfer rather than of instruction quality alone.

Circularity Check

1 steps flagged · score 6.0 of 10

Semantic-consistency comparison is circular: baselines are judged against NavComposer's own extracted entities while NavComposer generates instructions from those same entities.

  1. self definitional [Section IV-A (NavInstrCritic-related setup); Section III-A.4, Algorithm 1 lines 27-31; superiority claim in Section IV-C1]
    ""For methods that do not explicitly extract semantic entities, we use unified results from our best-performing variant." ... "Then, the instruction synthesis module integrates A, S, and O to generate a fluent language instruction I.""

    NavInstrCritic's semantic-consistency scores (ASC/SSC/OSC/MSC) evaluate alignment between the reference entities A, S, O and the instruction I. For NavComposer, I is produced by Algorithm 1 as LLM(prom, desc), where desc is exactly the serialized A, S, O lists extracted by its own modules (lines 27-31). Thus NavComposer's instruction is constructed to contain those reference entities, making its high semantic-consistency scores largely self-agreement. For all compared methods without explicit entity extraction, the reference is "unified results from our best-performing variant," i.e., NavComposer's own extraction, so valid alternative entities or phrasings used by baselines are penalized against NavComposer's choices.

full rationale

The paper's main claim of superiority passes through NavInstrCritic, whose semantic-consistency dimension is circular by construction. Section IV-A states that methods without explicit entity extraction are scored against "unified results from our best-performing variant," and Section III-A.4 shows that NavComposer's instruction synthesis consumes exactly those A, S, O entity lists. Consequently, high ASC/SSC/OSC/MSC for NavComposer reflect that its own instructions were generated from the same entities used as the scoring reference, while baselines that describe trajectories with different valid landmarks or phrasings are measured against NavComposer's entity choices. This bias also influences module selection in Tables II-V, where the same circular semantic-consistency scores help choose the "best-performing variant" that then supplies the reference entities for all other methods. The contrastive matching metrics, based on a CLIP-initialized matcher trained on VLN-CE, and the linguistic diversity metrics, based on standard text statistics, are not circular and provide some independent support. No load-bearing self-citation or imported uniqueness argument appears. Overall score 6: partial circularity because a major evaluation dimension is self-referential, while the remaining evidence retains independent content.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The framework relies on off-the-shelf perception models and LLMs whose accuracy is taken for granted. The most consequential assumption is that the semantic consistency evaluator can use NavComposer's own entity extractions as ground truth for all methods, which directly biases the comparison. The hand-crafted probabilities and correction rules are additional free choices that influence the output.

free parameters (4)
  • Enter/pass/leave probability thresholds = Z <= 1/6 enter; Z <= 1/3 pass; else leave
    Algorithm 1 lines 8-14. These manually chosen probabilities map 'move forward' actions to 'enter', 'pass', or 'leave' descriptions, affecting instruction content and length.
  • Temporal correction rules = ABA -> AAA; AAB -> AAA
    Section III-A.1. Hand-crafted rules for smoothing action sequences based on 'unlikely' patterns. They are not learned or validated against data.
  • Nine-square grid for object position mapping = 3x3 grid over the image
    Section III-A.3. The image is divided into a fixed 3x3 grid to map bounding boxes to spatial phrases; the resolution is arbitrary.
  • Last frame action override = stop
    Equation (1). The action at t=T is set to 'stop' manually, with no visual evidence, which anchors the instruction's ending.
assumptions (5)
  • domain assumption The action space is the four-class VLN-CE set {stop, move forward, turn left, turn right}.
    Section III-A.1: 'Similar to VLN-CE [42], we use the action space ...' This restricts the generator to a predefined vocabulary.
  • domain assumption Visual odometry with SIFT, FLANN, and RecoverPose can recover relative motion between frames and classify navigation actions.
    Section III-A.1 Equations (6)-(10). The VO-based action module assumes epipolar geometry yields reliable motion estimates across diverse domains.
  • domain assumption Multimodal LLMs provide accurate, grounded scene and object descriptions from single egocentric frames.
    Sections III-A.2 and III-A.3. The scene and object modules rely on off-the-shelf models such as GPT-4o-mini and Qwen to produce semantic entities.
  • domain assumption The Qwen-14B LLM can reliably score action, scene, and object consistency on a 0-10 scale.
    Section IV-A. No validation of the LLM judge's accuracy is provided.
  • domain assumption The CLIP-based contrastive matcher trained on VLN-CE is a valid measure of trajectory-instruction alignment, including for out-of-domain data.
    Section IV-A and IV-D. The matcher is trained only on VLN-CE; the paper itself notes that it performs poorly on divergent datasets.

how reviews work

0 comments
Cite this review

Pith. "Pith review of NavComposer: Composing Language Instructions for Navigation Trajectories through Action-Scene-Object Modularization." pith.science (2026). https://pith.science/paper/44AM6IZN

@misc{pith2026250710894,
  author       = {Pith},
  title        = {Pith review of: NavComposer: Composing Language Instructions for Navigation Trajectories through Action-Scene-Object Modularization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/44AM6IZN}},
  note         = {Machine review of arXiv:2507.10894}
}
read the original abstract

Language-guided navigation is a cornerstone of embodied AI, enabling agents to interpret language instructions and navigate complex environments. However, expert-provided instructions are limited in quantity, while synthesized annotations often lack quality, making them insufficient for large-scale research. To address this, we propose NavComposer, a novel framework for automatically generating high-quality navigation instructions. NavComposer explicitly decomposes semantic entities such as actions, scenes, and objects, and recomposes them into natural language instructions. Its modular architecture allows flexible integration of state-of-the-art techniques, while the explicit use of semantic entities enhances both the richness and accuracy of instructions. Moreover, it operates in a data-agnostic manner, supporting adaptation to diverse navigation trajectories without domain-specific training. Complementing NavComposer, we introduce NavInstrCritic, a comprehensive annotation-free evaluation system that assesses navigation instructions on three dimensions: contrastive matching, semantic consistency, and linguistic diversity. NavInstrCritic provides a holistic evaluation of instruction quality, addressing limitations of traditional metrics that rely heavily on expert annotations. By decoupling instruction generation and evaluation from specific navigation agents, our method enables more scalable and generalizable research. Extensive experiments provide direct and practical evidence for the effectiveness of our method.

Figures

Figures reproduced from arXiv: 2507.10894 by the authors.

Figure 1
Figure 1. Overview of our method. NavComposer extracts semantic entities from [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. NavComposer processes visual observations from a navigation trajectory, extracts semantic entities, and synthesizes a fluent language instruction, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Details of the action classification, scene recognition, and object detection modules. Each module supports two types of architectures with various [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Example outputs of different object detection modules. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Instruction lengths and examples generated by NavComposer and [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Visualization of navigation instructions generated by NavComposer [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

77 extracted references · 62 canonical work pages

  1. [1]

    A survey of embodied ai: From simulators to research tasks,

    J. Duan et al., “A survey of embodied ai: From simulators to research tasks,” IEEE Trans. Emerg. Topics Comput. Intell. , vol. 6, no. 2, pp. 230–244, 2022

    work page 2022

  2. [2]

    Behavior-1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation,

    C. Li et al. , “Behavior-1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation,” in Conf. Robot Learn. PMLR, 2023, pp. 80–93

    work page 2023

  3. [3]

    Aligning cyber space with physical world: A comprehen- sive survey on embodied ai,

    Y . Liu et al., “Aligning cyber space with physical world: A comprehen- sive survey on embodied ai,” arXiv preprint arXiv:2407.06886 , 2024

    arXiv 2024

  4. [4]

    Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments,

    P. Anderson et al. , “Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. , 2018, pp. 3674–3683

    work page 2018

  5. [5]

    Room-across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding,

    A. Ku et al. , “Room-across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding,” in Proc. 2020 Conf. Empirical Methods Natural Lang. Process. , Nov. 2020, pp. 4392–4412

    work page 2020

  6. [6]

    Talk2nav: Long-range vision-and-language navigation with dual attention and spatial memory,

    A. B. Vasudevan et al. , “Talk2nav: Long-range vision-and-language navigation with dual attention and spatial memory,” Int. J. Comput. Vis., vol. 129, no. 1, pp. 246–266, 2021

    work page 2021

  7. [7]

    Reverie: Remote embodied visual referring expression in real indoor environments,

    Y . Qi et al. , “Reverie: Remote embodied visual referring expression in real indoor environments,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2020, pp. 9979–9988

    work page 2020

  8. [8]

    Speaker-follower models for vision-and-language nav- igation,

    D. Fried et al., “Speaker-follower models for vision-and-language nav- igation,” in Adv. Neural Inf. Process. Syst. , 2018, pp. 3318–3329

    work page 2018

Show all 77 references
  1. [9]

    Learning to navigate unseen environments: Back trans- lation with environmental dropout,

    H. Tan et al. , “Learning to navigate unseen environments: Back trans- lation with environmental dropout,” in Proc. 2019 Conf. North Amer. Chapter Assoc. Comput. Linguistics: Human Lang. Technol., Vol. 1 , 2019, pp. 2610–2621

    2019

  2. [10]

    Visual landmark selection for generating grounded and interpretable navigation instructions,

    S. Agarwal et al. , “Visual landmark selection for generating grounded and interpretable navigation instructions,” in CVPR workshop , vol. 3, 2019, p. 7

    2019

  3. [11]

    Crossmap transformer: A crossmodal masked path transformer using double back-translation for vision-and-language navigation,

    A. Magassouba et al. , “Crossmap transformer: A crossmodal masked path transformer using double back-translation for vision-and-language navigation,” IEEE Robot. Autom. Lett. , vol. 6, no. 4, pp. 6258–6265, 2021

    2021

  4. [12]

    Improved speaker and navigator for vision-and-language navigation,

    Z. Wu et al., “Improved speaker and navigator for vision-and-language navigation,” IEEE MultiMedia, vol. 28, no. 4, pp. 55–63, 2021

    2021

  5. [13]

    Res-sts: Referring expression speaker via self-training with scorer for goal-oriented vision-language navigation,

    L. Wang et al., “Res-sts: Referring expression speaker via self-training with scorer for goal-oriented vision-language navigation,” IEEE Trans. Circuits Syst. Video Technol., vol. 33, no. 7, pp. 3441–3454, 2023

    2023

  6. [14]

    Touchdown: Natural language navigation and spatial reasoning in visual street environments,

    H. Chen et al. , “Touchdown: Natural language navigation and spatial reasoning in visual street environments,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. , 2019, pp. 12 530–12 539

    2019

  7. [15]

    Vision-language navigation with self-supervised auxil- iary reasoning tasks,

    F. Zhu et al. , “Vision-language navigation with self-supervised auxil- iary reasoning tasks,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2020, pp. 10 012–10 022

    2020

  8. [16]

    Towards navigation by reasoning over spatial config- urations,

    Y . Zhang et al., “Towards navigation by reasoning over spatial config- urations,” in Proc. 2nd Int. Combined Workshop Spatial Lang. Under- standing Grounded Commun. Robot. , 2021, pp. 42–52

    2021

  9. [17]

    Language-guided navigation via cross-modal ground- ing and alternate adversarial learning,

    W. Zhang et al., “Language-guided navigation via cross-modal ground- ing and alternate adversarial learning,” IEEE Trans. Circuits Syst. Video Technol., vol. 31, no. 9, pp. 3469–3481, 2021

    2021

  10. [18]

    A dual semantic-aware recurrent global-adaptive net- work for vision-and-language navigation,

    L. Wang et al. , “A dual semantic-aware recurrent global-adaptive net- work for vision-and-language navigation,” in Proc. 32nd Int. Joint Conf. Artif. Intell., ser. IJCAI ’23, 2023

    2023

  11. [19]

    Vision-and-language navigation via causal learning,

    L. Wang et al., “Vision-and-language navigation via causal learning,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. , June 2024, pp. 13 139–13 150

    2024

  12. [20]

    Waypoint models for instruction-guided navigation in continuous environments,

    J. Krantz et al., “Waypoint models for instruction-guided navigation in continuous environments,” in Proc. IEEE/CVF Int. Conf. Comput. Vis. , 2021, pp. 15 162–15 171

    2021

  13. [21]

    Bridging the gap between learning in discrete and continuous environments for vision-and-language navigation,

    Y . Hong et al. , “Bridging the gap between learning in discrete and continuous environments for vision-and-language navigation,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. , 2022, pp. 15 439– 15 449

    2022

  14. [22]

    Instruction-aligned hierarchical waypoint planner for vision-and-language navigation in continuous environments,

    Z. He et al. , “Instruction-aligned hierarchical waypoint planner for vision-and-language navigation in continuous environments,” Pattern Anal. Appl., vol. 27, no. 4, p. 132, 2024

    2024

  15. [23]

    Improving vision-and-language navigation with image-text pairs from the web,

    A. Majumdar et al. , “Improving vision-and-language navigation with image-text pairs from the web,” in Computer Vision – ECCV 2020 , 2020, pp. 259–274

    2020

  16. [24]

    Vln bert: A recurrent vision-and-language bert for navigation,

    Y . Hong et al. , “Vln bert: A recurrent vision-and-language bert for navigation,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. , 2021, pp. 1643–1653

    2021

  17. [25]

    Think global, act local: Dual-scale graph transformer for vision-and-language navigation,

    S. Chen et al. , “Think global, act local: Dual-scale graph transformer for vision-and-language navigation,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2022, pp. 16 537–16 547

    2022

  18. [26]

    Multimodal evolutionary encoder for continuous vision- language navigation,

    Z. He et al. , “Multimodal evolutionary encoder for continuous vision- language navigation,” in 2024 IEEE/RSJ Int. Conf. Intell. Robots Syst. , 2024, pp. 1443–1450

    2024

  19. [27]

    Lana: A language-capable navigator for instruction following and generation,

    X. Wang et al. , “Lana: A language-capable navigator for instruction following and generation,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., June 2023, pp. 19 048–19 058. PREPRINT VERSION 12

    2023

  20. [28]

    Pasts: Progress-aware spatio-temporal transformer speaker for vision-and-language navigation,

    L. Wang et al. , “Pasts: Progress-aware spatio-temporal transformer speaker for vision-and-language navigation,” Eng. Appl. Artif. Intell. , vol. 128, p. 107487, 2024

    2024

  21. [29]

    Envedit: Environment editing for vision-and-language navigation,

    J. Li et al. , “Envedit: Environment editing for vision-and-language navigation,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. , 2022, pp. 15 407–15 417

    2022

  22. [30]

    Less is more: Generating grounded navigation instruc- tions from landmarks,

    S. Wang et al., “Less is more: Generating grounded navigation instruc- tions from landmarks,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2022, pp. 15 428–15 438

    2022

  23. [31]

    Learning vision-and-language navigation from youtube videos,

    K. Lin et al. , “Learning vision-and-language navigation from youtube videos,” in Proc. IEEE/CVF Int. Conf. Comput. Vis. , 2023, pp. 8317– 8326

    2023

  24. [32]

    Video captioning: A comparative review of where we are and which could be the route,

    D. Moctezuma et al. , “Video captioning: A comparative review of where we are and which could be the route,” Comput. Vis. Image Understanding, vol. 231, p. 103671, 2023

    2023

  25. [33]

    A review of deep learning for video captioning,

    M. Abdar et al., “A review of deep learning for video captioning,” IEEE Trans. Pattern Anal. Mach. Intell. , 2024

    2024

  26. [34]

    A survey of video datasets for grounded event understanding,

    K. Sanders and B. Van Durme, “A survey of video datasets for grounded event understanding,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2024, pp. 7314–7327

    2024

  27. [35]

    Dual-stream recurrent neural network for video caption- ing,

    N. Xu et al., “Dual-stream recurrent neural network for video caption- ing,” IEEE Trans. Circuits Syst. Video Technol., vol. 29, no. 8, pp. 2482– 2493, 2019

    2019

  28. [36]

    Video captioning using global-local representation,

    L. Yan et al., “Video captioning using global-local representation,” IEEE Trans. Circuits Syst. Video Technol. , vol. 32, no. 10, pp. 6642–6656, 2022

    2022

  29. [37]

    Evcap: Element-aware video captioning,

    S. Liu et al. , “Evcap: Element-aware video captioning,” IEEE Trans. Circuits Syst. Video Technol., vol. 34, no. 10, pp. 9718–9731, 2024

    2024

  30. [38]

    Internvideo2.5: Empowering video mllms with long and rich context modeling,

    Y . Wang et al. , “Internvideo2.5: Empowering video mllms with long and rich context modeling,” arXiv preprint arXiv:2501.12386 , 2025

    2025 arXiv

  31. [39]

    Qwen2-vl: Enhancing vision-language model’s percep- tion of the world at any resolution,

    P. Wang et al., “Qwen2-vl: Enhancing vision-language model’s percep- tion of the world at any resolution,” arXiv preprint arXiv:2409.12191 , 2024

    2024 arXiv

  32. [40]

    Msr-vtt: A large video description dataset for bridging video and language,

    J. Xu et al. , “Msr-vtt: A large video description dataset for bridging video and language,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2016, pp. 5288–5296

    2016

  33. [41]

    Frozen in time: A joint video and image encoder for end-to-end retrieval,

    M. Bain et al. , “Frozen in time: A joint video and image encoder for end-to-end retrieval,” in Proc. IEEE/CVF Int. Conf. Comput. Vis. , 2021, pp. 1728–1738

    2021

  34. [42]

    Beyond the nav-graph: Vision-and-language navigation in continuous environments,

    J. Krantz et al., “Beyond the nav-graph: Vision-and-language navigation in continuous environments,” in Computer Vision – ECCV 2020 , 2020, pp. 104–120

    2020

  35. [43]

    Deep residual learning for image recognition,

    K. He et al. , “Deep residual learning for image recognition,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. , 2016, pp. 770–778

    2016

  36. [44]

    Object recognition from local scale-invariant features,

    D. Lowe, “Object recognition from local scale-invariant features,” in Proc. 7th IEEE Int. Conf. Comput. Vis. , vol. 2, 1999, pp. 1150–1157 vol.2

    1999

  37. [45]

    Fast approximate nearest neighbors with au- tomatic algorithm configuration,

    M. Muja and D. G. Lowe, “Fast approximate nearest neighbors with au- tomatic algorithm configuration,” in VISAPP (1), INSTICC. SciTePress, 2009, pp. 331–340

    2009

  38. [46]

    Revisiting weakly supervised pre-training of visual perception models,

    M. Singh et al. , “Revisiting weakly supervised pre-training of visual perception models,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., June 2022, pp. 804–814

    2022

  39. [47]

    BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,

    J. Li et al. , “BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” in Proc. 40th Int. Conf. Mach. Learn. , vol. 202, 23–29 Jul 2023, pp. 19 730–19 742

    2023

  40. [48]

    End-to-end object detection with transformers,

    N. Carion et al. , “End-to-end object detection with transformers,” in Computer Vision – ECCV 2020, A. Vedaldi et al., Eds. Cham: Springer International Publishing, 2020, pp. 213–229

    2020

  41. [49]

    Visual instruction tuning,

    H. Liu et al., “Visual instruction tuning,” Adv. Neural Inf. Process. Syst., vol. 36, pp. 34 892–34 916, 2023

    2023

  42. [50]

    Bleu: a method for automatic evaluation of machine translation,

    K. Papineni et al., “Bleu: a method for automatic evaluation of machine translation,” in Proc. 40th Annu. Meeting Assoc. Comput. Linguistics , 2002, pp. 311–318

    2002

  43. [51]

    Learning transferable visual models from natural language supervision,

    A. Radford et al. , “Learning transferable visual models from natural language supervision,” in Proc. 38th Int. Conf. Mach. Learn. , vol. 139. PMLR, 18–24 Jul 2021, pp. 8748–8763

    2021

  44. [52]

    Cutting the gordian knot: The moving-average type–token ratio (mattr),

    M. A. Covington and J. D. McFall, “Cutting the gordian knot: The moving-average type–token ratio (mattr),” J. Quantitative Linguistics , vol. 17, no. 2, pp. 94–100, 2010. [Online]. Available: https: //doi.org/10.1080/09296171003643098

    2010 doi

  45. [53]

    Locally typical sampling,

    C. Meister et al. , “Locally typical sampling,” Trans. Assoc. Comput. Linguistics, vol. 11, pp. 102–121, 01 2023. [Online]. Available: https://doi.org/10.1162/tacl a 00536

    2023 doi

  46. [54]

    Texygen: A benchmarking platform for text generation models,

    Y . Zhu et al. , “Texygen: A benchmarking platform for text generation models,” in 41st Int. ACM SIGIR Conf. Res. & Develop. Inf. Retrieval , New York, NY , USA, 2018, p. 1097–1100

    2018

  47. [55]

    Standardizing the measurement of text diversity: A tool and a comparative analysis of scores,

    C. Shaib et al., “Standardizing the measurement of text diversity: A tool and a comparative analysis of scores,” arXiv preprint arXiv:2403.00553, 2024

    2024

  48. [56]

    Dinov2: Learning robust visual features without supervision,

    M. Oquab et al. , “Dinov2: Learning robust visual features without supervision,” arXiv:2304.07193, 2023

    2023 arXiv

  49. [57]

    Masked autoencoders are scalable vision learners,

    K. He et al. , “Masked autoencoders are scalable vision learners,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. , June 2022, pp. 16 000–16 009

    2022

  50. [58]

    Places: A 10 million image database for scene recognition,

    B. Zhou et al. , “Places: A 10 million image database for scene recognition,” IEEE Trans. Pattern Anal. Mach. Intell. , vol. 40, no. 6, pp. 1452–1464, 2018

    2018

  51. [59]

    Llava-next: Improved reasoning, ocr, and world knowledge,

    H. Liu et al. , “Llava-next: Improved reasoning, ocr, and world knowledge,” January 2024. [Online]. Available: https://llava-vl.github. io/blog/2024-01-30-llava-next/

    2024

  52. [60]

    Qwen2.5-vl technical report,

    S. Bai et al. , “Qwen2.5-vl technical report,” arXiv preprint arXiv:2502.13923, 2025

    2025 arXiv

  53. [61]

    GPT-4o mini: advancing cost-efficient in- telligence

    OpenAI, “GPT-4o mini: advancing cost-efficient in- telligence.” [Online]. Available: https://openai.com/index/ gpt-4o-mini-advancing-cost-efficient-intelligence/

  54. [62]

    The llama 3 herd of models,

    A. Grattafiori et al. , “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  55. [63]

    Gemma 2: Improving open language models at a practical size,

    G. Team et al. , “Gemma 2: Improving open language models at a practical size,” arXiv preprint arXiv:2408.00118 , 2024

    2024 arXiv

  56. [64]

    Qwen2.5 technical report,

    A. Yang et al. , “Qwen2.5 technical report,” arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  57. [65]

    Openclip,

    G. Ilharco et al. , “Openclip,” Jul. 2021. [Online]. Available: https://doi.org/10.5281/zenodo.5143773

    2021 doi

  58. [66]

    Torchmetrics - measuring reproducibility in pytorch,

    N. S. Detlefsen et al. , “Torchmetrics - measuring reproducibility in pytorch,” J. Open Source Softw., vol. 7, no. 70, p. 4101, 2022. [Online]. Available: https://doi.org/10.21105/joss.04101

    2022 doi

  59. [67]

    Coca: Contrastive captioners are image-text foundation models,

    J. Yu et al. , “Coca: Contrastive captioners are image-text foundation models,” Trans. Mach. Learn. Res. , 2022

    2022

  60. [68]

    Llava-next: A strong zero-shot video understanding model,

    Y . Zhang et al. , “Llava-next: A strong zero-shot video understanding model,” April 2024. [Online]. Available: https://llava-vl.github.io/blog/ 2024-04-30-llava-next-video/

    2024

  61. [69]

    Habitat-matterport 3d dataset (HM3d): 1000 large-scale 3d environments for embodied AI,

    S. K. Ramakrishnan et al., “Habitat-matterport 3d dataset (HM3d): 1000 large-scale 3d environments for embodied AI,” inProc. 35th Adv. Neural Inf. Process. Syst. Datasets and Benchmarks Track , 2021

    2021

  62. [70]

    Scenenet rgb-d: Can 5m synthetic images beat generic imagenet pre-training on indoor segmentation?

    J. McCormac et al. , “Scenenet rgb-d: Can 5m synthetic images beat generic imagenet pre-training on indoor segmentation?” in Proc. IEEE Int. Conf. Comput. Vis. , 2017, pp. 2678–2687

    2017

  63. [71]

    Grutopia: Dream general robots in a city at scale,

    H. Wang et al. , “Grutopia: Dream general robots in a city at scale,” arXiv preprint arXiv:2407.10943 , 2024

    2024 arXiv

  64. [72]

    Sun3d: A database of big spaces reconstructed using sfm and object labels,

    J. Xiao et al. , “Sun3d: A database of big spaces reconstructed using sfm and object labels,” in Proc. IEEE Int. Conf. Comput. Vis., 2013, pp. 1625–1632

    2013

  65. [73]

    Scannet: Richly-annotated 3d reconstructions of indoor scenes,

    A. Dai et al. , “Scannet: Richly-annotated 3d reconstructions of indoor scenes,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2017, pp. 5828–5839

    2017

  66. [74]

    A benchmark for the evaluation of rgb-d slam systems,

    J. Sturm et al., “A benchmark for the evaluation of rgb-d slam systems,” in 2012 IEEE/RSJ Int. Conf. Intell. Robots Syst. IEEE, 2012, pp. 573– 580

    2012

  67. [75]

    Learning to navigate the energy landscape,

    J. Valentin et al., “Learning to navigate the energy landscape,” in 2016 4th Int. Conf. 3D Vis. IEEE, 2016, pp. 323–332

    2016

  68. [76]

    DIODE: A Dense Indoor and Outdoor DEpth Dataset,

    I. Vasiljevic et al. , “DIODE: A Dense Indoor and Outdoor DEpth Dataset,” CoRR, vol. abs/1908.00463, 2019

    1908 arXiv

  69. [77]

    Vision meets robotics: The kitti dataset,

    A. Geiger et al., “Vision meets robotics: The kitti dataset,” Int. J. Robot. Res., 2013

    2013

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.