ImagineNav: Prompting Vision-Language Models as Embodied Navigator through Scene Imagination
Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel pith:FXZGQA4Urecord.jsonopen to challenge →
read the original abstract
Visual navigation is an essential skill for home-assistance robots, providing the object-searching ability to accomplish long-horizon daily tasks. Many recent approaches use Large Language Models (LLMs) for commonsense inference to improve exploration efficiency. However, the planning process of LLMs is limited within texts and it is difficult to represent the spatial occupancy and geometry layout only by texts. Both are important for making rational navigation decisions. In this work, we seek to unleash the spatial perception and planning ability of Vision-Language Models (VLMs), and explore whether the VLM, with only on-board camera captured RGB/RGB-D stream inputs, can efficiently finish the visual navigation tasks in a mapless manner. We achieve this by developing the imagination-powered navigation framework ImagineNav, which imagines the future observation images at valuable robot views and translates the complex navigation planning process into a rather simple best-view image selection problem for VLM. To generate appropriate candidate robot views for imagination, we introduce the Where2Imagine module, which is distilled to align with human navigation habits. Finally, to reach the VLM preferred views, an off-the-shelf point-goal navigation policy is utilized. Empirical experiments on the challenging open-vocabulary object navigation benchmarks demonstrates the superiority of our proposed system.
This paper has not been read by Pith yet.
Forward citations
Cited by 5 Pith papers
-
EAGOR: Embodied Reasoning in Omni-direction
EAGOR reformulates embodied 360-degree directional reasoning as recursive Bayesian estimation on a spherical manifold using spherical harmonics, achieving training-free, rotation-equivariant target tracking.
-
ProCompNav: Proactive Instance Navigation with Comparative Judgment for Ambiguous User Queries
ProCompNav disambiguates ambiguous instance navigation queries via candidate-pool construction followed by attribute-based comparative binary questions that prune distractors, yielding higher success rates and shorter...
-
ProCompNav: Proactive Instance Navigation with Comparative Judgment for Ambiguous User Queries
ProCompNav improves success rate and shortens user responses in ambiguous instance navigation by using comparative binary questions that prune a candidate pool rather than requesting detailed descriptions.
-
Hierarchical 3D Scene Graph Construction and Belief-based Planning for Semantic Navigation
Proposes online hierarchical 3D scene graph construction paired with belief-based planning to improve zero-shot semantic navigation performance in unseen environments.
-
ProCompNav: Proactive Instance Navigation with Comparative Judgment for Ambiguous User Queries
ProCompNav builds a candidate pool from ambiguous queries then uses pool-splitting binary questions for disambiguation, improving success rate and shortening responses on CoIN-Bench and TextNav.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.