REVIEW 21 cited by
ViNT: A Foundation Model for Visual Navigation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
General-purpose pre-trained models ("foundation models") have enabled practitioners to produce generalizable solutions for individual machine learning problems with datasets that are significantly smaller than those required for learning from scratch. Such models are typically trained on large and diverse datasets with weak supervision, consuming much more training data than is available for any individual downstream application. In this paper, we describe the Visual Navigation Transformer (ViNT), a foundation model that aims to bring the success of general-purpose pre-trained models to vision-based robotic navigation. ViNT is trained with a general goal-reaching objective that can be used with any navigation dataset, and employs a flexible Transformer-based architecture to learn navigational affordances and enable efficient adaptation to a variety of downstream navigational tasks. ViNT is trained on a number of existing navigation datasets, comprising hundreds of hours of robotic navigation from a variety of different robotic platforms, and exhibits positive transfer, outperforming specialist models trained on singular datasets. ViNT can be augmented with diffusion-based subgoal proposals to explore novel environments, and can solve kilometer-scale navigation problems when equipped with long-range heuristics. ViNT can also be adapted to novel task specifications with a technique inspired by prompt-tuning, where the goal encoder is replaced by an encoding of another task modality (e.g., GPS waypoints or routing commands) embedded into the same space of goal tokens. This flexibility and ability to accommodate a variety of downstream problem domains establishes ViNT as an effective foundation model for mobile robotics. For videos, code, and model checkpoints, see our project page at https://visualnav-transformer.github.io.
Forward citations
Cited by 21 Pith papers
-
Weights or Skills? A Survey of Robot-Learning Techniques: from Action-Predicting Weights to Robots that Write their Own Skills
A taxonomy of robot learning on a weights-versus-skills axis, with a five-rung self-improvement ladder whose top cell (feedback plus memory plus search) holds only a few recent systems.
-
ACME: A Multi-Cultural, Multi-Embodiment Social-Navigation Dataset
ACME is a socially navigated robot and pedestrian trajectory dataset covering 8 sites in 5 countries with 7 robot embodiments, including human-verified BEV tracks and robot speech annotations.
-
G2-Nav: Grounded and Guarded Vision-Language Costmaps for Robot Social Navigation
G2-Nav turns vision-language reasoning about social scenes into a weighted costmap with a safety reflex layer, tested in recorded and live real-world trials.
-
SeeSE3: Emergence of 3D Space in Vision Features
Self-supervised vision features, especially DINOv2, contain a subspace that a small trained adapter can map to 3D camera motion, enabling pose estimation and latent-space navigation without explicit 3D reconstruction.
-
Learning to Navigate Efficiently with Only 0.58M Trainable Parameters
Decomposed navigation with analytical geometry interfaces and three small learned modules (0.58M trainable params) approaches SOTA point-goal performance at 50 Hz with lowest collisions.
-
RAE-NWM: Navigation World Model in Dense Visual Representation Space
Navigation world models trained in dense DINOv2 space with flow-matching CDiT-DH and time-gated action injection improve structural stability and planning over VAE baselines.
-
Approximate Imitation Learning for Event-based Quadrotor Flight in Cluttered Environments
Approximate imitation learning trains event-to-control quadrotor policies 28× faster by freezing a pretrained event encoder and fine-tuning a shared action decoder via a state-based approximate student, matching onlin...
-
Learning to Localize Reference Trajectories in Image-Space for Visual Navigation
LoTIS localizes a reference RGB trajectory in the robot's current view, predicting image-space coordinates, visibility, and distance to provide robot-agnostic guidance for navigation.
-
CostNav: A Navigation Benchmark for Real-World Economic-Cost Evaluation of Physical AI Agents
A navigation benchmark that converts simulated collisions, energy use, and delivery outcomes into dollar costs and revenue, and finds rule-based delivery robot baselines unprofitable per run.
-
MM-Nav: Multi-View VLA Model for Robust Visual Navigation via Multi-Expert Learning
A four-camera VLA navigation model trained by distilling multiple RL experts achieves strong simulation performance and qualitative real-world transfer.
-
CAST: Counterfactual Labels Improve Instruction Following in Vision-Language-Action Models
Counterfactual language-action relabeling raises instruction-following success from about 26% to 53% in real-world navigation tests.
-
HALO: Human Preference Aligned Offline Reward Learning for Robot Navigation
HALO learns a vision-based navigation reward from human preference rankings on egocentric video, and an IQL policy using it beats several baselines in 10-trial real-world tests.
-
Narrate2Nav: Real-Time Visual Navigation with Implicit Language Reasoning in Human-Centric Environments
Narrate2Nav uses Barlow Twins alignment to distill language-based reasoning from a large teacher into a small RGB-only navigation model, reporting lower trajectory error and higher goal-reaching success than four baselines.
-
UniNav: A Unified World-Action Diffusion Model for Visual Navigation
A unified diffusion transformer that jointly denoises future frames, waypoints, and geometry tokens improves image-goal navigation accuracy and can omit image tokens at test time for 0.1s latency.
-
DUViN: Diffusion-Based Underwater Visual Navigation via Knowledge-Transferred Depth Features
An in-air trained diffusion navigation policy, whose depth encoder is fine-tuned on underwater depth estimation, drives a BlueROV2 through cluttered water using only camera images.
-
From Seeing to Experiencing: Scaling Navigation Foundation Models with Reinforcement Learning
Pretraining a navigation foundation model on videos and then fine-tuning only a residual attention module with reinforcement learning improves success rate and collision avoidance compared to behavior cloning or super...
-
Skill-Nav: Enhanced Navigation with Versatile Quadrupedal Locomotion via Waypoint Interface
A waypoint-based interface between planners and a trained quadrupedal locomotion policy enables navigation over diverse obstacles in simulation and on a real robot.
-
OBSER: Object-Based Sub-Environment Recognition for Zero-Shot Environmental Inference
Object-based environment inference via kernel density estimates on learned object features achieves zero-shot room retrieval and beats scene-based CLIP.
-
ThermalLoc: A Vision Transformer-Based Approach for Robust Thermal Camera Relocalization in Large-Scale Environments
ThermalLoc combines EfficientNet and a Vision Transformer to regress 6-DoF camera poses from single thermal images, reporting improved relocalization accuracy on city-scale thermal datasets.
-
PIG-Nav: Key Insights for Pretrained Image Goal Navigation Models
A pretrained image-goal navigation model combining early-fusion ViT, auxiliary objectives, and game-video data reports higher success than GNM, ViNT, and NoMaD, though zero-shot generalization is clouded by possible p...
-
MOSU: Autonomous Long-range Robot Navigation with Multi-modal Scene Understanding
MOSU improves outdoor robot traversability by about 10% over prior state-of-the-art on the GND benchmark by fusing geometric, semantic, and vision-language-model scores for candidate trajectories.
Discussion (0). Sign in to comment.