REVIEW 6 cited by
Guiding Long-Horizon Task and Motion Planning with Vision Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Guiding Long-Horizon Task and Motion Planning with Vision Language Models
read the original abstract
Vision-Language Models (VLM) can generate plausible high-level plans when prompted with a goal, the context, an image of the scene, and any planning constraints. However, there is no guarantee that the predicted actions are geometrically and kinematically feasible for a particular robot embodiment. As a result, many prerequisite steps such as opening drawers to access objects are often omitted in their plans. Robot task and motion planners can generate motion trajectories that respect the geometric feasibility of actions and insert physically necessary actions, but do not scale to everyday problems that require common-sense knowledge and involve large state spaces comprised of many variables. We propose VLM-TAMP, a hierarchical planning algorithm that leverages a VLM to generate goth semantically-meaningful and horizon-reducing intermediate subgoals that guide a task and motion planner. When a subgoal or action cannot be refined, the VLM is queried again for replanning. We evaluate VLM- TAMP on kitchen tasks where a robot must accomplish cooking goals that require performing 30-50 actions in sequence and interacting with up to 21 objects. VLM-TAMP substantially outperforms baselines that rigidly and independently execute VLM-generated action sequences, both in terms of success rates (50 to 100% versus 0%) and average task completion percentage (72 to 100% versus 15 to 45%). See project site https://zt-yang.github.io/vlm-tamp-robot/ for more information.
Forward citations
Cited by 6 Pith papers
-
VoLo: A Physical Orchestrator for Open-Vocabulary Long-Horizon Manipulation
VoLoAgent uses a VLM to steer heterogeneous robot capabilities as interruptible tools for long-horizon manipulation and introduces the RoboVoLo benchmark, claiming substantial outperformance over single VLA/VLM or too...
-
LENS: LLM-guided Environment Simplification for Planning and Control in Clutter
A vision-language-model-based prune-and-merge abstraction improves success and runtime for TAMP, contact-implicit MPC, and a VLA policy in cluttered tabletop manipulation.
-
APIVOT: Adaptive Planning with Interleaved Vision-Language Thoughts
A VLM planner that adaptively inserts latent visual thoughts of future states into its reasoning trace beats language-only and prior VLM planners on long-horizon kitchen tasks, especially under tight free space.
-
TiPToP: A Modular Open-Vocabulary Robot Manipulation System That Plans
TiPToP, a zero-training modular planner using pretrained vision-language models and GPU-accelerated TAMP, achieves 74.6% success over 165 trials versus 52.4% for the 350-hour-trained pi0.5-DROID baseline across 28 man...
-
Robix: A Unified Model for Robot Interaction, Reasoning and Planning
A three-stage-trained VLM unifies robot planning and dialogue, and beats commercial VLMs on the authors' interactive-task benchmarks.
-
A Scalable Embodied Intelligence Platform for Seamless Real-to-Sim-to-Real Transfer of Household Mobile Manipulation Tasks
BestMan is a robotics platform with ASG for scene reconstruction, simulation-guided skill learning, and HUM middleware to enable seamless real-to-sim-to-real transfer in household mobile manipulation.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.