REVIEW 7 cited by
Robot Utility Models: General Policies for Zero-Shot Deployment in New Environments
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Robot models, particularly those trained with large amounts of data, have recently shown a plethora of real-world manipulation and navigation capabilities. Several independent efforts have shown that given sufficient training data in an environment, robot policies can generalize to demonstrated variations in that environment. However, needing to finetune robot models to every new environment stands in stark contrast to models in language or vision that can be deployed zero-shot for open-world problems. In this work, we present Robot Utility Models (RUMs), a framework for training and deploying zero-shot robot policies that can directly generalize to new environments without any finetuning. To create RUMs efficiently, we develop new tools to quickly collect data for mobile manipulation tasks, integrate such data into a policy with multi-modal imitation learning, and deploy policies on-device on Hello Robot Stretch, a cheap commodity robot, with an external mLLM verifier for retrying. We train five such utility models for opening cabinet doors, opening drawers, picking up napkins, picking up paper bags, and reorienting fallen objects. Our system, on average, achieves 90% success rate in unseen, novel environments interacting with unseen objects. Moreover, the utility models can also succeed in different robot and camera set-ups with no further data, training, or fine-tuning. Primary among our lessons are the importance of training data over training algorithm and policy class, guidance about data scaling, necessity for diverse yet high-quality demonstrations, and a recipe for robot introspection and retrying to improve performance on individual environments. Our code, data, models, hardware designs, as well as our experiment and deployment videos are open sourced and can be found on our project website: https://robotutilitymodels.com
Forward citations
Cited by 7 Pith papers
-
FLOWER: Democratizing Generalist Robot Policies with Efficient Vision-Language-Action Flow Policies
A compact 950-million-parameter robot policy trained in about 200 GPU-hours matches or beats multi-billion-parameter baselines on most manipulation benchmarks, including a new best score on CALVIN ABC.
-
Advancing Learnable Multi-Agent Pathfinding Solvers with Active Fine-Tuning
A 2M-parameter MAPF-GPT model fine-tuned with delta-driven data generation matches or exceeds much larger models on MAPF benchmarks and scales to over a million agents.
-
ParticleFormer: A 3D Point Cloud World Model for Multi-Object, Multi-Material Robotic Manipulation
ParticleFormer uses a Transformer over point-cloud particles and a hybrid Chamfer-Hausdorff loss to predict multi-material object dynamics, and it reports lower errors than GNN and image-based baselines in simulation ...
-
Vision in Action: Learning Active Perception from Human Demonstrations
ViA trains bimanual manipulation policies from human demonstrations that include active head-camera movement, using a 6-DoF robot neck and a VR interface with point-cloud rendering, reporting large gains on three occl...
-
Touch begins where vision ends: Generalizable policies for contact-rich manipulation
A localize-then-execute policy that combines vision-language reaching, semantic background augmentation, and residual reinforcement learning with tactile sensing reaches about 90% success on millimeter-precision manip...
-
eFlesh: Highly customizable Magnetic Touch Sensing using Cut-Cell Microstructures
eFlesh is a customizable 3D-printed magnetic tactile sensor that localizes contact to 0.5 mm, estimates force within 0.27 N, and boosts precise robot manipulation success to 91%.
-
Vision Language Action (VLA) Models for Unmanned Aerial Robotics and Bimanual Manipulation: A Review
Bimanual VLA coordination strategies, training recipes, and continuous action chunking transfer to unmanned aerial systems; the survey maps 183 works and lists fourteen shared research directions.
Discussion (0). Sign in to comment.