REVIEW 17 cited by
RoboCat: A Self-Improving Generalist Agent for Robotic Manipulation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The ability to leverage heterogeneous robotic experience from different robots and tasks to quickly master novel skills and embodiments has the potential to transform robot learning. Inspired by recent advances in foundation models for vision and language, we propose a multi-embodiment, multi-task generalist agent for robotic manipulation. This agent, named RoboCat, is a visual goal-conditioned decision transformer capable of consuming action-labelled visual experience. This data spans a large repertoire of motor control skills from simulated and real robotic arms with varying sets of observations and actions. With RoboCat, we demonstrate the ability to generalise to new tasks and robots, both zero-shot as well as through adaptation using only 100-1000 examples for the target task. We also show how a trained model itself can be used to generate data for subsequent training iterations, thus providing a basic building block for an autonomous improvement loop. We investigate the agent's capabilities, with large-scale evaluations both in simulation and on three different real robot embodiments. We find that as we grow and diversify its training data, RoboCat not only shows signs of cross-task transfer, but also becomes more efficient at adapting to new tasks.
Forward citations
Cited by 17 Pith papers
-
Weights or Skills? A Survey of Robot-Learning Techniques: from Action-Predicting Weights to Robots that Write their Own Skills
A taxonomy of robot learning on a weights-versus-skills axis, with a five-rung self-improvement ladder whose top cell (feedback plus memory plus search) holds only a few recent systems.
-
DeVA: Decoupled Video-Action Model with physical guidance for robot policy learning
Decoupling video dynamics from action prediction with multi-level feature transfer and affordance/depth guidance yields more data-efficient robot policies than coupled video-action models.
-
Try Once, Then Optimal: De-Redundified Procedure Memory for Cross-Episode Exploration Amortization
Object-centric procedure memory amortizes hidden-state exploration across encounters, cutting robot manipulation operations 16–30% at non-regressing success.
-
WCM: World-Cognition Model for Generalizable Human-Robot Interaction
WCM, a modular robot system with asynchronous execution and chain-of-thought distillation, reports 73.8% success across nine interactive tasks and learns a long-horizon task from natural-language teaching.
-
TempoVLA: Learning Speed-Controllable Vision-Language-Action Policies
TempoVLA learns a single VLA policy with controllable execution speed via variable-speed trajectory augmentation and explicit speed conditioning.
-
Humanoid Everyday: A Comprehensive Robotic Dataset for Open-World Humanoid Manipulation
A 10,300-demonstration, 260-task multimodal humanoid manipulation dataset with baseline policy evaluations and a cloud evaluation platform.
-
FLOWER: Democratizing Generalist Robot Policies with Efficient Vision-Language-Action Flow Policies
A compact 950-million-parameter robot policy trained in about 200 GPU-hours matches or beats multi-billion-parameter baselines on most manipulation benchmarks, including a new best score on CALVIN ABC.
-
Latent Policy Barrier: Learning Robust Visuomotor Policies by Staying In-Distribution
Latent Policy Barrier improves behavior-cloned visuomotor policies by using a latent dynamics model trained on expert and rollout data to guide actions back toward in-distribution expert states.
-
VLM-TDP: VLM-guided Trajectory-conditioned Diffusion Policy for Robust Long-Horizon Manipulation
VLM-TDP guides a diffusion-based robot policy with VLM-generated voxel trajectories, improving success rates by roughly 30-44% and adding robustness to noise and scene changes.
-
AnyBody: A Benchmark Suite for Cross-Embodiment Manipulation
AnyBody is a benchmark suite that tests cross-embodiment manipulation generalization along interpolation, extrapolation, and composition axes, and finds zero-shot generalization to unseen robot bodies remains difficult.
-
Towards Human-like Physical Intelligence: Lifelong Vision-Language-Action Learning for Robotic Manipulation
LifelongVLA pairs dual-timescale LoRA gating with stochastic cached-prefix replay to cut catastrophic forgetting in VLA policies, reporting 83.2% average success and 11.4% forgetting on a 10-task LIBERO stream.
-
DatasetAgent: A Novel Multi-Agent System for Auto-Constructing Datasets from Real-World Images
DatasetAgent is an LLM-powered multi-agent pipeline that automatically constructs image classification, detection, and segmentation datasets from web images, with modest downstream gains shown but weak experimental controls.
-
Combining Pre-Trained Models for Enhanced Feature Representation in Reinforcement Learning
A lightweight attention module that weights embeddings from multiple pre-trained models achieves comparable Atari RL performance to end-to-end training, with improved robustness to visual changes.
-
RoboPearls: Editable Video Simulation for Robot Manipulation
RoboPearls is a 3D Gaussian Splatting based framework that edits demonstration videos into varied photorealistic simulations, and training on them improves robot manipulation success rates on RLBench and COLOSSEUM.
-
Large Model Empowered Embodied AI: A Survey on Decision-Making and Embodied Learning
A review that categorizes large-model-empowered embodied AI into hierarchical and end-to-end decision-making, imitation and reinforcement learning, and world models.
-
Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunities
A tutorial reviewing LLM alignment through the lens of inverse reinforcement learning, arguing that neural reward models learned from human data are central to post-training.
-
Perspective on Utilizing Foundation Models for Laboratory Automation in Materials Research
A perspective article reviews the state of using foundation models for laboratory automation and proposes a roadmap for fully autonomous experiments.
Discussion (0). Continue with ORCID to comment.