REVIEW 12 cited by
Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We present relay policy learning, a method for imitation and reinforcement learning that can solve multi-stage, long-horizon robotic tasks. This general and universally-applicable, two-phase approach consists of an imitation learning stage that produces goal-conditioned hierarchical policies, and a reinforcement learning phase that finetunes these policies for task performance. Our method, while not necessarily perfect at imitation learning, is very amenable to further improvement via environment interaction, allowing it to scale to challenging long-horizon tasks. We simplify the long-horizon policy learning problem by using a novel data-relabeling algorithm for learning goal-conditioned hierarchical policies, where the low-level only acts for a fixed number of steps, regardless of the goal achieved. While we rely on demonstration data to bootstrap policy learning, we do not assume access to demonstrations of every specific tasks that is being solved, and instead leverage unstructured and unsegmented demonstrations of semantically meaningful behaviors that are not only less burdensome to provide, but also can greatly facilitate further improvement using reinforcement learning. We demonstrate the effectiveness of our method on a number of multi-stage, long-horizon manipulation tasks in a challenging kitchen simulation environment. Videos are available at https://relay-policy-learning.github.io/
Forward citations
Cited by 12 Pith papers
-
Weights or Skills? A Survey of Robot-Learning Techniques: from Action-Predicting Weights to Robots that Write their Own Skills
A taxonomy of robot learning on a weights-versus-skills axis, with a five-rung self-improvement ladder whose top cell (feedback plus memory plus search) holds only a few recent systems.
-
Learning Task-Sufficient World Models by Synergizing Agentic Exploration and Structured Modeling
Closed-loop agentic probing plus minimality/sufficiency masking recovers compact task-sufficient world-model latents that improve sample-efficient policy learning and cross-task generalization.
-
DASIP: Dynamic Test-Time Compute Scaling for Robot Control with Stochastic Interpolant Policies
A difficulty classifier adaptively selects step count, solver, and ODE/SDE mode for stochastic-interpolant robot policies, reporting 2.6–4.4x compute savings with roughly unchanged success rates.
-
Learning Upper Lower Value Envelopes to Shape Online RL: A Principled Approach
A two-stage RL framework learns value-function envelopes from offline data and uses them to shape online exploration, yielding regret bounds that improve as offline data grows.
-
D3P: Dynamic Denoising Diffusion Policy via Reinforcement Learning
D3P learns to adaptively allocate denoising steps per robot action, matching fixed-step diffusion policy performance at 2.2x lower inference cost.
-
Learning Temporal Abstractions via Variational Homomorphisms in Option-Induced Abstract MDPs
A variational option-critic algorithm with latent option embeddings and an implicit chain-of-thought cold-start is presented; the central optimality-preservation proof has a gap and some reported benchmark wins are in...
-
FOUNDER: Grounding Foundation Models in World Models for Open-Ended Embodied Decision Making
FOUNDER maps foundation-model embeddings of text or video prompts into world-model goal states and rewards policies by predicted temporal distance to those goals, improving reward-free multi-task offline control.
-
Relative Value Learning
A critic that learns antisymmetric value differences ∆(s_i,s_j)=V(s_i)−V(s_j) has a provably contracting Bellman operator and an unbiased advantage estimator, and PPO with this critic matches standard PPO on Atari.
-
Dual-Process Atomic Skill Learning: Decoupling Semantic Reasoning and Real-Time Control
Asynchronous dual-frequency hierarchical imitation learning with VQ skills and training-only latent diffusion improves compositional language-conditioned robot control and reduces skill codebook collapse.
-
VQ-VLA: Improving Vision-Language-Action Models via Scaling Vector-Quantized Action Tokenizers
A convolutional residual VQ-VAE action tokenizer trained on over 100x more data than prior work improves OpenVLA success rates and inference speed on several manipulation tasks.
-
Steering Robots with Inference-Time Interactions
Frozen imitation policies can be steered at inference time via user interactions, with a diffusion-sampling method and a constraint-enforcing framework that provides formal task guarantees.
-
Data Pyramid for Embodied Manipulation
Embodied training data form a five-layer pyramid—real-robot, UMI, ego/exo, simulation, general V–L—ordered by the trade-off between scale and robot alignment, and model capabilities track how those layers are mixed.
Discussion (0). Sign in to comment.