REVIEW 6 cited by
Imitation Bootstrapped Reinforcement Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Despite the considerable potential of reinforcement learning (RL), robotic control tasks predominantly rely on imitation learning (IL) due to its better sample efficiency. However, it is costly to collect comprehensive expert demonstrations that enable IL to generalize to all possible scenarios, and any distribution shift would require recollecting data for finetuning. Therefore, RL is appealing if it can build upon IL as an efficient autonomous self-improvement procedure. We propose imitation bootstrapped reinforcement learning (IBRL), a novel framework for sample-efficient RL with demonstrations that first trains an IL policy on the provided demonstrations and then uses it to propose alternative actions for both online exploration and bootstrapping target values. Compared to prior works that oversample the demonstrations or regularize RL with an additional imitation loss, IBRL is able to utilize high quality actions from IL policies since the beginning of training, which greatly accelerates exploration and training efficiency. We evaluate IBRL on 6 simulation and 3 real-world tasks spanning various difficulty levels. IBRL significantly outperforms prior methods and the improvement is particularly more prominent in harder tasks.
Forward citations
Cited by 6 Pith papers
-
Do You Really Need to Pretrain Q-Functions for Online RL Fine-Tuning?
On six robot-manipulation tasks, offline Q-pretraining does not accelerate online RL fine-tuning from a pretrained policy, while seeding the replay buffer with rollouts from an ensemble of policies (IPE) improves fina...
-
From Prior to Pro: Efficient Skill Mastery via Distribution Contractive RL Finetuning
Residual off-policy RL with selective BC regularization and value-guided sampling contracts a pretrained generative robot policy around successful actions, reaching high success on hard long-horizon tasks from pixels ...
-
SERNF: Sample-Efficient Real-World Dexterous Policy Fine-Tuning via Action-Chunked Critics and Normalizing Flows
SERNF fine-tunes dexterous manipulation policies on real hardware by pairing normalizing-flow policies with action-chunked critics and conservative off-policy RL.
-
Reinforcement Learning via Implicit Imitation Guidance
A reinforcement learning method that learns a state-dependent covariance from expert-policy action differences and uses it as exploration noise, improving sample efficiency on sparse-reward continuous control tasks.
-
Rapidly Adapting Policies to the Real World via Simulation-Guided Fine-Tuning
SGFT uses a simulation-trained value function to guide real-world exploration via potential-based reward shaping and short-horizon objectives, substantially improving fine-tuning sample efficiency.
-
SimLauncher: Launching Sample-Efficient Real-world Robotic Reinforcement Learning via Simulation Pre-training
Simulation-pretrained policies, with digital-twin demos for critic bootstrapping and action proposals, cut real-world RL training time while reaching near-perfect success on three manipulation tasks.
Discussion (0). Continue with ORCID to comment.