Pith. sign in

REVIEW 10 cited by

PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2106.05091 v1 pith:EOQKHRN2 submitted 2021-06-09 cs.LG cs.AI

Kimin Lee , Laura Smith , Pieter Abbeel This is my paper · ORCID
classification cs.LGcs.AI
keywords learningrewardfeedbackagentsoff-policyagentdifficultexperience
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Conveying complex objectives to reinforcement learning (RL) agents can often be difficult, involving meticulous design of reward functions that are sufficiently informative yet easy enough to provide. Human-in-the-loop RL methods allow practitioners to instead interactively teach agents through tailored feedback; however, such approaches have been challenging to scale since human feedback is very expensive. In this work, we aim to make this process more sample- and feedback-efficient. We present an off-policy, interactive RL algorithm that capitalizes on the strengths of both feedback and off-policy learning. Specifically, we learn a reward model by actively querying a teacher's preferences between two clips of behavior and use it to train an agent. To enable off-policy learning, we relabel all the agent's past experience when its reward model changes. We additionally show that pre-training our agents with unsupervised exploration substantially increases the mileage of its queries. We demonstrate that our approach is capable of learning tasks of higher complexity than previously considered by human-in-the-loop methods, including a variety of locomotion and robotic manipulation skills. We also show that our method is able to utilize real-time human feedback to effectively prevent reward exploitation and learn new behaviors that are difficult to specify with standard reward functions.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. RuleEdit: Failure-Guided Human-AI Model Editing with Prospective Impact Preview

    cs.HC 2026-04 conditional novelty 6.0 of 10

    Rule-guided mismatch cues raise Human+AI rehab-assessment accuracy by 14% and cut harmful reliance; prospective embedding previews raise local model-edit gains from 11.5% to 36%, with global transfer often regressing.

  2. HALO: Human Preference Aligned Offline Reward Learning for Robot Navigation

    cs.RO 2025-08 reject novelty 6.0 of 10

    HALO learns a vision-based navigation reward from human preference rankings on egocentric video, and an IQL policy using it beats several baselines in 10-trial real-world tests.

  3. CueLearner: Bootstrapping and local policy adaptation from relative feedback

    cs.RO 2025-07 conditional novelty 6.0 of 10

    CueLearner learns a relative-feedback model from a small number of human directional corrections and uses it to guide off-policy RL exploration or refine a deployed policy.

  4. CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queries

    cs.LG 2025-05 conditional novelty 6.0 of 10

    CLARIFY uses contrastive learning on preference data to embed trajectories, then rejection-samples queries that humans can distinguish clearly, improving offline preference-based RL.

  5. Efficiently Generating Expressive Quadruped Behaviors via Language-Guided Preference Learning

    cs.RO 2025-02 conditional novelty 6.0 of 10

    A hybrid method uses LLM-generated candidate gaits and then refines them with a few human preference rankings, achieving quadruped behaviors aligned with user intent in as few as four queries.

  6. Learning from Active Human Involvement through Proxy Value Propagation

    cs.AI 2025-02 conditional novelty 6.0 of 10

    A reward-free human-in-the-loop RL method that labels human demonstrations with high Q values and intervened agent actions with low Q values, then propagates these values through TD learning to train policies across d...

  7. Reward Evolution with Graph-of-Thoughts: A Bi-Level Language Model Framework for Reinforcement Learning

    cs.RO 2025-09 conditional novelty 5.0 of 10

    RE-GoT combines graph-of-thoughts planning in LLMs with VLM feedback from rollout videos to automatically write and refine RL reward functions, beating prior LLM-based reward design on RoboGen and ManiSkill2.

  8. Residual Reward Models for Preference-based Reinforcement Learning

    cs.LG 2025-07 conditional novelty 5.0 of 10

    Combining a hand-designed or learned prior reward with a preference-trained residual improves sample efficiency and final performance in preference-based reinforcement learning.

  9. Speaking the Language of Teamwork: LLM-Guided Credit Assignment in Multi-Agent Reinforcement Learning

    cs.MA 2025-02 conditional novelty 5.0 of 10

    LLM-generated, agent-specific potential-based rewards accelerate sparse-reward MARL training in grid world and pistonball benchmarks.

  10. Active Query Selection for Crowd-Based Reinforcement Learning

    cs.LG 2025-08 conditional novelty 4.0 of 10

    Extending the Advise algorithm with variational crowd modelling and entropy-based query selection yields faster learning in small tabular RL tasks, especially highly constrained ones.

Pith tools