Pith. sign in

REVIEW 3 major objections 5 minor 59 references

Residual Reward Models for Preference-based Reinforcement Learning

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a Residual Reward Model—a fixed prior reward plus a learned residual fit to preference data—substantially improves the success rate and feedback efficiency of preference-based reinforcement learning on simulated and…

desk verdict Useful empirical study, but the residual architecture's benefit is not isolated from the prior-reward injection; needs a PEBBLE+prior baseline. read the letter →

arxiv 2507.00611 v1 pith:WZ2SSDSH submitted 2025-07-01 cs.LG cs.AIcs.RO

Chenyang Cao , Miguel Rogel-García , Mohamed Nabail , Xueqian Wang , Nicholas Rhinehart This is my paper
classification cs.LGcs.AIcs.RO
keywords residualrewardmodelpreference-basedreinforcementlearningfromhumanfeedbacksampleefficiencyinverseroboticmanipulationsim-to-realtransferproxyrewards
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Preference-based reinforcement learning normally learns a reward function from human pairwise comparisons, a process that is slow because the reward must be inferred from scratch. This paper proposes to supply a fixed “prior reward”—a best-guess reward from any source, such as a hand-designed proxy or inverse reinforcement learning—and to learn only a residual correction on top of it, trained from the same preference data. Across five simulated robotic manipulation tasks and three real-robot tasks, the resulting Residual Reward Model raises success rates and cuts the amount of feedback needed compared with the base PbRL algorithm. The prior does not need to be accurate: even a negated version of the true reward is corrected by the residual, provided the prior gives the agent some usable signal early in training.

What carries the argument

The central object is the decomposition $\hat{r}^{\mathrm{RRM}}_\psi = r_0 + r'_\psi$, where $r_0$ can be a hand-designed proxy reward, an IRL-learned reward, or even a negated version of such a reward, and $r'_\psi$ is a neural-network residual whose output is bounded by $\tanh$ so that it stays small relative to the prior. The paper motivates this structure as maximum a posteriori reward estimation with a Gaussian prior concentrated at $r_0$. The prior term gives the agent an immediate shaping signal, making early preference queries informative, while the residual term is fit by the Bradley-Terry cross-entropy loss and can grow to dominate where the prior is wrong; the paper shows it actively inverting an opposite-signed prior within roughly the first 50,000 steps.

What would settle it

Train RRM on the same Meta-World tasks using a shuffled prior reward—the identical values of the proxy reward randomly reassigned to state-action pairs—under the same feedback budget as the main experiments; if RRM still outperforms PEBBLE the improvement is not explained by the prior's task-relevant structure, whereas if it does not, the paper's prior-guided residual correction mechanism is upheld.

Watch

Extended reading notes

Core claim

The paper’s central claim is that the true reward can be modeled as $\hat{r}^{\mathrm{RRM}}_\psi(s,a) = r_0(s,a) + r'_\psi(s,a, r_0(s,a))$, where $r_0$ is a prior reward fixed before training and $r'_\psi$ is a residual learned from preference data. Keeping the prior frozen and learning only the residual avoids the optimization instability the authors observe when a neural reward is pretrained and then fine-tuned with a different loss. Empirically, RRM raises the average IQM success rate of the PEBBLE baseline on Meta-World from 64.4% to 77.8% with a complete proxy prior, keeps high success rates when feedback is reduced tenfold, and on a real Franka Panda Pick-and-Reach task reaches 85% success at 600k steps where the baseline reaches 0%. The paper further shows that a prior rewarding the first subtask of a sequential manipulation is the most useful type of prior.

Load-bearing premise

The gains reported for RRM presuppose that the task is supplied with a nonzero prior reward before training, because the paper's own ablation shows RRM with a zero prior performs no better than its PEBBLE baseline; if no such prior can be obtained, the method offers no advantage.

Editorial extensions

If this is right

  • Built on PEBBLE, RRM lifts the average IQM success rate from 64.4% to 77.8% across five Meta-World tasks, and the same construction improves the SURF and MRN algorithms.
  • With a good prior, RRM holds roughly 90% or higher success on Button-press even when the preference batch size drops to 10 or feedback arrives only every 20,000 steps, while PEBBLE falls below 60% under the same cuts.
  • A prior rewarding the first subgoal of the task is the most effective; a negated prior is still workable because the residual learns to correct it, but a zero prior yields no advantage over the baseline.
  • Under stochastic (inconsistent) preference feedback RRM performs well, and it slightly improves on PEBBLE under mistaken (10% flipped) feedback, while the paper notes that fully resolving incorrect feedback remains an open limitation.
  • On a real Franka Panda, RRM reaches 85% success on Pick-and-Reach at 600k steps while PEBBLE achieves 0%, indicating faster policy acquisition under sim-to-real transfer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The prior-plus-residual split should transfer wherever a coarse reward estimate already exists, for instance using a rule-based reward or a small demonstration set as the fixed prior in language-model alignment, so that preference data only corrects deviations rather than specifying the whole objective.
  • The paper's conclusion that first-step priors work best suggests an automated recipe: decompose a task into its first subgoal and use the distance to that subgoal as the prior, which could be constructed from a task sketch or a few demonstrations and would directly address the paper's stated open problem of obtaining high-quality priors.
  • RRM with a zero prior performing no better than PEBBLE sets a clear boundary on the method: its value is conditional on prior availability, so future comparisons on new tasks should report the quality of the prior used.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces Residual Reward Models (RRMs) for preference-based RL, parameterizing the reward as a fixed prior reward r0 plus a learned residual r'_psi trained on pairwise preferences. It instantiates RRM on top of PEBBLE (and SURF/MRN), evaluates state-based and image-based Meta-World tasks, ablates the prior and residual components, studies reduced/noisy feedback, and reports sim-to-real Franka experiments. The central empirical claim is that adding a fixed prior reward and learning only the residual substantially improves the convergence speed and final success of PEBBLE across tasks.

Significance. If the empirical comparison is unbiased, the residual reward structure is a simple and potentially valuable plug-in for existing PbRL algorithms, and the results span multiple tasks, backbones, feedback regimes, and a real robot. The paper also provides code pointers and reports IQM with confidence intervals, which is a strength. However, the headline comparison is currently confounded by simultaneously removing PEBBLE's unsupervised pretraining and injecting a task-specific prior, so the central attribution of the gains to residual preference learning is not yet established.

major comments (3)
  1. [§4.1–4.2, Table 1, Algorithm 1] The main comparison in Table 1 varies two factors at once: RRM removes PEBBLE's unsupervised pretraining (Algorithm 1, line 2: 'Unsupervised pretaining (do not perform)') and injects a task-specific prior reward r0 into both the stored transitions and the relabeled reward. The paper never reports the natural control of adding the same prior reward to PEBBLE's reward signal while keeping its warmup. The ablation in Fig. 4b shows that RRM with a zero prior is 'slightly lower than PEBBLE,' which the authors attribute to the missing warmup; this makes the attribution of the Table 1 gains to the residual preference model unsupported. Please add PEBBLE+prior with and without unsupervised pretraining, and report these controls alongside Table 1.
  2. [§A, Eq. (6); §F.2] The MAP derivation in Appendix A concludes with an explicit L2 penalty on the residual r'_psi, but the implemented residual network uses a tanh output bound instead, and the paper states that tanh is used 'instead of the squared penalty term.' The claimed connection to MAP estimation is therefore not realized by the implemented objective. Please either implement the L2 penalty or explicitly frame the tanh bound as a heuristic alternative, and report whether the output scale of the residual affects the results.
  3. [§D.10, Fig. 22] In the human-in-the-loop preference study, the interface displays the cumulative reward of each trajectory segment at the top of the animation for the labeler to reference. This exposes the labeler to the reward signal that PbRL is designed to elicit, and it is unclear how the displayed reward influences the stated preferences. The claim that RRM improves PEBBLE with real human feedback should be supported either by hiding the cumulative reward or by analyzing the relationship between displayed reward and labelling choices.
minor comments (5)
  1. [Table 1 (right)] The RRM-ID row reads '74.9±6,92.6'; this appears to be a formatting error where the success-rate confidence interval and return value are merged, and should be corrected to separate columns.
  2. [Algorithms 1 and 2] The comment on line 2 of both algorithms contains a typo: 'Unsupervised pretaining' should be 'Unsupervised pretraining.'
  3. [Figure 2 vs. §4.1] The selection rule for the evaluation step is described differently in the caption of Fig. 2 ('the first step when a non-oracle method's 4/5 runs reach 100% success') and in §4.1 ('the first step at which the mean success rate over the top 75% of runs reaches its maximum'); these definitions should be aligned.
  4. [§4.1] The constants k1, k2, k3, k4 are called normalization hyperparameters but are always set to 1, so the proxy rewards are simply unscaled negative distances; consider reporting the actual scale or removing the word 'normalized.'
  5. [§4.7] The 'first step' prior-selection rule is derived post hoc from the five Meta-World tasks and is not validated on held-out tasks or environments; the text should make clear that this is an explanatory principle rather than a tested design rule.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the residual reward is learned from preference data against a fixed prior reward, and no central prediction in the paper reduces by construction to its fitted inputs.

full rationale

The paper's central mechanism is Eq. (3), which defines the residual reward model as r0(s,a) + r'_psi(s,a,r0(s,a)), where r0 is a fixed input available before training and r'_psi is trained by minimizing the preference cross-entropy loss in Eq. (2). No parameter of the model is fitted to the evaluation metric (success rate or true return) and then reported as a prediction; the preference labels are generated from the environment's true reward by a scripted teacher or collected from humans, and the same preference loss is used for all variants. The prior reward is an external input, not a relabeled version of the learned output, and the paper explicitly reports that removing the prior (zero proxy reward) leaves performance slightly below PEBBLE, confirming that the prior is an independent ingredient rather than a disguised result of the preference fit. The 'first-step' prior-selection rule in Sec. 4.7 is derived after the experiments as an explanatory principle, and the paper itself flags that obtaining high-quality priors remains an open problem; it is not a fitted constant inserted into the model. There are no load-bearing self-citations, no imported uniqueness theorem, and no ansatz smuggled in via citation. The main experimental comparison does lack a PEBBLE-plus-prior control, which is a real confound for attributing gains to the residual structure, but that is a question of experimental design and causal attribution, not circularity of the derivation. Under the stated criteria, no step in the derivation chain is equivalent to its inputs by construction.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical entities. Its central claim rests on the structural assumption that a prior reward plus a learned residual can represent the true reward, and on the availability of a useful prior. No constants are fitted to the experimental results.

free parameters (2)
  • Proxy reward normalization constants k1, k2, k3, k4
    Set to 1 in all experiments; they scale the priors but are not fitted to the outcome metric.
  • Residual output bound (tanh scale)
    The residual network is bounded to (-1,1) by tanh; the scale is not tuned.
assumptions (3)
  • ad hoc to paper The true reward can be decomposed as the sum of a prior reward and a learned residual.
    This is the central structural assumption of RRM (Eq. 3); it is not derived from the environment.
  • standard math Preferences follow the Bradley-Terry model.
    Standard in PbRL; used in the loss (Eq. 2).
  • domain assumption The residual reward network is universal enough to capture the correction to the prior.
    Assumed implicitly by using a three-layer MLP with tanh output; no capacity analysis is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Residual Reward Models for Preference-based Reinforcement Learning." pith.science (2026). https://pith.science/paper/WZ2SSDSH

@misc{pith2026250700611,
  author       = {Pith},
  title        = {Pith review of: Residual Reward Models for Preference-based Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WZ2SSDSH}},
  note         = {Machine review of arXiv:2507.00611}
}
read the original abstract

Preference-based Reinforcement Learning (PbRL) provides a way to learn high-performance policies in environments where the reward signal is hard to specify, avoiding heuristic and time-consuming reward design. However, PbRL can suffer from slow convergence speed since it requires training in a reward model. Prior work has proposed learning a reward model from demonstrations and fine-tuning it using preferences. However, when the model is a neural network, using different loss functions for pre-training and fine-tuning can pose challenges to reliable optimization. In this paper, we propose a method to effectively leverage prior knowledge with a Residual Reward Model (RRM). An RRM assumes that the true reward of the environment can be split into a sum of two parts: a prior reward and a learned reward. The prior reward is a term available before training, for example, a user's ``best guess'' reward function, or a reward function learned from inverse reinforcement learning (IRL), and the learned reward is trained with preferences. We introduce state-based and image-based versions of RRM and evaluate them on several tasks in the Meta-World environment suite. Experimental results show that our method substantially improves the performance of a common PbRL method. Our method achieves performance improvements for a variety of different types of prior rewards, including proxy rewards, a reward obtained from IRL, and even a negated version of the proxy reward. We also conduct experiments with a Franka Panda to show that our method leads to superior performance on a real robot. It significantly accelerates policy learning for different tasks, achieving success in fewer steps than the baseline. The videos are presented at https://sunlighted.github.io/RRM-web/.

Figures

Figures reproduced from arXiv: 2507.00611 by the authors.

Figure 1
Figure 1. Residual Reward Model. An agent interacts with the reward-free environment and generates trajectories. In order to generate rewards for reinforcement learning, our method assumes access to a “prior” reward that conveys some information about the task, but generally may be different than the true task reward function. These prior rewards form part of a reward function that is trained to be aligned with preference pai… view at source ↗
Figure 2
Figure 2. IQM success rate across 5 tasks. We choose the first step when a non-oracle method’s 4/5 runs reach 100% success rate for each task and compute IQM success rate across these steps. The vertical line in the middle of the box represents the mean of the data, which is also labeled with a number. The edges of the box indicate the 25th and 75th per￾centiles. The whisker lines represent the data points within a specific r… view at source ↗
Figure 3
Figure 3. Applying RRM to other PbRL methods. The shaded area indicates the interquartile range (IQR), which means the range between the 25% and 75% percentiles. (a) Comparison between RRM based on SURF and SURF. (b) Comparison between RRM based on MRN and MRN. (a) Ablation on residuals. (b) Ablation on priors [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (20 more)
Figure 4
Figure 4. Figure 4: Ablation studies. The shaded area indicates the interquartile range (IQR). (a) Contribution of residual rewards in RRM with different prior rewards. (b) Robustness of RRM to unexpected prior rewards. for Door-open and 100 for Button-press. The training curves are illus…
Figure 6
Figure 6. Figure 6: Training curves with stochastic and mistaken feedback. We train our method and baselines on Button-press in 10% mistaken and stochastic settings. The oracle is indicated by a dotted line with blue color. The shaded area indicates the interquartile range (IQR). IQM succ…
Figure 5
Figure 5. Figure 5: Results with less feed￾back. The curves represent the mean of the IQM success rate of algorithms with limited feedback. In real human-provided feedback, unlike preferences directly obtained from the true reward, there is often some level of irra￾tionality. Therefore, i…
Figure 7
Figure 7. Figure 7: Real world setup. (a) We use a Franka Research 3 arm, Robotiq 2F-85 gripper and two Intel D435 realsense cameras in a multi-view setting. We calibrate the cameras and obtain the cube’s state using ArUco markers. (b) We train the policy on the simulation built by Pybull…
Figure 8
Figure 8. Figure 8: Image-based Residual Reward Model. In image-based settings, RRM obtains images and proprioceptive states from the environment rather than states. An encoder is used for extracting representations from images and is jointly trained with the RL agent. C An Illustration o…
Figure 9
Figure 9. Figure 9: Issues with fine-tuning. Left and middle: When different loss functions are used during pre-training and fine-tuning, their gradients are generally different, which can lead to abrupt changes in the training dynamics. Abrupt changes can make the training unstable. Righ…
Figure 10
Figure 10. Figure 10: Learning curves on 5 tasks. We train RRM and its baselines on 5 tasks for 1 million steps, across five seeds. To clearly illustrate our results, we choose training clips of different lengths for different tasks. The oracle is indicated by a dotted line in blue [PITH_…
Figure 11
Figure 11. Figure 11: Learning curves on image-based tasks. RRM with different prior rewards and its baselines are trained on 2 visual tasks for 1 million steps. The oracle is indicated by a dotted line in blue. RRM and its baselines across five state-based tasks as the training progresses…
Figure 12
Figure 12. Figure 12: Observing from these results, it is clear that the performance of RRM far exceeds that of [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 12
Figure 12. Figure 12: IQM success rate across image-based tasks. For each visual task, we select the earliest step at which a non-oracle visual method achieves 100% success in at least 4 out of 5 runs. If no method meets this criterion, we use the final step instead. The box plot shows the…
Figure 13
Figure 13. Figure 13: Compare methods with encoder across image-based tasks. We apply the pre-trained encoder to each method for fair comparison. The oracle is indicated by a dotted line in blue. experiments. However, it is feasible because, before training RRM with imitation reward, we mu…
Figure 14
Figure 14. Figure 14: Ablation studies on more tasks. We conduct ablation studies on more tasks and run them for 1 million steps with 5 seeds [PITH_FULL_IMAGE:figures/full_fig_p018_14.png]
Figure 15
Figure 15. Figure 15: Ablation studies on image-based tasks. We conduct ablation studies on 2 visual tasks and run them for 1 million steps with 5 seeds. goals, leading to the agent learning only a part of the task and failing to complete it. These results can demonstrate the significant i…
Figure 16
Figure 16. Figure 16: Training curves with smaller amounts of feedback. 5000, 2000, 1000 feedbacks correspond to 25, 10, 5 reward batch size. They maintain the feedback frequency that provides preferences every 5000 steps. All the methods run 5 seeds [PITH_FULL_IMAGE:figures/full_fig_p019…
Figure 17
Figure 17. Figure 17: Training curves with less frequency of feedback. The reward batch size is fixed at 50 in these settings. Their total number of feedback is 5000, 25000, respectively. All the methods run 5 seeds. (a) Results with smaller amounts of feedback. (b) Results with less frequ…
Figure 18
Figure 18. Figure 18: IQM results across less feedback settings. For each task with less feedback, we select the earliest step at which a non-oracle method achieves 100% success rate in at least 4 out of 5 runs. If no method meets this condition, we use the final step instead. At the selec…
Figure 19
Figure 19. Figure 19: More details on opposite reward experiments. (a) The left plot shows the training curves of the opposite proxy reward, residual reward, and total estimated reward, where the total estimated reward is the sum of the proxy reward and the residual reward. The right plot …
Figure 20
Figure 20. Figure 20: PPOpref vs RRM-PPOpref. We train these two algorithms on Sweep-into for 3 million steps. Although they both show poor performances, PPOpref with RRM can achieve 20% success rate while PPOpref reaches around 10%. D.9 Real world experiments We implement the Reach, Push …
Figure 21
Figure 21. Figure 21: Experiment results of human preference study. RRM is compared with its base￾line PEBBLE on Window-open with real human feedback. The training curves represent the smoothed success rate (%). To further validate the effect of RRM, we design a simple in￾terface to collec…
Figure 22
Figure 22. Figure 22: Real-user study. A simple interface is made for collecting feedback from real humans. Users need to press the left arrow or the right arrow on the keyboard to choose their performance. They should try to avoid giving equal feedback based on pressing the space bar, so …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

59 extracted references · 30 canonical work pages

  1. [1]

    Residual policy learning,

    T. Silver, K. Allen, J. Tenenbaum, and L. Kaelbling, “Residual policy learning,” 2019. [Online]. Available: https://arxiv.org/abs/1812.06298

    arXiv 2019

  2. [2]

    Magnetic control of tokamak plasmas through deep reinforcement learning,

    J. Degrave, F. Felici, J. Buchli, M. Neunert, B. Tracey, F. Carpanese, T. Ewalds, R. Hafner, A. Abdolmaleki, D. de Las Casas et al., “Magnetic control of tokamak plasmas through deep reinforcement learning,” Nature, vol. 602, no. 7897, pp. 414–419, 2022

    work page 2022

  3. [3]

    Designing Rewards for Fast Learning

    H. Sowerby, Z. Zhou, and M. L. Littman, “Designing rewards for fast learning,”arXiv preprint arXiv:2205.15400, 2022

    work page Pith review arXiv 2022

  4. [4]

    Where do rewards come from,

    S. Singh, R. L. Lewis, and A. G. Barto, “Where do rewards come from,” in Proceedings of the annual conference of the cognitive science society. Cognitive Science Society, 2009, pp. 2601–2606

    work page 2009

  5. [5]

    On the expressivity of markov reward,

    D. Abel, W. Dabney, A. Harutyunyan, M. K. Ho, M. Littman, D. Precup, and S. Singh, “On the expressivity of markov reward,”Advances in Neural Information Processing Systems, vol. 34, pp. 7799–7812, 2021

    work page 2021

  6. [6]

    Reward (mis)design for autonomous driving,

    W. B. Knox, A. Allievi, H. Banzhaf, F. Schmitt, and P. Stone, “Reward (mis)design for autonomous driving,” 2022. [Online]. Available: https://arxiv.org/abs/2104.13906

    arXiv 2022

  7. [7]

    Inverse reward design,

    D. Hadfield-Menell, S. Milli, P. Abbeel, S. Russell, and A. Dragan, “Inverse reward design,”

    work page

  8. [8]

    Avoiding side effects in complex environments,

    A. M. Turner, N. Ratzlaff, and P. Tadepalli, “Avoiding side effects in complex environments,”

    work page

Show all 59 references
  1. [9]

    Challenges of real-world reinforcement learning: definitions, benchmarks and analysis,

    G. Dulac-Arnold, N. Levine, D. J. Mankowitz, J. Li, C. Paduraru, S. Gowal, and T. Hester, “Challenges of real-world reinforcement learning: definitions, benchmarks and analysis,” Machine Learning, vol. 110, no. 9, pp. 2419–2468, Sep 2021. [Online]. Available: https://doi.org/1...

    2021 doi

  2. [10]

    Available: https://arxiv.org/abs/2006.06547

    [Online]. Available: https://arxiv.org/abs/2006.06547

    2006 arXiv

  3. [11]

    The ingredients of real-world robotic reinforcement learning,

    H. Zhu, J. Yu, A. Gupta, D. Shah, K. Hartikainen, A. Singh, V . Kumar, and S. Levine, “The ingredients of real-world robotic reinforcement learning,” 2020. [Online]. Available: https://arxiv.org/abs/2004.12570

    2020 arXiv

  4. [12]

    Defining and characterizing reward hacking,

    J. Skalse, N. H. R. Howe, D. Krasheninnikov, and D. Krueger, “Defining and characterizing reward hacking,” 2022. [Online]. Available: https://arxiv.org/abs/2209.13085

    2022 arXiv

  5. [13]

    Deep reinforcement learning from human preferences,

    P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei, “Deep reinforcement learning from human preferences,” Advances in neural information processing systems, vol. 30, 2017

    2017

  6. [14]

    B-pref: Benchmarking preference-based rein- forcement learning,

    K. Lee, L. Smith, A. Dragan, and P. Abbeel, “B-pref: Benchmarking preference-based rein- forcement learning,” arXiv preprint arXiv:2111.03026, 2021

    2021 arXiv

  7. [15]

    Surf: Semi-supervised reward learning with data augmentation for feedback-efficient preference-based reinforcement learning,

    J. Park, Y . Seo, J. Shin, H. Lee, P. Abbeel, and K. Lee, “Surf: Semi-supervised reward learning with data augmentation for feedback-efficient preference-based reinforcement learning,” arXiv preprint arXiv:2203.10050, 2022

    2022 arXiv

  8. [16]

    Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training,

    K. Lee, L. Smith, and P. Abbeel, “Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training,” 2021. [Online]. Available: https://arxiv.org/abs/2106.05091

    2021 arXiv

  9. [17]

    A bayesian approach for policy learning from trajectory preference queries,

    A. Wilson, A. Fern, and P. Tadepalli, “A bayesian approach for policy learning from trajectory preference queries,” Advances in neural information processing systems, vol. 25, 2012

    2012

  10. [18]

    Reward uncertainty for exploration in preference- based reinforcement learning,

    X. Liang, K. Shu, K. Lee, and P. Abbeel, “Reward uncertainty for exploration in preference- based reinforcement learning,” arXiv preprint arXiv:2205.12401, 2022

    2022 arXiv

  11. [19]

    Keyframe-based learning from demonstration,

    B. Akgün, M. Cakmak, K. Jiang, and A. L. Thomaz, “Keyframe-based learning from demonstration,” International Journal of Social Robotics, vol. 4, pp. 343 – 355, 2012. [Online]. Available: https://api.semanticscholar.org/CorpusID:10004846

    2012

  12. [20]

    Apprenticeship learning via inverse reinforcement learning,

    P. Abbeel and A. Y . Ng, “Apprenticeship learning via inverse reinforcement learning,” in Proceedings of the twenty-first international conference on Machine learning, 2004, p. 1. 10

    2004

  13. [21]

    Learning human objectives from sequences of physical corrections,

    M. Li, A. Canberk, D. P. Losey, and D. Sadigh, “Learning human objectives from sequences of physical corrections,” 2021. [Online]. Available: https://arxiv.org/abs/2104.00078

    2021 arXiv

  14. [22]

    Learning complicated manipulation skills via deterministic policy with limited demonstrations,

    L. Haofeng, C. Yiwen, T. Jiayi, and M. H. Ang, “Learning complicated manipulation skills via deterministic policy with limited demonstrations,” 2023. [Online]. Available: https://arxiv.org/abs/2303.16469

    2023 arXiv

  15. [23]

    Land: Learning to navigate from disengagements,

    G. Kahn, P. Abbeel, and S. Levine, “Land: Learning to navigate from disengagements,” 2020. [Online]. Available: https://arxiv.org/abs/2010.04689

    2020 arXiv

  16. [24]

    Guiding policies with language via meta-learning,

    J. D. Co-Reyes, A. Gupta, S. Sanjeev, N. Altieri, J. Andreas, J. DeNero, P. Abbeel, and S. Levine, “Guiding policies with language via meta-learning,” 2019. [Online]. Available: https://arxiv.org/abs/1811.07882

    2019 arXiv

  17. [26]

    Mile: Model-based intervention learning,

    Y . Korkmaz and E. Bıyık, “Mile: Model-based intervention learning,” 2025. [Online]. Available: https://arxiv.org/abs/2502.13519

    2025 arXiv

  18. [27]

    State regularized policy optimization on data with dynamics shift,

    Z. Xue, Q. Cai, S. Liu, D. Zheng, P. Jiang, K. Gai, and B. An, “State regularized policy optimization on data with dynamics shift,” Advances in neural information processing systems, vol. 36, pp. 32 926–32 937, 2023

    2023

  19. [28]

    Learning reward functions from diverse sources of human feedback: Optimally integrating demonstrations and preferences,

    E. Bıyık, D. P. Losey, M. Palan, N. C. Landolfi, G. Shevchuk, and D. Sadigh, “Learning reward functions from diverse sources of human feedback: Optimally integrating demonstrations and preferences,” The International Journal of Robotics Research, vol. 41, no. 1, pp. 45–67, 2022

    2022

  20. [29]

    The vanishing gradient problem during learning recurrent neural nets and problem solutions,

    S. Hochreiter, “The vanishing gradient problem during learning recurrent neural nets and problem solutions,” International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, vol. 6, no. 02, pp. 107–116, 1998

    1998

  21. [30]

    Learning from random demonstrations: Offline reinforcement learning with importance-sampled diffusion models,

    Z. Fang and T. Lan, “Learning from random demonstrations: Offline reinforcement learning with importance-sampled diffusion models,” arXiv preprint arXiv:2405.19878, 2024

    2024 arXiv

  22. [31]

    Identity mappings in deep residual networks,

    K. He, X. Zhang, S. Ren, and J. Sun, “Identity mappings in deep residual networks,” inComputer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14. Springer, 2016, pp. 630–645

    2016

  23. [32]

    Loss jump during loss switch in solving pdes with neural networks,

    Z. Wang, L. Zhang, Z. Zhang, and Z.-Q. J. Xu, “Loss jump during loss switch in solving pdes with neural networks,” arXiv preprint arXiv:2405.03095, 2024

    2024 arXiv

  24. [33]

    Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning,

    T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine, “Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning,” inConference on robot learning. PMLR, 2020, pp. 1094–1100

    2020

  25. [34]

    Residual reinforcement learning for robot control,

    T. Johannink, S. Bahl, A. Nair, J. Luo, A. Kumar, M. Loskyll, J. A. Ojea, E. Solowjow, and S. Levine, “Residual reinforcement learning for robot control,” in2019 international conference on robotics and automation (ICRA). IEEE, 2019, pp. 6023–6029

    2019

  26. [35]

    Few-shot preference learning for human-in-the-loop rl,

    J. Hejna and D. Sadigh, “Few-shot preference learning for human-in-the-loop rl,” 2022. [Online]. Available: https://arxiv.org/abs/2212.03363

    2022 arXiv

  27. [36]

    Meta-reward-net: Implicitly differentiable reward learning for preference-based reinforcement learning,

    R. Liu, F. Bai, Y . Du, and Y . Yang, “Meta-reward-net: Implicitly differentiable reward learning for preference-based reinforcement learning,” Advances in Neural Information Processing Systems, vol. 35, pp. 22 270–22 284, 2022

    2022

  28. [37]

    Query-policy misalignment in preference- based reinforcement learning,

    X. Hu, J. Li, X. Zhan, Q.-S. Jia, and Y .-Q. Zhang, “Query-policy misalignment in preference- based reinforcement learning,” arXiv preprint arXiv:2305.17400, 2023

    2023 arXiv

  29. [38]

    Benchmarks and algorithms for offline preference- based reward learning,

    D. Shin, A. D. Dragan, and D. S. Brown, “Benchmarks and algorithms for offline preference- based reward learning,” arXiv preprint arXiv:2301.01392, 2023. 11

    2023 arXiv

  30. [39]

    Active preference-based gaussian process regression for reward learning,

    E. Bıyık, N. Huynh, M. J. Kochenderfer, and D. Sadigh, “Active preference-based gaussian process regression for reward learning,” arXiv preprint arXiv:2005.02575, 2020

    2005 arXiv

  31. [40]

    Batch active preference-based learning of reward functions,

    E. Biyik and D. Sadigh, “Batch active preference-based learning of reward functions,” in Conference on robot learning. PMLR, 2018, pp. 519–528

    2018

  32. [41]

    Learning reward functions by integrating human demonstrations and preferences,

    M. Palan, N. C. Landolfi, G. Shevchuk, and D. Sadigh, “Learning reward functions by integrating human demonstrations and preferences,” 2019. [Online]. Available: https://arxiv.org/abs/1906.08928

    2019 arXiv

  33. [42]

    Autonomous robotic reinforcement learning with asynchronous human feedback,

    M. B. I. Pamies, M. T. Villasevil, Z. Wang, S. Desai, P. Agrawal, and A. Gupta, “Autonomous robotic reinforcement learning with asynchronous human feedback,” in 7th Annual Conference on Robot Learning, 2023

    2023

  34. [43]

    Discor: Corrective feedback in reinforcement learning via distribution correction,

    A. Kumar, A. Gupta, and S. Levine, “Discor: Corrective feedback in reinforcement learning via distribution correction,” Advances in neural information processing systems, vol. 33, pp. 18 560–18 572, 2020

    2020

  35. [44]

    Reinforcement learning from imperfect corrective actions and proxy rewards,

    Z. Jiang, X. Feng, P. Weng, Y . Zhu, Y . Song, T. Zhou, Y . Hu, T. Lv, and C. Fan, “Reinforcement learning from imperfect corrective actions and proxy rewards,”arXiv preprint arXiv:2410.05782, 2024

    2024 arXiv

  36. [45]

    Residual reinforcement learning from demonstrations,

    M. Alakuijala, G. Dulac-Arnold, J. Mairal, J. Ponce, and C. Schmid, “Residual reinforcement learning from demonstrations,” 2021. [Online]. Available: https://arxiv.org/abs/2106.08050

    2021 arXiv

  37. [46]

    Reward-rational (implicit) choice: A unifying formalism for reward learning,

    H. J. Jeon, S. Milli, and A. Dragan, “Reward-rational (implicit) choice: A unifying formalism for reward learning,” in Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020,...

    2020

  38. [47]

    From imitation to refinement– residual rl for precise assembly,

    L. Ankile, A. Simeonov, I. Shenfeld, M. Torne, and P. Agrawal, “From imitation to refinement– residual rl for precise assembly,”arXiv preprint arXiv:2407.16677, 2024

    2024 arXiv

  39. [48]

    Teach a robot to fish: Versatile imitation from one minute of demonstrations,

    S. Haldar, J. Pari, A. Rai, and L. Pinto, “Teach a robot to fish: Versatile imitation from one minute of demonstrations,” arXiv preprint arXiv:2303.01497, 2023

    2023 arXiv

  40. [49]

    Advances in preference-based reinforcement learning: A review,

    Y . Abdelkareem, S. Shehata, and F. Karray, “Advances in preference-based reinforcement learning: A review,” in2022 IEEE international conference on systems, man, and cybernetics (SMC). IEEE, 2022, pp. 2527–2532

    2022

  41. [50]

    Data-informed residual reinforcement learning for high-dimensional robotic tracking control,

    C. Li, F. Liu, Y . Wang, and M. Buss, “Data-informed residual reinforcement learning for high-dimensional robotic tracking control,” IEEE/ASME Transactions on Mechatronics, 2024

    2024

  42. [51]

    Deep reinforcement learning at the edge of the statistical precipice,

    R. Agarwal, M. Schwarzer, P. S. Castro, A. C. Courville, and M. Bellemare, “Deep reinforcement learning at the edge of the statistical precipice,” Advances in neural information processing systems, vol. 34, pp. 29 304–29 320, 2021

    2021

  43. [52]

    Rank analysis of incomplete block designs: I. the method of paired comparisons,

    R. A. Bradley and M. E. Terry, “Rank analysis of incomplete block designs: I. the method of paired comparisons,” Biometrika, vol. 39, no. 3/4, pp. 324–345, 1952

    1952

  44. [53]

    Learning robust rewards with adversarial inverse reinforcement learning,

    J. Fu, K. Luo, and S. Levine, “Learning robust rewards with adversarial inverse reinforcement learning,” arXiv preprint arXiv:1710.11248, 2017

    2017 arXiv

  45. [54]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,

    T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” inInternational conference on machine learning. Pmlr, 2018, pp. 1861–1870

    2018

  46. [55]

    Drm: Mastering visual reinforcement learning through dormant ratio minimization,

    G. Xu, R. Zheng, Y . Liang, X. Wang, Z. Yuan, T. Ji, Y . Luo, X. Liu, J. Yuan, P. Huaet al., “Drm: Mastering visual reinforcement learning through dormant ratio minimization,” arXiv preprint arXiv:2310.19668, 2023

    2023 arXiv

  47. [56]

    Mastering visual continuous control: Improved data-augmented reinforcement learning,

    D. Yarats, R. Fergus, A. Lazaric, and L. Pinto, “Mastering visual continuous control: Improved data-augmented reinforcement learning,” arXiv preprint arXiv:2107.09645, 2021. 12

    2021 arXiv

  48. [58]

    Pybullet, a python module for physics simulation for games, robotics and machine learning,

    E. Coumans and Y . Bai, “Pybullet, a python module for physics simulation for games, robotics and machine learning,” http://pybullet.org, 2016–2021. 13 Appendix A Motivating RRMs with MAP Estimation Our goal is to estimate a reward function from data using a reward prior: p(r|...

    2016

  49. [59]

    +y(1) · log Pψ(σi 1 ≻ σi 0) − 1 2σ2 (r′ ψ)2 = argmaxr 1 N NX i=1 y(0) · log Pψ(σi 0 ≻ σi

  50. [60]

    The car needs to drive around obstacles and reach the goal location

    +y(1) · log Pψ(σi 1 ≻ σi 0) − 1 2σ2 (r′ ψ)2 ≈ argminrLr + 1 2σ2 (r′ ψ)2, (6) where the approximation results from a finite-sample estimate of the expectation in Lr. In practice, we use tanh to enforce small values of r′ ψ (see Appx. F.2), instead of the squared penalty term. T...

    2000

  51. [2020]

    Available: https://arxiv.org/abs/1711.02827

    [Online]. Available: https://arxiv.org/abs/1711.02827

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.