Pith. sign in

REVIEW 4 major objections 5 minor 63 references

Learning from Active Human Involvement through Proxy Value Propagation

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Proxy Value Propagation turns human interventions and demonstrations into a reward-free training signal for value-based reinforcement learning, and the resulting policies learn faster and safer than reward-driven baselines across driving…

desk verdict A solid reward-free human-in-the-loop RL method with a neat TD-propagation trick; the empirical and theoretical claims could be tightened, but it deserves real refereeing. read the letter →

arxiv 2502.03369 v1 pith:B3FZO6O3 submitted 2025-02-05 cs.AI cs.RO

Zhenghao Peng , Wenjie Mo , Chenda Duan , Quanyi Li , Bolei Zhou This is my paper · ORCID
classification cs.AIcs.RO
keywords human-in-the-loopreinforcementlearningreward-freepolicyoptimizationproxyvaluepropagationactivehumaninvolvementintervention-basedvalue-basedRLAIalignmentdriving
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that a human actively intervening in an AI agent's trial-and-error exploration—taking over control to demonstrate good actions and flag bad ones—can replace the reward function entirely. It introduces Proxy Value Propagation (PVP), which labels the human's demonstrated actions with high Q-values and the agent's intervened-on actions with low Q-values, then spreads those labels to unexplored transitions through temporal-difference learning. The result, the authors claim, is a value-based reinforcement learning agent that learns to emulate the human's behavior with far fewer environment interactions than reward-driven baselines, and with fewer safety violations during training. This matters because hand-designed rewards are often misaligned with what humans actually want, and trial-and-error exploration is unsafe in settings like driving.

What carries the argument

The central object is the proxy value function, a Q-function trained without rewards. For human-involved transitions (state $s$, agent action $a_n$, human action $a_h$), the value loss pushes $Q(s,a_h)$ toward $+1$ and $Q(s,a_n)$ toward $-1$; for all other transitions sampled from the novice buffer, a reward-free TD loss propagates these values forward through $Q(s,a) \approx \gamma \max_{a'} Q(s',a')$. Two design details carry the argument: the balanced buffer, which samples equal-size batches from human and novice data so scarce demonstrations are not overwhelmed, and the L2 regularization that keeps proxy values bounded and separates human from agent actions. Together these turn any value-based RL algorithm into a reward-free human-in-the-loop learner.

What would settle it

Run a controlled study with a scripted human whose demonstration error rate and intervention miss rate are known and varied, for example a simulated supervisor that deliberately produces wrong actions a fixed fraction of the time. If PVP's test-time success and its measured intent-violation rate do not degrade as those error rates increase in the way the paper's bound predicts, or if the method collapses even at small error rates, the claim that the proxy value function faithfully emulates human behavior would be called into question.

Watch

Extended reading notes

Core claim

PVP's central claim is that a proxy value function trained from human intervention signals can induce a policy that faithfully emulates human behaviors without any access to the environment reward. In every state where the human takes over, the Q-value of the human action is regressed to +1 and the Q-value of the agent's own action is regressed to −1; these labels then propagate through the Bellman backup into the agent's unlabeled exploratory transitions, so the value function encodes both what to do and what not to do. The learned policy is the argmax of this proxy Q-function, as in standard value-based RL. The authors show that this reward-free scheme outperforms reward-driven RL baselines and prior human-in-the-loop methods on driving tasks in MetaDrive, CARLA, and GTA V, and on discrete MiniGrid tasks.

Load-bearing premise

The method assumes the human almost always demonstrates wanted behavior when taking over and reliably intervenes on every unwanted agent action; if the human is occasionally wrong or inattentive, the proxy values encode those mistakes and the learned policy degrades.

Editorial extensions

If this is right

  • If PVP is correct, reward engineering is unnecessary for a broad class of control tasks where a human can intervene and demonstrate; the human's own behavior supplies the training signal.
  • The method inherits the sample efficiency of off-policy value-based RL, so a human can teach a policy from tens of thousands of interactions—about 16 minutes in the GTA V driving case—where reward-based TD3 needs roughly 300K steps for comparable performance.
  • Because the agent policy is deterministic and Q-values are bounded, the learned behavior is smoother and less fatiguing for the human supervisor, which the user study reports as lower stress and higher compliance.
  • The same proxy-value recipe transfers across continuous and discrete action spaces and across gamepad, wheel, and keyboard input devices, suggesting it is a general interface between human intent and policy optimization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Extending the logic beyond the tested environments, PVP should apply to any value-based RL setting where an external oracle can occasionally take over control, including robotics teleoperation where demonstrations are already collected.
  • The paper's connection to Conservative Q-Learning suggests PVP is a form of online preference constraint; an untested corollary is that the same proxy values could fine-tune an offline pretrained policy without any reward, provided the human's intervention policy stays consistent.
  • Since the safety bound depends on the human's error rates, a practical deployment would need to monitor those rates online and pause training if they drift, which the present method does not do.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Proxy Value Propagation (PVP), a reward-free human-in-the-loop method for value-based RL. During training, a human subject can intervene and demonstrate actions; human actions are assigned proxy Q-value +1 and intervened novice actions are assigned -1, and these labels are propagated to unlabeled novice-buffer transitions through a TD backup. The method is tested in MetaDrive, CARLA, GTA V, and MiniGrid, with gamepad, keyboard, and wheel controllers, and compared to RL, safe-RL, imitation, and other human-in-the-loop baselines. The authors report higher learning efficiency, better test performance, lower training safety cost, and better user-study scores than baselines, and they provide code and a demo video.

Significance. If the claims hold, PVP is a simple, practical way to turn TD3/DQN into reward-free active-learning-from-human-feedback agents, with broad empirical coverage across continuous and discrete control and multiple input devices. The connection to CQL and the balanced-buffer design are useful and the ablations are informative. The paper also ships code and a video, which strengthens reproducibility. However, the central alignment/safety claim rests on unverified assumptions about human reliability, and the theoretical guarantee as stated applies to the training-time behavior policy rather than the deployed policy. The empirical evidence is also partly reported via best checkpoints and without error bars for several human-in-the-loop baselines. These issues are fixable but are load-bearing for the strongest claims, so the paper needs revision rather than acceptance in its current form.

major comments (4)
  1. [Appendix D, Theorem D.3 and Eq. (12)] The intent-violation bound is derived for the behavior policy π_b used during training, not for the final deployed novice policy μ_n that maximizes the learned proxy Q-values. As a result, the theorem does not actually support the abstract's claim that PVP 'induces a policy that faithfully emulates human behaviors,' nor the broader safety/alignment benefit at deployment. The authors should either extend the bound to μ_n under explicit assumptions about the learned Q-function, or clearly state that Theorem D.3 applies only to the training-time behavior policy and temper the deployment-related claims accordingly.
  2. [Section 4.1, Eq. (5), and Assumption D.2] The second term of the loss in Eq. (5) propagates proxy values through novice-buffer transitions, treating non-intervention as evidence that the agent's action is aligned with human preferences. This is load-bearing for the method's correctness, but the paper does not measure or validate the key quantity κ in Assumption D.2: the probability that the human fails to intervene on an undesired agent action. Inattention, slow reaction, or a missed signal would enter the TD backup as positive evidence for undesired actions. The Limitations section itself admits that the method assumes humans are 'attentive throughout the entire training.' Without at least an indirect estimate of κ or an ablation that varies human attentiveness, the alignment claim in Section 4.1 is unsupported.
  3. [Section 5.1, Tables 1, 2, 4, 6, and Table 3] The empirical comparison is weakened by reporting the best checkpoint as the final result and by presenting many human-in-the-loop rows without standard deviations. For example, Table 2 reports PVP route completion 0.92 ± 0.05 but HG-DAgger, IWR, and HACO are given only as point estimates, and Table 3's user study is based on five subjects with no statistical test. Because the claimed superior learning efficiency and user experience are central, the authors should report mean performance across subjects/seeds with error bars (or at least explain why this is infeasible for human experiments), and should plot or tabulate full learning curves rather than only the best checkpoint.
  4. [Section 4.3 and Appendix G (Tables 7–10)] The method is described as a 'minimal modification' to value-based RL, but it introduces several hand-chosen quantities: the Q-value bound (set to 1), the balanced-buffer ratio (fixed at 1:1), and the proxy label values (±1). No sensitivity analysis is provided for any of these. Since the Q-value bound and buffer ratio directly control the strength of the propagated human intent, the paper should include an ablation varying at least one of them, or justify why the chosen default values are not critical.
minor comments (5)
  1. [Abstract and Section 6] The phrase 'faithfully emulates human behaviors' overstates what the experiments and theory can support, given that the Limitations section concedes suboptimal human demonstrations will damage learning; consider hedging this wording.
  2. [Section 4.2, Eq. (6)] The notation in Eq. (6) is slightly confusing: the brace groups 'TD loss' together with the CQL-like term, but the TD loss itself is not written out in the equation. Please expand or explicitly reference Eq. (5) for clarity.
  3. [Section 4.1, paragraph 4] The sentence 'The transitions in the novice buffer are not intervened by the human subject, meaning they are aligned with human preferences' should be softened to reflect that non-intervention is only a proxy for alignment, pending the validation of Assumption D.2.
  4. [General] There are minor typographical issues, e.g., 'a efficient policy learning method' in Section 6 and inconsistent capitalization of 'Human-in-the-loop Learning' in Section 2; a careful proofread would help.
  5. [Table 3] The user study reports Likert-style scores but no statistical comparison across methods; adding at least a paired test or confidence intervals would substantially strengthen the claim that PVP is more user-friendly.

Circularity Check

1 steps flagged · score 2.0 of 10

Proxy-value labeling makes the policy prefer demonstrated actions by construction, but TD propagation and held-out evaluation carry the independent content.

  1. self definitional [Sec. 4.1, Eq. 4 and Eq. 3; Abstract]
    "The key insight of this work is that we can manipulate the Q values to induce desired behaviors, given that value-based RL has the nature to seek value-maximizing policy as Eq. 3. ... we sample data (s, an, ah) from the human buffer and label the Q value of the human action ah with +1 and the novice action an with −1. ... µn(s) = arg max_a Q(s, a)."

    The PV loss in Eq. 4 directly fits Q(s, ah) toward +1 and Q(s, an) toward −1 on every intervened transition, while Eq. 3 defines the policy as the argmax of Q. Therefore, on any state where both the human action and the intervened novice action are present, the policy's preference for the demonstrated action is entailed by the loss construction. The abstract's statement that 'the proxy value function thus induces a policy that faithfully emulates human behaviors' is, on this labeled subset, a restatement of the training objective rather than an independent empirical result. The genuinely independent content is the TD propagation in Eq. 5 to unlabeled novice-buffer transitions and the held-out task-metric evaluations, which is why this is a mild, partial circularity rather than a fatal one.

full rationale

Most of PVP's derivation is not circular: the central mechanism is the TD backup in Eq. 5, which propagates the hand-assigned proxy values from human-involved states to unlabelled novice-buffer states, and the reported performance is measured on held-out environments or objective task metrics (e.g., MetaDrive and GTA V train/test splits, success rate, route completion). These evaluations give independent grounding, and the ablation 'PVP w/o TD' confirms that the propagation step is load-bearing. The only by-construction element is that the +1/−1 PV loss plus argmax policy forces the policy to prefer demonstrated actions on the labeled states; this is the stated mechanism, not a hidden prediction. The safety analysis in Theorem D.3 bounds the training-time behavior policy under unverified assumptions D.1 and D.2, and the paper's own Limitations section concedes that suboptimal or inattentive humans will damage learning; this is a correctness risk rather than circularity. Self-citations to MetaDrive and HACO provide the environment and a baseline from the same group, but they are open-source and not used to justify PVP's mechanism, so they do not raise the circularity score.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method is an algorithmic contribution; its central claim relies on a set of hand-chosen constants (proxy labels, balanced ratio, Q bound) and on unverified assumptions about human behavior. No new physical entities are introduced.

free parameters (3)
  • Q value bound = 1.0
    Proxy Q labels for human actions are set to +1 and for intervened agent actions to -1; this magnitude (Q Value Bound 1 in Tables 7-10) is a manually chosen constant that controls the scale of the value function and the strength of the L2 regularization.
  • Balanced buffer ratio = 1:1 (human:novice batch)
    The paper samples N/2 transitions from each buffer in every SGD batch (Section 4.3 'Balanced Buffers'); this ratio is chosen by hand and is shown to be important in the ablation.
  • Proxy label values = +1 / -1
    The assignment of +1 to human actions and -1 to intervened agent actions is an ad hoc encoding of preference; the paper does not derive these values from first principles.
assumptions (4)
  • standard math Bellman backup and TD learning are valid for the learned value function
    Section 4 relies on the standard TD loss without reward (Eq. 2) and the max operator for deterministic policies.
  • domain assumption Human demonstrations are aligned with human preferences
    Stated in Limitations: 'We assume that human always demonstrates desired actions.' This underlies the +1 labeling.
  • domain assumption Human intervention policy catches most undesired agent actions
    Appendix D, Assumption D.2: the probability of missing an undesired action is bounded by κ. This is needed for the safety bound Theorem D.3 and for the meaning of the -1 labels.
  • ad hoc to paper The proxy value function can be propagated through the reward-free TD target without reward
    Equation 2 removes the reward term entirely; the paper does not prove that the resulting fixed point corresponds to a useful value function, only that it works empirically.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning from Active Human Involvement through Proxy Value Propagation." pith.science (2026). https://pith.science/paper/B3FZO6O3

@misc{pith2026250203369,
  author       = {Pith},
  title        = {Pith review of: Learning from Active Human Involvement through Proxy Value Propagation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B3FZO6O3}},
  note         = {Machine review of arXiv:2502.03369}
}
read the original abstract

Learning from active human involvement enables the human subject to actively intervene and demonstrate to the AI agent during training. The interaction and corrective feedback from human brings safety and AI alignment to the learning process. In this work, we propose a new reward-free active human involvement method called Proxy Value Propagation for policy optimization. Our key insight is that a proxy value function can be designed to express human intents, wherein state-action pairs in the human demonstration are labeled with high values, while those agents' actions that are intervened receive low values. Through the TD-learning framework, labeled values of demonstrated state-action pairs are further propagated to other unlabeled data generated from agents' exploration. The proxy value function thus induces a policy that faithfully emulates human behaviors. Human-in-the-loop experiments show the generality and efficiency of our method. With minimal modification to existing reinforcement learning algorithms, our method can learn to solve continuous and discrete control tasks with various human control devices, including the challenging task of driving in Grand Theft Auto V. Demo video and code are available at: https://metadriverse.github.io/pvp

Figures

Figures reproduced from arXiv: 2502.03369 by the authors.

Figure 1
Figure 1. Illustration of Proxy Value Propagation. [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Evaluation of PVP under four different environments with human control devices. For each [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. We visualize the action sequences generated by HACO and PVP agents in the same MetaDrive map who are trained to 40K steps. PVP has much smoother actions. episode. The success rate is the ratio of episodes in which agents reach the destination to the total test episodes. In CARLA, we report route completion and success rate. Route completion is the ratio of the traveled distance to the length of the complete route. G… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Evolution of proxy values. performance. In MiniGrid tasks, our method successfully solves the tasks while vanilla DQN fails, showing that PVP can learn an exploratory solution and can be incorporated into discrete action space. We also show experiments on one easier an…
Figure 6
Figure 6. Figure 6: CARLA Town01 [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 8
Figure 8. Figure 8: MiniGrid (Four Room) [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Control signals in a straight road in CARLA. [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: In MetaDrive task, we use the top-down view to plot the trajectories of human-agent [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]
Figure 11
Figure 11. Figure 11: MiniGrid results. In [PITH_FULL_IMAGE:figures/full_fig_p022_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

63 extracted references · 47 canonical work pages

  1. [1]

    Agent-agnostic human-in-the-loop reinforcement learning

    David Abel, John Salvatier, Andreas Stuhlmüller, and Owain Evans. Agent-agnostic human-in-the-loop reinforcement learning. ArXiv preprint, abs/1701.04079, 2017

    arXiv 2017

  2. [2]

    Constrained policy optimization

    Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In Doina Precup and Yee Whye Teh, editors,Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 22–31. PMLR, 2017

    work page 2017

  3. [3]

    An interactive framework for learning continuous actions policies based on corrective feedback

    Carlos Celemin and Javier Ruiz-del Solar. An interactive framework for learning continuous actions policies based on corrective feedback. Journal of Intelligent & Robotic Systems, 95(1):77–97, 2019

    work page 2019

  4. [4]

    Minimalistic gridworld environment for openai gym

    Maxime Chevalier-Boisvert, Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai gym. https://github.com/maximecb/gym-minigrid, 2018

    work page 2018

  5. [5]

    Christiano, Jan Leike, Tom B

    Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V . N. Vishwanathan, and Roman Garnett, editors,Advances in Neural Information Processing Systems 30: Annual Conference on Neural ...

    2017

  6. [6]

    Open problems in cooperative ai

    Allan Dafoe, Edward Hughes, Yoram Bachrach, Tantum Collins, Kevin R McKee, Joel Z Leibo, Kate Larson, and Thore Graepel. Open problems in cooperative ai. arXiv preprint arXiv:2012.08630, 2020

    arXiv 2012

  7. [7]

    Magnetic control of tokamak plasmas through deep reinforcement learning

    Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897):414–419, 2022

    work page 2022

  8. [8]

    CARLA: An open urban driving simulator

    Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. In Proceedings of the 1st Annual Conference on Robot Learning , pages 1–16, 2017

    2017

Show all 63 references
  1. [9]

    Learning robust rewards with adverserial inverse reinforcement learning

    Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse reinforcement learning. In International Conference on Learning Representations, 2018

    2018

  2. [10]

    Addressing function approximation error in actor-critic methods

    Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden...

    2018

  3. [11]

    Widening the pipeline in human-guided reinforcement learning with explanation and context-aware data augmentation

    Lin Guan, Mudit Verma, Sihang Guo, Ruohan Zhang, and Subbarao Kambhampati. Widening the pipeline in human-guided reinforcement learning with explanation and context-aware data augmentation. Advances in Neural Information Processing Systems, 34, 2021

    2021

  4. [12]

    Learning to walk in the real world with minimal human effort, 2020

    Sehoon Ha, Peng Xu, Zhenyu Tan, Sergey Levine, and Jie Tan. Learning to walk in the real world with minimal human effort, 2020

    2020

  5. [13]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learnin...

    2018

  6. [14]

    Generative adversarial imitation learning

    Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, Advances in Neural In- formation Processing Systems 29: Annual Conference on Neural Information Process...

    2016

  7. [15]

    Waslander

    Braden Hurl, Krzysztof Czarnecki, and Steven L. Waslander. Precise synthetic image and lidar (presil) dataset for autonomous vehicle perception. CoRR, abs/1905.00160, 2019

    1905 arXiv

  8. [16]

    Learning to share autonomy across repeated interaction

    Ananth Jonnavittula and Dylan P Losey. Learning to share autonomy across repeated interaction. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1851–1858. IEEE, 2021

    2021

  9. [17]

    Hg-dagger: Interactive imitation learning with human experts

    Michael Kelly, Chelsea Sidrane, Katherine Driggs-Campbell, and Mykel J Kochenderfer. Hg-dagger: Interactive imitation learning with human experts. In 2019 International Conference on Robotics and Automation (ICRA), pages 8077–8083. IEEE, 2019. 11

    2019

  10. [18]

    Learning to drive in a day

    Alex Kendall, Jeffrey Hawke, David Janz, Przemyslaw Mazur, Daniele Reda, John-Mark Allen, Vinh-Dieu Lam, Alex Bewley, and Amar Shah. Learning to drive in a day. In 2019 International Conference on Robotics and Automation (ICRA), pages 8248–8254. IEEE, 2019

    2019

  11. [19]

    Reinforcement learning from human reward: Discounting in episodic tasks

    W Bradley Knox and Peter Stone. Reinforcement learning from human reward: Discounting in episodic tasks. In 2012 IEEE RO-MAN: The 21st IEEE international symposium on robot and human interactive communication, pages 878–885. IEEE, 2012

    2012

  12. [20]

    Specification gaming: the flip side of ai ingenuity

    Victoria Krakovna, Jonathan Uesato, Vladimir Mikulik, Matthew Rahtz, Tom Everitt, Ramana Kumar, Zac Kenton, Jan Leike, and Shane Legg. Specification gaming: the flip side of ai ingenuity. DeepMind Blog, 2020

    2020

  13. [21]

    Conservative q-learning for offline reinforcement learning

    Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems...

    2020

  14. [22]

    Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training

    Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. arXiv preprint arXiv:2106.05091, 2021

    2021 arXiv

  15. [23]

    Scalable agent alignment via reward modeling: a research direction

    Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: a research direction. arXiv preprint arXiv:1811.07871, 2018

    2018 arXiv

  16. [24]

    Offline reinforcement learning: Tutorial, review, and perspectives on open problems

    Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. ArXiv preprint, abs/2005.01643, 2020

    2005 arXiv

  17. [25]

    Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning

    Quanyi Li, Zhenghao Peng, Lan Feng, Qihang Zhang, Zhenghai Xue, and Bolei Zhou. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning. IEEE transactions on pattern analysis and machine intelligence, 2022

    2022

  18. [26]

    Efficient learning of safe driving policy via human-ai copilot optimization

    Quanyi Li, Zhenghao Peng, and Bolei Zhou. Efficient learning of safe driving policy via human-ai copilot optimization. In International Conference on Learning Representations, 2022

    2022

  19. [27]

    Interactive learning from policy-dependent human feedback

    James MacGlashan, Mark K Ho, Robert Loftin, Bei Peng, Guan Wang, David L Roberts, Matthew E Taylor, and Michael L Littman. Interactive learning from policy-dependent human feedback. In International conference on machine learning, pages 2285–2294. PMLR, 2017

    2017

  20. [28]

    Where to add actions in human-in- the-loop reinforcement learning

    Travis Mandel, Yun-En Liu, Emma Brunskill, and Zoran Popovic. Where to add actions in human-in- the-loop reinforcement learning. In Satinder P. Singh and Shaul Markovitch, editors, Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, ...

    2017

  21. [29]

    Human- in-the-loop imitation learning using remote teleoperation

    Ajay Mandlekar, Danfei Xu, Roberto Martín-Martín, Yuke Zhu, Li Fei-Fei, and Silvio Savarese. Human- in-the-loop imitation learning using remote teleoperation. ArXiv preprint, abs/2012.06733, 2020

    2012 arXiv

  22. [30]

    Ensembledagger: A bayesian approach to safe imitation learning

    Kunal Menda, Katherine Driggs-Campbell, and Mykel J Kochenderfer. Ensembledagger: A bayesian approach to safe imitation learning. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5041–5048. IEEE, 2019

    2019

  23. [31]

    Human-level control through deep reinforcement learning

    V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529–533, 2015

    2015

  24. [32]

    Interactively shaping robot behaviour with unlabeled human instructions

    Anis Najar, Olivier Sigaud, and Mohamed Chetouani. Interactively shaping robot behaviour with unlabeled human instructions. Autonomous Agents and Multi-Agent Systems, 34(2):1–35, 2020

    2020

  25. [33]

    Deep exploration via bootstrapped DQN

    Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped DQN. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Confer...

    2016

  26. [34]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155, 2022

    2022 arXiv

  27. [35]

    Deeptake: Prediction of driver takeover behavior using multimodal data

    Erfan Pakdamanian, Shili Sheng, Sonia Baee, Seongkook Heo, Sarit Kraus, and Lu Feng. Deeptake: Prediction of driver takeover behavior using multimodal data. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1–14, 2021. 12

    2021

  28. [36]

    Learning reward functions by integrating human demonstrations and preferences

    Malayandi Palan, Gleb Shevchuk, Nicholas Charles Landolfi, and Dorsa Sadigh. Learning reward functions by integrating human demonstrations and preferences. In Robotics: Science and Systems, 2019

    2019

  29. [37]

    Stable-baselines3: Reliable reinforcement learning implementations

    Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1–8, 2021

    2021

  30. [38]

    Shared autonomy via deep reinforcement learning

    Siddharth Reddy, Anca D Dragan, and Sergey Levine. Shared autonomy via deep reinforcement learning. Robotics: Science and Systems, 2018

    2018

  31. [39]

    Efficient reductions for imitation learning

    Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 661–668. JMLR Workshop and Conference Proceedings, 2010

    2010

  32. [40]

    Human compatible: Artificial intelligence and the problem of control

    Stuart Russell. Human compatible: Artificial intelligence and the problem of control. Penguin, 2019

    2019

  33. [41]

    Active preference-based learning of reward functions

    Dorsa Sadigh, Anca D Dragan, Shankar Sastry, and Sanjit A Seshia. Active preference-based learning of reward functions. UC Berkeley, 2017

    2017

  34. [42]

    The starcraft multi-agent challenge

    Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. ArXiv preprint, abs/1902.04043, 2019

    1902 arXiv

  35. [43]

    Trial without error: Towards safe reinforcement learning via human intervention

    William Saunders, Girish Sastry, Andreas Stuhlmueller, and Owain Evans. Trial without error: Towards safe reinforcement learning via human intervention. InProceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pages 2067–2069. Internation...

    2018

  36. [44]

    Prioritized experience replay

    Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv:1511.05952, 2015

    2015 arXiv

  37. [45]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv preprint, abs/1707.06347, 2017

    2017 arXiv

  38. [46]

    Mastering the game of go with deep neural networks and tree search

    David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484–489, 2016

    2016

  39. [47]

    Learning from interventions

    Jonathan Spencer, Sanjiban Choudhury, Matthew Barnes, Matthew Schmittle, Mung Chiang, Peter Ra- madge, and Siddhartha Srinivasa. Learning from interventions. In Robotics: Science and Systems (RSS), 2020

    2020

  40. [48]

    Responsive safety in reinforcement learning by PID lagrangian methods

    Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by PID lagrangian methods. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learni...

    2020

  41. [49]

    Intervention aided reinforcement learning for safe and practical policy optimization in navigation

    Fan Wang, Bo Zhou, Ke Chen, Tingxiang Fan, Xi Zhang, Jiangyong Li, Hao Tian, and Jia Pan. Intervention aided reinforcement learning for safe and practical policy optimization in navigation. In Conference on Robot Learning, pages 410–421. PMLR, 2018

    2018

  42. [50]

    Appli: Adaptive planner parameter learning from interventions

    Zizhao Wang, Xuesu Xiao, Bo Liu, Garrett Warnell, and Peter Stone. Appli: Adaptive planner parameter learning from interventions. In 2021 IEEE international conference on robotics and automation (ICRA), pages 6079–6085. IEEE, 2021

    2021

  43. [51]

    Apple: Adaptive planner parameter learning from evaluative feedback

    Zizhao Wang, Xuesu Xiao, Garrett Warnell, and Peter Stone. Apple: Adaptive planner parameter learning from evaluative feedback. IEEE Robotics and Automation Letters, 6(4):7744–7749, 2021

    2021

  44. [52]

    Waytowich, Vernon Lawhern, and Peter Stone

    Garrett Warnell, Nicholas R. Waytowich, Vernon Lawhern, and Peter Stone. Deep TAMER: interactive agent shaping in high-dimensional state spaces. In Sheila A. McIlraith and Kilian Q. Weinberger, editors, Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligenc...

    2018

  45. [53]

    A survey of preference-based reinforcement learning methods

    Christian Wirth, Riad Akrour, Gerhard Neumann, Johannes Fürnkranz, et al. A survey of preference-based reinforcement learning methods. Journal of Machine Learning Research, 18(136):1–46, 2017. 13

    2017

  46. [54]

    Look before you leap: Safe model-based reinforcement learning with human intervention

    Yunkun Xu, Zhenyu Liu, Guifang Duan, Jiangcheng Zhu, Xiaolong Bai, and Jianrong Tan. Look before you leap: Safe model-based reinforcement learning with human intervention. In Conference on Robot Learning, pages 332–341. PMLR, 2022

    2022

  47. [55]

    How to leverage unlabeled data in offline reinforcement learning

    Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Chelsea Finn, and Sergey Levine. How to leverage unlabeled data in offline reinforcement learning. arXiv preprint arXiv:2202.01741, 2022

    2022 arXiv

  48. [56]

    Query-efficient imitation learning for end-to-end simulated driving

    Jiakai Zhang and Kyunghyun Cho. Query-efficient imitation learning for end-to-end simulated driving. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017. 14 A Ethics Statement Human subjects get paid to participate in the experiments. They can pau...

    2017

  49. [57]

    We also compare the behavior of agents learned from PVP and TD3 baseline

    The first section shows how we learn the driving policy in CARLA task within 20 minutes. We also compare the behavior of agents learned from PVP and TD3 baseline

  50. [58]

    We present the behavior comparison between PVP and TD3 baseline

    In the second section, we show the footage of MetaDrive human experiment where the human subject uses a gamepad as the control device. We present the behavior comparison between PVP and TD3 baseline

  51. [59]

    PVP performs well in GTA V and can drive smoothly on the highway

    In the third section, we show the applicability of our method to other tasks. PVP performs well in GTA V and can drive smoothly on the highway. In the discrete control tasks, the behavior comparison between PVP and DQN baseline in MiniGrid Empty Room and Four Room are provided...

  52. [60]

    If the agent drives in the wrong way then the displace- ment reward will be multiplied by −1

    The displacement reward: Rdisp = dt − dt−1, wherein the dt and dt−1 denotes the longitu- dinal movement in meters of the target vehicle in Frenet coordinates of the target trajectory between two consecutive time steps. If the agent drives in the wrong way then the displace- me...

  53. [61]

    If the agent drives in wrong way then the speed reward will be multiplied by −1

    The speed reward: Rspeed = vt/vmax, where vt, vmax denotes current speed and maximum allow speed in current road in km/h, respectively. If the agent drives in wrong way then the speed reward will be multiplied by −1. We set cspeed = 0.1

  54. [62]

    Otherwise, it is 0

    The collision reward: Rcollision = 1if a collision with a vehicle, human, or object happens. Otherwise, it is 0. The coefficient ccollision = 5. 18

  55. [63]

    At that step, we set Rdisp = Rspeed = Rcollision = 0and assign Rterm according to the terminal state

    The terminal reward: Rterm is non-zero only at the last time step. At that step, we set Rdisp = Rspeed = Rcollision = 0and assign Rterm according to the terminal state. Rterm is set to +10 if the vehicle reaches the destination (successes) and −5 if the vehicle drives out of t...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.