Pith. sign in

REVIEW 5 major objections 5 minor 75 references

Learning Safe Agent Behaviour from Human Preferences and Justifications via World Models

T0 review · 5 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a human can train a safe agent without any handcrafted reward or real-world exploration, by playing once inside a learned world model and giving justified preference feedback; one-shot human play cuts training cost and

desk verdict DROPJ is a practical step forward—one-shot dream trajectories and justification-weighted preferences are genuinely new—but the proof is weak, and the safe-deployment claim leans on a world model whose fidelity is only partially demonstrated. read the letter →

arxiv 2607.13172 v1 pith:D44KJPFM submitted 2026-07-14 cs.AI cs.LG

Ilias Kazantzidis , Timothy J. Norman , Yali Du , Christopher T. Freeman This is my paper
classification cs.AIcs.LG
keywords safereinforcementlearningpreference-basedrewardworldmodelshumanfeedbackjustificationsmodelpredictivecontrolexplorationone-shot
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces DROPJ, a human-centred method that trains and deploys a safe agent when environment dynamics are unknown and no reward function is available. It first learns a world model—a learned simulator—from past reward-free trajectories, then lets a human play inside that simulator to collect diverse trajectories in one shot, and finally learns a reward model from the human's preferences along with justifications for those preferences. The agent is deployed directly with model predictive control, so no real-world exploration is needed and training has zero safety violations. The paper's core claims are that the one-shot generation of queries cuts computational cost and improves performance compared to iterative techniques, and that safety justifications encoded as severity-weighted preference labels steer the deployed agent toward user-prescribed safety aspects, at some cost in raw performance.

What carries the argument

The mechanism that carries the safety argument is the justification-weighted preference label µ. For each query the user states whether each segment is safe and which segment they prefer; the label is then computed from the most severe active justification, with weights w_1 ≥ … ≥ w_N (Equation 4), so safety-justified answers take values near 0 or 1 while default-justified answers sit closer to 0.5. Feeding these labels into the Bradley–Terry cross-entropy loss makes safety-justified queries exert larger gradients, effectively importance-weighting the safety objective in the learned reward model. Deployment then uses the same world model for sampling-based model predictive control, so the rew

What would settle it

Repeat the Section 7.3 comparison of one-shot (DROS) versus iterative (ReQueST) query generation with a deliberately degraded world model, e.g. trained on only 25 trajectories; the paper's ablation shows DROPJ remains resilient at that level, but if ReQueST surpasses DROS there, the one-shot advantage depends on world-model quality rather than being intrinsic. Alternatively, test the deployment claim directly by changing both the grass and road colours simultaneously, which the paper reports causes significant degradation; if safety gains from justifications disappear under that combined shift

Watch

Extended reading notes

Core claim

The central discovery is that human feedback collected entirely inside a learned world model—a short play session followed by one batch of preference-and-justification queries—is enough to learn a reward model that, combined with the same world model for planning, deploys a safe agent directly in the real environment. In Car Racing and Obstacle Car Racing experiments with real users, the one-shot technique outperforms the iterative ReQueST pipeline in deployment return while cutting the waiting time the human spends between queries from hours to minutes, and preference feedback beats sparse-label feedback for similar human burden. Adding a safety justification, implemented by mapping the use

Load-bearing premise

The entire pipeline rests on the world model learned from past trajectories being accurate enough that the human's judgments on reconstructed dream frames and the planner's imagined rollouts transfer to the real environment; the paper itself notes that significant distribution shifts or hallucinations degrade performance.

Editorial extensions

If this is right

  • All learning happens inside the dream world: the agent never takes an unsafe action in the real environment during training, whereas model-free RL (DRQV2) incurred roughly 50,000 grass steps.
  • Generating queries by one-shot human play lowers wall-clock training cost dramatically: ReQueST needed about 75 trajectory-optimisation segments (~242 minutes of waiting) versus ~5 minutes of play for the one-shot user.
  • Preference feedback with the one-shot technique reaches near-oracle deployment return with about 500 queries (~60 minutes of feedback time), whereas sparse-label methods lag for the same human burden.
  • Attaching a safety justification to preferences cuts the deployed grass rate substantially (from ~0.2 to ~0.002 at 300+ queries) at the price of some return; with multiple justifications, weights let the user dial which hazard to avoid most.
  • Justification-based reward learning is robust to noise in non-critical answers: corrupting 60% of non-critical queries barely degrades DROPJ, because safety labels stay pinned to the extremes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the world model is accurate enough, the method removes the need for a handcrafted reward function entirely; the practical bottleneck then shifts from reward design to world-model fidelity and to the human's ability to judge reconstructed dream frames.
  • The severity-ordered label formula is a general recipe for multi-objective reward learning from humans: a user only needs to supply a ranked list of hazards, not numerical trade-off weights; testing whether summing or averaging active justifications instead of taking the max improves sample efficiency is a natural extension.
  • The robustness to non-critical errors suggests justifications could act as a guard against reward hacking or noisy label sources, since ordinary preference noise is damped; a testable prediction is that DROPJ tolerates label noise on easy queries better than any preference-only baseline.
  • Because the same teleoperation setup used to collect real-world trajectories can be reused in the dream, the one-shot idea transfers to physical robot tasks where simulation is cheap; the claim would be directly testable in a lab-based robot navigation task, as the paper suggests.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes DROPJ, a pipeline for safe reward learning and deployment in environments with unknown dynamics and no handcrafted reward. Step 1 learns a VAE + MDN-RNN world model from an offline dataset of reward-free trajectories. Step 2 has a human 'play the game' inside this learned simulator to generate a diverse set of dream trajectories in one shot. Step 3 samples pairs of dream segments, elicits preferences together with safety justifications, maps them to Bradley-Terry labels via Eq. 4, and trains a reward model. Step 4 deploys the agent with sampling-based MPC. Experiments in Car Racing and a new Obstacle Car Racing compare DROPJ against ReQueST, sparse-label and preference-only variants, and an oracle-reward RL baseline, measuring return, crash rates, human burden, and computational cost.

Significance. If the empirical claims hold, DROPJ fills a practical gap: it learns a reward model and deploys a policy without real-world exploration and without a handcrafted reward function. The one-shot dream-trajectory collection is a simple and potentially useful alternative to iterative trajectory optimization, and the justification-weighted label construction provides a concrete mechanism for injecting safety priorities at training time. The paper's strengths are its real-user experiments, multiple environment trials, non-parametric significance tests, world-model ablations, distribution-shift checks, and released code. The main caveats are the weak statistical support for the one-shot performance advantage over ReQueST, the limited user sample, and the absence of a quantitative link between world-model fidelity and deployment safety.

major comments (5)
  1. [Section 6, Theorem 1] The proof does not establish the stated theorem. The observation that w_s > w_def makes safety labels more extreme, and hence increases the per-sample gradient magnitude, does not imply that the learned reward model will 'prioritise safe behaviour' without assumptions on optimization convergence, data distribution, model capacity, and the MPC planner. The argument is essentially a property of the label construction in Eq. 4, not a theorem about the learned reward or deployment behaviour. Please reframe this as informal motivation or provide a formal statement with explicit assumptions; the empirical results in Section 7.5 remain the primary evidence.
  2. [Section 7.6, Table 4] The first core contribution claims that the one-shot technique 'improves both computational cost and performance compared to previous techniques.' The computational advantage is clear from Section 7.3, but the performance part is weakly supported: ReQueST vs. DROS return p = 0.0605 and grass rate p = 1.0000, i.e., not significant at the 0.05 level. Either report more trials/effect sizes, or soften the abstract and conclusion to state that the one-shot technique reduces compute and 'can' improve performance, with appropriate caveats.
  3. [Section 7.7] The criterion for a 'sufficiently good world model' is not quantitative: M is selected from validation-loss plateaus and visual inspection, and the distribution-shift results are reported as qualitative bullets. The claim that performance degrades 'significantly' when both grass and road colors change is not backed by numbers or significance tests. Given that Section 7.8 reports 172/600 queries skipped due to hallucinations (≈29%), the paper should quantify how world-model errors propagate to reward-model and planning performance, or explicitly restrict the deployment-safety claim to near-training-distribution conditions. This is load-bearing because the method's purpose is deployment where the offline dataset is the only prior.
  4. [Section 7.9 and Table 3] The multiple-justification results in OCR are presented with box plots and qualitative statements but no significance tests. Claims such as '(0.75, 1, 1) yields better safety' or that justification weights can prioritise safety aspects need statistical support or clear labelling as exploratory observations. In addition, Table 3's 'best model' selection criterion is not defined; it appears to be chosen post hoc based on both return and crash rate. Please state the criterion explicitly.
  5. [Sections 7.3-7.9, user study] The real-user experiments appear to rely on one primary labeller; the additional labeller in Section 7.8 annotates only DROPJ fully and a subset of DROP/DROPe. The reported p-values are over 10 environment trials, not over multiple users. Since the method is human-centred, the claims about human burden and the generalizability of justification benefits should either include more labellers or be framed as a case study with the primary labeller. This is important for evaluating the human-feedback contributions.
minor comments (5)
  1. [Section 7.3 / Appendix / Table 1] Notation is inconsistent: the number of dream trajectories is T in Algorithm 1 and Table 1, but called N in Section 7.3 ('N=10 episodes') and in the Appendix ('N=10 dream user trajectories'). N is also used for the number of justifications. Please unify.
  2. [Section 7.8] The symbol M is used both for the number of real-world trajectories (Table 1) and for the number of reward ensembles in the automatic error-detection method. Please rename one of them.
  3. [Section 6] The heading 'Theorem 1' and 'Proof' promise more than the argument delivers. If the content remains a heuristic rationale, label it as such.
  4. [Section 7.7] The distribution-shift bullets would be more convincing with at least mean ± standard deviation for each condition and the number of episodes tested, rather than qualitative descriptions.
  5. [Figure 5] The sentence 'these likely correspond to rare task situations that the agent does not need to prioritise' is speculative. Either provide evidence or soften the claim.

Circularity Check

1 steps flagged · score 2.0 of 10

Core empirical claims are independent; only Theorem 1 restates the justification-weight encoding.

  1. self definitional [Section 6, Theorem 1 (proof around Eq. 5)]
    "Since w_s > w_def > 0.5, safety-justified preferences assign labels further from 0.5 than non-safety ones. Around uncertain predictions, this introduces a larger gradient magnitude for the binary cross-entropy loss, |p[P=1st]−µ|, for samples with a safety justification. Thus, the loss puts more pressure on the reward model to fit such preferences, effectively behaving like an importance-weighted training objective. This means that the learned reward model will prioritise safety over other performance aspects, which leads to the safer behaviour observed in the experiments."

    The theorem's conclusion restates the construction in Table 2 and Eqs. 2-4: safety-justified queries are assigned µ = w_s or 1 - w_s with w_s > w_def, so they are farther from 0.5 by definition. The 'larger gradient magnitude' is a direct algebraic consequence of that label encoding, so the claimed prioritisation is built into the inputs rather than derived from an independent mechanism. The theorem does not by itself establish generalization to unseen states; that is supplied by the deployment experiments, which are non-circular. This is a minor overformalization, not a fatal circularity.

full rationale

The paper's two central claims are supported by non-circular empirical comparisons. The one-shot technique (Q1) is tested by comparing ReQueST with DROS under matched feedback protocols, with computational cost measured directly and trajectory diversity quantified via DPP and t-SNE; no fitted parameter is renamed as a prediction. The preference-over-sparse-labels claim (Q2) is measured by deployment return against human feedback time, again an external comparison. The safety-justification claim (Q3, Q4) is evaluated by deploying the trained reward model with MPC on randomized tracks not used in training; the reduction in grass/chuckhole/car rates is measured in the real environment, not read off from the preference labels. The user-set justification weights are inputs, and the crash rates are outputs, so the observed trade-offs are genuine empirical findings, even though the direction of the effect is plausibly related to the encoding. The only step that reduces to its own construction is Theorem 1, which 'proves' safety prioritisation from the very label weighting (w_s > w_def) that defines the safety signal; this is a self-definitional statement about the training objective rather than evidence of deployment behavior. The paper does not rely on a self-citation chain for its core correctness: the method is described in full, and the baselines are external or independently implemented. Accordingly, the overall circularity is low: one minor self-definitional theorem, while the central empirical contributions remain self-contained and externally evaluated.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim does not rest on many invented entities, but it does rest on several hand-chosen parameters and domain assumptions. Most important are the justification weights, which directly define the preference labels µ (Eq. 2-4) and therefore largely encode the safety priority the paper reports; and the world-model quality assumption, which is acknowledged but not guaranteed. The multiple-justification combination rule is ad hoc. No new physical entities are introduced.

free parameters (5)
  • Justification weights (w_s, w_def) = CR: (1.0, 0.75); OCR: e.g. (0.75,1,1), (1,1,1,1), (1,1,0.98,0.98)
    Hand-chosen before training; define preference label µ via Eq. 2-4, directly controlling how strongly safety dominates the reward model. Figure 8 sweeps them.
  • MPC planning horizon H and replan period R = H=15 for preference methods, 25 for ReQueST/DROS; R=4
    Tuned by 'short experimentation' (Appendix MPC); affects which trajectories are compared and final performance/safety.
  • Number of dream trajectories N and queries K = N=10 (CR), 60 (OCR); K=500 (CR), 1000/1500 (OCR)
    Chosen based on complexity until 'best model' per method; not independently justified.
  • Number of sampled action sequences S for MPC = S=15
    Fixed; with hand-crafted sensible sequences (drive straight, turn left, etc.), which limits planning coverage.
  • Segment length k and sparse-label reward constants = k=20 (preferences), 50 (sparse); R_good=10, R_unsafe=-1, R_neutral=0
    Taken from prior work but arbitrary; segment length affects query informativeness.
assumptions (5)
  • domain assumption Offline dataset R has enough coverage to train a sufficiently good world model.
    Step 1 (Section 5.1) states M×L should be large enough and R should include past failures; authors stop at M=600/800 based on validation-loss plateau (Section 7.7). This is not guaranteed by the method.
  • domain assumption Human users can accurately perceive safety and task progress from reconstructed dream frames and provide consistent annotations.
    Steps 2-3; paper documents hallucinations and skips, and a second labeller disagreed on 23% of non-critical queries (Section 7.8), showing this assumption is fragile.
  • domain assumption The reward model trained on dream segments transfers to real deployment under distribution shifts.
    Step 4 and Section 7.7; authors find significant degradation when both road and grass colors change, showing limited transfer.
  • ad hoc to paper Most severe active justification formula (Eq. 4) with weight ordering is a valid way to combine safety factors.
    The authors say 'a straightforward and reasonable setup'; no principle given for why max-severity combination is correct.
  • standard math Standard ML machinery (Bradley-Terry, VAE, MDN, cross-entropy) is appropriate and correctly implemented.
    Standard background; not proven in the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning Safe Agent Behaviour from Human Preferences and Justifications via World Models." pith.science (2026). https://pith.science/paper/D44KJPFM

@misc{pith2026260713172,
  author       = {Pith},
  title        = {Pith review of: Learning Safe Agent Behaviour from Human Preferences and Justifications via World Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D44KJPFM}},
  note         = {Machine review of arXiv:2607.13172}
}
read the original abstract

We address the problem of safely training an agent policy and deploying a good and safe policy, in settings where the environment dynamics are unknown and no suitable reward function is available. In the context of safety-critical environments, we consider traditional reinforcement learning impractical and resort to the resource of human input. We introduce DROPJ, a human-centred method for both safe training and deployment. We first learn a world model (a learned simulator) from a dataset of prior real-world trajectories. A human then plays the game in this learned simulator to extract several informative simulated trajectories. From these, we sample pairs of simulated trajectory segments and elicit from a human their preference over these segments, as well as a reason (justification) for their choice. We then train a reward model from these justified preferences and use it, together with the world model, to directly deploy the agent using model predictive control. Running real-user experiments, we find that generating informative simulated trajectories from a user significantly reduces the computational cost during training compared to other strategies, and can also improve the performance during deployment. In the context of training within a learned simulator, we show that the use of preferences rather than other types of feedback substantially improves the performance during deployment. We further demonstrate that safety justifications accompanying preferences can significantly enhance safety or prioritise user-prescribed aspects of safety associated with them during deployment.

Figures

Figures reproduced from arXiv: 2607.13172 by the authors.

Figure 1
Figure 1. The DROPJ framework [39]. In Step 3, we show the case with a single safety justification (associated with unsafe steps of the car driving onto grass). user expresses their preference and a justification for it. A reward model is then learned with preferences and justifications similar to [14]. In Step 4, equipped with the learned dynamics and reward models, we directly deploy the agent with MPC. Thus, DROPJ solves t… view at source ↗
Figure 2
Figure 2. Environments used in this research [39]. Our problem setting is based on the following assumptions. The unsafe states (driving over grass or chuckholes, or hitting other cars) are unknown to the agent, and we have access to neither the state transition dynamics func￾tion of the environment (i.e. there is no way to sample experience for free), nor to a suitable (oracle) reward function. The state and action spaces ar… view at source ↗
Figure 3
Figure 3. GUI of Obstacle Car Racing with multiple justifications [ [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗
Figures from the paper (15 more)
Figure 4
Figure 4. Figure 4: Q1: Impact of the one-shot technique [39]. We confirmed this not only by visual inspection of trajectory segments, but also in more systematic ways. We compared the diversity of DROS and ReQueST segments, using the Determinantal Point Process (Equation 7 from [16]). Th…
Figure 5
Figure 5. Figure 5: Q1: Inspecting the diversity of ReQueST versus DROS trajectory seg￾ments with the t-SNE of state representations [39]. Moreover, for the computational cost in our experiments in CR, we mea￾sured that the time ReQueST required to generate one trajectory segment of 50 st…
Figure 6
Figure 6. Figure 6: Q2: Impact of using preferences instead of sparse labels [39] [PITH_FULL_IMAGE:figures/full_fig_p020_6.png]
Figure 7
Figure 7. Figure 7: Q3: Impact of including a safety justification in preferences [39]. Given that, [PITH_FULL_IMAGE:figures/full_fig_p021_7.png]
Figure 8
Figure 8. Figure 8: Mean return (left) and mean grass rate (right) for various [PITH_FULL_IMAGE:figures/full_fig_p022_8.png]
Figure 9
Figure 9. Figure 9: Return and grass rate distributions of the best models in Car Racing [ [PITH_FULL_IMAGE:figures/full_fig_p023_9.png]
Figure 10
Figure 10. Figure 10: Encoder (VAE) and dynamics model (MDN-RNN) Validation Mean [PITH_FULL_IMAGE:figures/full_fig_p024_10.png]
Figure 11
Figure 11. Figure 11: World model ablation for different numbers of real-world trajectories [ [PITH_FULL_IMAGE:figures/full_fig_p025_11.png]
Figure 12
Figure 12. Figure 12: Effect on best-trained models via synthetic erroneous feedback [ [PITH_FULL_IMAGE:figures/full_fig_p026_12.png]
Figure 13
Figure 13. Figure 13: Q4: Impact of including multiple justifications in preferences (OCR with only chuckholes) [39]. the chuckhole rate, but at the cost of more steps on the grass. Interestingly, with this combination the car tends to keep a safe distance from the chuckholes, even if that…
Figure 14
Figure 14. Figure 14: Q4: Impact of including multiple justifications in preferences (OCR with both chuckholes and cars) [39]. However, as mentioned before, justification weights should be used with con￾sideration if all safety aspects are critical or one affects another. For instance, wit…
Figure 15
Figure 15. Figure 15: GUI of the sparse reward labels feedback in ReQueST and DROS [ [PITH_FULL_IMAGE:figures/full_fig_p034_15.png]
Figure 16
Figure 16. Figure 16: GUIs in Car Racing of DROP (b), DROPe (c) and DROPJ (d–f) on the [PITH_FULL_IMAGE:figures/full_fig_p036_16.png]
Figure 17
Figure 17. Figure 17: GUI in Obstacle Car Racing of DROPJ for the multiple justifications [PITH_FULL_IMAGE:figures/full_fig_p037_17.png]
Figure 18
Figure 18. Figure 18: MPC dreams of a set of sensible random action sequences [ [PITH_FULL_IMAGE:figures/full_fig_p038_18.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

75 extracted references · 2 canonical work pages

  1. [1]

    In: International Conference on Machine Learning

    Achiam, J., Held, D., Tamar, A., Abbeel, P.: Constrained Policy Optimization. In: International Conference on Machine Learning. pp. 22–31 (2017)

    2017

  2. [2]

    CRC Press (1999)

    Altman, E.: Constrained Markov Decision Processes. CRC Press (1999)

    1999

  3. [3]

    arXiv:2201.09802 (2022)

    As, Y., Usmanova, I., Curi, S., Krause, A.: Constrained Policy Optimization via Bayesian world models. arXiv:2201.09802 (2022)

    arXiv 2022

  4. [4]

    arXiv:2506.09985 (2025)

    Assran, M., Bardes, A., Fan, D., Garrido, Q., Howes, R., Muckley, M., Rizvi, A., Roberts, C., Sinha, K., Zholus, A., et al.: V-JEPA 2: Self-supervised video models enable understanding, prediction and planning. arXiv:2506.09985 (2025)

    arXiv 2025

  5. [5]

    Bishop, C.M.: Mixture Density Networks. Tech. rep., Aston University (1994)

    1994

  6. [6]

    In: Conference on Robot Learning

    Biyik, E., Sadigh, D.: Batch active preference-based learning of reward functions. In: Conference on Robot Learning. pp. 519–528 (2018) Learning Safe Behaviour from Preferences & Justifications via World Models 39

    2018

  7. [7]

    In: Handbook of statistics, vol

    Botev, Z.I., Kroese, D.P., Rubinstein, R.Y., L’Ecuyer, P.: The cross-entropy method for optimization. In: Handbook of statistics, vol. 31, pp. 35–59. Elsevier (2013)

    2013

  8. [8]

    The method of paired comparisons.Biometrika39(3/4), 324–345 (1952)

    Bradley, R.A., Terry, M.E.: Rank analysis of incomplete block designs: I. The method of paired comparisons.Biometrika39(3/4), 324–345 (1952)

    1952

Show all 75 references
  1. [9]

    arXiv:1606.01540 (2016)

    Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., Zaremba, W.: OpenAI Gym. arXiv:1606.01540 (2016)

    2016 arXiv

  2. [10]

    In: International Conference on Machine Learning

    Brown, D., Goo, W., Nagarajan, P., Niekum, S.: Extrapolating beyond subop- timal demonstrations via Inverse Reinforcement Learning from observations. In: International Conference on Machine Learning. pp. 783–792 (2019)

    2019

  3. [11]

    In: Conference on Robot Learning

    Brown, D.S., Cui, Y., Niekum, S.: Risk-aware active Inverse Reinforcement Learn- ing. In: Conference on Robot Learning. pp. 362–372 (2018)

    2018

  4. [12]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Cheng, R., Orosz, G., Murray, R.M., Burdick, J.W.: End-to-end safe Reinforce- mentLearningthroughbarrierfunctionsforsafety-criticalcontinuouscontroltasks. In: Proceedings of the AAAI Conference on Artificial Intelligence. pp. 3387–3395 (2019)

    2019

  5. [13]

    In: Advances in Neural Information Processing Systems

    Chow,Y.,Nachum,O.,Duenez-Guzman,E.,Ghavamzadeh,M.:ALyapunov-based approach to Safe Reinforcement Learning. In: Advances in Neural Information Processing Systems. pp. 8092–8101 (2018)

    2018

  6. [14]

    In: Advances in Neural Information Processing Systems

    Christiano, P.F., Leike, J., Brown, T., Martic, M., Legg, S., Amodei, D.: Deep Re- inforcement Learning from human preferences. In: Advances in Neural Information Processing Systems. pp. 4299–4307 (2017)

    2017

  7. [15]

    In: The Twelfth International Conference on Learning Representations (2024)

    Dai, J., Pan, X., Sun, R., Ji, J., Xu, X., Liu, M., Wang, Y., Yang, Y.: Safe RLHF: Safe Reinforcement Learning from human feedback. In: The Twelfth International Conference on Learning Representations (2024)

    2024

  8. [16]

    In: Pacific Rim Inter- national Conference on Artificial Intelligence

    Dai, T., Liu, H., Arulkumaran, K., Ren, G., Bharath, A.A.: Diversity-based tra- jectory and goal selection with hindsight experience replay. In: Pacific Rim Inter- national Conference on Artificial Intelligence. pp. 32–45 (2021)

    2021

  9. [17]

    Dunn, O.J.: Multiple comparisons using rank sums.Technometrics6(3), 241–252 (1964)

    1964

  10. [18]

    arXiv:1902.06766 (2019)

    Frye, C., Feige, I.: Parenting: Safe Reinforcement Learning from human input. arXiv:1902.06766 (2019)

    1902 arXiv

  11. [19]

    Automatica25(3), 335–348 (1989)

    Garcia, C.E., Prett, D.M., Morari, M.: Model Predictive Control: Theory and prac- tice—a survey. Automatica25(3), 335–348 (1989)

    1989

  12. [20]

    Journal of Machine Learning Research16(1), 1437–1480 (2015)

    Garcıa, J., Fernández, F.: A comprehensive survey on safe Reinforcement Learning. Journal of Machine Learning Research16(1), 1437–1480 (2015)

    2015

  13. [21]

    IEEE Access7, 165007–165017 (2019)

    Ge, Y., Zhu, F., Ling, X., Liu, Q.: Safe Q-learning method based on constrained markov decision processes. IEEE Access7, 165007–165017 (2019)

    2019

  14. [22]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Goecks, V.G., Gremillion, G.M., Lawhern, V.J., Valasek, J., Waytowich, N.R.: Efficiently combining human demonstrations and interventions for safe training of autonomous systems in real-time. In: Proceedings of the AAAI Conference on Artificial Intelligence. pp. 2462–2470 (2019)

    2019

  15. [23]

    Frontiers in Neurorobotics17, 1280341 (2023)

    Gu, S., Kshirsagar, A., Du, Y., Chen, G., Peters, J., Knoll, A.: A human-centered safe robot Reinforcement Learning framework with interactive behaviors. Frontiers in Neurorobotics17, 1280341 (2023)

    2023

  16. [24]

    IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)

    Gu, S., Yang, L., Du, Y., Chen, G., Walter, F., Wang, J., Knoll, A.: A review of safe Reinforcement Learning: Methods, theories and applications. IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)

    2024

  17. [25]

    In: International Conference on Principles and Practice of Multi-Agent Systems

    Guo, Z., Norman, T.J., Gerding, E.H.: MTIRL: Multi-trainer interactive Reinforce- ment Learning system. In: International Conference on Principles and Practice of Multi-Agent Systems. pp. 227–242 (2022) 40 I. Kazantzidis et al

    2022

  18. [26]

    In: Proceedings of the 32nd International Conference on Neural Information Process- ing Systems

    Ha, D., Schmidhuber, J.: Recurrent world models facilitate policy evolution. In: Proceedings of the 32nd International Conference on Neural Information Process- ing Systems. pp. 2455–2467 (2018)

    2018

  19. [27]

    arXiv:1912.01603 (2019)

    Hafner, D., Lillicrap, T., Ba, J., Norouzi, M.: Dream to control: Learning behaviors by latent imagination. arXiv:1912.01603 (2019)

    1912 arXiv

  20. [28]

    In: International Conference on Machine Learning

    Hafner, D., Lillicrap, T., Fischer, I., Villegas, R., Ha, D., Lee, H., Davidson, J.: Learning latent dynamics for planning from pixels. In: International Conference on Machine Learning. pp. 2555–2565 (2019)

    2019

  21. [29]

    arXiv:2010.02193 (2020)

    Hafner, D., Lillicrap, T., Norouzi, M., Ba, J.: Mastering Atari with discrete world models. arXiv:2010.02193 (2020)

    2010 arXiv

  22. [30]

    Nature640(8059), 647–653 (2025)

    Hafner, D., Pasukonis, J., Ba, J., Lillicrap, T.: Mastering diverse control tasks through world models. Nature640(8059), 647–653 (2025)

    2025

  23. [31]

    Evolutionary Computation9(2), 159–195 (2001)

    Hansen, N., Ostermeier, A.: Completely derandomized self-adaptation in evolution strategies. Evolutionary Computation9(2), 159–195 (2001)

    2001

  24. [32]

    Neural Computation 9(8), 1735–1780 (1997)

    Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Computation 9(8), 1735–1780 (1997)

    1997

  25. [33]

    65–70 (1979)

    Holm, S.: A simple sequentially rejective multiple test procedure.Scandinavian Journal of Statisticspp. 65–70 (1979)

    1979

  26. [34]

    In: International Conference on Learning Represen- tations

    Huang, W., Ji, J., Xia, C., Zhang, B., Yang, Y.: Safedreamer: Safe Reinforcement Learning with world models. In: International Conference on Learning Represen- tations. vol. 2024, pp. 53839–53869 (2024)

    2024

  27. [35]

    IEEE Access (2025)

    Karlaus, J., Schwenker, F.: TEMPO: Timestep explanations for modeling prefer- ences in online preference-based RL. IEEE Access (2025)

    2025

  28. [36]

    Kazantzidis, I.: Towards safe autonomous agents via human input. Ph.D. thesis, University of Southampton (2026)

    2026

  29. [37]

    In: Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems

    Kazantzidis, I., Norman, T.J., Du, Y., Freeman, C.T.: How to train your agent: Active learning from human preferences and justifications in safety-critical envi- ronments. In: Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems. pp. 165...

    2022

  30. [38]

    Kazantzidis, I., Norman, T.J., Du, Y., Freeman, C.T.: Learning safe behaviour via justified human preferences and hypothetical queries (2022).https://doi.org/ 10.21203/rs.3.rs-2406802/v2, Preprint

    2022 doi

  31. [39]

    In: Proceedings of the 18th International Conference on Agents and Artificial Intelligence

    Kazantzidis, I., Norman, T.J., Du, Y., Freeman, C.T.: Safe reward learning from human preferences and justifications. In: Proceedings of the 18th International Conference on Agents and Artificial Intelligence. pp. 489–503 (2026).https:// doi.org/10.5220/0014408700004052

    2026 doi

  32. [40]

    In: International Confer- ence on Learning Representations (2023)

    Kim, C., Park, J., Shin, J., Lee, H., Abbeel, P., Lee, K.: Preference transformer: Modeling human preferences using transformers for RL. In: International Confer- ence on Learning Representations (2023)

    2023

  33. [41]

    arXiv:1312.6114 (2013)

    Kingma, D.P.: Auto-encoding variational Bayes. arXiv:1312.6114 (2013)

    2013 arXiv

  34. [42]

    In: Proceedings of the fifth International Conference on Knowledge Capture

    Knox, W.B., Stone, P.: Interactively shaping agents via human reinforcement: The TAMER framework. In: Proceedings of the fifth International Conference on Knowledge Capture. pp. 9–16 (2009)

    2009

  35. [43]

    Jour- nal of the American statistical Association47(260), 583–621 (1952)

    Kruskal, W.H., Wallis, W.A.: Use of ranks in one-criterion variance analysis. Jour- nal of the American statistical Association47(260), 583–621 (1952)

    1952

  36. [44]

    2, 2022-06-27

    LeCun, Y., et al.: A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review62(1), 1–62 (2022)

    2022

  37. [45]

    In: Inter- national Conference on Machine Learning

    Lee, K., Smith, L., Abbeel, P.: PEBBLE: Feedback-efficient interactive Reinforce- ment Learning via relabeling experience and unsupervised pre-training. In: Inter- national Conference on Machine Learning. pp. 6152–6163 (2021) Learning Safe Behaviour from Preferences & Justific...

    2021

  38. [46]

    arXiv:2111.03026 (2021)

    Lee, K., Smith, L., Dragan, A., Abbeel, P.: B-pref: Benchmarking preference-based Reinforcement Learning. arXiv:2111.03026 (2021)

    2021 arXiv

  39. [47]

    arXiv:1811.07871 (2018)

    Leike, J., Krueger, D., Everitt, T., Martic, M., Maini, V., Legg, S.: Scalable agent alignment via reward modeling: a research direction. arXiv:1811.07871 (2018)

    2018 arXiv

  40. [48]

    Cognitive Computation17(5), 1–16 (2025)

    Li, W., Liu, H., Huang, K., Hussain, A.: Reinforcement Learning for Human-AI col- laboration: Challenges, mechanisms, and methods. Cognitive Computation17(5), 1–16 (2025)

    2025

  41. [49]

    IEEE Transactions on Vehicular Technology (2026)

    Liu, T., Li, S., Tian, Z., Geng, J., Wang, T., Li, T., Yuan, K., Huang, Y., Yu, Z.: Safe RL-based decision-making of autonomous driving with human preference alignment. IEEE Transactions on Vehicular Technology (2026)

    2026

  42. [50]

    In: 2023 IEEE International Conference on Robotics and Automation (ICRA)

    Liu, Y., Datta, G., Novoseller, E., Brown, D.S.: Efficient preference-based Rein- forcement Learning using learned dynamics models. In: 2023 IEEE International Conference on Robotics and Automation (ICRA). pp. 2921–2928 (2023)

    2023

  43. [51]

    In: Proceedings of the 23rd International Conference on Autonomous Agents and Mul- tiagent Systems

    Lou, X., Zhang, J., Wang, Z., Huang, K., Du, Y.: Safe Reinforcement Learning with free-form natural language constraints and pre-trained language models. In: Proceedings of the 23rd International Conference on Autonomous Agents and Mul- tiagent Systems. pp. 1274–1282 (2024)

    2024

  44. [52]

    Advances in Neural Information Processing Systems pp

    Luo, Y., Ma, T.: Learning barrier certificates: Towards safe Reinforcement Learn- ing with zero training-time violations. Advances in Neural Information Processing Systems pp. 25621–25632 (2021)

    2021

  45. [53]

    Journal of Machine Learning research9(Nov), 2579–2605 (2008)

    Maaten, L.v.d., Hinton, G.: Visualizing data using t-SNE. Journal of Machine Learning research9(Nov), 2579–2605 (2008)

    2008

  46. [54]

    In: The Fourteenth International Conference on Learning Representations (2026)

    Mani, K., Pequignot, Y., Mai, V., Paull, L.: SHAPO: Sharpness-aware policy op- timization for safe exploration. In: The Fourteenth International Conference on Learning Representations (2026)

    2026

  47. [55]

    50–60 (1947)

    Mann, H.B., Whitney, D.R.: On a test of whether one of two random variables is stochastically larger than the other.The annals of mathematical statisticspp. 50–60 (1947)

    1947

  48. [56]

    Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K., Ostrovski, G., et al.: Human-level control through Deep Reinforcement Learning.Nature518(7540), 529–533 (2015)

    2015

  49. [57]

    In: Pro- ceedings of the Seventeenth International Conference on Machine Learning

    Ng, A.Y., Russell, S.J.: Algorithms for inverse reinforcement learning. In: Pro- ceedings of the Seventeenth International Conference on Machine Learning. pp. 663–670 (2000)

    2000

  50. [58]

    Advances in Neural Information Processing Systems pp

    Ouyang,L.,Wu,J.,Jiang,X.,Almeida,D.,Wainwright,C.,Mishkin,P.,Zhang,C., Agarwal, S., Slama, K., Ray, A., et al.: Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems pp. 27730–27744 (2022)

    2022

  51. [59]

    In: 10th International Conference on Learning Representa- tions, ICLR 2022 (2022)

    Park, J., Seo, Y., Shin, J., Lee, H., Abbeel, P., Lee, K.: SURF: Semi-supervised reward learning with data augmentation for feedback-efficient preference-based Re- inforcement Learning. In: 10th International Conference on Learning Representa- tions, ICLR 2022 (2022)

    2022

  52. [60]

    Advances in neural information processing systems36, 53728–53741 (2023)

    Rafailov, R., Sharma, A., Mitchell, E., Manning, C.D., Ermon, S., Finn, C.: Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems36, 53728–53741 (2023)

    2023

  53. [61]

    In: Learning for Dynamics and Control

    Rafailov, R., Yu, T., Rajeswaran, A., Finn, C.: Offline Reinforcement Learning from images with latent space models. In: Learning for Dynamics and Control. pp. 1154–1168 (2021)

    2021

  54. [62]

    arXiv:2201.08102 (2022) 42 I

    Rahtz, M., Varma, V., Kumar, R., Kenton, Z., Legg, S., Leike, J.: Safe Deep RL in 3D environments using human feedback. arXiv:2201.08102 (2022) 42 I. Kazantzidis et al

    2022 arXiv

  55. [63]

    In: International Conference on Machine Learn- ing

    Reddy, S., Dragan, A., Levine, S., Legg, S., Leike, J.: Learning human objectives by evaluating hypothetical behavior. In: International Conference on Machine Learn- ing. pp. 8020–8029 (2020)

    2020

  56. [64]

    In: Proceedings of the 17th Interna- tional Conference on Autonomous Agents and MultiAgent Systems

    Saunders, W., Sastry, G., Stuhlmueller, A.: Trial without error: Towards safe Re- inforcement Learning via human intervention. In: Proceedings of the 17th Interna- tional Conference on Autonomous Agents and MultiAgent Systems. pp. 2067–2069 (2018)

  57. [65]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Shi, Z., Fang, M., Chen, L., Du, Y., Wang, J.: Human-guided moral decision mak- ing in text-based games. In: Proceedings of the AAAI Conference on Artificial Intelligence. pp. 21574–21582 (2024)

    2024

  58. [66]

    Silver, D., Huang, A., Maddison, C.J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al.: Master- ing the game of Go with Deep Neural Networks and tree search.Nature529(7587), 484 (2016)

    2016

  59. [67]

    12151–12162 (2020)

    Turchetta, M., Kolobov, A., Shah, S., Krause, A., Agarwal, A.: Safe Reinforcement Learning via curriculum induction.Advances in Neural Information Processing Systemspp. 12151–12162 (2020)

    2020

  60. [68]

    Vinyals, O., Babuschkin, I., Czarnecki, W.M., Mathieu, M., Dudzik, A., Chung, J., Choi, D.H., Powell, R., Ewalds, T., Georgiev, P., et al.: Grandmaster level in starcraft II using multi-agent Reinforcement Learning.Nature575(7782), 350–354 (2019)

    2019

  61. [69]

    Advances in Neural Information Processing Systems36, 29252–29272 (2023)

    Wachi, A., Hashimoto, W., Shen, X., Hashimoto, K.: Safe exploration in Rein- forcement Learning: A generalized formulation and algorithms. Advances in Neural Information Processing Systems36, 29252–29272 (2023)

    2023

  62. [70]

    Advances in Neural Information Processing Systems34, 20759–20771 (2021)

    Wachi, A., Wei, Y., Sui, Y.: Safe policy optimization with local generalized linear function approximations. Advances in Neural Information Processing Systems34, 20759–20771 (2021)

    2021

  63. [71]

    In: Proceedings of the AAAI conference on artificial intelligence

    Warnell, G., Waytowich, N., Lawhern, V., Stone, P.: Deep tamer: Interactive agent shaping in high-dimensional state spaces. In: Proceedings of the AAAI conference on artificial intelligence. vol. 32 (2018)

    2018

  64. [72]

    In: 9th International Conference on Learning Representations, ICLR 2021 (2021)

    Yarats, D., Fergus, R., Kostrikov, I.: Image augmentation is all you need: Regular- izing Deep Reinforcement Learning from pixels. In: 9th International Conference on Learning Representations, ICLR 2021 (2021)

    2021

  65. [73]

    In: Deep RL Workshop NeurIPS 2021 (2021)

    Yarats, D., Fergus, R., Lazaric, A., Pinto, L.: Mastering visual continuous con- trol: Improved data-augmented Reinforcement Learning. In: Deep RL Workshop NeurIPS 2021 (2021)

    2021

  66. [74]

    Advances in Neural Information Processing Systems35, 2608–2621 (2022)

    Yu, H., Xu, W., Zhang, H.: Towards safe Reinforcement Learning with a safety editor policy. Advances in Neural Information Processing Systems35, 2608–2621 (2022)

    2022

  67. [75]

    In: International Conference on Machine Learning

    Zhao, H., Sylvain, T., Laroche, R., Precup, D., Bengio, Y.: Rejecting hallucinated state targets during planning. In: International Conference on Machine Learning. pp. 77677–77702 (2025)

    2025

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.