Pith. sign in

REVIEW 5 major objections 5 minor 38 references

SPLASH! Sample-efficient Preference-based inverse reinforcement learning for Long-horizon Adversarial tasks from Suboptimal Hierarchical demonstrations

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces SPLASH, a preference-based inverse reinforcement learning method that recovers reward functions for long-horizon adversarial tasks from suboptimal hierarchical demonstrations.

desk verdict A sensible extension of D-REX to long-horizon adversarial tasks, but the preference-labeling step is under-specified and the evaluation doesn't close the IRL loop. read the letter →

arxiv 2507.08707 v1 pith:4ONGHOOZ submitted 2025-07-11 cs.LG cs.RO

Peter Crowley , Zachary Serlin , Tyler Paine , Makai Mann , Michael Benjamin , Calin Belta This is my paper · ORCID
classification cs.LGcs.RO
keywords inversereinforcementlearningpreference-basedIRLsuboptimaldemonstrationshierarchicaloptionsframeworklong-horizontasksadversarialmaritimecapture-the-flag
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SPLASH is a preference-based inverse reinforcement learning method for recovering the reward function of a long-horizon adversarial task from demonstrations that are suboptimal and collected at the level of high-level behaviors, called options. The paper's central claim is that this is the first IRL approach designed for suboptimal demonstrations of long-horizon and adversarial tasks, and that it significantly outperforms the existing D-REX method at reward learning in the maritime capture-the-flag setting. If the claim is right, reward learning becomes usable for field robotics problems where expert demonstrations are unavailable and where handcrafted or sparse rewards are unreliable. The paper supports the claim with simulated capture-the-flag experiments, ablations, and a sim-to-real deployment of the learned options-level policy on unmanned surface vehicles.

What carries the argument

The load-bearing object is the SPLASH loss function, $L(\theta) = L_{\mathrm{PBIRL}}(\theta) + \lambda_{\mathrm{IF}} L_{\mathrm{IF}}(\theta) + L_{\mathrm{dR}}(\theta)$, which combines three mechanisms. $L_{\mathrm{PBIRL}}$ is a binary cross-entropy preference loss over pairs of full, downsampled trajectories labeled by a D-REX-style noise schedule; pairs are kept only when the noise ranking agrees with the task's score and success criteria. $L_{\mathrm{IF}}$ is an initial-final state comparison that encodes the zero-sum structure of adversarial games by requiring the final reward of a winning trajectory to lie above its initial reward and the final reward of a losing trajectory to lie below it. $L_{\mathrm{dR}}$ penalizes large first and second differences of reward along a trajectory, smoothing the learned reward. Around this sits the options-level demonstration scheme: trajectories are sequences of high-level behaviors such as attack, defend, and retreat, with predefined control primitives, which makes behavioral cloning sample-efficient enough to generate rollouts for preference labeling.

What would settle it

Roll out the cloned options policy under the exact training noise schedule and compare mean ground-truth score per noise bin; if mean score does not strictly decrease as $\epsilon$ increases, the automatic preference labels are wrong and the recovered reward cannot be trusted. A second check is to compute whether SPLASH's predicted returns rank-order with ground-truth returns on held-out trajectories better than D-REX's; if not, the claimed advantage is absent.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that a preference-based IRL recipe can turn suboptimal hierarchical demonstrations into a reward function that tracks both task performance and moment-to-moment progress in an adversarial game. The paper presents SPLASH as the first inverse reinforcement learning approach designed to learn from suboptimal demonstrations of long-horizon and adversarial tasks. Its objective is a pairwise preference loss over full downsampled trajectories, with labels generated by a noise schedule and filtered by score and success criteria, plus an initial-final state comparison that forces winning trajectories to end above their start and losing trajectories to end below it, plus temporal consistency regularization. In the maritime capture-the-flag domain, SPLASH's predicted returns align with ground-truth returns better than D-REX's, its per-step reward marks captures by both teams, and the options-level policy learned from fifty demonstrations transfers to real unmanned surface vehicles.

Load-bearing premise

The load-bearing premise is that injecting more noise into the cloned options policy produces strictly worse trajectories, so the automatically generated preference labels are mostly correct; SPLASH's score and success checks remove only the obvious contradictions and do not test this premise.

Editorial extensions

If this is right

  • SPLASH extends reward learning from suboptimal demonstrations to long-horizon and adversarial tasks, where snippet-based ranking fails because partial trajectories can demonstrate different subtasks.
  • The reward functions learned by SPLASH extrapolate beyond the performance of the demonstrations, so a downstream reinforcement learning agent should be able to exceed the demonstrator's level.
  • Options-level demonstration collection is dramatically more sample-efficient than low-level behavioral cloning: fifty options-level demonstrations produced a capture-the-flag policy that won 99 of 100 simulated games, while low-level cloning won 2 of 100.
  • The score and success filters plus temporal regularization clean the reward signal enough to show captures and losses as clear rises and falls during a game.
  • The options-level policy that carried the learned reward in simulation also competed on real unmanned surface vehicles, supporting sim-to-real transfer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: because SPLASH prunes pairs whose noise-based labels contradict measured score or success, it may tolerate some violations of the noise-degradation assumption; deliberately running a non-monotonic noise schedule would quantify that tolerance.
  • Inference: the initial-final state comparison loss encodes zero-sum progress in a compact way that should port to other competitive tasks with a defined win, loss, or draw condition.
  • Inference: options-level data collection shifts the human burden from fine-grained low-level control to choosing high-level behaviors, which could make demonstration collection cheaper and safer on physical systems.
  • Inference: a natural next test is to close the loop by training an agent on SPLASH's learned reward and measuring wins against the baseline, rather than only measuring how well the reward tracks ground-truth returns.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces SPLASH, a preference-based inverse reinforcement learning method that extends D-REX to long-horizon, adversarial tasks with suboptimal demonstrations. The main components are options-level demonstrations with behavioral cloning, comparisons over downsampled full trajectories, automatic preference labeling from a noise schedule followed by pruning using task score and success criteria, initial-final state reward constraints, and first/second-order temporal consistency regularization. The method is evaluated on a maritime capture-the-flag simulation and with three hardware games on autonomous surface vehicles, with the claim that SPLASH significantly outperforms D-REX in reward learning from suboptimal demonstrations.

Significance. If the reported results are robust, SPLASH provides a useful practical recipe for reward learning in long-horizon adversarial tasks, an underexplored regime for preference-based IRL. The options-level behavioral cloning result is a concrete and potentially valuable empirical finding: the options-BC policy wins 99/100 simulated games with average score 1.89, whereas vanilla BC wins 2/100 with average score -1.86. The paper also demonstrates sim-to-real transfer of the cloned policy on physical vehicles, and the use of an open-source CTF environment supports reproducibility. However, the strength of the central claim is limited by the evaluation design: the reward is never used to train a downstream RL policy, the reported comparisons lack statistical quantification, and the score/success signal used for training also serves as the evaluation ground truth.

major comments (5)
  1. [Algorithm 1, lines 6-11; Section III-C, Eq. (1)] As printed, the preference direction in Algorithm 1 is inverted relative to the paper's own notation. Section III-C defines tau_i ≺ tau_j to mean that tau_j is preferential to tau_i, and Eq. (1) trains the reward to assign higher return to tau_j. Algorithm 1 loops over epsilon_i < epsilon_j (so tau_i is the lower-noise, supposedly better rollout) and adds (tau_i ≺ tau_j) whenever psi(tau_i) <= psi(tau_j) and eta(tau_i) <= eta(tau_j). Under the stated convention this labels the higher-noise trajectory as preferred, the opposite of D-REX's rule and of the prose in Section IV-B. If the implementation follows the pseudocode, the training labels are inverted and Fig. 3 would not reflect the intended algorithm; if the implementation follows the prose, the pseudocode must be corrected. This is a load-bearing point because Eq. (11) is trained entirely on these automatically generated labels.
  2. [Section V-C, Fig. 3] The evaluation in Fig. 3 is partially circular. The 'ground truth returns' axis is the same score function eta used in Algorithm 1 to prune preference pairs and in Eqs. (5)-(7) to define the success constraint psi. The training loss therefore already encourages consistency with this metric, and the pruning removes the pairs that most strongly violate it. A positive correlation on this axis is partly a self-consistency check rather than an external validation of the learned reward. Please report results against an independent ground truth — for example, human preference labels on held-out trajectory pairs, or the performance of an RL policy trained on the learned reward — and also report the alignment separately for pairs that passed the pruning filter versus those that were removed.
  3. [Section III-C and Section IV-B, Eq. (4)] The central monotonicity assumption of D-REX — that higher epsilon produces strictly worse trajectories — is not validated for the CTF task. SPLASH relies on this assumption to generate preference labels from the noise schedule, but the paper reports only aggregate scores for options-BC (1.89) and vanilla BC (-1.86) and never reports eta(tau) by noise bin for epsilon in {0.5, 0.67, 0.83, 1.0}. In a long-horizon adversarial game, larger noise could occasionally lead to better exploration outcomes, and the pruning filter in Algorithm 1 removes only pairs whose score/success labels directly contradict the noise-based label, not pairs that are mislabeled but happen to satisfy the filter. Please provide the per-bin score distributions and, if feasible, validate a sample of the automatically generated preference labels against human judgments.
  4. [Section V-D and Section V-C] The IRL loop is never closed: the learned reward function is not used to train a downstream RL policy. The hardware experiments in Section V-D use the options-BC policy directly, so they demonstrate sim-to-real transfer of behavioral cloning, not of the reward learned by SPLASH. To support the claim that SPLASH produces rewards that are useful for RL, the paper should include at least one experiment in which a policy is trained with the SPLASH reward, compared against a policy trained with the D-REX reward and against the demonstrator's performance. Without this, the central motivation of recovering a reward for downstream policy optimization remains untested.
  5. [Section V, Figs. 3-4, Table I] The empirical support for 'significantly outperform' is statistically thin. Fig. 3 and Fig. 4 show no confidence intervals, no multiple-seed results, and no significance tests, and Table I reports only three hardware games with a 1-1-1 record. The phrase 'significantly lower deviation' and the abstract's 'significantly outperform the state-of-the-art' are not supported by any statistical procedure. Please report means and variances over multiple seeds for the simulated experiments, add a paired test or effect-size measure for the SPLASH versus D-REX comparison, and describe how the single trajectory in Fig. 4 was selected.
minor comments (5)
  1. [Eq. (8)] The denominator in Eq. (8) reads '|τj| + |τj| − 2' and '|τj| + |τj| − 4'; this appears to be a typo for |τi| + |τj| − 2 and |τi| + |τj| − 4, and it should be corrected.
  2. [Eq. (4)] In Eq. (4), the condition 'if πΩBC(s) = a' uses an action variable a where an option variable ω would be consistent with the option-level notation; please clarify.
  3. [Section V-B] The success criterion for demonstrations is described as η(τ) > 0 in Section V-B, while the formal success function ψ is defined earlier; the relationship between ψ and η should be stated explicitly so that 'successful demonstration' is unambiguous.
  4. [Section V-C] The paper states that D-REX was trained with the same number of trajectory pairs (256,000). Given M=100 rollouts per noise level and four noise levels, the number of cross-noise pairs is 60,000 before any augmentation, so the source of 256,000 pairs should be explained (for example, whether pairs are counted per agent or with repetition).
  5. [Section V-C] Fig. 3's normalization of predicted returns is described only briefly; please specify the normalization procedure and whether it is applied identically to SPLASH and D-REX.

Circularity Check

2 steps flagged · score 6.0 of 10

Evaluation is partly circular: the same score η used to prune preference pairs and define the IF loss is used as the 'ground truth' in Fig. 3, and the printed Algorithm 1 derives preference direction from η rather than from the noise schedule.

  1. fitted input called prediction [Section IV-B (trajectory pair pruning) and Section V-C (Fig. 3 evaluation)]
    "To qualify for the training dataset, SPLASH requires that a trajectory pair τi ≺ τj satisfies ψ(τi) ≤ ψ(τj) and η(τi) ≤ η(τj). That is, the higher-ranked trajectory must have a score and degree of success greater than or equal to that of the lower-ranked trajectory. ... In Fig. 3, we compare the alignment of SPLASH and D-REX to the ground truth reward."

    The score η is an input to training-data construction: only pairs whose η (and ψ) ordering matches the intended preference are kept, and the PBIRL loss (Eq. 11) is trained on those pairs. The evaluation in Fig. 3 then uses the same η as 'Ground Truth Returns' for the blue training rollouts. Thus SPLASH's agreement with η on training pairs is enforced by construction rather than discovered, and the comparison with D-REX, which never receives η, is not a fair test of reward learning. The green held-out lower-noise extrapolation points are less affected, but the headline figure pools them with in-sample points, making the reported 'alignment' partly self-consistent rather than externally validated.

  2. other [Algorithm 1, lines 6–11, with the preference convention defined in Section III-C]
    "τi ≺ τj indicates that τj is preferential to τi. ... for ϵi ∈ E do ... for ϵj ∈ E s.t. ϵi < ϵj do ... if ψ(τi) ≤ ψ(τj) ∧ η(τi) ≤ η(τj) then Add (τi ≺ τj) to P"

    Under the paper's own convention, adding (τ_i ≺ τ_j) declares τ_j preferred. The loop takes ϵ_i < ϵ_j, so τ_j is the higher-noise rollout; the guard admits the pair exactly when the higher-noise rollout has η(τ_j) ≥ η(τ_i) and ψ(τ_j) ≥ ψ(τ_i). The preference direction is therefore read off from η and ψ, not from D-REX's noise-based ranking rule. Rewards trained with Eq. (11) on these pairs are fit to rank by η, and Fig. 3 then scores predicted returns against η as 'ground truth.' If the implementation instead follows the prose in Section IV-B, the pseudocode's preference direction is inverted; either way, the printed algorithm makes η the effective label source, so the reported alignment is partly by construction.

full rationale

SPLASH is a genuine engineering contribution: it adds options-level behavioral cloning, full-trajectory downsampling, score/success filtering, an initial-final progress loss, and smoothness regularization to D-REX. The noise-based preference generation from a trained BC policy is an independent signal, and the green held-out lower-noise rollouts in Fig. 3 provide some non-circular extrapolation evidence. However, the central evaluation is partially circular. The score function η is fed into the training-data construction via the pair-pruning condition in Section IV-B and Algorithm 1, and the same η is then used as the ground truth against which predicted returns are plotted in Fig. 3. D-REX is never given η, so SPLASH's better alignment on training rollouts is partly a self-consistency check rather than evidence of superior reward learning. The printed Algorithm 1 aggravates this: under the paper's own '≺' convention, the guard η(τ_i) ≤ η(τ_j) causes the preference label to be assigned to the higher-η, higher-noise rollout, so η, not the noise schedule, determines the label direction. The paper also never validates D-REX's monotonicity assumption for the CTF task; that is a correctness risk rather than a circularity. No load-bearing self-citation or uniqueness-theorem circularity is present: the cited environment is an open benchmark and is not doing argumentative work. On balance, the central claim is partially undermined rather than fully forced, because the extrapolation to withheld noise levels retains independent content; a moderate circularity score of 6 is therefore appropriate.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The algorithm contributes no new physical entities. It does rely on several hand-set losses, a manually chosen noise schedule, and domain assumptions about noise preferences, score signals, and BC coverage. These choices are the practical price of the method and should be stress-tested in follow-up work.

free parameters (5)
  • lambda_1 (first-order temporal consistency weight) = 10
    Set by hand in Section V-C; no sensitivity analysis or principled selection is reported.
  • lambda_2 (second-order temporal consistency weight) = 20
    Set by hand in Section V-C; no sensitivity analysis is reported.
  • lambda_IF (initial-final constraint weight) = 4
    Set by hand in Section V-C; no sensitivity analysis is reported.
  • Noise schedule E = (0.5, 0.67, 0.83, 1.0)
    Chosen in Section V-C with lower noise withheld to test extrapolation; this design choice shapes the reported extrapolation gap.
  • Trajectory downsampling rate = 40 timesteps per sample
    Hand-selected in Section V-C to balance trajectory length and information; affects which states the reward is trained on.
assumptions (4)
  • domain assumption Noise injection into the BC policy produces monotonically decreasing trajectory quality (D-REX assumption).
    Section III-C Eq. 2 and Section IV-B Eq. 4: all preference labels derive from the epsilon schedule; SPLASH prunes contradictions but does not test monotonicity.
  • domain assumption The task score eta and success criterion psi are reliable proxies for the demonstrator's true intent.
    Section IV-B: pairs are kept only when eta(τ_i) <= eta(τ_j) and psi(τ_i) <= psi(τ_j); if eta and psi miss the true objective, pruning removes useful demonstrations.
  • domain assumption Behavioral cloning on 50 options-level demonstrations captures a representative policy-over-options.
    Section V-B: the options-BC policy wins 99 out of 100 games against the heuristic, but this does not guarantee coverage of the full state space or reliability under noise.
  • domain assumption Full trajectories of equal length can be compared meaningfully after downsampling.
    Section IV-B: SPLASH relies on full-length trajectory pairs; variable rollout lengths due to early termination or different capture times can distort preference comparisons, and no handling is described.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SPLASH! Sample-efficient Preference-based inverse reinforcement learning for Long-horizon Adversarial tasks from Suboptimal Hierarchical demonstrations." pith.science (2026). https://pith.science/paper/4ONGHOOZ

@misc{pith2026250708707,
  author       = {Pith},
  title        = {Pith review of: SPLASH! Sample-efficient Preference-based inverse reinforcement learning for Long-horizon Adversarial tasks from Suboptimal Hierarchical demonstrations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4ONGHOOZ}},
  note         = {Machine review of arXiv:2507.08707}
}
read the original abstract

Inverse Reinforcement Learning (IRL) presents a powerful paradigm for learning complex robotic tasks from human demonstrations. However, most approaches make the assumption that expert demonstrations are available, which is often not the case. Those that allow for suboptimality in the demonstrations are not designed for long-horizon goals or adversarial tasks. Many desirable robot capabilities fall into one or both of these categories, thus highlighting a critical shortcoming in the ability of IRL to produce field-ready robotic agents. We introduce Sample-efficient Preference-based inverse reinforcement learning for Long-horizon Adversarial tasks from Suboptimal Hierarchical demonstrations (SPLASH), which advances the state-of-the-art in learning from suboptimal demonstrations to long-horizon and adversarial settings. We empirically validate SPLASH on a maritime capture-the-flag task in simulation, and demonstrate real-world applicability with sim-to-real translation experiments on autonomous unmanned surface vehicles. We show that our proposed methods allow SPLASH to significantly outperform the state-of-the-art in reward learning from suboptimal demonstrations.

Figures

Figures reproduced from arXiv: 2507.08707 by the authors.

Figure 1
Figure 1. Autonomous Heron surface vehicles playing capture-the-flag on [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. A CTF game played out in the Python simulator where the numbers [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Reward extrapolation plots comparing SPLASH and D-REX pre [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: GPS visualization of real-world Heron surface vehicles 824 seconds [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 4
Figure 4. Figure 4: Blue team reward signals of SPLASH and D-REX for a game of [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 21 canonical work pages

  1. [1]

    Human-level control through deep reinforcement learning,

    V . Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. , “Human-level control through deep reinforcement learning,” nature, vol. 518, no. 7540, pp. 529–533, 2015

    2015

  2. [2]

    R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction. MIT press, 2018

    2018

  3. [3]

    Prioritized experi- ence replay,

    T. Schaul, J. Quan, I. Antonoglou, and D. Silver, “Prioritized experi- ence replay,” arXiv preprint arXiv:1511.05952 , 2015

    arXiv 2015

  4. [4]

    Deep reinforcement learning with double q-learning,

    H. Van Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with double q-learning,” in Proceedings of the AAAI conference on artificial intelligence, vol. 30, no. 1, 2016

    2016

  5. [5]

    Dueling network architectures for deep reinforcement learning,

    Z. Wang, T. Schaul, M. Hessel, H. Hasselt, M. Lanctot, and N. Freitas, “Dueling network architectures for deep reinforcement learning,” in International conference on machine learning . PMLR, 2016, pp. 1995–2003

    2016

  6. [6]

    Rainbow: Combining improvements in deep reinforcement learning,

    M. Hessel, J. Modayil, H. Van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver, “Rainbow: Combining improvements in deep reinforcement learning,” inProceed- ings of the AAAI conference on artificial intelligence , vol. 32, no. 1, 2018

    work page 2018

  7. [7]

    Agent57: Outperforming the atari human benchmark,

    A. P. Badia, B. Piot, S. Kapturowski, P. Sprechmann, A. Vitvitskyi, Z. D. Guo, and C. Blundell, “Agent57: Outperforming the atari human benchmark,” in International conference on machine learning . PMLR, 2020, pp. 507–517

    work page 2020

  8. [8]

    Human-level atari 200x faster,

    S. Kapturowski, V . Campos, R. Jiang, N. Raki ´cevi´c, H. van Hasselt, C. Blundell, and A. P. Badia, “Human-level atari 200x faster,” arXiv preprint arXiv:2209.07550, 2022

    arXiv 2022

Show all 38 references
  1. [9]

    Hind- sight experience replay,

    M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welin- der, B. McGrew, J. Tobin, O. Pieter Abbeel, and W. Zaremba, “Hind- sight experience replay,” Advances in neural information processing systems, vol. 30, 2017

    2017

  2. [10]

    Policy invariance under reward transformations: Theory and application to reward shaping,

    A. Y . Ng, D. Harada, and S. Russell, “Policy invariance under reward transformations: Theory and application to reward shaping,” in Icml, vol. 99. Citeseer, 1999, pp. 278–287

    1999

  3. [11]

    Concrete problems in ai safety,

    D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Man ´e, “Concrete problems in ai safety,” arXiv preprint arXiv:1606.06565, 2016

    2016 arXiv

  4. [12]

    Algorithms for inverse reinforcement learning

    A. Y . Ng, S. Russell, et al. , “Algorithms for inverse reinforcement learning.” in Icml, vol. 1, 2000, p. 2

    2000

  5. [13]

    A survey of inverse reinforcement learning: Challenges, methods and progress,

    S. Arora and P. Doshi, “A survey of inverse reinforcement learning: Challenges, methods and progress,” Artificial Intelligence, vol. 297, p. 103500, 2021

    2021

  6. [14]

    Maximum entropy inverse reinforcement learning

    B. D. Ziebart, A. L. Maas, J. A. Bagnell, A. K. Dey, et al., “Maximum entropy inverse reinforcement learning.” in Aaai, vol. 8. Chicago, IL, USA, 2008, pp. 1433–1438

    2008

  7. [15]

    Relative entropy inverse reinforcement learning,

    A. Boularias, J. Kober, and J. Peters, “Relative entropy inverse reinforcement learning,” in Proceedings of the fourteenth international conference on artificial intelligence and statistics . JMLR Workshop and Conference Proceedings, 2011, pp. 182–189

    2011

  8. [16]

    Guided cost learning: Deep inverse optimal control via policy optimization,

    C. Finn, S. Levine, and P. Abbeel, “Guided cost learning: Deep inverse optimal control via policy optimization,” in International conference on machine learning . PMLR, 2016, pp. 49–58

    2016

  9. [17]

    Preference-learning based inverse reinforcement learning for dialog control,

    H. Sugiyama, T. Meguro, and Y . Minami, “Preference-learning based inverse reinforcement learning for dialog control,” in Thirteenth An- nual Conference of the International Speech Communication Associ- ation, 2012

    2012

  10. [18]

    Model-free preference- based reinforcement learning,

    C. Wirth, J. F ¨urnkranz, and G. Neumann, “Model-free preference- based reinforcement learning,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 30, no. 1, 2016

    2016

  11. [19]

    Reward learning from human preferences and demonstrations in atari,

    B. Ibarz, J. Leike, T. Pohlen, G. Irving, S. Legg, and D. Amodei, “Reward learning from human preferences and demonstrations in atari,” Advances in neural information processing systems , vol. 31, 2018

    2018

  12. [20]

    Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations,

    D. Brown, W. Goo, P. Nagarajan, and S. Niekum, “Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations,” in International conference on machine learning . PMLR, 2019, pp. 783–792

    2019

  13. [21]

    Learning reward functions by integrating human demonstrations and preferences,

    M. Palan, N. C. Landolfi, G. Shevchuk, and D. Sadigh, “Learning reward functions by integrating human demonstrations and preferences,” CoRR, vol. abs/1906.08928, 2019. [Online]. Available: http://arxiv.org/abs/1906.08928

    1906 arXiv

  14. [22]

    Better-than-demonstrator imitation learning via automatically-ranked demonstrations,

    D. S. Brown, W. Goo, and S. Niekum, “Better-than-demonstrator imitation learning via automatically-ranked demonstrations,” in Con- ference on robot learning . PMLR, 2020, pp. 330–359

    2020

  15. [23]

    Learning from suboptimal demonstration via self-supervised reward regression,

    L. Chen, R. Paleja, and M. Gombolay, “Learning from suboptimal demonstration via self-supervised reward regression,” in Conference on robot learning . PMLR, 2021, pp. 1262–1277

    2021

  16. [24]

    Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning,

    R. S. Sutton, D. Precup, and S. Singh, “Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning,” Artificial intelligence, vol. 112, no. 1-2, pp. 181–211, 1999

    1999

  17. [25]

    Pyquaticus capture the flag gymnasium,

    Z. Serlin, M. Mann, P. Crowley, T. Gonsalves, and J. Kleim, “Pyquaticus capture the flag gymnasium,” https://github.com/ mit-ll-trusted-autonomy/pyquaticus, 2024

    2024

  18. [26]

    Nested autonomy for unmanned marine vehicles with moos-ivp,

    M. R. Benjamin, H. Schmidt, P. M. Newman, and J. J. Leonard, “Nested autonomy for unmanned marine vehicles with moos-ivp,” Journal of Field Robotics , vol. 27, no. 6, pp. 834–875, 2010

    2010

  19. [27]

    Efficient training of artificial neural networks for autonomous navigation,

    D. A. Pomerleau, “Efficient training of artificial neural networks for autonomous navigation,” Neural computation, vol. 3, no. 1, pp. 88–97, 1991

    1991

  20. [28]

    Behavioral cloning from observation,

    F. Torabi, G. Warnell, and P. Stone, “Behavioral cloning from observation,” in Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18 . International Joint Conferences on Artificial Intelligence Organization, 7 2018, pp. 4950–

    2018

  21. [29]

    Generative adversarial nets,

    I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y . Bengio, “Generative adversarial nets,” Advances in neural information processing systems , vol. 27, 2014

    2014

  22. [30]

    Generative adversarial imitation learning,

    J. Ho and S. Ermon, “Generative adversarial imitation learning,” Advances in neural information processing systems , vol. 29, 2016

    2016

  23. [31]

    Learning robust rewards with adversarial inverse reinforcement learning,

    J. Fu, K. Luo, and S. Levine, “Learning robust rewards with adversarial inverse reinforcement learning,” arXiv preprint arXiv:1710.11248 , 2017

    2017 arXiv

  24. [32]

    Inverse reinforcement learning for video games,

    A. Tucker, A. Gleave, and S. Russell, “Inverse reinforcement learning for video games,” arXiv preprint arXiv:1810.10593 , 2018

    2018 arXiv

  25. [33]

    A survey of preference-based reinforcement learning methods,

    C. Wirth, R. Akrour, G. Neumann, J. F ¨urnkranz, et al. , “A survey of preference-based reinforcement learning methods,” Journal of Ma- chine Learning Research , vol. 18, no. 136, pp. 1–46, 2017

    2017

  26. [34]

    Optiongan: Learning joint reward-policy options using generative adversarial inverse reinforcement learning,

    P. Henderson, W.-D. Chang, P.-L. Bacon, D. Meger, J. Pineau, and D. Precup, “Optiongan: Learning joint reward-policy options using generative adversarial inverse reinforcement learning,” in Proceedings of the AAAI conference on artificial intelligence , vol. 32, no. 1, 2018

    2018

  27. [35]

    Hierarchical relative entropy policy search,

    C. Daniel, G. Neumann, and J. Peters, “Hierarchical relative entropy policy search,” in Artificial Intelligence and Statistics . PMLR, 2012, pp. 273–281

    2012

  28. [36]

    Deep reinforcement learning from human preferences,

    P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei, “Deep reinforcement learning from human preferences,” Advances in neural information processing systems , vol. 30, 2017

    2017

  29. [37]

    Inverse reinforcement learning from failure,

    K. Shiarlis, J. Messias, and S. Whiteson, “Inverse reinforcement learning from failure,” 2016

    2016

  30. [4957]

    Available: https://doi.org/10.24963/ijcai.2018/687

    [Online]. Available: https://doi.org/10.24963/ijcai.2018/687

    2018 doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.