Pith. sign in

REVIEW 4 major objections 5 minor 60 references

Beyond-Expert Performance with Limited Demonstrations: Efficient Imitation Learning with Double Exploration

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ILDE trains an imitation agent with two exploration signals—an optimism bonus for uncertain state-action pairs and a curiosity reward for deviations from the demonstrations—and the paper claims this lets a policy exceed the expert's…

desk verdict The empirical ILDE recipe is genuinely interesting and honestly reported, but the central theoretical claim as stated does not deliver a sublinear regret bound because the additive eps_E T term makes it linear for any fixed demonstration set. read the letter →

arxiv 2506.20307 v1 pith:4E3CJZ7I submitted 2025-06-25 cs.LG cs.AI

Heyang Zhao , Xingrui Yu , David M. Bossens , Ivor W. Tsang , Quanquan Gu This is my paper · ORCID
classification cs.LGcs.AI MSC 68T0590C4068Q32
keywords imitationlearningdoubleexplorationcuriosity-drivenoptimisticpolicyoptimizationbonusgeneralizedEluderdimensionregretboundbeyond-expertperformance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ILDE is an imitation-learning algorithm that trains a policy from expert demonstrations using two exploration signals at once: an optimism bonus for state-action pairs whose values are hard to predict from the data collected so far, and a curiosity reward for visiting states that deviate from the demonstration trajectories. The paper tries to establish that this double exploration makes imitation learning sample-efficient enough that an agent can surpass, not just copy, the expert: it reports an average return 5.33 times the expert's across six Atari games from only 10% of a one-life demonstration. The paper further derives a regret bound for a natural-policy-gradient version of ILDE, which it claims is the first theoretical guarantee for imitation learning in MDPs with nonlinear function approximation. If these claims hold, sparse demonstrations are not a ceiling for imitation learning, and curiosity-driven exploration can be combined with principled optimism inside a single algorithm.

What carries the argument

The load-bearing object is the uncertainty-regularized discrepancy $\ell(\pi, r) = J(\pi_E, r) - J(\pi, r) - \lambda \cdot \mathrm{Int}(\pi; \tau^E)$, where $\pi_E$ is the expert policy, $J$ is expected cumulative reward under reward $r$, and $\mathrm{Int}(\pi; \tau^E)$ is the expected cumulative intrinsic reward derived from a transition model trained on the demonstrations. The algorithm alternates between updating $r$ by projected gradient descent on an empirical estimate of this discrepancy and updating the policy by natural mirror descent on an optimistic Q-estimate. The exploration bonus in that Q-estimate is the $D_{\mathcal{F}}$-uncertainty, a generalized-Eluder notion of how surprising a state-action pair remains given the collected data; this supplies the optimism. The curiosity term $\lambda\,\mathrm{Int}$ supplies the second exploration signal by rewarding transitions that the demonstration model fails to predict, which is what lets the policy move beyond the demonstration support. The theory controls the cumulative effect of these bonuses via covering numbers and the generalized Eluder dimension, yielding the regret bound.

What would settle it

Run ILDE on a game-like environment whose reward is explicitly inverse to novelty (the expert's high-scoring path is a narrow safe corridor surrounded by lethal states that look novel to the demonstration model). If the agent still exceeds the expert's return, the alignment premise is not needed; if it falls below, the beyond-expert claim fails exactly where the paper's stated limitation predicts.

Watch

Extended reading notes

Core claim

On the paper's own terms, ILDE solves a min-max objective: minimize, over policies, the worst-case uncertainty-regularized discrepancy $\ell(\pi, r) = J(\pi_E, r) - J(\pi, r) - \lambda \cdot \mathrm{Int}(\pi; \tau^E)$, where $J$ is expected cumulative reward under reward $r$, $\pi_E$ is the expert, and $\mathrm{Int}$ is the expected cumulative intrinsic reward from a demonstration-trained transition model. Two exploration signals carry the argument: a bonus proportional to the generalized-Eluder uncertainty of a state-action pair, which makes the policy optimistically seek under-explored pairs, and a curiosity term that rewards deviation from demonstration trajectories. The paper proves that its ILDE-NPG algorithm attains regret $\widetilde{O}\big(H^{8/3}(\dim_T(\mathcal{F}) \log N_{\epsilon_F}(\mathcal{F}))^{1/3} T^{2/3} + \epsilon_E T\big)$ against the optimal policy for the uncertainty-regularized objective, and reports that the practical PPO-based version exceeds the expert in all six Atari games tested, with an average return 5.33 times the expert's, using only 10% of a one-life demonstration.

Load-bearing premise

The empirical claim collapses if exploring novel states does not lead to high scores in the target task, and the regret bound additionally requires that the chosen function class can represent the expected next-state value for every possible value function.

Editorial extensions

If this is right

  • ILDE shows that imitation learning can produce a policy that exceeds the demonstrator's own average return, so expert trajectories stop being an upper bound on achievable performance.
  • The sample-efficiency result implies that a few hundred transitions (10% of a one-life demonstration) can substitute for the many full episodes that adversarial imitation methods typically need.
  • The regret bound transfers the optimism principle from reward-based RL to reward-free imitation learning, with sublinear regret in the number of episodes.
  • Because the analysis is stated with generalized Eluder dimension and covering numbers, the guarantee covers linear MDPs and other Eluder-bounded MDPs, extending provable imitation learning beyond tabular settings.
  • The ablation study indicates that the two exploration terms are complementary: the bonus contributes most where coverage is hard, and the imitation reward contributes most where the curiosity signal is unreliable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: one could test whether a per-state adaptive curiosity weight, set to the recent correlation between exploring that state and true return, would close the MuJoCo gap the paper reports.
  • Editorial extension: because the regret proof rests on the same value-difference machinery as optimistic reward-based RL, future sublinear bounds in that literature should transfer directly to ILDE by replacing the bonus terms.
  • Editorial extension: the alignment assumption suggests that ILDE's beyond-expert gains should be most reproducible in games where high scores are reachable only by visiting under-represented states; games with dense safe corridors should show smaller gains.
  • Editorial extension: the paper's reliance on a demonstration-pretrained model to score curiosity means the method inherits the pretraining distribution; retraining on noisier demonstrations should degrade the curiosity term before the optimism term, a prediction the robustness experiment partially checks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Imitation Learning with Double Exploration (ILDE), which combines a GAIL/VAIL-style imitation reward with two exploration signals: an optimism-based state-action uncertainty bonus and a curiosity/intrinsic reward from a generative model pretrained on the demonstration data. The authors claim that this combination achieves beyond-expert performance on six Atari games and four MuJoCo tasks with very sparse demonstrations, and they support the framework with a theoretical regret bound (Theorem 4.7) for a natural-policy-gradient version (Algorithm 1) under a generalized Eluder dimension assumption. A stated limitation is that the analysis does not directly cover the practical Algorithm 3 used in the experiments.

Significance. If the empirical findings are robust, the paper provides a practical and interesting recipe for imitation learning from extremely sparse demonstrations, with a plausible mechanism for exceeding expert scores through curiosity and optimistic exploration. The theoretical section is a genuine derivation against an explicit surrogate objective and is commendable for being explicit about its assumptions. However, the advertised sublinear-regret conclusion is not actually supported because of the additive epsilon_E T term, and the theory applies to Algorithm 1 rather than the evaluated Algorithm 3. The empirical headline rests on mean performances with large standard deviations. These issues materially weaken the claims as currently stated, although the underlying ideas and experiments remain valuable.

major comments (4)
  1. [Theorem 4.7 / Eq. (4.5) and Section 3] The abstract and Section 3 claim that the regret grows sublinearly in the number of episodes, but Theorem 4.7 proves only a bound of eO(H^{8/3}(dim_T(F) log N_{epsilon_F}(F))^{1/3} T^{2/3} + epsilon_E T). Under Assumption 4.2, epsilon_E > 0 is a fixed constant determined by the n demonstration trajectories, which are inputs independent of T. Lemma C.7 indeed produces O(epsilon_E K), and the final substitution multiplies by N/m to give epsilon_E T. For fixed epsilon_E > 0, the additive term is Theta(T), so the stated sublinear-regret claim does not follow. The convergence statement following the regret definition also fails unless one additionally assumes epsilon_E = o(1) as T grows, which is neither stated nor used. The authors should correct the claim, for example by stating the bound as eO(T^{2/3} + epsilon_E T) and removing the sublinear claim, or by adding an explicit assumption that the demonstration set grows with T so that epsilon_E -> 0.
  2. [Section 7 / Algorithm 3] The theoretical guarantee in Theorem 4.7 is for Algorithm 1 (ILDE-NPG), but Section 7 states that the analysis cannot be directly applied to the practical version implemented in experiments. Algorithm 3 differs from Algorithm 1 in substantial ways: it uses PPO rather than the analyzed mirror-descent update, a VAIL-style discriminator rather than the projected gradient reward update, a state-entropy bonus rather than the D_F bonus, and the GIRIL intrinsic reward. Therefore the claim that the theory provides a justification for the empirically evaluated ILDE is not supported. This is acknowledged in the limitations, but it should be reflected in the abstract and in the statement of contributions, since the 'first theoretical guarantee' is for a different algorithm from the one whose results are reported.
  3. [Table 1 / Table 9] The headline claims that ILDE 'outperforms the expert in all of the 6 Atari games' and achieves 5.33 times expert performance are based only on mean returns. For several games the standard deviations are large enough to make the comparison inconclusive: in StarGunner the expert mean is 22,495 +/- 11,516 and ILDE is 25,361 +/- 38,557, while in BattleZone the expert is 22,410 +/- 5,839 and ILDE is 59,109 +/- 25,575. In addition, the 'Improve vs Expert' ratio is an average of per-game normalized means without uncertainty propagation, and the ablation ILDE w/o r_k also outperforms the expert in 5/6 games. The authors should provide significance tests or per-seed ratios, and should temper the beyond-expert claim to what the data support.
  4. [Assumption 4.5 / Lemma C.3] Assumption 4.5 is a strong realizability condition: for every h and every bounded function V_{h+1}: S -> [-H,H], P_h V_{h+1} must belong to the function class F_h. This is much stronger than standard realizability over the values induced by policies and rewards, and it is load-bearing in Lemma C.3, where the regression error is measured through D_{F_h} after identifying P_h V_{k,h+1} with an element of F_h. Since Remark 4.8 claims that the setting captures linear MDPs, the authors should either verify Assumption 4.5 for linear MDPs under the stated boundedness requirements, or qualify the claim that the result covers standard linear MDPs. Without this, the scope of the 'first theoretical guarantee for nonlinear function approximation' is unclear.
minor comments (5)
  1. [Remark 4.8] The word 'Algorthm' is a typo for 'Algorithm'.
  2. [Section 6.1] The text refers to 'IDLE' in the baseline description; this should be 'ILDE'.
  3. [Eq. (4.4) and Lemma C.3] The placement of the term gamma * D_{F_h}((s,a);D_h) relative to the square root is inconsistent between Eq. (4.4) and Lemma C.3. Please clarify whether this term is inside or outside the square root, since the proof and the final regret bound depend on the correct reading.
  4. [Notation across Section 3, Section 4, and Definition 4.3] The symbol lambda is used for the curiosity reward weight in Eq. (3.3), for the KL regularization parameter in Algorithm 1, and lambda_ED appears in Definition 4.3. These uses should be disambiguated to avoid confusion.
  5. [Table 1] The column labeled 'ILDE w/o lambda L_tauE' is not mentioned in the text's ablation list; the text discusses 'ILDE w/o b' and 'ILDE w/o r_k'. Please make the table and text consistent and define each ablation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the regret analysis is a genuine derivation and the beyond-expert results are empirical, not derived predictions.

full rationale

The paper's claimed derivations are self-contained against the stated objective. The regret bound (Theorem 4.7) is proved by decomposing Regret(T) into I1+I3 (optimistic policy optimization, Lemma C.6) and I2 (online reward learning, Lemma C.7), each bounded by standard value-difference, concentration, Eluder-dimension, and online-gradient-descent arguments; no step solves the target bound from an equation that already contains it. The exploration bonus in Eq. (4.4) is justified by Lemma C.3, a high-probability bound on the OPE regression error, rather than by assuming the regret conclusion. The generalized Eluder dimension is imported from Agarwal et al. (2023), an external source, and the self-citation to Zhao et al. (2023) in Remark 4.8 is illustrative rather than load-bearing for the proof. The beyond-expert empirical claim is presented as a measured benchmark result, and Section 7 explicitly flags the theory-practice gap ('our analysis cannot be directly applied to the practical version implemented in experiments') and the novelty-alignment assumption ('Our method assumes that the target task to be imitated is aligned with the concept of seeking novelty or curiosity'), which weakens but does not circularize the empirical conclusion. The additive ε_E T term in Theorem 4.7 is a possible correctness or consistency issue with the abstract's 'sublinear' wording, but it is not a circularity: no parameter is fitted and then renamed as a prediction, and no asserted result is equivalent by construction to the inputs. Overall, the derivation chain is genuine and the paper is not circular.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The central empirical claim rests on the choice of the curiosity weight and the implicit premise that the benchmark tasks reward novelty; the central theoretical claim rests on strong function-class assumptions (4.5, 4.1, 4.2) and the generalized Eluder dimension machinery. No new physical entities are introduced.

free parameters (3)
  • lambda (curiosity reward weight) = 10.0 (both domains); parametric study over {20, 10, 5, 2} shows BeamRider score increases as lambda decreases (12,600…
    Weights the GIRIL intrinsic reward in the aggregated reward (Algorithm 3, line 5); it controls the balance between imitation and beyond-expert exploration and affects the headline results, per Table 8.
  • per-domain curiosity scale constant = unspecified (described only as 'a multiplicative constant based on the range of the curiosity')
    B.2.4 states the same hyperparameters are used within a domain 'apart from a multiplicative constant based on the range of the curiosity', an undisclosed per-domain fit that changes the effective intrinsic reward scale.
  • VAIL information constraint I_c = 0.2
    VDB bottleneck constraint (B.2.1, Table 7) controls how much information the discriminator keeps, shaping the imitation reward that anchors the policy; inherited from VAIL but a chosen constant here.
assumptions (6)
  • domain assumption Assumption 4.5 (strong realizability): for all h and all V_{h+1}: S -> [-H,H], there exists f_h in F_h with f_h = P_h V_{h+1}.
    Needed for the OPE analysis (Lemma C.3) to control the error of the fitted value function; much stronger than one-step Bellman closure over the encountered data.
  • domain assumption Assumption 4.1: reward function parameterization is convex and Lipschitz (gradient norm O(1)).
    Used to get the O(sqrt(H^2 K)) reward-learning regret in Lemma C.7; real discriminator losses are generally non-convex.
  • domain assumption Assumption 4.2: demonstration average reward error bounded by eps_E for every r in R.
    Controls the additional eps_E T term in the regret; the paper argues it follows from concentration, but it requires the demonstrations to be i.i.d. from pi_E.
  • ad hoc to paper Novelty-alignment premise: the environment's true reward is aligned with seeking novelty or curiosity.
    Stated in Section 7 (limitation) as the condition for beyond-expert performance; the six Atari games appear to satisfy it, and the authors note MuJoCo improvements are weaker.
  • standard math Generalized Eluder dimension as complexity measure (Agarwal et al. 2023, adapted unweighted in Remark 4.4).
    The regret bound is expressed in terms of dim_T(F); the unweighted redefinition in Definition 4.3 is a modification the paper claims broadens the class, and its properties are inherited from the cited machinery.
  • ad hoc to paper Conjecture (Section 3): uncertainty regularization helps the learner attain performance beyond expert level.
    The paper labels this a conjecture rather than a theorem; it is the unproved bridge between the regularized objective and the beyond-expert interpretation of the results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond-Expert Performance with Limited Demonstrations: Efficient Imitation Learning with Double Exploration." pith.science (2026). https://pith.science/paper/4E3CJZ7I

@misc{pith2026250620307,
  author       = {Pith},
  title        = {Pith review of: Beyond-Expert Performance with Limited Demonstrations: Efficient Imitation Learning with Double Exploration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4E3CJZ7I}},
  note         = {Machine review of arXiv:2506.20307}
}
read the original abstract

Imitation learning is a central problem in reinforcement learning where the goal is to learn a policy that mimics the expert's behavior. In practice, it is often challenging to learn the expert policy from a limited number of demonstrations accurately due to the complexity of the state space. Moreover, it is essential to explore the environment and collect data to achieve beyond-expert performance. To overcome these challenges, we propose a novel imitation learning algorithm called Imitation Learning with Double Exploration (ILDE), which implements exploration in two aspects: (1) optimistic policy optimization via an exploration bonus that rewards state-action pairs with high uncertainty to potentially improve the convergence to the expert policy, and (2) curiosity-driven exploration of the states that deviate from the demonstration trajectories to potentially yield beyond-expert performance. Empirically, we demonstrate that ILDE outperforms the state-of-the-art imitation learning algorithms in terms of sample efficiency and achieves beyond-expert performance on Atari and MuJoCo tasks with fewer demonstrations than in previous work. We also provide a theoretical justification of ILDE as an uncertainty-regularized policy optimization method with optimistic exploration, leading to a regret growing sublinearly in the number of episodes.

Figures

Figures reproduced from arXiv: 2506.20307 by the authors.

Figure 1
Figure 1. Average return vs. number of simulation steps on Atari games. The solid lines show the [PITH_FULL_IMAGE:figures/full_fig_p018_1.png] view at source ↗
Figure 2
Figure 2. Average return vs. the number of simulation steps on MuJoCo tasks. The solid lines show [PITH_FULL_IMAGE:figures/full_fig_p022_2.png] view at source ↗
Figure 3
Figure 3. Ablations on MuJoCo tasks. Average return vs. the number of simulation steps on MuJoCo [PITH_FULL_IMAGE:figures/full_fig_p023_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

60 extracted references · 54 canonical work pages

  1. [1]

    VO Q L: Towards Optimal Regret in Model-free RL with Nonlinear Function Approximation

    Alekh Agarwal, Yujia Jin, and Tong Zhang. VO Q L: Towards Optimal Regret in Model-free RL with Nonlinear Function Approximation. InAnnual Conference on Learning Theory (COLT 2023), pp. 987–1063,

    work page 2023

  2. [2]

    Hence the right-hand side of the above inequality is a martingale difference sequence

    23 Published as a conference paper at ICLR 2025 Then it follows that for allh∈[H], X (sh,ah)∈Dk h bfk,h(sh, ah)−P hVk,h+1(sh, ah) 2 ≤2 X (sh,ah)∈Dk h bfk,h(sh, ah)−P hVk,h+1(sh, ah) · PhVk,h+1(sh, ah)−V k,h+1(sh+1) .(C.2) Note that Vk,h+1 is a function only depends on data in Dh+1,D h+2,· · ·,DH. Hence the right-hand side of the above inequality is a mart...

    work page 2025

  3. [7]

    Mitigating covariate shift in imitation learning via offline data without great coverage.arXiv preprint arXiv:2106.03207,

    Jonathan D Chang, Masatoshi Uehara, Dhruv Sreenivas, Rahul Kidambi, and Wen Sun. Mitigating covariate shift in imitation learning via offline data without great coverage.arXiv preprint arXiv:2106.03207,

    arXiv

  4. [9]

    Deep reinforcement learning from human preferences

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. InAdvances in neural information processing systems (NeurIPS 2017), pp. 4302–4310,

    work page 2017

  5. [11]

    Guided cost learning: Deep inverse optimal control via policy optimization

    Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. InInternational Conference on Machine Learning (ICML 2016), pp. 49–58. PMLR,

    work page 2016

  6. [12]

    Efficient Bias-Span-Constrained Exploration-Exploitation in Reinforcement Learning

    Ronan Fruit, Matteo Pirotta, Alessandro Lazaric, and Ronald Ortner. Efficient Bias-Span-Constrained Exploration-Exploitation in Reinforcement Learning. InInternational Conference on Machine Learning (ICML 2018), pp. 1578–1586,

    work page 2018

  7. [13]

    Ess- InfoGAIL: Semi-supervised Imitation Learning from Imbalanced Demonstrations

    11 Published as a conference paper at ICLR 2025 Huiqiao Fu, Kaiqiang Tang, Yuanyang Lu, Yiming Qi, Guizhou Deng, and Chunlin Chen. Ess- InfoGAIL: Semi-supervised Imitation Learning from Imbalanced Demonstrations. InAdvances in Neural Information Processing Systems (NeurIPS 2023), pp. 60048–60059,

    work page 2025

  8. [16]

    Deep Q-learning from Demonstrations

    Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et al. Deep Q-learning from Demonstrations. In Proceedings of the AAAI conference on artificial intelligence (AAAI 2018), pp. 3223–3230,

    work page 2018

Show all 60 references
  1. [17]

    Generative adversarial imitation learning

    Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. InAdvances in Neural Information Processing Systems (NeurIPS 2016), pp. 4572–4580,

    2016

  2. [19]

    3D Perception based Imitation Learning under Limited Demonstration for Laparoscope Control in Robotic Surgery

    Bin Li, Ruofeng Wei, Jiaqi Xu, Bo Lu, Chi Hang Yee, Chi Fai Ng, Pheng Ann Heng, Qi Dou, and Yun Hui Liu. 3D Perception based Imitation Learning under Limited Demonstration for Laparoscope Control in Robotic Surgery. InIEEE International Conference on Robotics and Automation (I...

    2022

  3. [21]

    Visual imitation learning with patch rewards

    Minghuan Liu, Tairan He, Weinan Zhang, Shuicheng Yan, and Zhongwen Xu. Visual imitation learning with patch rewards. InInternational Conference on Learning Representations (ICLR 2023), 2023a. Qinghua Liu, Gell´ert Weisz, Andr´as Gy¨orgy, Chi Jin, and Csaba Szepesv´ari. Optimis...

    2023

  4. [22]

    A modern introduction to online learning.arXiv preprint arXiv:1912.13213,

    Francesco Orabona. A modern introduction to online learning.arXiv preprint arXiv:1912.13213,

    1912 arXiv

  5. [23]

    Online Trajectory Planning in Dynamic Environments for Surgical Task Automation

    Takayuki Osa, Naohiko Sugita, and Mamoru Mitsuishi. Online Trajectory Planning in Dynamic Environments for Surgical Task Automation. InRobotics: Science and Systems (RSS 2014), pp. 1–9,

    2014

  6. [25]

    Variational discriminator bottleneck: Improving imitation learning, inverse RL, and GANs by constraining information flow

    Xue Bin Peng, Angjoo Kanazawa, Sam Toyer, Pieter Abbeel, and Sergey Levine. Variational discriminator bottleneck: Improving imitation learning, inverse RL, and GANs by constraining information flow. InInternational Conference on Learning Representations (ICLR 2019), pp. 1–27,

    2019

  7. [26]

    Alvinn: An autonomous land vehicle in a neural network

    Dean A Pomerleau. Alvinn: An autonomous land vehicle in a neural network. InAdvances in Neural Information Processing Systems (NeurIPS 1998), pp. 305–313,

    1998

  8. [29]

    Eluder dimension and the sample complexity of optimistic ex- ploration

    Daniel Russo and Benjamin Van Roy. Eluder dimension and the sample complexity of optimistic ex- ploration. InAdvances in Neural Information Processing Systems (NeurIPS 2013), pp. 2256–2264,

    2013

  9. [31]

    State entropy maximization with random encoders for efficient exploration

    Younggyo Seo, Lili Chen, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. State entropy maximization with random encoders for efficient exploration. InInternational Conference on Machine Learning (ICML 2021), pp. 9443–9454,

    2021

  10. [32]

    Optimistic policy optimization with bandit feedback

    Lior Shani, Yonathan Efroni, Aviv Rosenberg, and Shie Mannor. Optimistic policy optimization with bandit feedback. InInternational Conference on Machine Learning (ICML 2020), pp. 8604–8613,

    2020

  11. [33]

    Online apprenticeship learning

    Lior Shani, Tom Zahavy, and Shie Mannor. Online apprenticeship learning. InProceedings of the AAAI Conference on Artificial Intelligence (AAAI 2022), pp. 8240–8248,

    2022

  12. [37]

    Error bounds of imitating policies and environments

    Tian Xu, Ziniu Li, and Yang Yu. Error bounds of imitating policies and environments. InAdvances in Neural Information Processing Systems (NeurIPS 2020), pp. 15737–15749,

    2020

  13. [38]

    Planning for sample efficient imitation learning

    Zhao-Heng Yin, Weirui Ye, Qifeng Chen, and Yang Gao. Planning for sample efficient imitation learning. InAdvances in Neural Information Processing Systems (NeurIPS 2022), pp. 2577–2589,

    2022

  14. [39]

    Intrinsic reward driven imitation learning via generative model

    Xingrui Yu, Yueming Lyu, and Ivor Tsang. Intrinsic reward driven imitation learning via generative model. InInternational Conference on Machine Learning (ICML 2020), pp. 10925–10935,

    2020

  15. [40]

    Confidence-Aware Imitation Learning from Demonstrations with Varying Optimality

    13 Published as a conference paper at ICLR 2025 Songyuan Zhang, Zhangjie Cao, Dorsa Sadigh, and Yanan Sui. Confidence-Aware Imitation Learning from Demonstrations with Varying Optimality. InAdvances in Neural Information Processing Systems (NeurIPS 2021), pp. 12340–12350,

    2025

  16. [41]

    Deep imitation learning for complex manipulation tasks from virtual reality teleoperation

    Tianhao Zhang, Zoe McCarthy, Owen Jow, Dennis Lee, Xi Chen, Ken Goldberg, and Pieter Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. InIEEE International Conference on Robotics and Automation (ICRA 2018), pp. 5628–5635,

    2018

  17. [42]

    Generative adversarial imitation learning with neural network parameterization: Global optimality and convergence rate

    Yufeng Zhang, Qi Cai, Zhuoran Yang, and Zhaoran Wang. Generative adversarial imitation learning with neural network parameterization: Global optimality and convergence rate. InInternational Conference on Machine Learning (ICML 2020), pp. 11044–11054,

    2020

  18. [43]

    A Nearly Optimal and Low-Switching Algorithm for Reinforcement Learning with General Function Approximation.arXiv preprint arXiv:2311.15238,

    Heyang Zhao, Jiafan He, and Quanquan Gu. A Nearly Optimal and Low-Switching Algorithm for Reinforcement Learning with General Function Approximation.arXiv preprint arXiv:2311.15238,

  19. [44]

    Self-adaptive imitation learning: Learning tasks with delayed rewards from sub-optimal demonstrations

    Zhuangdi Zhu, Kaixiang Lin, Bo Dai, and Jiayu Zhou. Self-adaptive imitation learning: Learning tasks with delayed rewards from sub-optimal demonstrations. InProceedings of the AAAI conference on artificial intelligence (AAAI 2022), pp. 9269–9277,

    2022

  20. [45]

    Maximum entropy inverse reinforcement learning

    Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. InAAAI Conference on Artificial Intelligence (AAAI 2008), pp. 1433–1438,

    2008

  21. [47]

    Table 3: Comparison of three reward components with three attributes

    A MOTIVATION TABLE Table 3 illustrates the strengths of each component in ILDE, which helps to explain the motivation of ILDE. Table 3: Comparison of three reward components with three attributes. Stability Exploration Imitation GIRIL reward ++ + + Bonusb + ++ - V AILrk - - ++...

    2020

  22. [48]

    14 Published as a conference paper at ICLR 2025 Table 4: Demonstration lengths in the Atari environment

    regards imitation learning as a distribution matching problem, and updates policy using adversarial learning (Goodfellow et al., 2020). 14 Published as a conference paper at ICLR 2025 Table 4: Demonstration lengths in the Atari environment. Atari Games One-life Demo. Full-epis...

    2020

  23. [49]

    VDB constrains the information flow in the discriminator using an information bottleneck

    improves GAIL by compressing the information via a variational information bottleneck (VDB). VDB constrains the information flow in the discriminator using an information bottleneck. By enforcing a constraint on the mutual information between the observations and the discrimin...

    2020

  24. [50]

    The module is composed of several neural networks, including recognition network qϕ(z|st, st+1), a generative network pθ(st+1|z, st), and prior network pθ(z|st)

    to combine a backward action encoding model and a forward dynamics model into a single generative model. The module is composed of several neural networks, including recognition network qϕ(z|st, st+1), a generative network pθ(st+1|z, st), and prior network pθ(z|st). GIRIL refe...

    2021

  25. [51]

    State entropy estimate as bonus.Following Seo et al

    +C k ∝ 1 N NX i=1 log∥x i −x k−NN i ∥2,(B.4) where xk−NN i is the k-nearest neighbor of xi within a set of N representations {x1, x2,· · ·, xN }, Γ is the gamma function, bπrefers to an estimate of the number π (as opposed to the policy), and log(k−ψ)whereψis the digamma funct...

    2021

  26. [53]

    We use the same hyperparameter setting for the different experiments within a domain (Atari vs MuJoCo), apart from a multiplicative constant based on the range of the curiosity

    Note that λ, which controls the curiosity bonus was set without tuning to λ= 10 in our experiments. We use the same hyperparameter setting for the different experiments within a domain (Atari vs MuJoCo), apart from a multiplicative constant based on the range of the curiosity....

    2025

  27. [54]

    Improve vs Expert

    On average, V AIL is worse than the expert, only outperforming the expert in Krull. GIRIL 17 Published as a conference paper at ICLR 2025 archives a performance that is 4.87 times the expert, performing the best in Q*bert. ILDE is the best, achieving a performance that is 5.33...

    2025

  28. [55]

    Improve vs GIRIL

    and HyPE (Ren et al., 2024). First, we compare IQ-Learn on our Atari task setting (with 10% of a one-life demonstration). The results in Table 12 show that IQ-Learn does not perform well on Atari tasks, likely because the 18 Published as a conference paper at ICLR 2025 Table 1...

    2024

  29. [57]

    (2023), and (3) the analysis leads to a sublinear batch-regret, which is stronger than a sample complexity bound

    as shown in Zhao et al. (2023), and (3) the analysis leads to a sublinear batch-regret, which is stronger than a sample complexity bound. First, we have the following extended value difference lemma. Lemma C.1(Value difference lemma).For any policy π, the value functions {V k ...

    2023

  30. [59]

    Then with probability at least1−2δ, KX k=1 J(π ∗,erk)−J(π,er k) ≤Hlog|A|/η+ηH 3K/2 + 2HK· p 8H 2 log(H· NF (ϵF )/δ) + 4ϵF N+γ·O s H N · H 2 +γ γ dimN (Fh) log(1/δ) !

    Suppose that Assumption 4.5 holds. Then with probability at least1−2δ, KX k=1 J(π ∗,erk)−J(π,er k) ≤Hlog|A|/η+ηH 3K/2 + 2HK· p 8H 2 log(H· NF (ϵF )/δ) + 4ϵF N+γ·O s H N · H 2 +γ γ dimN (Fh) log(1/δ) ! . Proof.Suppose that the high-probability events in Lemma C.3 and Lemma C.5 ...

    2025

  31. [60]

    Lemma D.2(Self-normalized bound for scalar-valued martingales).Consider random variables (vn|n∈N) adapted to the filtration (Hn :n= 0,1, ...)

    Then for any0< δ <1, with probability at least1−δ, we have nX i=1 xi ≤M p 2nlog(1/δ). Lemma D.2(Self-normalized bound for scalar-valued martingales).Consider random variables (vn|n∈N) adapted to the filtration (Hn :n= 0,1, ...) . Let {ηi}∞ i=1 be a sequence of real-valued rand...

    2019

  32. [100]

    Table 16 shows the MLP architectures, i.e., GIRIL’s encoder and decoder and V AIL’s discriminator

    Table 15 shows the architecture details of the actor-critic network for MuJoCo tasks. Table 16 shows the MLP architectures, i.e., GIRIL’s encoder and decoder and V AIL’s discriminator. Table 15: Architectures of the actor-critic network for MuJoCo tasks. dim(S) and dim(A) repr...

    2025

  33. [1988]

    Toward the fundamental limits of imitation learning

    Nived Rajaraman, Lin Yang, Jiantao Jiao, and Kannan Ramchandran. Toward the fundamental limits of imitation learning. InAdvances in Neural Information Processing Systems (NeurIPS 2020), pp. 2914–2924,

    2020

  34. [2002]

    Relative entropy inverse reinforcement learning

    Abdeslam Boularias, Jens Kober, and Jan Peters. Relative entropy inverse reinforcement learning. In International Conference on Artificial Intelligence and Statistics (AISTATS 2011), pp. 182–189,

    2011

  35. [2003]

    Learning structured output representation using deep conditional generative models

    Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. InAdvances in Neural Information Processing Systems (NeurIPS 2015), pp. 3483–3491,

    2015

  36. [2008]

    and Bagnell, J

    Ziebart, Brian D. and Bagnell, J. Andrew and Dey, Anind K. Modeling interaction via the principle of maximum causal entropy. InInternational Conference on Machine Learning (ICML 2010), pp. 1255–1262,

    2010

  37. [2010]

    Provably efficient reinforcement learning with linear function approximation

    Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan. Provably efficient reinforcement learning with linear function approximation. InConference on learning theory (COLT 2020), pp. 2137–2143. PMLR,

    2020

  38. [2011]

    OpenAI Gym.arXiv preprint arXiv:1606.01540,

    Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAI Gym.arXiv preprint arXiv:1606.01540,

    arXiv

  39. [2012]

    Proximal point imitation learning

    Luca Viano, Angeliki Kamoutsi, Gergely Neu, Igor Krawczuk, and V olkan Cevher. Proximal point imitation learning. InAdvances in Neural Information Processing Systems (NeurIPS 2022), pp. 24309–24326,

    2022

  40. [2013]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    arXiv

  41. [2014]

    Curiosity-driven exploration by self-supervised prediction

    12 Published as a conference paper at ICLR 2025 Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. InInternational Conference on Machine Learning (ICML 2017), pp. 2778–2787,

    2025

  42. [2015]

    Mujoco: A physics engine for model-based control

    Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In2012 IEEE/RSJ international conference on intelligent robots and systems (IROS 2012), pp. 5026–5033. IEEE,

    2012

  43. [2016]

    Extrapolating beyond sub- optimal demonstrations via inverse reinforcement learning from observations

    Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond sub- optimal demonstrations via inverse reinforcement learning from observations. InInternational Conference on Machine Learning (ICML 2019), pp. 783–792,

    2019

  44. [2017]

    End-to- end driving via conditional imitation learning

    Felipe Codevilla, Matthias M¨uller, Antonio L´opez, Vladlen Koltun, and Alexey Dosovitskiy. End-to- end driving via conditional imitation learning. InIEEE International Conference on Robotics and Automation (ICRA 2018), pp. 4693–4700,

    2018

  45. [2018]

    On the global convergence of imitation learning: A case for linear quadratic regulator.arXiv preprint arXiv:1901.03674,

    Qi Cai, Mingyi Hong, Yongxin Chen, and Zhaoran Wang. On the global convergence of imitation learning: A case for linear quadratic regulator.arXiv preprint arXiv:1901.03674,

    1901 arXiv

  46. [2019]

    Large-scale study of curiosity-driven learning.arXiv preprint arXiv:1808.04355,

    Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros. Large-scale study of curiosity-driven learning.arXiv preprint arXiv:1808.04355,

    arXiv

  47. [2020]

    Hybrid inverse reinforcement learning.arXiv preprint arXiv:2402.08848,

    Juntao Ren, Gokul Swamy, Zhiwei Steven Wu, J Andrew Bagnell, and Sanjiban Choudhury. Hybrid inverse reinforcement learning.arXiv preprint arXiv:2402.08848,

    arXiv

  48. [2021]

    On Computation and Generalization of Generative Adversarial Imitation Learning

    Minshuo Chen, Yizhou Wang, Tianyi Liu, Zhuoran Yang, Xingguo Li, Zhaoran Wang, and Tuo Zhao. On Computation and Generalization of Generative Adversarial Imitation Learning. InInternational Conference on Learning Representations (ICLR 2019),

    2019

  49. [2022]

    Imitation Learning from Imperfection: Theoretical Justifications and Algorithms

    Ziniu Li, Tian Xu, Zeyu Qin, Yang Yu, and Zhi-Quan Luo. Imitation Learning from Imperfection: Theoretical Justifications and Algorithms. InAdvances in Neural Information Processing Systems (NeurIPS 2023), pp. 18404–18443,

    2023

  50. [2023]

    IQ-Learn: Inverse soft-Q Learning for Imitation

    Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. IQ-Learn: Inverse soft-Q Learning for Imitation. InAdvances in Neural Information Processing Systems (NeurIPS 2021), pp. 4028–4039,

    2021

  51. [2024]

    Exploration via Elliptical Episodic Bonuses.arXiv preprint arXiv:2210.05805,

    Mikael Henaff, Roberta Raileanu, Minqi Jiang, and Tim Rockt ¨aschel. Exploration via Elliptical Episodic Bonuses.arXiv preprint arXiv:2210.05805,

    arXiv

  52. [5184]

    For a fair comparison, we used an identical policy network for all methods

    In the state feature space, we use 3-layer MLPs to implement the discriminator for V AIL and the encoder and decoder for GIRIL. For a fair comparison, we used an identical policy network for all methods. We used the actor-critic approach for training PPO policy for all imitati...

    2018

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.