Pith. sign in

REVIEW 3 major objections 5 minor 53 references

Off-Dynamics Reinforcement Learning via Domain Adaptation and Reward Augmented Imitation

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Reward-shaping RL fails on the target; imitation learning fixes it

desk verdict The core idea is useful and the main experiments are solid, but the proof's key step is an unproven equality and one appendix table directly contradicts the headline claim. read the letter →

arxiv 2411.09891 v1 pith:5GMU5YTI submitted 2024-11-15 cs.LG cs.AIcs.RO

Yihong Guo , Yixuan Wang , Yuanyuan Shi , Pan Xu , Anqi Liu This is my paper · ORCID
classification cs.LGcs.AIcs.RO
keywords off-dynamicsreinforcementlearningdomainadaptationimitationfromobservationgenerativeadversarialrewardmodificationdoublyrobustestimationimportanceweighting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that existing off-dynamics RL methods such as DARC only match the source-domain behavior of a policy to the target-optimal trajectory distribution, and that directly deploying the resulting policy in the target domain suffers a systematic performance drop. DARAIL, the proposed method, adds imitation learning from observation: it uses DARC's source trajectories as expert demonstrations and trains a new policy, rolled out mostly in the source domain, to reproduce those state transitions in the target domain. To make this imitation stable, the paper introduces the reward augmented estimator (RAE), which blends the discriminator's learned reward with the true source reward through an importance weight, in the spirit of doubly robust estimation. The paper proves a finite-sample error bound for DARAIL and reports on four MuJoCo environments that it outperforms DARC and six baselines.

What carries the argument

The reward augmented estimator (RAE), defined as $RAE(s_t, s_{t+1}) = -\log D_\omega(s_t, s_{t+1}) + \rho(s_t, s_{t+1})(r_{\mathrm{src}}(s_t, a_t, s_{t+1}) + \log D_\omega(s_t, s_{t+1}))$, is the central object. It combines the discriminator signal from generative adversarial imitation learning from observation (GAIfO) with the true source reward, reweighted by the importance weight $\rho(s_t, s_{t+1}) = p_{\mathrm{trg}}(s_{t+1}|s_t,a_t)/p_{\mathrm{src}}(s_{t+1}|s_t,a_t)$, so that the biased discriminator reward is corrected by the ground-truth reward, analogous to doubly robust estimation. The DARC-modified policy provides the expert demonstrations, and importance weighting lets the imitation objective be trained almost entirely from source rollouts, with only occasional target rollouts to estimate the classifiers.

What would settle it

Run DARAIL on the broken-source Ant or Walker2d with freezing probability $p_f = 1.0$, a setting the paper reports where DARC cannot obtain near-optimal source reward; if DARAIL then does not beat DARC's target evaluation reward, the method's dependence on high-quality DARC source trajectories is confirmed.

Watch

Extended reading notes

Core claim

The central claim is that the off-dynamics RL failure mode of reward-modification methods can be cured by a second stage that imitates the modified-reward policy's trajectories: instead of deploying the DARC policy directly, DARAIL learns a new policy whose target-domain state-transition distribution matches the state-transition distribution DARC produced in the source domain. The theoretical result upper-bounds the target-domain suboptimality of DARAIL by the DARC source error plus an imitation-learning generalization term, thereby relaxing DARC's restrictive assumption that the target-optimal policy is good in the source domain. Empirically, DARAIL's target reward exceeds DARC's target reward on all four tested environments and generally exceeds the other baselines.

Load-bearing premise

The method relies on DARC producing high-quality, near-optimal trajectories in the source domain; if the dynamics shift is so large that DARC itself fails, the imitation stage ends up copying suboptimal behavior and DARAIL can underperform DARC.

Editorial extensions

If this is right

  • A practical recipe for off-dynamics RL: first run a reward-modification method like DARC, then imitate its source trajectories instead of deploying it directly, requiring no target rewards.
  • The error bound relaxes the no-exploit assumption of DARC: the target-optimal policy no longer needs to be near-optimal in the source domain; only the importance weight needs to stay bounded.
  • Because RAE downweights the discriminator reward where the dynamics shift is large, DARAIL is less sensitive to the importance-weight clipping interval than importance-sampling baselines.
  • The method inherits the quality of the DARC source trajectories; when DARC achieves near-optimal source reward, DARAIL can match or exceed DARC's target evaluation reward across the tested environments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • RAE is a template that can be read as a doubly robust correction for any biased reward estimator in imitation or offline RL, not just for the DARC discriminator; this connection is implied by the paper's analogy but not developed into a general algorithm.
  • A testable extension would be to replace DARC with another reward-modification method or with a distributionally robust policy in the first stage, to see whether the imitation stage compensates for different kinds of source suboptimality.
  • The bounded-importance-weight assumption suggests that extremely large dynamics shifts, where $\rho$ becomes unbounded in practice, remain the hard regime; DARAIL's advantage over DARC should shrink or invert there.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper addresses off-dynamics reinforcement learning, where source and target domains share the same reward but have different transition dynamics and target reward is unavailable. The proposed method, DARAIL, first trains DARC in the source domain, collects its source-domain trajectories as expert demonstrations, and then runs generative adversarial imitation from observation with an importance-weighted discriminator and a "reward augmented estimator" (RAE) that combines the discriminator reward with the source reward. The paper claims a finite-sample target-domain error bound (Theorem 4.1) that relaxes DARC's no-exploit assumption, and reports experiments on HalfCheetah, Ant, Walker2d, and Reacher under broken-action, gravity, and density shifts, concluding that DARAIL outperforms both DARC and other baselines.

Significance. If the theoretical and empirical claims were fully established, DARAIL would be a practical and useful recipe for off-dynamics RL: it combines a reward-modification stage with an imitation stage, and the paper supplies a reproducible implementation, multiple-seed results with standard deviations, and several ablations (importance-weight clipping, discriminator update frequency, per-step versus cumulative importance weights, and matched target-rollout comparisons). However, the central theoretical claim is not proven as written, and the abstract's blanket empirical claim is contradicted by the paper's own appendix. The method may still be valuable in specific regimes, but the contributions need to be restated and the theory repaired before the claims can be accepted.

major comments (3)
  1. [Appendix B, proof of Theorem B.5] The proof sets the term I1 to zero with the assertion that "Based on the formulation, π*_DARC can generate optimal trajectories for the target domain in the source domain so that I1 = 0." This equality does not follow from the reverse-KL objective in Eq. (3.3), which matches the distribution q(τ_src^{πDARC}) to p(τ_trg^{π*}); distribution matching does not imply equality of expected true rewards between the target-optimal policy in the target domain and some source policy in the source domain. The asserted equality is essentially the strong "no-exploit" condition that Section 4.1 claims to avoid, and it is especially implausible in the broken-source regime that motivates the paper. With I1 left nonzero, Theorem 4.1 contains an additional unquantified term, so the claimed relaxation of DARC's assumption is not established. The notation π*_DARC is also never defined.
  2. [Appendix Table 5 and Table 9] The abstract's statement that DARAIL outperforms DARC in target domains is contradicted by the paper's own appendix. In Table 5 (Ant, 0.5 gravity), DARAIL reports 990±12 while DARC Evaluation reports 2058±553. In Table 9 (Ant, 1.5 density), DARAIL reports 5193±463 while DARC Evaluation reports 5961±970. Table 6 further shows DARAIL essentially tied with DAIL and IS-ACL on 0.5-gravity Ant. The abstract, Section 3.2, and Section 5.2 should be revised to state the settings in which DARAIL outperforms DARC, and these tables should be presented in the main comparison rather than only in the appendix if they are part of the evaluation.
  3. [Theorem 4.1 and Section 4.1] The advertised error bound is a consistency decomposition rather than a formal comparison with DARC's target-domain performance. The first term is DARC's source-domain training error, the second term contains an unquantified approximation error inf_ζ d_D(τ̂_src^{πDARC}, τ̂_trg_ζ), and the I1 term is set to zero without justification. Consequently, the theorem does not imply that DARAIL's target reward exceeds DARC Evaluation. Moreover, Appendix E concedes that the method inherits the quality of DARC's source trajectories; for example, in Table 2 the Ant 1.5-gravity setting has DARC Training 2170±258 versus Optimal 5320±429, and DARAIL reaches only 3472±771. This dependence should be stated as a limitation of the theoretical guarantee, not only of the experimental setup.
minor comments (5)
  1. [Section 2 and 3.2] The acronym is spelled inconsistently as both "GAIfO" and "GAILfo"; please use one form throughout.
  2. [Appendix C.3, Figure 5 caption] The caption contains a copy-paste error: it says "(a) and (c) shows the performance of DARC under pf = 0.8, and (a) and (c) shows the performance of DARC under pf = 1.0," but panels (b) and (d) are the pf = 1.0 cases.
  3. [Table 9] The "Optimal" column reports single numbers without standard deviations, unlike all other tables; please clarify whether these are single-run estimates or provide the same statistics as elsewhere.
  4. [Algorithm 1] The pseudocode contains formatting artifacts such as "S ROLLOUT" and undefined notation such as "DπDARC src S ROLLOUT(πDARC, Msrc)"; please clean up the pseudocode and define all symbols used.
  5. [Appendix E] "Mujoco" should be "MuJoCo" for consistency with the rest of the text.

Circularity Check

1 steps flagged · score 6.0 of 10

Theorem 4.1's proof zeroes I1 by fiat, assuming the target-optimality transfer it claims to prove.

  1. self definitional [Appendix B, proof of Theorem B.5 (main-text Theorem 4.1), I1 decomposition]
    "Based on the formulation, π∗ DARC can generate optimal trajectories for the target domain in the source domain so that I1 = 0."

    In the decomposition I1 = E_{ptrg,π*}[Σ r] − E_{psrc,π*_DARC}[Σ r + H], setting I1 = 0 asserts that the source-domain DARC-optimal policy's entropy-regularized reward equals the target-optimal expected reward. This is exactly the no-exploit-style transfer guarantee that Section 4.1 claims DARAIL avoids assuming, and it is not listed among Theorem 4.1's assumptions. The reverse-KL matching objective in Eq. (3.3) matches trajectory distributions; it does not imply equality of these expected rewards, especially when π* is not in the no-exploit set. With I1 left nonzero, the bound contains an unquantified extra term, so the claimed relaxation of DARC's assumption rests on an equality the paper asserts without proof rather than derives.

full rationale

The only load-bearing circular move is the I1 = 0 step in the proof of Theorem B.5/4.1. The proof decomposes target regret into I1 + I2 + I3 and then removes I1 by saying that π*_DARC can generate optimal trajectories for the target domain in the source domain. That equality is the core transfer guarantee the paper is trying to establish; it is not among the theorem's stated assumptions (m, B, W, discriminator boundedness, empirical training error) and does not follow from the reverse-KL distribution-matching objective in Eq. (3.3). Thus the theoretical claim that DARAIL relaxes DARC's no-exploit assumption is conditional on an unproven equality, i.e., the result is partly definitional. The empirical evaluation is independent and self-contained, and there is no load-bearing self-citation; however, Appendix Table 5 reports DARAIL Ant reward 990±12 versus DARC Evaluation 2058±553 at 0.5 gravity, which is an empirical discrepancy with the abstract's general claim that DARAIL outperforms DARC in target domains, and Appendix E concedes that an overly large dynamics shift may prevent obtaining high-quality state-space data to imitate. These are correctness risks rather than additional circularity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests on DARC's ability to produce near-optimal source trajectories, a bounded importance weight, and the expressivity of the discriminator class. I list these as axioms; no new entities are postulated.

free parameters (4)
  • Importance weight clipping interval = [0.01, 100]
    Applied to DARAIL, DAIL, IS-R, IS-ACL; chosen by hand for fair comparison and affects all results (Section 5.1, Figure 2).
  • Discriminator update steps = 50 (HalfCheetah, Walker2d), 1000 (Reacher)
    Tuned per environment; smaller steps preferred (Appendix D.2).
  • Target rollout ratio = 1% (rollout every 100 source steps)
    Determines the amount of target data used; chosen in Algorithm 1 and experiments.
  • Broken probability pf = 0.8 for Ant and Walker2d
    Chosen instead of 1.0 because DARC cannot achieve optimal source reward at pf=1.0 (Appendix C.3), making the benchmark favorable to DARAIL.
assumptions (5)
  • domain assumption Optimal policy trajectory distribution is proportional to exp(sum of rewards) (pi* proportional to exp(Sum r))
    Used in Eq (3.2) to derive DARC's modified reward, and invoked in the I1=0 step of Theorem B.5.
  • domain assumption Importance weight is bounded, rho(s_t,s_{t+1}) <= W
    Assumed in Theorem 4.1 and Lemma B.4; in practice enforced by clipping to [0.01, 100].
  • domain assumption Ground truth reward lies in the linear span of the discriminator class D
    Definition B.3 and Lemma B.4 assume r in span(D) so that ||r||_D is finite.
  • ad hoc to paper The imitation learning optimizer achieves empirical neural network distance error epsilon_hat
    Assumed in Theorem 4.1; not derived from the GAN training dynamics.
  • standard math Standard concentration inequalities (McDiarmid, Rademacher complexity) hold
    Used in the proof of Lemma B.4.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Off-Dynamics Reinforcement Learning via Domain Adaptation and Reward Augmented Imitation." pith.science (2026). https://pith.science/paper/5GMU5YTI

@misc{pith2026241109891,
  author       = {Pith},
  title        = {Pith review of: Off-Dynamics Reinforcement Learning via Domain Adaptation and Reward Augmented Imitation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5GMU5YTI}},
  note         = {Machine review of arXiv:2411.09891}
}
read the original abstract

Training a policy in a source domain for deployment in the target domain under a dynamics shift can be challenging, often resulting in performance degradation. Previous work tackles this challenge by training on the source domain with modified rewards derived by matching distributions between the source and the target optimal trajectories. However, pure modified rewards only ensure the behavior of the learned policy in the source domain resembles trajectories produced by the target optimal policies, which does not guarantee optimal performance when the learned policy is actually deployed to the target domain. In this work, we propose to utilize imitation learning to transfer the policy learned from the reward modification to the target domain so that the new policy can generate the same trajectories in the target domain. Our approach, Domain Adaptation and Reward Augmented Imitation Learning (DARAIL), utilizes the reward modification for domain adaptation and follows the general framework of generative adversarial imitation learning from observation (GAIfO) by applying a reward augmented estimator for the policy optimization step. Theoretically, we present an error bound for our method under a mild assumption regarding the dynamics shift to justify the motivation of our method. Empirically, our method outperforms the pure modified reward method without imitation learning and also outperforms other baselines in benchmark off-dynamics environments.

Figures

Figures reproduced from arXiv: 2411.09891 by the authors.

Figure 1
Figure 1. (a) Training reward in the source domain, i.e. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Performance of DARAIL and IPS-ACL on HalfCheetah and Walker2d under different [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Performance of DARC and DARAIL under different off-dynamics shifts on Ant. Action [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Optimization objective of DARC. DARC minimizes the reverse KL divergence of the [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: Training reward in the source domain, i.e. [PITH_FULL_IMAGE:figures/full_fig_p021_5.png]
Figure 6
Figure 6. Figure 6: Upper horizon line: DARC reward in the source domain. Lower horizon line: DARC [PITH_FULL_IMAGE:figures/full_fig_p021_6.png]
Figure 7
Figure 7. Figure 7: Training Curve of changing gravity setting. Top: target domain gravity [PITH_FULL_IMAGE:figures/full_fig_p022_7.png]
Figure 8
Figure 8. Figure 8: Training Curve of changing density setting. Top: target domain density [PITH_FULL_IMAGE:figures/full_fig_p022_8.png]
Figure 9
Figure 9. Figure 9: Training reward in the source domain, i.e. [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]
Figure 10
Figure 10. Figure 10: Experiments of DARC and DARAIL on the intact source and broken target setting. We [PITH_FULL_IMAGE:figures/full_fig_p024_10.png]
Figure 11
Figure 11. Figure 11: Experiments on using source optimal policy as the expert demonstration instead of the [PITH_FULL_IMAGE:figures/full_fig_p025_11.png]
Figure 12
Figure 12. Figure 12: Experiment on how cumulative n-step importance weight performs on DARAIL. Per-step [PITH_FULL_IMAGE:figures/full_fig_p026_12.png]
Figure 13
Figure 13. Figure 13: Experiment on how cumulative n-step importance weight performs on IS-R in broken [PITH_FULL_IMAGE:figures/full_fig_p026_13.png]
Figure 14
Figure 14. Figure 14: Experiment on the performance of DARAIL under different update steps of the discrimi [PITH_FULL_IMAGE:figures/full_fig_p027_14.png]
Figure 15
Figure 15. Figure 15: Experiment of different η in the modified reward r(st, at)+η+∆(st, at, st+1) for DARC in broken source environment setting. Top row: η = 1, middle row: η = 1.5 and button row: η = 2. We observe that increasing the η will reduce the performance degradation in most case…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

53 extracted references · 37 canonical work pages

  1. [1]

    Deep rein- forcement learning for dynamic treatment regimes on medical registry data

    Ying Liu, Brent Logan, Ning Liu, Zhiyuan Xu, Jian Tang, and Yangzhi Wang. Deep rein- forcement learning for dynamic treatment regimes on medical registry data. In 2017 IEEE international conference on healthcare informatics (ICHI), pages 380–385. IEEE, 2017

    work page 2017

  2. [2]

    Deep reinforcement learning for autonomous driving: A survey

    B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems, 23(6):4909–4926, 2021

    2021

  3. [3]

    Off-dynamics reinforcement learning: Training for transfer with domain classifiers

    Benjamin Eysenbach, Swapnil Asawa, Shreyas Chaudhari, Sergey Levine, and Ruslan Salakhut- dinov. Off-dynamics reinforcement learning: Training for transfer with domain classifiers. arXiv preprint arXiv:2006.13916, 2020

    arXiv 2006

  4. [4]

    Sim-to-real interactive recommendation via off-dynamics reinforcement learning

    Junda Wu, Zhihui Xie, Tong Yu, Qizhi Li, and Shuai Li. Sim-to-real interactive recommendation via off-dynamics reinforcement learning. In 2rd Offline Reinforcement Learning Workshop Advances at NeurIPS, 2021

    work page 2021

  5. [5]

    Dara: Dynamics-aware reward augmentation in offline reinforcement learning

    Jinxin Liu, Hongyin Zhang, and Donglin Wang. Dara: Dynamics-aware reward augmentation in offline reinforcement learning. arXiv preprint arXiv:2203.06662, 2022

    arXiv 2022

  6. [6]

    Unsupervised domain adaptation with dynamics-aware rewards in reinforcement learning

    Jinxin Liu, Hao Shen, Donglin Wang, Yachen Kang, and Qiangxing Tian. Unsupervised domain adaptation with dynamics-aware rewards in reinforcement learning. Advances in Neural Information Processing Systems, 34:28784–28797, 2021

    work page 2021

  7. [7]

    Generative adversarial imitation learning

    Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016

    2016

  8. [8]

    Generative adversarial imitation from observa- tion

    Faraz Torabi, Garrett Warnell, and Peter Stone. Generative adversarial imitation from observa- tion. arXiv preprint arXiv:1807.06158, 2018

    arXiv 2018

Show all 53 references
  1. [9]

    Offline imitation learning with a misspecified simulator

    Shengyi Jiang, Jingcheng Pang, and Yang Yu. Offline imitation learning with a misspecified simulator. Advances in neural information processing systems, 33:8510–8520, 2020

    2020

  2. [10]

    An imitation from observation approach to transfer learning with dynamics mismatch

    Siddharth Desai, Ishan Durugkar, Haresh Karnan, Garrett Warnell, Josiah Hanna, and Peter Stone. An imitation from observation approach to transfer learning with dynamics mismatch. Advances in Neural Information Processing Systems, 33:3917–3929, 2020

    2020

  3. [11]

    State-only imitation with transition dynamics mismatch

    Tanmay Gangwani and Jian Peng. State-only imitation with transition dynamics mismatch. arXiv preprint arXiv:2002.11879, 2020

    2002 arXiv

  4. [12]

    Doubly robust policy evaluation and learning

    Miroslav Dudík, John Langford, and Lihong Li. Doubly robust policy evaluation and learning. arXiv preprint arXiv:1103.4601, 2011

    2011 arXiv

  5. [13]

    Doubly robust off-policy value evaluation for reinforcement learning

    Nan Jiang and Lihong Li. Doubly robust off-policy value evaluation for reinforcement learning. In International Conference on Machine Learning, pages 652–661. PMLR, 2016

    2016

  6. [14]

    Doubly robust off-policy evaluation with shrinkage

    Yi Su, Maria Dimakopoulou, Akshay Krishnamurthy, and Miroslav Dudík. Doubly robust off-policy evaluation with shrinkage. In International Conference on Machine Learning, pages 9167–9176. PMLR, 2020

    2020

  7. [15]

    Doubly robust off-policy actor-critic: Convergence and optimality

    Tengyu Xu, Zhuoran Yang, Zhaoran Wang, and Yingbin Liang. Doubly robust off-policy actor-critic: Convergence and optimality. In International Conference on Machine Learning, pages 11581–11591. PMLR, 2021. 11

    2021

  8. [16]

    Doubly robust distribu- tionally robust off-policy evaluation and learning

    Nathan Kallus, Xiaojie Mao, Kaiwen Wang, and Zhengyuan Zhou. Doubly robust distribu- tionally robust off-policy evaluation and learning. In International Conference on Machine Learning, pages 10598–10632. PMLR, 2022

    2022

  9. [17]

    Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861–1870. PMLR, 2018

    2018

  10. [18]

    On the off-dynamics approach to reinforcement learning

    Joshua Arvind Holla. On the off-dynamics approach to reinforcement learning . McGill University (Canada), 2021

    2021

  11. [19]

    When to trust your model: Model-based policy optimization

    Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. Advances in neural information processing systems , 32, 2019

    2019

  12. [20]

    Mutual alignment transfer learning

    Markus Wulfmeier, Ingmar Posner, and Pieter Abbeel. Mutual alignment transfer learning. In Conference on Robot Learning, pages 281–290. PMLR, 2017

    2017

  13. [21]

    Domain adaptation for reinforcement learning on the atari

    Thomas Carr, Maria Chli, and George V ogiatzis. Domain adaptation for reinforcement learning on the atari. arXiv preprint arXiv:1812.07452, 2018

    2018 arXiv

  14. [22]

    Domain adaptation in reinforcement learning via latent unified state representation

    Jinwei Xing, Takashi Nagata, Kexin Chen, Xinyun Zou, Emre Neftci, and Jeffrey L Krichmar. Domain adaptation in reinforcement learning via latent unified state representation. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10452–10459, 2021

    2021

  15. [23]

    Transfer learning in deep reinforce- ment learning: A survey

    Zhuangdi Zhu, Kaixiang Lin, Anil K Jain, and Jiayu Zhou. Transfer learning in deep reinforce- ment learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023

    2023

  16. [24]

    Offline reinforcement learning: Tutorial, review, and perspectives on open problems

    Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020

    2005 arXiv

  17. [25]

    State regularized policy optimization on data with dynamics shift

    Zhenghai Xue, Qingpeng Cai, Shuchang Liu, Dong Zheng, Peng Jiang, Kun Gai, and Bo An. State regularized policy optimization on data with dynamics shift. Advances in neural informa- tion processing systems, 36, 2024

    2024

  18. [26]

    Generative adversarial nets

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014

    2014

  19. [27]

    Distributionally robust off-dynamics reinforcement learning: Prov- able efficiency with linear function approximation

    Zhishuai Liu and Pan Xu. Distributionally robust off-dynamics reinforcement learning: Prov- able efficiency with linear function approximation. In International Conference on Artificial Intelligence and Statistics, pages 2719–2727. PMLR, 2024

    2024

  20. [28]

    Learning robust rewards with adversarial inverse reinforcement learning

    Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprint arXiv:1710.11248, 2017

    2017 arXiv

  21. [29]

    Imitation learning via kernel mean embedding

    Kee-Eung Kim and Hyun Soo Park. Imitation learning via kernel mean embedding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018

    2018

  22. [30]

    Variational discriminator bottleneck: Improving imitation learning, inverse rl, and gans by constraining information flow

    Xue Bin Peng, Angjoo Kanazawa, Sam Toyer, Pieter Abbeel, and Sergey Levine. Variational discriminator bottleneck: Improving imitation learning, inverse rl, and gans by constraining information flow. arXiv preprint arXiv:1810.00821, 2018

    arXiv

  23. [31]

    Task transfer by preference-based cost learning

    Mingxuan Jing, Xiaojian Ma, Wenbing Huang, Fuchun Sun, and Huaping Liu. Task transfer by preference-based cost learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 2471–2478, 2019

    2019

  24. [32]

    Imitation learning from video by leveraging proprioception

    Faraz Torabi, Garrett Warnell, and Peter Stone. Imitation learning from video by leveraging proprioception. arXiv preprint arXiv:1905.09335, 2019

    1905 arXiv

  25. [33]

    Imitation from observation: Learning to imitate behaviors from raw video via context translation

    YuXuan Liu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Imitation from observation: Learning to imitate behaviors from raw video via context translation. In2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1118–1125. IEEE, 2018. 12

    2018

  26. [34]

    Behavioral cloning from observation

    Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. arXiv preprint arXiv:1805.01954, 2018

    2018 arXiv

  27. [35]

    Recent advances in imitation learning from observation

    Faraz Torabi, Garrett Warnell, and Peter Stone. Recent advances in imitation learning from observation. arXiv preprint arXiv:1905.13566, 2019

    1905 arXiv

  28. [36]

    Domain adaptive imitation learning

    Kuno Kim, Yihong Gu, Jiaming Song, Shengjia Zhao, and Stefano Ermon. Domain adaptive imitation learning. In International Conference on Machine Learning, pages 5286–5295. PMLR, 2020

    2020

  29. [37]

    Generalization and equilibrium in generative adversarial nets (gans)

    Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, and Yi Zhang. Generalization and equilibrium in generative adversarial nets (gans). In International conference on machine learning, pages 224–232. PMLR, 2017

    2017

  30. [38]

    vf+MOWCXlXkD7CB/Zj9tqm3hyT0=

    Tian Xu, Ziniu Li, and Yang Yu. Error bounds of imitating policies and environments.Advances in Neural Information Processing Systems, 33:15737–15749, 2020. 13 A Analysis of DARC A.1 DARC Objective Figure 4 shows the objective of DARC, which minimizes the reverse KL divergence...

    2020

  31. [39]

    And in the introduc- tion section, we have a contribution list

    Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: Abstract and Introduction section states the contribution. And in the introduc- tion section, we have a contribution l...

  32. [40]

    Limitations

    Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We talk about the limitation of our method in the Appendix E. Guidelines: • The answer NA means that the paper has no limitation while the answer No ...

  33. [41]

    We present our theoretical result in Section 4 and the proof is in Appendix B

    Theory Assumptions and Proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] 29 Justification: Yes. We present our theoretical result in Section 4 and the proof is in Appendix B. Guide...

  34. [42]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experimental Result Reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and da...

  35. [43]

    Guidelines: • The answer NA means that paper does not include experiments requiring code

    Open access to data and code 30 Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We provide a GitHub repo...

  36. [44]

    We also describe the hyperparameter tuning in the Appendix D.4

    Experimental Setting/Details Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: We provide the details of the...

  37. [45]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experiment Statistical Significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: We have multiple runs of each experiment and repor...

  38. [46]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experiments Compute Resources Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: We provide the GPU/CPU ...

  39. [47]

    Guidelines: • The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics

    Code Of Ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: Our data is open source benchmarks in the RL research field. Guidelines: • The ...

  40. [48]

    Guidelines: • The answer NA means that there is no societal impact of the work performed

    Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: In the conclusion, we briefly mentioned that our method avoids directly training a policy in a high-risk ...

  41. [49]

    Guidelines: • The answer NA means that the paper poses no such risks

    Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: We run the expe...

  42. [50]

    Guidelines: • The answer NA means that the paper does not use existing assets

    Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We provide citat...

  43. [51]

    Also, details about the implementation are included in the paper

    New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: We include the code in our paper. Also, details about the implementation are included in the paper. Guidelines: • Th...

  44. [52]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Crowdsourcing and Research with Human Subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? A...

  45. [53]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.