Pith. sign in

REVIEW 3 major objections 5 minor 15 references

Recursive Deep Inverse Reinforcement Learning

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper derives recursive Kalman-style updates for deep inverse RL by bounding the maximum-entropy objective with a moment-matching loss, recovering expert costs online and outperforming batch baselines on the tested benchmarks.

desk verdict A new and plausibly useful online deep IRL algorithm whose advertised theoretical guarantee — the upper-bound derivation — doesn't hold as written; worth reviewing, but Section 4 needs real repair. read the letter →

arxiv 2504.13241 v6 pith:4V6PBFUP submitted 2025-04-17 cs.LG cs.AI

Paul Ghanem , Owen Howell , Michael Potter , Pau Closas , Alireza Ramezani , Deniz Erdogmus , Tales Imbiriba This is my paper · ORCID
classification cs.LGcs.AI
keywords inversereinforcementlearningmaximumentropyIRLonlinemomentmatchingextendedKalmanfiltersecond-orderNewtonmethodmodelpredictivepathintegralcontrolcognitiveradar
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper seeks to make deep inverse reinforcement learning — recovering an expert's cost or reward function from demonstrated behavior — work online, updating on each arriving demonstration instead of waiting for large trajectory batches. Its key move is to prove that the maximum-entropy IRL objective, whose log-of-a-sum term normally blocks incremental optimization, is bounded above by a moment-matching loss: the accumulated difference between the learned cost of expert trajectories and the learned cost of trajectories sampled from the current policy. The authors minimize that loss with sequential second-order Newton updates patterned on the extended Kalman filter, so every expert sample immediately updates the cost network and the sampling policy. If the claims hold, the result is an online deep IRL algorithm that converges in far fewer episodes than batch methods such as GCL, AIRL, GAIL, SQIL, and ML-IRL on the tested continuous-control and cognitive-radar benchmarks, and a maximum-entropy justification for the moment-matching family of IRL losses.

What carries the argument

Two objects carry the argument. The first is the moment-matching upper bound (Eq. 10): a Jensen-type inequality for convex functions, applied with $f=-\log$, bounds the log-of-a-sum in the MaxEnt objective by a plain sum of per-trajectory costs minus $K=\log a+\log b-2\log((a+b)/2)$. That constant, argued to be independent of the parameters, is what turns a batch-only objective into an additive form that can be minimized incrementally. The second is the Kalman-style recursive update of Theorem 5.1: $\hat{\theta}(t_i)=\hat{\theta}(t_{i-1})-P_{\theta i}\bigl(\nabla c_\theta(\tau^i_{\mathrm{demo}})-\nabla c_\theta(\tau^i_{\mathrm{samp}})\bigr)$ with the covariance recursion $P_{\theta i}=\bigl[(P_{\theta i-1}+Q_\theta)^{-1}+\bigl(\nabla^2 c_\theta(\tau^i_{\mathrm{demo}})-\nabla^2 c_\theta(\tau^i_{\mathrm{samp}})\bigr)\bigr]^{-1}$, where $P_{\theta i}$ acts as an adaptively sized, uncertainty-aware learning rate and the Hessian difference carries the second-order information that drives fast convergence.

What would settle it

On a single benchmark task, record at every RDIRL parameter update both the true maximum-entropy negative log-likelihood $L_{\mathrm{IRL}}(\theta)$ from Eq. 2 and the moment-matching loss $L_{\mathrm{UB-MM}}(\theta)$ from Eq. 10. If the gap $L_{\mathrm{UB-MM}}-L_{\mathrm{IRL}}$ dips below a fixed constant at any point along the optimization trajectory — that is, if the moment-matching loss ever falls below the true objective by more than a constant offset — the upper-bound claim is refuted, since a valid bound must hold for all $\theta$ with one fixed constant.

Watch

Extended reading notes

Core claim

Section 4 establishes the paper's theoretical load-bearing claim: the log-sum term in the maximum-entropy IRL negative log-likelihood, $L_{\mathrm{IRL}}(\theta)=\frac{1}{N}\sum_{i}c_\theta(\tau^i_{\mathrm{demo}})+\log\frac{1}{M}\sum_{j}\frac{\exp(-c_\theta(\tau^j_{\mathrm{samp}}))}{q(\tau^j_{\mathrm{samp}})}$, can be replaced by a sum of per-trajectory terms minus a constant $K$, via a Jensen-type inequality for convex functions applied to $f=-\log$. Minimizing the resulting moment-matching loss $L_{\mathrm{UB-MM}}(\theta)=\sum_i\bigl[c_\theta(\tau^i_{\mathrm{demo}})-c_\theta(\tau^i_{\mathrm{samp}})\bigr]$ is therefore equivalent to minimizing the true maximum-entropy objective, up to a constant independent of $\theta$. Theorem 5.1 then derives the recursive update $\hat{\theta}(t_i)=\hat{\theta}(t_{i-1})-P_{\theta i}\bigl(\nabla c_\theta(\tau^i_{\mathrm{demo}})-\nabla c_\theta(\tau^i_{\mathrm{samp}})\bigr)$ and the covariance recursion $P_{\theta i}=\bigl[(P_{\theta i-1}+Q_\theta)^{-1}+\bigl(\nabla^2 c_\theta(\tau^i_{\mathrm{demo}})-\nabla^2 c_\theta(\tau^i_{\mathrm{samp}})\bigr)\bigr]^{-1}$ — a sequential second-order Newton step obtained along the same lines as the extended Kalman filter. The paper's overall claim is that this recursion, with MPPI as the inner sampling policy, learns an expert's cost or reward function from demonstrations as they arrive and surpasses GCL, AIRL, GAIL, SQIL, and ML-IRL in the reported continuous-control and cognitive-radar benchmarks.

Load-bearing premise

The derivation in Section 4 assumes every ratio $\exp(-c_\theta(\tau^j_{\mathrm{samp}}))/q(\tau^j_{\mathrm{samp}})$ lies inside a fixed interval $[a,b]$ whose endpoints — and therefore the constant $K$ — are independent of the network parameters $\theta$; the paper asserts this without constructing $a$ and $b$, and a neural-network cost that grows without bound as $\theta$ varies can violate the requirement, so the upper bound is not guaranteed to hold in general.

Editorial extensions

If this is right

  • IRL training can proceed online: each expert demonstration immediately updates the cost network and re-optimizes the inner sampling policy, enabling real-time use in latency-constrained settings such as counterplanning and adversarial radar tracking.
  • The recursion replaces a fixed learning rate with an uncertainty-adaptive step $P_{\theta i}$, which the paper identifies as the reason RDIRL converges in one to a few dozen episodes on the tested benchmarks while batch baselines lag or fail to converge.
  • Moment-matching IRL losses acquire a principled status: because they bound the maximum-entropy negative log-likelihood, they inherit the MaxEnt model's interpretation rather than standing as ad hoc objectives.
  • The scheme is not tied to one control solver: the inner sampling policy $q(\tau)$ can be updated by any policy optimization method, and the paper uses MPPI throughout for both RDIRL and the adapted baselines.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same Jensen-variant bound could convert other log-sum objectives — partition functions in energy-based models or maximum-entropy policy evaluation — into additive form, potentially yielding online second-order optimizers beyond inverse RL.
  • A direct stress test of the theory would monitor the gap between $L_{\mathrm{UB-MM}}$ and the true $L_{\mathrm{IRL}}$ during training; if the bound is loose on harder tasks, the Newton interpretation would degrade even while the updates remain well-defined.
  • The cognitive-radar experiment points to a broader capability: an observer who sees only another agent's position traces might infer that agent's sensing objective (here, Fisher-information-based tracking), which is precisely the input that counterplanning against sensor-aware adversaries would need.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Recursive Deep Inverse Reinforcement Learning (RDIRL), an online algorithm for recovering a neural-network cost/reward function from expert demonstrations. The authors claim that the moment-matching loss in Eq. (10) is an upper bound of the maximum-entropy IRL negative log-likelihood in Eq. (2) up to a constant independent of the model parameters, and that this bound can be minimized recursively with sequential second-order Newton updates inspired by the extended Kalman filter (Theorem 5.1, Eqs. (12)-(13)). The method is evaluated on CartPole, MountainCar, HalfCheetah, Hopper, Walker2d, and a cognitive radar task, where it is reported to outperform GAIL, GCL, AIRL, SQIL, and ML-IRL when all methods use MPPI as the inner policy.

Significance. If the upper-bound claim were correct, the paper would offer a principled connection between maximum-entropy IRL and moment-matching objectives, and the recursive second-order update would be a meaningful step toward online deep IRL. The authors make a fair effort to compare against baselines by adapting all of them to the same MPPI inner policy and using the same network architecture, and they provide code in the supplementary material. However, the central theoretical derivation has a load-bearing gap: the claimed θ-independent upper bound is not established for the settings considered, and the covariance update in Eq. (13) lacks a positive-definiteness guarantee. These issues undermine the main theoretical contribution, although the algorithm may still work as a heuristic with additional assumptions or empirical justification.

major comments (3)
  1. [§4, Eq. (9)] The derivation of the upper bound requires that y_n = exp(−c_θ(τ_samp_i))/q(τ_samp_i) lie in a fixed interval [a,b] independent of θ, so that K = log a + log b − 2 log((a+b)/2) is a constant. The paper neither constructs such an interval nor restricts the parameter space. For a neural-network cost with unbounded output, exp(−c_θ) can be arbitrarily large or arbitrarily close to zero as θ varies, and q itself is updated using c_θ (Algorithm 1, line 11), so log q also depends on θ. Therefore K and C in Eq. (9) are not constants independent of θ, and dropping them to obtain Eq. (10) changes the objective by a θ-dependent term. Consequently, Eq. (10) is not proven to be an upper bound of Eq. (2), and the central claim that RDIRL minimizes an upper bound of the GCL objective is unsupported.
  2. [§5, Theorem 5.1, Eq. (13)] The recursive covariance update P_θ_i = [(P_θ_{i−1} + Q_θ)^{-1} + C^2_τdemo(t_i) − C^2_τsamp(t_i)]^{-1} has no positive-definiteness guarantee. The difference of Hessians C^2_τdemo(t_i) − C^2_τsamp(t_i) is not necessarily positive definite for a nonconvex neural-network cost, and the proof only cites Lemma B.3 of (Humpherys et al., 2012) without verifying the needed invertibility or definite-ness conditions. If P_θ_i is not positive definite, the Newton-like update in Eq. (12) may not be a descent direction, and P_θ_i cannot be interpreted as a posterior covariance. Since the convergence behavior of RDIRL is one of the main advertised benefits, this gap needs to be closed or the claim substantially softened.
  3. [§6, Tables 1–4 and Figures 1–4] The paper claims that RDIRL consistently outperforms all baselines, but the empirical support is incomplete. Standard deviations are reported, yet no statistical significance tests, confidence intervals, or effect sizes are provided, and in several tasks the intervals overlap substantially (e.g., MountainCar in Table 1: RDIRL 0.68±0.32 vs. GCL 0.247±0.19). The cognitive radar experiment is based on only five Monte Carlo runs. Additionally, the convergence comparison is in terms of episodes, not wall-clock time, so the claim of speed is not fully verified given that RDIRL computes Hessians at every step. The authors should either add proper statistical analysis or temper the comparative claims.
minor comments (5)
  1. [§4, Eq. (7)] The notation '[a,b]∈R' should be '[a,b]⊂R' to indicate an interval in the real line.
  2. [§6.2] In the cognitive radar section, the text refers to 'the reward function in Figure 1' when it should refer to Figure 3, and 'Table equation 3' should be 'Table 3'.
  3. [§B.1, Table 3] The grid-search ranges used to select hyperparameters for the adapted baselines are not reported, making it difficult to assess whether the baselines were given a fair tuning budget.
  4. [§6.1] The statement that the method 'requires no fixed learning rate' is presented as an advantage, but the covariance update in Eq. (13) itself has initialization parameters P_θ0 and Q_θ that need tuning, and the sensitivity to these parameters is not studied.
  5. [§B.3, Algorithm 1] Computing exact Hessians of a deep network at every sample may be expensive, but the paper does not discuss Hessian approximation, computational overhead, or the practical cost of this choice relative to first-order baselines.

Circularity Check

0 steps flagged · score 2.0 of 10

No material circularity: the moment-matching upper bound is derived from an external Jensen inequality, and the empirical claims are checked against true expert rewards.

full rationale

The central claimed derivation is Eq. (10) as an upper bound of the MaxEnt IRL negative log-likelihood Eq. (2). The chain is: Eq. (5)-(6) is an external Jensen-inequality variant from Matkovic & Pecaric (2007); substituting p_n=1/N and y_n=exp(-c_theta(tau_samp))/q(tau_samp) gives Eq. (8); substituting Eq. (8) into Eq. (2) gives Eq. (9); dropping the term C then gives Eq. (10). None of these steps defines c_theta in terms of the quantity being predicted, fits a parameter to the data used for evaluation, or imports a uniqueness conclusion from the authors' own prior work. The moment-matching objective is explicitly credited to Swamy et al. (2021), and the paper's contribution is the upper-bound interpretation plus a recursive Newton/EKF-style update; acknowledging that Eq. (10) is a known moment-matching loss is a reinterpretation, not circular renaming. Theorem 5.1 is proved by invoking Lemma B.3 of Humpherys et al. (2012), an independent external result; the self-citations (Ghanem et al. 2023, 2025; Imbiriba et al. 2022) are used only for analogies to Bayesian filtering regularization, and Potter et al. (2024) supplies the cognitive-radar testbed, neither of which is load-bearing for the upper-bound claim. Experiments evaluate policies trained with the recovered reward against the environment's true expert reward, so the reported improvements are not forced by construction. The possible failure of the [a,b] interval assumption for unbounded neural-network costs, and the dependence of q on theta across updates, is a correctness/validity concern about the upper bound, but it is not circularity: the derivation would be unsupported, not tautological. No step reduces by definition to its own input, so no circular step is identified. The score reflects only the presence of minor, non-load-bearing self-citations.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central derivation rests on a Jensen inequality variant and, crucially, on an unstated fixed-interval assumption that makes the bounding constant independent of θ. No free parameters are fitted to data in a circular way, but Pθ0, Qθ, and MPPI hyperparameters are hand-picked. There are no invented entities.

free parameters (3)
  • Pθ0 (initial parameter covariance) = 1e-2 I
    Chosen by hand; controls initial step size and uncertainty of the parameter estimate in the recursive update (Appendix B.1).
  • Qθ (process noise covariance) = 1e-4 I
    Chosen by hand; regularizes consecutive parameter estimates in Eq. (11), analogous to a random-walk prior.
  • MPPI temperature, horizon, number of trajectories = Table 3 per environment
    Hyperparameters selected via grid search for the adapted baselines and reused for RDIRL; they shape the sampling distribution q(τ).
assumptions (4)
  • standard math Jensen inequality variant (Matkovic and Pecaric 2007)
    Used in Section 4 to bound the log-sum term; treated as a black box, with no verification that its preconditions hold for the chosen a,b.
  • domain assumption Expert trajectories follow p(τ) ∝ exp(−cθ(τ))
    The MaxEnt IRL model (Eq. 1) underlies the negative log-likelihood objective; assumes near-optimal stochastic expert behavior.
  • ad hoc to paper Fixed interval [a,b] exists with yn = exp(−cθ(τsamp_i))/q(τsamp_i) in [a,b] for all θ and all samples
    Required for K = log a + log b − 2 log((a+b)/2) to be independent of θ and for Eq. (10) to be the stated upper bound. The paper never constructs a,b, and this is unlikely for unbounded neural-network costs. See Section 4, around Eqs. (7)-(9).
  • ad hoc to paper Hessian difference keeps Pθi positive definite
    Eq. (13) updates Pθi via inversion of [ (Pθi−1+Qθ)−1 + C²τdemo − C²τsamp ]; no proof is given that this matrix remains positive definite, which the Newton step requires.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Recursive Deep Inverse Reinforcement Learning." pith.science (2026). https://pith.science/paper/4V6PBFUP

@misc{pith2026250413241,
  author       = {Pith},
  title        = {Pith review of: Recursive Deep Inverse Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4V6PBFUP}},
  note         = {Machine review of arXiv:2504.13241}
}
read the original abstract

Inferring an adversary's goals from exhibited behavior is crucial for counterplanning and non-cooperative multi-agent systems in domains like cybersecurity, military, and strategy games. Deep Inverse Reinforcement Learning (IRL) methods based on maximum entropy principles show promise in recovering adversaries goals but are typically offline, require large batch sizes with gradient descent, and rely on first-order updates, limiting their applicability in real-time scenarios. We propose an online Recursive Deep Inverse Reinforcement Learning (RDIRL) approach to recover the cost function governing the adversary actions and goals. Specifically, we minimize an upper bound on the standard Guided Cost Learning (GCL) objective using sequential second-order Newton updates, akin to the Extended Kalman Filter (EKF), leading to a fast (in terms of convergence) learning algorithm. We demonstrate that RDIRL is able to recover cost and reward functions of expert agents in standard and adversarial benchmark tasks. Experiments on benchmark tasks show that our proposed approach outperforms several leading IRL algorithms.

Figures

Figures reproduced from arXiv: 2504.13241 by the authors.

Figure 1
Figure 1. Learning curves for RDIRL and other methods. [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Learning curves for for Walker2d [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Learning curves for RDIRL and other methods. In all algorithms, we used the same neural net￾work architecture to parameterize the radar’s FIM reward function: one hidden layer of 128 units, with a RELU activation function All networks were always initialized randomly at the start of each experiment and all experiments are run on on an intel core i7 CPU. 9 [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Learning curves for RDIRL and online adaptation methods. [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 8 canonical work pages

  1. [1]

    The goal is to iteratively updateθ such that trajectories generated from the current policyq(τ) match the expert demonstrations

    The algorithm maintains a cost functioncθ(τ) parameterized byθ, which maps trajectoriesτ to scalar costs. The goal is to iteratively updateθ such that trajectories generated from the current policyq(τ) match the expert demonstrations. At each outer iteration (episode), we initialize the sampling policyq(τ) which can be a stochastic policy optimized with m...

    work page 2012

  2. [3]

    Additionally, we usePθ0 = 1e−2I and Qθ = 1e−4IwhereIis the identity matrix. B.2 REWARDFUNCTION ANDDISCRIMINATORNETWORKARCHITECTURES We use the same neural network architecture to parameterize the cost-function/reward- func- tion/discriminator for all methods. For continuous control task with raw state input, i.e. Cart- pole,MountainCar, and the MuJoCo tas...

    work page 2000

  3. [5]

    Jeffrey Humpherys, Preston Redd, and Jeremy West

    Accessed: 2025-05-14. Jeffrey Humpherys, Preston Redd, and Jeremy West. A fresh look at the kalman filter.SIAM review, 54(4):801–823,

    work page 2025

  4. [10]

    Online inverse reinforcement learning for systems with disturbances

    Ryan Self, Moad Abudia, and Rushikesh Kamalapurkar. Online inverse reinforcement learning for systems with disturbances. In2020 American control conference (ACC), pp. 1118–1123. IEEE, 2020a. Ryan Self, Kevin Coleman, He Bai, and Rushikesh Kamalapurkar. Online observer-based inverse reinforcement learning.IEEE Control Systems Letters, 5(6):1922–1927, 2020b...

    work page 1922

  5. [13]

    MPPIis a probabilistic model predictive control policy that estimates an optimal action distribution that minimizes an agent’s objective cost function

    as inner policy q(τ) in our baseline methods. MPPIis a probabilistic model predictive control policy that estimates an optimal action distribution that minimizes an agent’s objective cost function. To do so, MPPI samples a number of trajectories and weighs these trajectories depending on how well they minimize the cost function, then updates the mean of i...

    work page 2021

  6. [2006]

    Sqil: Imitation learning via reinforcement learning with sparse rewards.arXiv preprint arXiv:1905.11108,

    Siddharth Reddy, Anca D Dragan, and Sergey Levine. Sqil: Imitation learning via reinforcement learning with sparse rewards.arXiv preprint arXiv:1905.11108,

    arXiv 1905

  7. [2011]

    Openai gym.arXiv preprint arXiv:1606.01540,

    G Brockman. Openai gym.arXiv preprint arXiv:1606.01540,

    arXiv

  8. [2014]

    Continuously Optimizing Radar Placement with Model Predictive Path Integrals

    Michael Potter, Shuo Tang, Paul Ghanem, Milica Stojanovic, Pau Closas, Murat Akcakaya, Ben Wright, Marius Necsoiu, Deniz Erdogmus, Michael Everett, et al. Continuously optimizing radar placement with model predictive path integrals.arXiv preprint arXiv:2405.18999,

    work page Pith review arXiv

Show all 15 references
  1. [2016]

    A connection between generative adversarial networks, inverse reinforcement learning, and energy-based models.arXiv preprint arXiv:1611.03852, 2016a

    Chelsea Finn, Paul Christiano, Pieter Abbeel, and Sergey Levine. A connection between generative adversarial networks, inverse reinforcement learning, and energy-based models.arXiv preprint arXiv:1611.03852, 2016a. Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost le...

    arXiv

  2. [2017]

    From language to goals: Inverse reinforcement learning for vision-based instruction following.arXiv preprint arXiv:1902.07742,

    Justin Fu, Anoop Korattikara, Sergey Levine, and Sergio Guadarrama. From language to goals: Inverse reinforcement learning for vision-based instruction following.arXiv preprint arXiv:1902.07742,

    1902 arXiv

  3. [2018]

    Bayesian estimation and kalman filtering: A unified framework for mobile robot localization

    Stergios I Roumeliotis and George A Bekey. Bayesian estimation and kalman filtering: A unified framework for mobile robot localization. InProceedings 2000 ICRA. Millennium conference. IEEE international conference on robotics and automation. Symposia proceedings (Cat. No. 00CH...

    2000

  4. [2020]

    Maximum entropy deep inverse reinforce- ment learning.arXiv preprint arXiv:1507.04888,

    Markus Wulfmeier, Peter Ondruska, and Ingmar Posner. Maximum entropy deep inverse reinforce- ment learning.arXiv preprint arXiv:1507.04888,

    arXiv

  5. [2021]

    Derivation of a constant velocity motion model for visual tracking.arXiv preprint arXiv:2005.00844,

    Nathanael L Baisa. Derivation of a constant velocity motion model for visual tracking.arXiv preprint arXiv:2005.00844,

    2005 arXiv

  6. [2023]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    arXiv

  7. [2025]

    Inverse optimal control for finite-horizon discrete-time linear quadratic regulator under noisy output

    12 Under review Han Zhang, Yibei Li, and Xiaoming Hu. Inverse optimal control for finite-horizon discrete-time linear quadratic regulator under noisy output. In2019 IEEE 58th conference on decision and control (CDC), pp. 6663–6668. IEEE, 2019a. Han Zhang, Jack Umenberger, and ...

    arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.