Pith. sign in

REVIEW 4 major objections 5 minor 67 references

Distributional Inverse Reinforcement Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that inverse reinforcement learning can recover the full distribution of rewards—not just a mean—by aligning return distributions with first-order stochastic dominance, yielding risk-aware policies.

desk verdict Real idea, wrong sign: the FSD/DRM combination is original and the dopamine validation is interesting, but Eq. (8)'s quantile identity is false and the main framework rests on it. read the letter →

arxiv 2510.03013 v4 pith:7J4XWRSD submitted 2025-10-03 cs.LG

Feiyang Wu , Ye Zhao , Anqi Wu This is my paper · ORCID
classification cs.LG
keywords distributionalinversereinforcementlearningrewarddistributionfirst-orderstochasticdominancedistortionriskmeasureofflineimitationrisk-awarepolicyquantileregressiondopamine
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's aim is to lift inverse reinforcement learning (IRL) past point estimates of a reward function. It proposes DistIRL, a framework in which each state-action reward is a random variable with a learnable distribution, and expert demonstrations are matched by their full return distribution rather than by expected return. The matching criterion is first-order stochastic dominance (FSD): the learner minimizes the positive area between the expert's and the agent's return CDFs. Because FSD compares entire distributions, the recovered reward captures variance and higher moments, and the induced policy is risk-sensitive when updated with a distortion risk measure such as CVaR. The paper demonstrates the method on synthetic gridworlds, on mouse dopamine data, and on MuJoCo benchmarks.

What carries the argument

The load-bearing devices are (1) the FSD violation loss $L_{\mathrm{FSD}}(\pi,r)=\int [F_{Z_\pi}(z)-F_{Z_E}(z)]_+ \, dz$, which the paper rewrites as a positive integral over quantile-function differences via Eq. (8); and (2) the distortion risk measure $M_\xi(Z_\pi)=\int F^{-1}_{Z_\pi}(v)\, d\tilde{\xi}(v)$, used as a tractable surrogate for the FSD-indicator weighting in the policy objective (Eq. 13). The first connects reward-distribution learning to higher-order moment matching; the second converts FSD-aware policy learning into a standard risk-sensitive RL update.

What would settle it

Evaluate Eq. (8) with X constant 0 and Y constant -1: the CDF-side positive integral is 0, while the quantile-side integral is 1. This mismatch shows the identity is false as written, and the paper's own proof in Appendix B.1 contains the step in which the inequality direction is reversed (the 'if and only if' between events $\{F_X(z) \ge v, v \ge F_Y(z)\}$ and $\{F^{-1}_X(v) \ge z, z \ge F^{-1}_Y(v)\}$ has the second event's quantile inequalities reversed).

Watch

Extended reading notes

Core claim

The paper's central claim is that the minimax IRL objective can be replaced by a distributional one: minimize over reward distributions and maximize over policies the FSD violation between agent and expert return distributions, plus policy entropy and a reward prior. Solving it yields both a posterior over rewards (via variational inference with a skew-normal family) and a risk-aware policy that optimizes a distortion risk measure (DRM), specifically CVaR in the experiments. The authors claim this is the first principled recovery of the full reward distribution, and they report better imitation performance than baselines on risk-sensitive D4RL tasks and a correlation of about 0.3 between inf

Load-bearing premise

The derivation of both the reward loss and the policy objective assumes the identity $\int [F_X-F_Y]_+ \, dz = \int [F^{-1}_X(v)-F^{-1}_Y(v)]_+ \, dv$ (Eq. 8); if that quantile-integral identity fails, the algorithm is not minimizing the FSD violation it claims to minimize.

Editorial extensions

If this is right

  • If the central claim holds, offline IRL can recover not just mean rewards but variance and skewness, making reward inference useful in settings with stochastic or contact-rich outcomes.
  • Demonstrations from humans or animals can in principle reveal risk preferences, since the recovered policy explicitly optimizes a distortion risk measure.
  • The method is stated to converge in O(ε^{-2}) iterations, so the algorithmic cost is comparable to standard distributional RL training.
  • Recovering reward distributions from behavior alone could help decode internal reward signals (e.g., dopamine variability) in neuroscience.
  • Because the framework accepts any differentiable parametric reward distribution, it is extensible beyond skew-normal to mixtures or heavy-tailed families.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If Eq. (8) fails, the actual minimized loss is not the FSD violation; with a corrected sign the algorithm would reward policies for decreasing quantile gaps, reversing the intended risk direction. This can be tested on constant random variables.
  • The reported correlation between inferred reward means and dopamine suggests a broader path: IRL could infer trial-to-trial variability in reward from behavior, potentially applicable to other neuromodulators.
  • The skew-normal reward family limits fidelity for multimodal distributions; a mixture-based parameterization would be a natural extension and could be tested on the same dopamine data.
  • The method's risk-aware policy uses a single DRM (CVaR); because the theory posits all distortion functions are needed for exact FSD, approximating with one DRM is an unverified gap.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes DistIRL, a distributional offline inverse reinforcement learning framework that models the reward as a variational distribution q_phi(r|s,a) and learns a risk-sensitive policy by matching the full return distribution. The method uses a first-order stochastic dominance (FSD) violation between agent and expert return CDFs as the reward-learning loss, a quantile-regression critic, and a distortion risk measure (DRM) for policy updates. Experiments are reported on a gridworld, a mouse dopamine dataset, and MuJoCo control tasks. The central theoretical step is Eq. (8), which claims an equivalence between a positive-part integral of CDF differences and a positive-part integral of quantile differences.

Significance. The problem is relevant: extending IRL to genuinely stochastic rewards and distribution-aware policies could matter for neuroscience and risk-sensitive imitation. The dopamine experiment is a genuine external benchmark, which is a strength. However, the paper's central identity (Eq. 8) is false, the DRM used for policy learning weights the wrong tail of the return distribution, and the abstract's convergence guarantee is explicitly disclaimed in Section 4.3. These are load-bearing issues: the algorithm as written optimizes a different objective from the one claimed, so the empirical results cannot be attributed to the proposed theoretical framework. The contribution is not established in its current form.

major comments (4)
  1. [§4.1, Eq. (8) and Appendix B.1] Eq. (8) is false as stated. For X=δ_0 and Y=δ_{−1}, F_X−F_Y = −1 on [−1,0), so the LHS is 0; F_X^{-1}(v)−F_Y^{-1}(v)=1 for v∈(0,1), so the RHS is 1. The correct identity is ∫[F_X−F_Y]_+ dz = ∫[F_Y^{-1}−F_X^{-1}]_+ dv. The proof in Appendix B inverts the quantile implication: F_X(z)≥v implies F_X^{-1}(v)≤z, not ≥z. Since Eq. (8) is used to form the FSD reward loss (Eq. 7) and the policy objective (Eq. 9), the sign of the optimized violation is reversed. This invalidates the central theoretical claim.
  2. [§4.2, Eq. (13) and Table 1] The CVaR distortion used in Eq. (13) is not the 'average of worst α-fraction of outcomes' claimed in Table 1. With ξ(v)=min(v/α,1) and dual distortion ξ~(v)=1−ξ(1−v), dξ~ is supported on [1−α,1], so M_ξ(X) = (1/α)∫_{1−α}^1 F_X^{-1}(v)dv, i.e., the conditional expectation of the best α-fraction of returns. Maximizing Eq. (13) is therefore risk-seeking, not risk-averse. This contradicts the text's interpretation and the risk-averse expert descriptions in Secs. 5.1 and 5.3. Additionally, the surrogate-objective argument uses one arbitrary DRM, so Proposition 4.6 does not justify equivalence with Eq. (9).
  3. [Abstract and §4.3] The abstract states: 'Theoretical analysis shows that the algorithm converges with O(ε^{-2}) iteration complexity.' Section 4.3 explicitly states: 'Regrettably, a rigorous convergence analysis is beyond the scope of this paper.' No theorem or proof of an O(ε^{-2}) bound appears anywhere in the manuscript. This is an unsupported and internally contradictory claim that must be removed or substantiated.
  4. [§5.2] The paper claims the dopamine distributions are 'often right-skewed and multimodal' and that S-DistIRL 'most accurately recovers the shape.' However, S-DistIRL uses Azzalini's skew-normal distribution, which is unimodal by construction and cannot represent multimodal densities. No multimodality test or mixture extension is provided. The quantitative evidence (Fig. 4B) is a single Wasserstein-1 distance without confidence intervals or per-subject statistical comparisons, so the neurobehavioral validation is weaker than claimed.
minor comments (5)
  1. [Table 2] The table header reads 'IPMD (ours)' while the text and the rest of the paper call the method DistIRL. This appears to be a copy-paste error.
  2. [§3.1, Eq. (2)] The quantile Huber loss is introduced as ρ^κ_τ but the LQR loss writes ρ_{τ_i} without κ; also θ_j(s',a') notation should be clarified (target quantile index) to match the standard QR loss.
  3. [§4.1] The notation 'empirical quantile ... (z_{(-N)},...,z_{(1)})' for order statistics is nonstandard and confusing; standard order-statistic notation would be clearer.
  4. [§5.3, Table 3] The text says DistIRL achieves 'competitive or superior performance' on deterministic D4RL, but on HalfCheetah DistIRL (7779±228) is clearly below Offline ML-IRL (11231±585). The wording should be adjusted.
  5. [Table 4] The ablation reports only 'scaled performance for clarity'; without the actual unscaled returns and standard deviations, the quantitative comparison is difficult to interpret.

Circularity Check

0 steps flagged · score 1.0 of 10

No material circularity: the reward distribution is fit to demonstrations and validated against external dopamine/returns benchmarks; remaining self-citations are not load-bearing.

full rationale

The central derivation is not circular. DistIRL defines an FSD-based objective (Eq. 4), derives the reward-learning loss (Eq. 7) and the DRM policy objective (Eq. 13) from stated definitions rather than from pre-existing conclusions, and then validates the learned reward distribution against external measurements (e.g., recorded dopamine traces in Sec. 5.2 and true returns in MuJoCo experiments). The reward distribution q_phi is the optimization variable, not a renamed input. The paper does cite prior work by its own authors (e.g., Wu et al. 2023 for the TTSA scheme), but this is used only as an algorithmic convenience and is not load-bearing for the paper's main claim; moreover, the paper explicitly disclaims a rigorous convergence analysis, despite the abstract's convergence assertion. Separately, the quantile-integral identity in Eq. 8/Prop. B.1 appears mathematically incorrect (the positive part reverses the quantile order), but that is a correctness concern, not a circularity: the identity is not obtained by assuming the conclusion. The arbitrary choice of CVaR with alpha=0.05 is a modeling choice, not a fitted parameter masquerading as a prediction. Therefore, under the circularity criteria, the paper's derivation chain is independent and no specific reduction to its own inputs can be exhibited.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The central claim rests on (i) the correctness of the quantile-integral identity in Eq. 8, (ii) a parametric family expressive enough to hold the true reward distribution, and (iii) a risk measure matching the expert's preferences. The first is false; the second is questionable for multimodal dopamine; the third is arbitrary (CVaR alpha=0.05).

free parameters (6)
  • CVaR risk parameter alpha = 0.05
    Table 6 sets the risk measure CVaR with parameter 0.05; the experts' true risk aversion is not reported and this choice is not justified, yet it shapes the policy objective.
  • Entropy coefficient = 0.1
    Table 6; weighting of H(pi) in Eq. 13; not tuned per domain.
  • Reward regularization = 0.01
    KL weight in Eq. 7; listed in Table 6 without sensitivity analysis.
  • Reward range = [-5,5] (gridworld [0,2], MuJoCo [-10,10])
    Applied via scaled tanh; bounds the learned reward support and affects return distribution shape.
  • Number of quantiles N = 200
    Critic quantile count; Table 6.
  • Skew-normal parameters (mu, sigma, alpha) = learned
    Outputs of neural network; the choice of skew-normal family is a modeling assumption, especially for multimodal dopamine.
assumptions (6)
  • standard math Fubini-Tonelli interchange in Eq. (8) is valid and the quantile-integral identity holds as written
    Used in Prop. B.1; the interchange is valid for positive integrands, but the identity's sign is wrong.
  • standard math Quantile function characterization of FSD (Hadar & Russell equivalence)
    Prop 4.2; standard.
  • domain assumption Distributional Bellman operator contraction
    Invoked in Sec 4.3 to suggest convergence; requires the distributional Bellman operator to be a contraction in the chosen metric, standard for distributional RL but not proved here.
  • domain assumption Expert demonstrations generated by an MDP with stochastic reward and Markov property holds for syllabic behavior
    Sec 5.2 models mouse behavior as MDP over discrete syllables; this is borrowed from prior work.
  • ad hoc to paper The variational family q_phi (skew-normal) contains a good approximation to the true reward posterior
    Needed for the ELBO to recover the reward distribution; contradicted by the paper's own statement that dopamine is multimodal (Sec 5.2), since skew-normal is unimodal.
  • ad hoc to paper Risk measure xi chosen (CVaR with alpha=0.05) reflects the expert's risk preference
    The policy objective Eq. 13 is a DRM over returns; without knowing the expert's distortion, the recovered policy and reward are sensitive to this arbitrary choice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Distributional Inverse Reinforcement Learning." pith.science (2026). https://pith.science/paper/7J4XWRSD

@misc{pith2026251003013,
  author       = {Pith},
  title        = {Pith review of: Distributional Inverse Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7J4XWRSD}},
  note         = {Machine review of arXiv:2510.03013}
}
abstract

We propose a distributional framework for offline Inverse Reinforcement Learning (IRL) that jointly models uncertainty over reward functions and full distributions of returns. Unlike conventional IRL approaches that recover a deterministic reward estimate or match only expected returns, our method captures richer structure in expert behavior, particularly in learning the reward distribution, by minimizing first-order stochastic dominance (FSD) violations and thus integrating distortion risk measures (DRMs) into policy learning, enabling the recovery of both reward distributions and distribution-aware policies. This formulation is well-suited for behavior analysis and risk-aware imitation learning. Theoretical analysis shows that the algorithm converges with $\mathcal{O}(\varepsilon^{-2})$ iteration complexity. Empirical results on synthetic benchmarks, real-world neurobehavioral data, and MuJoCo control tasks demonstrate that our method recovers expressive reward representations and achieves state-of-the-art performance.

Figures

Figures reproduced from arXiv: 2510.03013 by the authors.

Figure 1
Figure 1. Illustration of quantile functions and first-order stochastic dominance (FSD). [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Inferring reward mean and variance in the gridworld example with 10 demonstrations. We begin with a 5 × 5 gridworld environment where the agent is trained to navigate from the starting state (2, 0) (left￾center) to rewarding goal locations. Two high-reward states are placed at (0, 4) (top-right) and (4, 4) (bottom-right), with the top-right reward modeled as a stochastic outcome drawn from N (1, 1). The first column… view at source ↗
Figure 3
Figure 3. Learned reward distribution versus recorded dopamine signals and their empirical CDFs. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Left: Pearson correlation of the reward mean and dopamine level. Right: W-1 loss be￾tween learned distribution and dopamine level. We also quantify the similarity between esti￾mated rewards and actual dopamine distribu￾tions. In Fig. 4A), we report the correlation betw…
Figure 5
Figure 5. Figure 5: Return distributions comparison in HalfCheetah. To further validate the fidelity of our inferred return distributions from DistIRL and compare with the BIRL framework that only matches the mean, we collect 200 trajectories and sample its learned return distribution for…
Figure 6
Figure 6. Figure 6: Reward recovery for state 1 action 7 2 0 2 4 Reward 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Density Reward Distribution State 3, Action 6 True S-DistIRL DistIRL S-BIRL BIRL Det 2 0 2 4 Reward 0.0 0.2 0.4 0.6 0.8 1.0 Cumulative Probability Empirical CDF State 3, Action 6 True S-Dis…
Figure 7
Figure 7. Figure 7: Reward recovery for state 3 action 6 18 [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: Reward recovery for state 4 action 8 4 2 0 2 4 Reward 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Density Reward Distribution State 5, Action 7 True S-DistIRL DistIRL S-BIRL BIRL Det 4 2 0 2 4 Reward 0.0 0.2 0.4 0.6 0.8 1.0 Cumulative Probability Empirical CDF State 5, Action 7 True S…
Figure 9
Figure 9. Figure 9: Reward recovery for state 5 action 7 E LLM USAGE AND REPRODUCIBILITY We use LLM to aid or polish writings only. Research ideation, retrieval and discovery (e.g., finding related work) are conducted by ourselves. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: Reward recovery for state 5 action 9 4 2 0 2 4 Reward 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Density Reward Distribution State 7, Action 8 True S-DistIRL DistIRL S-BIRL BIRL Det 4 2 0 2 4 Reward 0.0 0.2 0.4 0.6 0.8 1.0 Cumulative Probability Empirical CDF State 7, Action 8 True …
Figure 11
Figure 11. Figure 11: Reward recovery for state 7 action 8 2 0 2 Reward 0.0 0.1 0.2 0.3 0.4 0.5 Density Reward Distribution State 9, Action 7 True S-DistIRL DistIRL S-BIRL BIRL Det 2 0 2 Reward 0.0 0.2 0.4 0.6 0.8 1.0 Cumulative Probability Empirical CDF State 9, Action 7 True S-DistIRL Di…
Figure 12
Figure 12. Figure 12: Reward recovery for state 9 action 7 20 [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

67 extracted references · 12 linked inside Pith

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Apprenticeship learning via inverse reinforcement learning

    Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, pp.\ 1, 2004

    2004

  3. [3]

    A survey of inverse reinforcement learning: Challenges, methods and progress

    Saurabh Arora and Prashant Doshi. A survey of inverse reinforcement learning: Challenges, methods and progress. Artificial Intelligence, 297: 0 103500, 2021

    2021

  4. [4]

    Dynamic inverse reinforcement learning for characterizing animal behavior

    Zoe Ashwood, Aditi Jha, and Jonathan W Pillow. Dynamic inverse reinforcement learning for characterizing animal behavior. Advances in neural information processing systems, 35: 0 29663--29676, 2022

    2022

  5. [5]

    The multivariate skew-normal distribution

    Adelchi Azzalini and A Dalla Valle. The multivariate skew-normal distribution. Biometrika, 83 0 (4): 0 715--726, 1996

    1996

  6. [6]

    Walking the values in bayesian inverse reinforcement learning

    Ondrej Bajgar, Alessandro Abate, Konstantinos Gatsis, and Michael A Osborne. Walking the values in bayesian inverse reinforcement learning. arXiv preprint arXiv:2407.10971, 2024

    arXiv 2024

  7. [7]

    A distributional perspective on reinforcement learning

    Marc G Bellemare, Will Dabney, and R \'e mi Munos. A distributional perspective on reinforcement learning. In International conference on machine learning, pp.\ 449--458. PMLR, 2017

    2017

  8. [8]

    Variational inference: A review for statisticians

    David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians. Journal of the American Statistical Association, 112 0 (518): 0 859--877, 2017

    2017

Show all 67 references
  1. [9]

    Scalable bayesian inverse reinforcement learning

    Alex J Chan and Mihaela van der Schaar. Scalable bayesian inverse reinforcement learning. arXiv preprint arXiv:2102.06483, 2021

    2021 arXiv

  2. [10]

    Eliciting risk aversion with inverse reinforcement learning via interactive questioning

    Ziteng Cheng, Anthony Coache, and Sebastian Jaimungal. Eliciting risk aversion with inverse reinforcement learning via interactive questioning. arXiv preprint arXiv:2308.08427, 2023

    2023

  3. [11]

    Map inference for bayesian inverse reinforcement learning

    Jaedeug Choi and Kee-Eung Kim. Map inference for bayesian inverse reinforcement learning. Advances in neural information processing systems, 24, 2011

    2011

  4. [12]

    Implicit quantile networks for distributional reinforcement learning

    Will Dabney, Georg Ostrovski, David Silver, and R \'e mi Munos. Implicit quantile networks for distributional reinforcement learning. In International conference on machine learning, pp.\ 1096--1105. PMLR, 2018 a

    2018

  5. [13]

    Distributional reinforcement learning with quantile regression

    Will Dabney, Mark Rowland, Marc Bellemare, and R \'e mi Munos. Distributional reinforcement learning with quantile regression. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018 b

    2018

  6. [14]

    Cortical substrates for exploratory decisions in humans

    Nathaniel D Daw, John P O'Doherty, Peter Dayan, Ben Seymour, and Raymond J Dolan. Cortical substrates for exploratory decisions in humans. Nature, 441 0 (7095): 0 876--879, 2006

    2006

  7. [15]

    Nonuniform random variate generation

    Luc Devroye. Nonuniform random variate generation. Handbooks in operations research and management science, 13: 0 83--121, 2006

    2006

  8. [16]

    Remarks on quantiles and distortion risk measures

    Jan Dhaene, Alexander Kukush, Dani \"e l Linders, and Qihe Tang. Remarks on quantiles and distortion risk measures. European Actuarial Journal, 2: 0 319--328, 2012

    2012

  9. [17]

    Distributional soft actor-critic: Off-policy reinforcement learning for addressing value estimation errors

    Jingliang Duan, Yang Guan, Shengbo Eben Li, Yangang Ren, Qi Sun, and Bo Cheng. Distributional soft actor-critic: Off-policy reinforcement learning for addressing value estimation errors. IEEE transactions on neural networks and learning systems, 33 0 (11): 0 6584--6598, 2021

    2021

  10. [18]

    D4rl: Datasets for deep data-driven reinforcement learning

    Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020

    2004 arXiv

  11. [19]

    Iq-learn: Inverse soft-q learning for imitation

    Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. Iq-learn: Inverse soft-q learning for imitation. Advances in Neural Information Processing Systems, 34: 0 4028--4039, 2021

    2021

  12. [20]

    Probability: a graduate course, volume 200

    Allan Gut and Allan Gut. Probability: a graduate course, volume 200. Springer, 2006

    2006

  13. [21]

    Rules for ordering uncertain prospects

    Josef Hadar and William R Russell. Rules for ordering uncertain prospects. The American economic review, 59 0 (1): 0 25--34, 1969

    1969

  14. [22]

    Idql: Implicit q-learning as an actor-critic method with diffusion policies

    Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304.10573, 2023

    2023 arXiv

  15. [23]

    Introduction to real analysis, volume 280

    Christopher Heil. Introduction to real analysis, volume 280. Springer, 2019

    2019

  16. [24]

    Generative adversarial imitation learning

    Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016

    2016

  17. [25]

    A bayesian approach to generative adversarial imitation learning

    Wonseok Jeon, Seokin Seo, and Kee-Eung Kim. A bayesian approach to generative adversarial imitation learning. Advances in neural information processing systems, 31, 2018

    2018

  18. [26]

    Rize: Regularized imitation learning via distributional reinforcement learning

    Adib Karimi and Mohammad Mehdi Ebadzadeh. Rize: Regularized imitation learning via distributional reinforcement learning. arXiv preprint arXiv:2502.20089, 2025

    2025

  19. [27]

    Inverse reinforcement learning with switching rewards and history dependency for characterizing animal behaviors

    Jingyang Ke, Feiyang Wu, Jiyi Wang, Jeffrey Markowitz, and Anqi Wu. Inverse reinforcement learning with switching rewards and history dependency for characterizing animal behaviors. arXiv preprint arXiv:2501.12633, 2025

    2025 arXiv

  20. [28]

    Imitation learning via off-policy distribution matching

    Ilya Kostrikov, Ofir Nachum, and Jonathan Tompson. Imitation learning via off-policy distribution matching. arXiv preprint arXiv:1912.05032, 2019

    1912 arXiv

  21. [29]

    Offline reinforcement learning with implicit q-learning

    Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169, 2021

    2021 arXiv

  22. [30]

    Risk-sensitive generative adversarial imitation learning

    Jonathan Lacotte, Mohammad Ghavamzadeh, Yinlam Chow, and Marco Pavone. Risk-sensitive generative adversarial imitation learning. In The 22nd International Conference on Artificial Intelligence and Statistics, pp.\ 2154--2163. PMLR, 2019

    2019

  23. [31]

    A tutorial on energy-based learning

    Yann LeCun, Sumit Chopra, Raia Hadsell, Marc'Aurelio Ranzato, and Fu Jie Huang. A tutorial on energy-based learning. Predicting structured data, 2006

    2006

  24. [32]

    Risk-sensitive mpcs with deep distributional inverse rl for autonomous driving

    Keuntaek Lee, David Isele, Evangelos A Theodorou, and Sangjae Bae. Risk-sensitive mpcs with deep distributional inverse rl for autonomous driving. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp.\ 7635--7642. IEEE, 2022

    2022

  25. [33]

    Nonlinear inverse reinforcement learning with gaussian processes

    Sergey Levine, Zoran Popovic, and Vladlen Koltun. Nonlinear inverse reinforcement learning with gaussian processes. Advances in neural information processing systems, 24, 2011

    2011

  26. [34]

    Internally rewarded reinforcement learning

    Mengdi Li, Xufeng Zhao, Jae Hee Lee, Cornelius Weber, and Stefan Wermter. Internally rewarded reinforcement learning. In International Conference on Machine Learning, pp.\ 20556--20574. PMLR, 2023

    2023

  27. [35]

    Bellman diffusion: Generative modeling as learning a linear operator in the distribution space

    Yangming Li, Chieh-Hsin Lai, Carola-Bibiane Sch \"o nlieb, Yuki Mitsufuji, and Stefano Ermon. Bellman diffusion: Generative modeling as learning a linear operator in the distribution space. arXiv preprint arXiv:2410.01796, 2024

    2024 arXiv

  28. [36]

    Distributional reinforcement learning for risk-sensitive policies

    Shiau Hong Lim and Ilyas Malik. Distributional reinforcement learning for risk-sensitive policies. Advances in Neural Information Processing Systems, 35: 0 30977--30989, 2022

    2022

  29. [37]

    Kernel density bayesian inverse reinforcement learning

    Aishwarya Mandyam, Didong Li, Diana Cai, Andrew Jones, and Barbara E Engelhardt. Kernel density bayesian inverse reinforcement learning. arXiv preprint arXiv:2303.06827, 2023

    2023 arXiv

  30. [38]

    Spontaneous behaviour is structured by reinforcement without explicit reward

    Jeffrey E Markowitz, Winthrop F Gillis, Maya Jay, Jeffrey Wood, Ryley W Harris, Robert Cieszkowski, Rebecca Scott, David Brann, Dorothy Koveal, Tomasz Kula, Caleb Weinreb, Mohammed Abdal Monium Osman, Sandra Romero Pinto, Naoshige Uchida, Scott W Linderman, Bernardo L Sabatini...

    2023

  31. [39]

    Spontaneous behaviour is structured by reinforcement without explicit reward

    Jeffrey E Markowitz, Winthrop F Gillis, Maya Jay, Jeffrey Wood, Ryley W Harris, Robert Cieszkowski, Rebecca Scott, David Brann, Dorothy Koveal, Tomasz Kula, et al. Spontaneous behaviour is structured by reinforcement without explicit reward. Nature, 614 0 (7946): 0 108--117, 2023 b

    2023

  32. [40]

    The kolmogorov-smirnov test for goodness of fit

    Frank J Massey Jr. The kolmogorov-smirnov test for goodness of fit. Journal of the American statistical Association, 46 0 (253): 0 68--78, 1951

    1951

  33. [41]

    Foraging for foundations in decision neuroscience: insights from ethology

    Dean Mobbs, Pete C Trimmer, Daniel T Blumstein, and Peter Dayan. Foraging for foundations in decision neuroscience: insights from ethology. Nature Reviews Neuroscience, 19 0 (6): 0 419--427, 2018

    2018

  34. [42]

    f-irl: Inverse reinforcement learning via state marginal matching

    Tianwei Ni, Harshit Sikchi, Yufei Wang, Tejus Gupta, Lisa Lee, and Ben Eysenbach. f-irl: Inverse reinforcement learning via state marginal matching. In Conference on Robot Learning, pp.\ 529--551. PMLR, 2021

    2021

  35. [43]

    Bayesian inverse reinforcement learning

    Deepak Ramachandran and Eyal Amir. Bayesian inverse reinforcement learning. In IJCAI, volume 7, pp.\ 2586--2591, 2007

    2007

  36. [44]

    Optimization of conditional value-at-risk

    R Tyrrell Rockafellar, Stanislav Uryasev, et al. Optimization of conditional value-at-risk. Journal of risk, 2: 0 21--42, 2000

    2000

  37. [45]

    Driving with style: Inverse reinforcement learning in general-purpose planning for automated driving

    Sascha Rosbach, Vinit James, Simon Gro johann, Silviu Homoceanu, and Stefan Roth. Driving with style: Inverse reinforcement learning in general-purpose planning for automated driving. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp.\ 2658...

    2019

  38. [46]

    Learning risk-aware quadrupedal locomotion using distributional reinforcement learning

    Lukas Schneider, Jonas Frey, Takahiro Miki, and Marco Hutter. Learning risk-aware quadrupedal locomotion using distributional reinforcement learning. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp.\ 11451--11458. IEEE, 2024

    2024

  39. [47]

    A neural substrate of prediction and reward

    Wolfram Schultz, Peter Dayan, and P Read Montague. A neural substrate of prediction and reward. Science, 275 0 (5306): 0 1593--1599, 1997

    1997

  40. [48]

    Distortion risk measures in portfolio optimization

    Ekaterina N Sereda, Efim M Bronshtein, Svetozar T Rachev, Frank J Fabozzi, Wei Sun, and Stoyan V Stoyanov. Distortion risk measures in portfolio optimization. Handbook of portfolio construction, pp.\ 649--673, 2010

    2010

  41. [49]

    Risk-sensitive inverse reinforcement learning via semi-and non-parametric methods

    Sumeet Singh, Jonathan Lacotte, Anirudha Majumdar, and Marco Pavone. Risk-sensitive inverse reinforcement learning via semi-and non-parametric methods. The International Journal of Robotics Research, 37 0 (13-14): 0 1713--1740, 2018

    2018

  42. [50]

    Reinforcement learning: An introduction, volume 1

    Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998

    1998

  43. [51]

    Risk-averse offline reinforcement learning

    N \'u ria Armengol Urp \' , Sebastian Curi, and Andreas Krause. Risk-averse offline reinforcement learning. arXiv preprint arXiv:2102.05371, 2021

    2021 arXiv

  44. [52]

    Inverse reinforcement learning algorithms and features for robot navigation in crowds: an experimental comparison

    Dizan Vasquez, Billy Okal, and Kai O Arras. Inverse reinforcement learning algorithms and features for robot navigation in crowds: an experimental comparison. In 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp.\ 1341--1346. IEEE, 2014

    2014

  45. [53]

    A bayesian approach to robust inverse reinforcement learning

    Ran Wei, Siliang Zeng, Chenliang Li, Alfredo Garcia, Anthony D McDonald, and Mingyi Hong. A bayesian approach to robust inverse reinforcement learning. In Conference on Robot Learning, pp.\ 2304--2322. PMLR, 2023

    2023

  46. [54]

    Foundations of multivariate distributional reinforcement learning

    Harley Wiltzer, Jesse Farebrother, Arthur Gretton, and Mark Rowland. Foundations of multivariate distributional reinforcement learning. Advances in Neural Information Processing Systems, 37: 0 101297--101336, 2024

    2024

  47. [55]

    Inverse reinforcement learning with the average reward criterion

    Feiyang Wu, Jingyang Ke, and Anqi Wu. Inverse reinforcement learning with the average reward criterion. Advances in Neural Information Processing Systems, 36: 0 69117--69129, 2023

    2023

  48. [56]

    Infer and adapt: Bipedal locomotion reward learning from demonstrations via inverse reinforcement learning

    Feiyang Wu, Zhaoyuan Gu, Hanran Wu, Anqi Wu, and Ye Zhao. Infer and adapt: Bipedal locomotion reward learning from demonstrations via inverse reinforcement learning. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp.\ 16243--16250. IEEE, 2024

    2024

  49. [57]

    Efficient sampling-based maximum entropy inverse reinforcement learning with application to autonomous driving

    Zheng Wu, Liting Sun, Wei Zhan, Chenyu Yang, and Masayoshi Tomizuka. Efficient sampling-based maximum entropy inverse reinforcement learning with application to autonomous driving. IEEE Robotics and Automation Letters, 5 0 (4): 0 5355--5362, 2020

    2020

  50. [58]

    Maximum entropy deep inverse reinforcement learning

    Markus Wulfmeier, Peter Ondruska, and Ingmar Posner. Maximum entropy deep inverse reinforcement learning. arXiv preprint arXiv:1507.04888, 2015

    2015 arXiv

  51. [59]

    Modeling, learning, perception, and control methods for deformable object manipulation

    Hang Yin, Anastasia Varava, and Danica Kragic. Modeling, learning, perception, and control methods for deformable object manipulation. Science Robotics, 6 0 (54): 0 eabd8803, 2021

    2021

  52. [60]

    Maximum-likelihood inverse reinforcement learning with finite-time guarantees

    Siliang Zeng, Chenliang Li, Alfredo Garcia, and Mingyi Hong. Maximum-likelihood inverse reinforcement learning with finite-time guarantees. Advances in Neural Information Processing Systems, 35: 0 10122--10135, 2022

    2022

  53. [61]

    When demonstrations meet generative world models: A maximum likelihood framework for offline inverse reinforcement learning

    Siliang Zeng, Chenliang Li, Alfredo Garcia, and Mingyi Hong. When demonstrations meet generative world models: A maximum likelihood framework for offline inverse reinforcement learning. Advances in Neural Information Processing Systems, 36: 0 65531--65565, 2023

    2023

  54. [62]

    From demonstrations to rewards: Alignment without explicit human preferences

    Siliang Zeng, Yao Liu, Huzefa Rangwala, George Karypis, Mingyi Hong, and Rasool Fakoor. From demonstrations to rewards: Alignment without explicit human preferences. arXiv preprint arXiv:2503.13538, 2025

    2025 arXiv

  55. [63]

    Maximum entropy inverse reinforcement learning

    Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pp.\ 1433--1438. Chicago, IL, USA, 2008

    2008

  56. [64]

    Modeling interaction via the principle of maximum causal entropy

    Brian D Ziebart, J Andrew Bagnell, and Anind K Dey. Modeling interaction via the principle of maximum causal entropy. In International conference on machine learning. Carnegie Mellon University, 2010

    2010

  57. [65]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  58. [66]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  59. [67]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.