Pith. sign in

REVIEW 3 major objections 5 minor 49 references

Reward Redistribution for CVaR MDPs using a Bellman Operator on L-infinity

T0 review · 3 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adding the negative part of the budget to the value function makes the static-CVaR Bellman operator a contraction on the full space of bounded functions, with dense per-step rewards.

desk verdict A useful reformulation of static CVaR that gives a contracting bounded Bellman operator, with discretization guarantees that are solid only under non-positive rewards. read the letter →

arxiv 2602.03778 v2 pith:M2DBXXGP submitted 2026-02-03 cs.LG cs.AI

Aneri Muni , Vincent Taboga , Esther Derman , Pierre-Luc Bacon , Erick Delage This is my paper · ORCID
classification cs.LGcs.AI MSC 90C4093E20
keywords staticCVaRconditionalvalue-at-riskMarkovdecisionprocessesstateaugmentationBellmanoperatorcontractionmappingdiscretizationerrorrisk-averseQ-learning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Static conditional value-at-risk in infinite-horizon MDPs has resisted dynamic programming: the standard augmented-state operator has zero as its only fixed point on bounded functions and pays rewards only at the end of the horizon. This paper argues these failures are not intrinsic to CVaR but artifacts of the chosen value-function parameterization. Adding the negative part of the accumulated-return variable to the value function, with per-step reward $z^- - (r+z)^-$, yields an equivalent fixed point whose Bellman operator is a $\gamma$-contraction on all bounded functions and gives informative rewards at every step. Discretizing the budget by rounding down or up produces lower and upper operators with certified performance gaps of order $\gamma(1-\gamma)^{-1} \Delta / \alpha$, and the lower-bound policy provably attains its stated CVaR. The payoff is that value iteration and Q-learning on a finite grid become provably convergent tools for a genuinely tail-sensitive objective.

What carries the argument

The central object is the shifted augmented value function $\bar{v}^*(s,z) = E[-(R+z)^- + z^-]$ and its Bellman operator $\bar{T}$ with augmented reward $\tilde{r}(s,z,a) = z^- - (r(s,a)+z)^-$, where $z^- = \max(0,-z)$ is the negative part of the budget. The shift by $z^-$ converts the terminal CVaR signal into a dense per-step signal, makes the operator a $\gamma$-contraction on $L^\infty$, and forces the fixed point to be constant outside the bounded interval $[-r_{\max}/(1-\gamma), r_{\max}/(1-\gamma)]$, so the continuous budget can be projected onto a compact interval and discretized by lower/upper rounding functions $l,u$. The rounding operators $\bar{T}^l$, $\bar{T}^u$ sandwich the exact operator whenever rewards are non-positi

What would settle it

Compute the true static-CVaR optimum by exhaustive policy enumeration in a tiny MDP (e.g., two states, one positive reward) and compare it to the upper and lower discretized fixed points on a fine grid; if the true optimum escapes the predicted sandwich, or if a non-decreasing value function stops being non-decreasing after one $\bar{T}^l$ update under positive rewards, Theorems 6-7 fail.

Watch

Extended reading notes

Core claim

The paper's central claim: the value function $\bar{v}^*(s,z) = \max_{\pi} E[-(R+z)^- + z^-]$ ($z^- = \max(0,-z)$) is the right object for static CVaR. It is bounded, 1-Lipschitz in $z$, and flat outside $[-r_\gamma, r_\gamma]$ with $r_\gamma = r_{\max}/(1-\gamma)$, unlike the classical $v^*(s,z) = E[-(R+z)^-]$, which is unbounded and whose only bounded fixed point is zero. The induced operator $\bar{T}$, with per-step reward $\tilde{r}(s,z,a) = z^- - (r(s,a)+z)^-$, is a $\gamma$-contraction on $L^\infty$ with unique fixed point $\bar{v}^*$, so value iteration converges from any bounded start. Discretizing $z$ by floor/ceil envelopes yields two contracting operators bracketing $\bar{v}^*$ within $\gamma(1-\gamma)^{-1}\Delta$; CVaR performance is w

Load-bearing premise

The whole sandwich argument rests on Assumption 1, that every reward is non-positive; the order preservation and error bounds are proved only under this sign condition.

Editorial extensions

If this is right

  • Value iteration on the finite grid converges to the fixed point of the rounded operator and returns optimal Q-values for every discrete budget level in a single run (Proposition 9).
  • The upper and lower CVaR estimates bracket the true optimum, and the gap shrinks linearly with grid granularity at rate gamma/(1-gamma) times Delta/alpha (Theorems 6-7).
  • The policy extracted from the lower-bound operator is certified: its true CVaR is at least Psi_l, so it cannot overstate safety (Theorem 7).
  • Q-learning converges with probability one under Robbins-Monro step sizes and infinite exploration of every state-action pair (Theorem 10), with no restrictions on initialization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the same z^- shift should apply to other static risk measures whose dual representation uses the hinge function (eta-X)^+, such as expectiles or spectral risk measures; the proof ingredients are measure-agnostic.
  • The 1/alpha factor in the error bound implies that strongly risk-averse regimes (small alpha) need disproportionately finer grids; one could test this by measuring the empirical bound gap at alpha = 0.05 versus alpha = 0.5.
  • The claim that the analysis adapts to non-negative rewards is not proven; a useful check is to run the bounding operators on a reward-shifted MDP and see whether order preservation and the Delta/alpha bound survive mixed signs.
  • The block update that reuses one transition to update all budget levels is portable: any augmented MDP with deterministic budget dynamics can use the same data-relabeling trick, which is likely where the sample-efficiency gain comes from.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes an augmented-state Bellman operator \bar T for infinite-horizon static CVaR maximization. It defines the bounded value function \bar v*(s,z) = max_π E[-(R(τ)+z)^- + z^-] and proves that \bar T is a γ-contraction on L∞(\bar S) with \bar v* as its unique fixed point (Theorem 4), before developing lower/upper discretized operators based on rounding maps. Under Assumption 1 (non-positive rewards), it derives monotonicity, fixed-point bounds, and performance guarantees (Propositions 5, Theorems 6--7), which are then used to design tabular Q-value iteration and Q-learning algorithms. A stochastic gridworld with only negative rewards validates the methods and shows that the discretization gap closes as the grid is refined.

Significance. If the results hold, the paper makes a useful contribution to static CVaR RL. The core Bellman operator operates on the full space of bounded functions, avoiding the restricted admissible classes of Bäuerle--Ott, and it provides a dense per-step reward signal. The proof of Theorem 4 is detailed and self-contained, and the explicit O(γ(1−γ)^{-1}Δ/α) discretization gap is a concrete, falsifiable prediction. The empirical section confirms the ordering and convergence of the upper/lower approximations. The main limitation is that the approximation guarantees are proved only for non-positive rewards; the claimed adaptation to non-negative rewards is not proved and is load-bearing for the paper's generality.

major comments (3)
  1. [Section 5, footnote 3, Theorems 6-7] The discretization analysis is conditional on Assumption 1. Proposition 5 and Lemma 15 use Lemma 14 with Δ = r(s,a) ≤ 0 to show that h(z)=min(0,z+r)−min(0,z) is non-decreasing. For r ≥ 0 the map is non-increasing; for mixed-sign rewards it is not monotone. The footnote asserting that the results 'can also be adapted to non-negative rewards' gives no proof, and the displayed complement notation is not specified. Since the empirical evaluation is cost-only, it does not exercise the claimed extension. The paper should either provide a full adaptation proof or explicitly restrict the approximation-error claims to Assumption 1.
  2. [Theorem 7 / Appendix B.8, Eq. (8)] The lower-bound policy guarantee in Theorem 7 also depends essentially on Assumption 1. In the proof of the operator \hat T^†, Lemma 14 is applied with Δ = Σ_{t=1}^k γ^{t-1} r(s_t,a_t), and the inequality invoked requires this sum to be non-positive. Thus the performance guarantee for the policy induced by the lower-discretized value function is established only in the non-positive reward regime. The text should be explicit that this is part of the scope of Theorem 7.
  3. [Abstract / Contributions] The paper's title and abstract promise general CVaR MDPs, while the algorithmic approximation results in Sections 5-6 are proved only for r ≤ 0. The core fixed-point theorem is sign-agnostic, but the discretization bounds, the lower/upper envelope ordering, and the policy guarantee are not. This mismatch between advertised scope and proved scope should be corrected, either by proving the extension or by stating the reward-sign restriction as a formal limitation of the approximation results.
minor comments (5)
  1. [Footnote 3] The notation 'lc := u = [lc]c' is unclear and under-specified. Define the complement operation explicitly so the reader can see how lower/upper rounding would be transformed for non-negative rewards.
  2. [Appendix B.8, proof of Theorem 7] In the chain of inequalities, the final expression '≤ Ψ* + α \barΔ_u/α' contains a redundant α; it should be '≤ Ψ* + \barΔ_u/α'.
  3. [Algorithm 2] The block update over all \tilde z ∈ Z_Δ is a useful data-relabeling idea, but the pseudo-code mixes two roles of z: the environmental budget z_k updated in Line 16 and the auxiliary \tilde z used only for the update. Add a sentence distinguishing these roles to avoid confusion.
  4. [Figure 4] The caption calls the 10k-bin result the 'true value'; since this is still a finite discretization, it should be described as a high-resolution numerical estimate rather than the exact continuous optimal value.
  5. [Section 7] Minor grammatical issue: 'the corresponding the optimal starting budget' should read 'the corresponding optimal starting budget'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the Bellman operator fixed point is independently defined and verified; discretization bounds are proved from stated assumptions, with only a non-circular missing-proof limitation under Assumption 1.

full rationale

The central chain is self-contained rather than circular. The paper defines the value function \bar v*(s,z) := max_{\pi} E[-(R(\tau)+z)^- + z^-] directly from the static CVaR objective, after the convex dual reformulation (Proposition 1, Eq. 3), and only then defines the operator \bar T with reward \tilde r(s,z,a)=z^- - (r(s,a)+z)^- (Eq. 4). Theorem 4 verifies, rather than assumes, that \bar T \bar v* = \bar v*, using the telescoping identity min(0,R(\tau)+z)-min(0,z), and proves \gamma-contraction on L^\infty(\bar S) from the discount factor. Thus the fixed point is not installed by construction; it is a checked consequence of the reward-redistribution identity. The discretization results (Propositions 5, 8; Theorems 6, 7) are likewise proved from contraction, 1-Lipschitzness of \bar v*, and the projection/rounding maps; no fitted parameter is later renamed as a prediction. The external citations [36] and [6] supply the standard CVaR dual form and the augmented-state setup, but the paper's load-bearing contraction and error-bound arguments do not reduce to those citations. The self-references in the reference list ([20], [30]) are not used as load-bearing evidence. One limitation is explicitly present and should be weighed as a correctness/completeness concern, not as circularity: Assumption 1 restricts rewards to r <= 0, and footnote 3 asserts without proof that the discretization results adapt to non-negative rewards ('Specifically, Propositions 5 to 7 extend to the non-negative rewards using \bar T^e \bar v(s,z) := ...'). The proof of Proposition 5 in Appendix B.6 indeed relies on the concavity of min and on r <= 0 to make z -> min(0,r+z)-min(0,z) non-decreasing, and the empirical section uses a cost-only gridworld, so the claimed extension is not exercised. However, the stated theorems are explicitly conditional on Assumption 1, and the unproved extension does not feed the conclusion back into the hypothesis. Consequently, no circular step is established under the quoted-evidence standard.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces the transformed reward \tilde r=z^- - (r+z)^- and the operator \bar T, but these are functions, not new physical entities. No fitted parameters appear; hyperparameters in experiments are standard. The main extra assumption is non-positive rewards, which is explicitly used in the discretization bounds.

assumptions (4)
  • standard math The convex dual representation CVaR_alpha[X]=sup_eta{eta - (1/alpha)E[(eta-X)_+]} (Eq. 1, from Rockafellar-Uryasev [36]) holds for bounded returns.
    Background theorem used in Proposition 1; not proved in the paper.
  • domain assumption The augmented state (s,z) with deterministic update z'=(r+z)/gamma is a sufficient statistic for the inner CVaR maximization, and the max over history-dependent policies can be solved by stationary policies in the augmented MDP (Bauerle & Ott [6]).
    Central setup is inherited from [6].
  • ad hoc to paper Assumption 1: r(s,a) <= 0 for all (s,a).
    Used in Proposition 5 and Theorems 6-7 for monotonicity and bound order; not required for Theorem 4 contraction.
  • domain assumption For Q-learning convergence: Robbins-Monro step sizes and infinite visitation of every nominal (s,a) (Theorem 10, conditions (1)-(2)).
    Standard stochastic approximation conditions; not verified in the practical beta schedule.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reward Redistribution for CVaR MDPs using a Bellman Operator on L-infinity." pith.science (2026). https://pith.science/paper/M2DBXXGP

@misc{pith2026260203778,
  author       = {Pith},
  title        = {Pith review of: Reward Redistribution for CVaR MDPs using a Bellman Operator on L-infinity},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M2DBXXGP}},
  note         = {Machine review of arXiv:2602.03778}
}
read the original abstract

Tail-end risk measures such as static conditional value-at-risk (CVaR) are used in safety-critical applications to prevent rare, yet catastrophic events. Unlike risk-neutral objectives, the static CVaR of the return depends on entire trajectories without admitting a recursive Bellman decomposition in the underlying Markov decision process. A classical resolution relies on state augmentation with a continuous variable. However, unless restricted to a specialized class of admissible value functions, this formulation induces sparse rewards and degenerate fixed points. In this work, we propose a novel formulation of the static CVaR objective based on augmentation. Our alternative approach leads to a Bellman operator with: (1) dense per-step rewards; (2) contracting properties on the full space of bounded value functions. Building on this theoretical foundation, we develop risk-averse value iteration and model-free Q-learning algorithms that rely on discretized augmented states. We further provide convergence guarantees and approximation error bounds due to discretization. Empirical results demonstrate that our algorithms successfully learn CVaR-sensitive policies and achieve effective performance-safety trade-offs.

Figures

Figures reproduced from arXiv: 2602.03778 by the authors.

Figure 1
Figure 1. Illustration of the reward schemes produced along a trajectory under the augmented MDP of [6] [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Stochastic Gridworld Environment. We validate the proposed static CVaR algorithms on a stochastic gridworld, where a robot has to traverse a 2D terrain map ( [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Performance comparison between the proposed static CVaR Q-learning and Q-VI algorithms for [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Comparing the effect of augmented state discretization resolution on the accuracy of CVaR perfor [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]
Figure 5
Figure 5. Figure 5: Illustration of the reward schemes produced along a trajectory under the augmented MDP of [6] [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

49 extracted references · 9 linked inside Pith

  1. [1]

    Andrychowicz, F

    M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, O. Pieter Abbeel, and W. Zaremba. Hindsight experience replay.Advances in neural information processing systems, 30, 2017

    2017

  2. [2]

    Predictive CVar q-learning

    Anonymous. Predictive CVar q-learning. InSubmitted to The Fourteenth International Conference on Learning Representations, 2025. under review

    2025

  3. [3]

    Artzner, F

    P. Artzner, F. Delbaen, J.-M. Eber, and D. Heath. Coherent measures of risk.Mathematical Finance, 9(3):203–228, 1999

    1999

  4. [4]

    Bastani, J

    O. Bastani, J. Y. Ma, E. Shen, and W. Xu. Regret bounds for risk-sensitive reinforcement learning.Advances in Neural Information Processing Systems, 35:36259–36269, 2022

    2022

  5. [5]

    B¨ auerle, T

    N. B¨ auerle, T. G¨ oll, and A. Ja´ skiewicz. Yet another distributional bellman equation.arXiv preprint arXiv:2505.21098, 2025

    arXiv 2025

  6. [6]

    B¨ auerle and J

    N. B¨ auerle and J. Ott. Markov decision processes with average-value-at-risk criteria.Math- ematical Methods of Operations Research, 74(3):361–379, 2011. 13

    2011

  7. [7]

    M. G. Bellemare, W. Dabney, and M. Rowland.Distributional Reinforcement Learning. The MIT Press, 05 2023

    2023

  8. [8]

    D. P. Bertsekas.Dynamic Programming and Optimal Control. Athena Scientific, 1st edition, 1995

    1995

Show all 49 references
  1. [9]

    D. P. Bertsekas and J. N. Tsitsiklis.Neuro-Dynamic Programming. Athena Scientific, 1996

    1996

  2. [10]

    L. Bisi, D. Santambrogio, F. Sandrelli, A. Tirinzoni, B. D. Ziebart, and M. Restelli. Risk-averse policy optimization via risk-neutral policy optimization.Artificial Intelligence, 311:103765, 2022

    2022

  3. [11]

    Bodnar, A

    C. Bodnar, A. Li, K. Hausman, P. Pastor, and M. Kalakrishnan. Quantile qt-opt for risk-aware vision-based robotic grasping.arXiv preprint arXiv:1910.02787, 2019

    1910 arXiv

  4. [12]

    Y. Chow, A. Tamar, S. Mannor, and M. Pavone. Risk-sensitive and robust decision-making: a cvar optimization approach.Advances in neural information processing systems, 28, 2015

    2015

  5. [13]

    Coache and S

    A. Coache and S. Jaimungal. Reinforcement learning with dynamic convex risk measures. Mathematical Finance, 34(2):557–587, 2024

    2024

  6. [14]

    Dabney, G

    W. Dabney, G. Ostrovski, D. Silver, and R. Munos. Implicit quantile networks for distri- butional reinforcement learning. InInternational conference on machine learning, pages 1096–1105. PMLR, 2018

    2018

  7. [15]

    Dabney, M

    W. Dabney, M. Rowland, M. Bellemare, and R. Munos. Distributional reinforcement learn- ing with quantile regression. InProceedings of the AAAI conference on artificial intelligence, volume 32, 2018

    2018

  8. [16]

    Eysenbach, X

    B. Eysenbach, X. Geng, S. Levine, and R. R. Salakhutdinov. Rewriting history with inverse rl: Hindsight inference for policy improvement.Advances in neural information processing systems, 33:14783–14795, 2020

    2020

  9. [17]

    Godbout and A

    M. Godbout and A. Durand. On the fundamental limitations of dual static cvar decompo- sitions in markov decision processes.arXiv preprint arXiv:2507.14005, 2025

    2025 arXiv

  10. [18]

    Greenberg, Y

    I. Greenberg, Y. Chow, M. Ghavamzadeh, and S. Mannor. Efficient risk-averse reinforce- ment learning. InProceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA, 2022. Curran Associates Inc

    2022

  11. [19]

    W. B. Haskell and R. Jain. A convex analytic approach to risk-aware markov decision processes.SIAM J. Control. Optim., 53:1569–1598, 2015

    2015

  12. [20]

    J. L. Hau, E. Delage, E. Derman, M. Ghavamzadeh, and M. Petrik. Q-learning for Quantile MDPs: A Decomposition, Performance, and Convergence Analysis, Oct. 2024. arXiv:2410.24128 [cs]

    2024 arXiv

  13. [21]

    J. L. Hau, E. Delage, M. Ghavamzadeh, and M. Petrik. On dynamic programming de- compositions of static risk measures in markov decision processes.Advances in Neural Information Processing Systems, 36:51734–51757, 2023

    2023

  14. [22]

    R. L. Karandikar and M. Vidyasagar. Convergence of batch asynchronous stochastic ap- proximation with applications to reinforcement learning.arXiv preprint arXiv:2109.03445, 2021. 14

    2021 arXiv

  15. [23]

    Keramati, C

    R. Keramati, C. Dann, A. Tamkin, and E. Brunskill. Being optimistic to be conserva- tive: Quickly learning a cvar policy. InProceedings of the AAAI conference on artificial intelligence, volume 34, pages 4436–4443, 2020

    2020

  16. [24]

    Kim and S

    J.-H. Kim and S. Min. Risk-sensitive policy optimization via predictive cvar policy gradient. InForty-first International Conference on Machine Learning, 2024

    2024

  17. [25]

    S. H. Lim and I. Malik. Distributional reinforcement learning for risk-sensitive policies. Advances in Neural Information Processing Systems, 35:30977–30989, 2022

    2022

  18. [26]

    Majumdar and M

    A. Majumdar and M. Pavone. How should a robot assess risk? towards an axiomatic theory of risk in robotics. In N. M. Amato, G. Hager, S. Thomas, and M. Torres-Torriti, editors, Robotics Research, pages 75–84, Cham, 2020. Springer International Publishing

    2020

  19. [27]

    H. Mead, C. Costen, B. Lacerda, and N. Hawes. Return capping: Sample-efficient cvar policy gradient optimisation.arXiv preprint arXiv:2504.20887, 2025

    2025 arXiv

  20. [28]

    M. Minsky. Steps toward artificial intelligence.Proceedings of the IRE, 49(1):8–30, 2007

    2007

  21. [29]

    Moghimi and H

    M. Moghimi and H. Ku. Beyond cvar: Leveraging static spectral risk measures for enhanced decision-making in distributional reinforcement learning, 2025

    2025

  22. [30]

    A. Muni, E. Derman, V. Taboga, P.-L. Bacon, and E. Delage. What matters when modeling human behavior using imitation learning? In2nd Workshop on Models of Human Feedback for AI Alignment, 2025

    2025

  23. [31]

    X. Ni, G. Liu, and L. Lai. Risk-sensitive reward-free reinforcement learning with cvar. In Forty-first International Conference on Machine Learning, 2024

    2024

  24. [32]

    G. C. Pflug and A. Pichler. Time-consistent decisions and temporal decomposition of coherent risk functionals.Math. Oper. Res., 41:682–699, 2016

    2016

  25. [33]

    Pignatelli, J

    E. Pignatelli, J. Ferret, M. Geist, T. Mesnard, H. van Hasselt, O. Pietquin, and L. Toni. A survey of temporal credit assignment in deep reinforcement learning.arXiv preprint arXiv:2312.01072, 2023

    2023 arXiv

  26. [34]

    B. ´A. Pires, M. Rowland, D. Borsa, Z. D. Guo, K. Khetarpal, A. Barreto, D. Abel, R. Munos, and W. Dabney. Optimizing return distributions with distributional dynamic programming.arXiv preprint arXiv:2501.13028, 2025

    2025 arXiv

  27. [35]

    M. L. Puterman.Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., USA, 1st edition, 1994

    1994

  28. [36]

    R. T. Rockafellar, S. Uryasev, et al. Optimization of conditional value-at-risk.Journal of risk, 2:21–42, 2000

    2000

  29. [37]

    Ruszczy´ nski

    A. Ruszczy´ nski. Risk-averse dynamic programming for markov decision processes.Mathe- matical programming, 125(2):235–261, 2010

    2010

  30. [38]

    Schneider, J

    L. Schneider, J. Frey, T. Miki, and M. Hutter. Learning risk-aware quadrupedal locomotion using distributional reinforcement learning. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 11451–11458. IEEE, 2024

    2024

  31. [39]

    A. Shapiro. On a time consistency concept in risk averse multistage stochastic program- ming.Operations Research Letters, 37(3):143–147, 2009. 15

    2009

  32. [40]

    Stanko and K

    S. Stanko and K. Macek. Risk-averse distributional reinforcement learning: A cvar opti- mization approach. InIJCCI, pages 412–423, 2019

    2019

  33. [41]

    Tamar, Y

    A. Tamar, Y. Chow, M. Ghavamzadeh, and S. Mannor. Policy gradient for coherent risk measures.Advances in neural information processing systems, 28, 2015

    2015

  34. [42]

    Tamar, Y

    A. Tamar, Y. Glassner, and S. Mannor. Optimizing the cvar via sampling. InProceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015

    2015

  35. [43]

    Y. C. Tang, J. Zhang, and R. Salakhutdinov. Worst cases policy gradients.arXiv preprint arXiv:1911.03618, 2019

    1911 arXiv

  36. [44]

    J. N. Tsitsiklis. Asynchronous stochastic approximation and q-learning.Mach. Learn., 16(3):185–202, Sept. 1994

    1994

  37. [45]

    K. Wang, N. Kallus, and W. Sun. Near-minimax-optimal risk-sensitive reinforcement learn- ing with cvar. InInternational Conference on Machine Learning, pages 35864–35907. PMLR, 2023

    2023

  38. [46]

    K. Wang, D. Liang, N. Kallus, and W. Sun. A reductions approach to risk-sensitive rein- forcement learning with optimized certainty equivalents.arXiv preprint arXiv:2403.06323, 2024

    2024 arXiv

  39. [47]

    Wang and M

    Y. Wang and M. P. Chapman. Risk-averse autonomous systems: A brief history and recent developments from the perspective of optimal control.Artificial Intelligence, 311:103743, 2022

    2022

  40. [48]

    − 0−1X t=0 γtr(st, at) +z ! − +z − # =−z − +z − = ˆv† 0(s, z). 29 We follow with atk+ 1: ˆv† k+1(s, z) =E τ∼P ¯π(·,z∗ l ) s

    C. J. C. H. Watkins and P. Dayan. Technical note: q -learning.Mach. Learn., 8(3–4):279–292, May 1992. 16 A Illustration of Various Reward Schemes Nominal MDP s1 s2 s3 s4 s4+1 −3 +1 0 0 Augmented MDP (B¨ auerle & Ott, 2011) s1 0 s2 2 s3 −2 s4 −2 s4 . . . 0 0 0 0 0 Transformed A...

    1992

  41. [49]

    These are used to calculate the empirical return distribution, the empirical CVaR performance as well as analyzing the number of times the robot enters the crater per episode

    We run the policy for eachαand collect 10k trajectories. These are used to calculate the empirical return distribution, the empirical CVaR performance as well as analyzing the number of times the robot enters the crater per episode. D.4 Static CVaR Q-learning The static CVaR Q...

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.