Pith. sign in

REVIEW 3 major objections 6 minor 43 references

Optimistic Exploration for Risk-Averse Constrained Reinforcement Learning

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Risk-averse constrained RL agents can escape sub-optimal policies by exploring optimistically toward high-reward, low-worst-case-cost actions without relaxing safety constraints.

desk verdict A solid, well-motivated combination of optimistic exploration and risk-averse constrained RL, but the failure to isolate optimism direction from raw exploration magnitude leaves the core claim under-supported. read the letter →

arxiv 2507.08793 v2 pith:NUGHDQOT submitted 2025-07-11 cs.LG cs.AI

James McCarthy , Radu Marinescu , Elizabeth Daly , Ivana Dusparic This is my paper
classification cs.LGcs.AI
keywords risk-averseconstrainedreinforcementlearningoptimisticexplorationconditionalvalueatriskdistributionalactor-criticsafeMarkovdecisionprocessepistemicuncertainty
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Risk-averse constrained reinforcement learning typically explores too cautiously, so agents converge to sub-optimal policies that leave reward on the table or miss the goal entirely. This paper proposes ORAC, an exploration strategy that biases every sampled action toward a local upper confidence bound of reward while keeping a local lower confidence bound of the risk-averse cost below the safety threshold. The result, if correct, is that risk aversion no longer has to come at the price of getting stuck: the agent discovers high-reward regions without abandoning worst-case safety. Experiments on a risky gridworld, Safety-Gymnasium, and the CityLearn building-energy simulator show ORAC reaching the correct risk-averse path more often and improving the reward-cost trade-off over the WCSAC baseline.

What carries the argument

The load-bearing object is the local optimistic exploratory action distribution $\pi_E(a|s) = \mathcal{N}(\mu_E, \sigma_T)$, built by moving the target policy's mean $\mu_T$ along the gradient $\nabla_a[\hat{Q}_R(s,a) - \bar{\lambda} \hat{Q}_{C,\alpha}(s,a)]$ evaluated at $a = \mu_T$, with step size $\delta$ and a KL-bound update. Here $\hat{Q}_R$ is a mean-plus-$\beta_R$ standard-deviation bound over two reward critics, and $\hat{Q}_{C,\alpha}$ is a CVaR-style expectation over per-quantile lower confidence bounds $\mu^k_{q_C} - \beta_C \sigma^k_{q_C}$ formed from an ensemble of $E$ distributional cost critics. The adaptive weight $\bar{\lambda} = \max(\lambda - (\bar{c} - \hat{Q}_{C,\alpha}), 0)$ strengthens or weakens the cost term depending on whether the optimistic cost estimate clears the constraint. This machinery converts epistemic uncertainty about reward and worst-case cost into a concrete action bias, so optimism enters through exploration rather than through relaxing the risk constraint.

What would settle it

Run ORAC in GuardedMaze with guard probability 0.1 over 20 seeds: if its long-path success rate is not clearly above WCSAC's 30% baseline, the central claim that optimistic exploration prevents getting stuck is refuted; equivalently, setting $\beta_R = \beta_C = 0$ should collapse ORAC's behaviour back to WCSAC's, showing the bounds themselves do the work.

Watch

Extended reading notes

Core claim

ORAC's central claim is that conservative exploration, not risk aversion itself, is what makes risk-averse constrained agents converge to sub-optimal policies, and that this conservatism can be corrected without relaxing safety constraints. The method constructs a separate exploratory policy at each step by shifting the mean of the actor's Gaussian action distribution along the gradient of an optimistic objective: the upper confidence bound of the reward value function $\hat{Q}_R(s,a) = \mu_{Q_R}(s,a) + \beta_R \sigma_{Q_R}(s,a)$ minus the Lagrangian-weighted optimistic lower confidence bound of the CVaR cost $\hat{Q}_{C,\alpha}$, whose per-quantile lower bounds come from an ensemble of distributional cost critics. The Lagrangian weight is adjusted upward or downward whenever the optimistic cost bound exceeds or falls below the safety threshold, so the shift favours reward when the constraint is comfortably satisfied. The paper argues this lets the agent explore uncertain regions that promise high reward while still steering around worst-case cost violations, and presents evidence that ORAC avoids the sub-optimal traps that WCSAC falls into.

Load-bearing premise

The load-bearing premise is that the disagreement among the cost-critic ensemble and between the two reward critics is a calibrated measure of epistemic uncertainty, so the local gradient step with fixed $\beta_R$ and $\beta_C$ moves exploration toward regions that are genuinely better and still safe.

Editorial extensions

If this is right

  • In the GuardedMaze gridworld, ORAC converges to the correct risk-averse long path in 100% of runs at guard probability 0.15 and 80% at 0.1, versus 60% and 30% for WCSAC, and it converges in fewer steps.
  • In Safety-Gymnasium PointGoal1, ORAC achieves higher episodic reward than WCSAC while keeping CVaR costs at or below threshold at both CVaR-0.5 and CVaR-0.25 risk levels.
  • In PointButton1, ORAC satisfies the CVaR-0.5 cost constraint that WCSAC marginally violates, while at CVaR-0.25 it achieves higher reward at comparable cost.
  • In CityLearn, ORAC raises mean episodic reward well above WCSAC while staying under the CVaR-0.05 cost threshold, and it cuts outage temperature violations below both baselines.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Our inference: if ensemble disagreement is a calibrated uncertainty signal, the same local mean-shift rule could be applied to other coherent risk measures, such as spectral risk, and to multi-constraint CMDPs by maintaining one lower bound per cost.
  • Our inference: the adaptive $\bar{\lambda}$ rule suggests a testable extension in which $\beta_R$ and $\beta_C$ are also made adaptive, scaling optimism with observed constraint slack; the paper's own ablations show performance is sensitive to these hand-set values.
  • Our inference: ORAC's benefit should be most visible when the optimistic action is one gradient step from the current mean; in settings with highly multi-modal value functions, the local update may need to be replaced by a multi-step or model-based search.
  • Our inference: because the paper concedes ORAC does not guarantee safe exploration, a natural deployment-level extension is to combine ORAC's exploration with a safety shield that blocks constraint-violating actions during training.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes ORAC (Optimistic Risk-averse Actor Critic), an exploration mechanism for risk-averse constrained reinforcement learning built on top of WCSAC. At each environment step, ORAC constructs an exploratory policy by shifting the mean of the target policy along the gradient of a locally optimistic upper confidence bound of the reward value function minus a Lagrangian-weighted locally optimistic lower confidence bound of the CVaR cost value function. The adaptive Lagrangian weight is increased or decreased depending on whether the lower confidence bound of cost is below or above the safety threshold. The paper evaluates ORAC against WCSAC and SAC-Lag in a GuardedMaze gridworld, Safety-Gymnasium PointGoal1 and PointButton1, and CityLearn. The reported results show that ORAC prevents the risk-averse agent from getting stuck in sub-optimal policies in GuardedMaze (100% vs 60% success at guard probability 0.15, 80% vs 30% at 0.1), and yields modest reward or cost improvements in continuous control and CityLearn, though several differences are within one standard deviation and the PointButton1 CVaR-0.25 setting violates the cost threshold for both ORAC and WCSAC. The paper includes hyperparameter ablations, ensemble-size ablations, and a comparison of IQN versus linear spectral-risk approximations of CVaR.

Significance. If the central claim holds, ORAC addresses a real problem in risk-averse constrained RL: conservative exploration caused by risk aversion can lead to sub-optimal policies, and optimistic exploration guided by both reward and cost confidence bounds is a principled, computationally cheap remedy. The GuardedMaze results are striking and clearly demonstrate that the additional exploration helps avoid a known failure mode. The paper is transparent: it provides algorithm pseudocode, environment-specific hyperparameters, ablations of the introduced βR, βC, and ensemble size, and a discussion of limitations including the concession that ORAC does not guarantee safe exploration. This transparency is a strength. However, the evidence does not currently isolate the optimism direction as the cause of the improvements, and several Safety-Gymnasium trade-off improvements are within noise; these gaps limit the strength of the paper's central attribution claim, although the underlying method remains plausible and useful. The contribution is appropriate for the RL/constrained-RL community if the attribution gap is addressed.

major comments (3)
  1. [Section 4.4, Eq. (18) and Tables 1–2] The experiments compare ORAC against WCSAC, which performs no explicit mean-shifting exploration. ORAC therefore differs from WCSAC in two coupled ways: it injects a larger directed perturbation (δ = 4, βR = 3–4, βC = 1–2) and it gives that perturbation an optimistic UCB direction. Because the paper reports no control with the same KL budget, the same δ schedule, and the same β magnitudes but a random or otherwise undirected perturbation direction, the GuardedMaze success rates (100% vs 60% and 80% vs 30%) and the Safety-Gymnasium differentials cannot be attributed specifically to optimism rather than to undirected extra exploration. This is load-bearing for the paper's central claim. A matched random-direction ablation, or an ablation using only the reward-gradient direction without the cost branch, is needed.
  2. [Section 5.3, Table 2] Several claims of improved reward-cost trade-off are not statistically supported. In PointGoal1, ORAC-0.5 reward (34.2 ± 1.2) overlaps WCSAC-0.5 (33.8 ± 0.5) and the cost metrics are essentially identical. In PointButton1 at α = 0.5, ORAC's reward (13.9 ± 2.7) is lower than WCSAC's (14.2 ± 3.5), although its CVaR is better. At α = 0.25, ORAC's CVaR0.25 (12.7 ± 4.7) exceeds the cost threshold of 10, as does WCSAC's (11.4 ± 2.8). The text states that ORAC 'satisfies the constraints at the 0.5 risk-level' but 'marginally violates' at 0.25; this needs error-bar-aware analysis or significance testing before claiming a significantly improved trade-off.
  3. [Section 6 and Appendix D] The paper concedes that ORAC 'does not guarantee safe exploration' and relies on hand-selected βR, βC, and δ values per environment. The ablations in Tables 7–9 show sensitivity to these choices: for example, in PointButton1 at CVaR 0.5, βR = 5 raises the CVaR0.5 cost to 13.5 ± 1.8 compared with 9.6 ± 3.5 at βR = 4, and ensemble size materially changes results. Since the cost branch's safety benefit depends on the ensemble spread being a calibrated measure of epistemic uncertainty, the paper should either provide a calibration check or explicitly restrict the claims to the tuned hyperparameter settings. As written, the abstract's generality claim ('whilst still satisfying safety constraints') is stronger than what Section 6 and the ablations support.
minor comments (6)
  1. [Eq. (19)] The displayed formula for λ̄ contains an unmatched bracket: λ̄ = λ − (c̄ − Q̂π_{C,α}(s,a)] should close with a parenthesis, not a bracket. The clipping operation should also be stated unambiguously, e.g., λ̄ = max(0, λ − (c̄ − Q̂π_{C,α}(s,a))).
  2. [Section 3.2] In the definition of Zπ, the subscript 'ss+1' should be 's_{t+1}'; the same typo occurs in the background discussion of discounted cost returns.
  3. [Before Eq. (13)] The text contains a stray 'fwhere' at the start of the sentence introducing Eq. (13); this should be corrected.
  4. [Tables 2 and 3] The column headers 'CVaR 0 .5' and 'CVaR 0 .25' have awkward spacing; use 'CVaR-0.5' and 'CVaR-0.25'. In Table 3, 'CVAR0.05' should be 'CVaR0.05' for consistency.
  5. [Section 4.1 and Table 1] The text says 'under the 0.1% at just 30%' when it means a guard probability of 0.1 (i.e., 10%), not 0.1%; this is confusing and should be reworded.
  6. [Section 5.1] The paper should state more prominently that the Safety-Gymnasium and CityLearn WCSAC baseline uses the linearized spectral-risk approximation from Kim et al. [21], not the original IQN-based WCSAC from Yang et al. [34]; the change is disclosed but the abstract and introduction still refer to 'WCSAC' without this qualification.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ORAC's construction uses learned critics as a standard optimism heuristic, and its central claims are validated by external empirical comparisons rather than by definition.

full rationale

The paper's derivation chain is self-contained in the relevant sense. The exploration policy is constructed from the learned reward and cost critics (Eqs. 14-18), but this is a mechanism, not a prediction that reduces to its inputs. The central claims—that ORAC prevents convergence to sub-optimal policies in GuardedMaze and improves the reward-cost trade-off in Safety-Gymnasium and CityLearn—are empirical results obtained by comparing against WCSAC and SAC-Lag, which are external baselines, not quantities fitted from the same data. The confidence-bound parameters βR, βC, and δ are disclosed hyperparameters with ablations (Appendix D), so they are not fitted predictions renamed as results. The paper explicitly extends WCSAC with the linear risk approximation of Kim et al. [21] for both WCSAC and ORAC in the complex environments, and Appendix C shows that the original IQN-based WCSAC is worse, so the comparison isolates the exploration mechanism rather than importing an unfair baseline. There is no load-bearing self-citation: the inspirations are Ciosek et al. [8] and Yang et al. [34], which are not the present authors' own work, and no uniqueness theorem is invoked. The disclosed environment modification (the +1 bonus in GuardedMaze) and the concession that ORAC 'does not guarantee safe exploration' are relevant to correctness or experimental design, but they do not make any equation or result equivalent to its inputs by construction. Any concern about a missing random-direction control is an empirical confound, not a circularity, and per the review rules such concerns belong under correctness risk rather than circularity score.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The central experimental claim rests on standard CMDP and CVaR modeling, on quantile critics approximating the cost-return distribution, and on ensemble disagreement being a usable uncertainty signal for exploration. The ensemble-as-uncertainty premise is not independently validated, and the adaptive lambda rule and fixed beta/delta values are introduced ad hoc to make the algorithm work.

free parameters (4)
  • beta_R (reward optimism bound) = 3.0 in GuardedMaze, 4.0 in Safety Gym and CityLearn
    Controls the magnitude of the reward upper-confidence-bound in Eq. 14; Appendix D ablation selects 4.0 as the best constraint-satisfying value in PointButton1.
  • beta_C (cost optimism bound) = 2.0 in GuardedMaze, 1.0 in Safety Gym and CityLearn
    Controls the cost lower-confidence-bound magnitude in Eq. 15; ablation shows 1.0 gives the best constraint satisfaction while lower and higher values increase cost.
  • delta (exploration step bound) = 4.0, linearly decayed to 0
    Bounds the shift from the target action mean to the exploratory mean in Eq. 18; it is not ablated.
  • Cost critic ensemble size E = 2 in GuardedMaze, 5 in Safety Gym and CityLearn
    Determines the epistemic uncertainty estimate in Eq. 15; Appendix D.3 shows performance varies with E and the main choice is not uniformly best.
assumptions (6)
  • domain assumption CMDP with expected discounted reward/cost and CVaR constraint (Eq. 6) is the correct problem formulation.
    Safety is encoded as R_sigma(Z^pi) <= c, and all algorithms optimize this Lagrangian objective.
  • domain assumption Primal-dual Lagrangian optimization (Eq. 7) converges to a policy satisfying the risk constraint.
    WCSAC and ORAC inherit this from Yang et al.; no convergence proof is provided for ORAC.
  • domain assumption Quantile critics with IQN or spectral-risk linearization approximate the true cost-return distribution.
    Needed to compute Q^pi_{C,alpha}; relies on distributional RL literature rather than new evidence.
  • domain assumption Ensemble standard deviation across E quantile critics is a valid epistemic uncertainty signal for cost, and the two reward critics' standard deviation is valid for reward.
    Core to Eqs. 14-15; no calibration analysis is provided.
  • domain assumption Local gradient of the optimistic objective at mu_T, with KL-bound delta, gives a safe and useful exploration direction.
    Inherited from Ciosek et al. for reward-only RL; assumed to transfer to the risk-averse constrained setting.
  • ad hoc to paper Adaptive Lagrangian weighting bar_lambda = lambda - (bar_c - Qhat_{C,alpha}) correctly balances reward and cost at each exploration step.
    Introduced in Eq. 19 and clipped at 0; no theoretical justification beyond intuition.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Optimistic Exploration for Risk-Averse Constrained Reinforcement Learning." pith.science (2026). https://pith.science/paper/NUGHDQOT

@misc{pith2026250708793,
  author       = {Pith},
  title        = {Pith review of: Optimistic Exploration for Risk-Averse Constrained Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NUGHDQOT}},
  note         = {Machine review of arXiv:2507.08793}
}
read the original abstract

Risk-averse Constrained Reinforcement Learning (RaCRL) aims to learn policies that minimise the likelihood of rare and catastrophic constraint violations caused by an environment's inherent randomness. In general, risk-aversion leads to conservative exploration of the environment which typically results in converging to sub-optimal policies that fail to adequately maximise reward or, in some cases, fail to achieve the goal. In this paper, we propose an exploration-based approach for RaCRL called Optimistic Risk-averse Actor Critic (ORAC), which constructs an exploratory policy by maximising a local upper confidence bound of the state-action reward value function whilst minimising a local lower confidence bound of the risk-averse state-action cost value function. Specifically, at each step, the weighting assigned to the cost value is increased or decreased if it exceeds or falls below the safety constraint value. This way the policy is encouraged to explore uncertain regions of the environment to discover high reward states whilst still satisfying the safety constraints. Our experimental results demonstrate that the ORAC approach prevents convergence to sub-optimal policies and improves significantly the reward-cost trade-off in various continuous control tasks such as Safety-Gymnasium and a complex building energy management environment CityLearn.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

43 extracted references · 32 canonical work pages

  1. [1]

    C. Acerbi. Spectral measures of risk: A coherent representation of sub- jective risk aversion. Journal of Banking & Finance, 26(7):1505–1518, July 2002. ISSN 03784266. doi: 10.1016/S0378-4266(02)00281-9. URL https://linkinghub.elsevier.com/retrieve/pii/S0378426602002819

    doi:10.1016/s0378-4266(02)00281-9 2002

  2. [2]

    Achiam, D

    J. Achiam, D. Held, A. Tamar, and P. Abbeel. Constrained Policy Optimization. In Proceedings of the 34th International Conference on Machine Learning , pages 22–31. PMLR, July 2017. URL https: //proceedings.mlr.press/v70/achiam17a.html. ISSN: 2640-3498

    work page 2017

  3. [3]

    E. Altman. Constrained Markov Decision Processes: Stochastic Model- ing. Routledge, Boca Raton, 1 edition, 1999. ISBN 978-1-315-14022-

    work page 1999

  4. [4]

    URL https://www.taylorfrancis.com/ books/9781315140223

    doi: 10.1201/9781315140223. URL https://www.taylorfrancis.com/ books/9781315140223

    work page arXiv

  5. [5]

    M. G. Bellemare, W. Dabney, and M. Rowland. Distributional rein- forcement learning. The MIT Press, Cambridge, Massachusetts, 2023. ISBN 978-0-262-37402-6. OCLC: 1350431122

    work page 2023

  6. [6]

    Bharadhwaj, A

    H. Bharadhwaj, A. Kumar, N. Rhinehart, S. Levine, F. Shkurti, and A. Garg. Conservative Safety Critics for Exploration, Apr. 2021. URL http://arxiv.org/abs/2010.14497. arXiv:2010.14497 [cs]

    arXiv 2021

  7. [7]

    P. J. Blokland and G. L. Reniers. The Concepts of Risk, Safety, and Security: A Fundamental Exploration and Understanding of Similar- ities and Differences. In C. Bieder and K. Pettersen Gould, editors, The Coupling of Safety and Security, pages 9–16. Springer International Publishing, Cham, 2020. ISBN 978-3-030-47228-3 978-3-030-47229-

    work page 2020

  8. [8]

    URL http://link.springer.com/ 10.1007/978-3-030-47229-0_2

    doi: 10.1007/978-3-030-47229-0_2. URL http://link.springer.com/ 10.1007/978-3-030-47229-0_2. Series Title: SpringerBriefs in Applied Sciences and Technology

    work page doi:10.1007/978-3-030-47229-0_2

Show all 43 references
  1. [9]

    S. Carr, N. Jansen, S. Junges, and U. Topcu. Safe Reinforcement Learn- ing via Shielding under Partial Observability. Proceedings of the AAAI Conference on Artificial Intelligence, 37(12):14748–14756, June 2023. ISSN 2374-3468, 2159-5399. doi: 10.1609/aaai.v37i12.26723. URL ht...

    2023 doi

  2. [10]

    Ciosek, Q

    K. Ciosek, Q. Vuong, R. Loftin, and K. Hofmann. Better Ex- ploration with Optimistic Actor Critic. In Advances in Neural In- formation Processing Systems , volume 32. Curran Associates, Inc.,

  3. [11]

    Dulac-Arnold, N

    G. Dulac-Arnold, N. Levine, D. J. Mankowitz, J. Li, C. Paduraru, S. Gowal, and T. Hester. Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning , 110(9):2419–2468, Sept. 2021. ISSN 0885-6125, 1573-0565. doi: 10.1007/s10994-021-...

    2021 doi

  4. [12]

    Dabney, G

    W. Dabney, G. Ostrovski, D. Silver, and R. Munos. Implicit Quantile Networks for Distributional Reinforcement Learning. In Proceed- ings of the 35th International Conference on Machine Learning , pages 1096–1105. PMLR, July 2018. URL https://proceedings.mlr.press/v80/ dabney18...

    2018

  5. [13]

    Dabney, M

    W. Dabney, M. Rowland, M. Bellemare, and R. Munos. Distributional Reinforcement Learning With Quantile Regression. Proceedings of the AAAI Conference on Artificial Intelligence , 32(1), Apr. 2018. ISSN 2374-3468, 2159-5399. doi: 10.1609/aaai.v32i1.11791. URL https: //ojs.aaai....

    2018 doi

  6. [14]

    Greenberg, Y

    I. Greenberg, Y . Chow, M. Ghavamzadeh, and S. Mannor. Efficient risk-averse reinforcement learning. Advances in Neural Information Processing Systems, 35:32639–32652, 2022

    2022

  7. [15]

    Fujimoto, H

    S. Fujimoto, H. Hoof, and D. Meger. Addressing Function Approxim- ation Error in Actor-Critic Methods. In Proceedings of the 35th Inter- national Conference on Machine Learning , pages 1587–1596. PMLR, July 2018. URL https://proceedings.mlr.press/v80/fujimoto18a.html. ISSN: 2640-3498

    2018

  8. [16]

    García and F

    J. García and F. Fernández. A Comprehensive Survey on Safe Rein- forcement Learning. Journal of Machine Learning Research , 16(42): 1437–1480, 2015. ISSN 1533-7928. URL http://jmlr.org/papers/v16/ garcia15a.html

    2015

  9. [17]

    P. J. Huber. Robust estimation of a location parameter. The Annals of Mathematical Statistics , 35(1):73 – 101, 1964. doi: 10.1214/aoms/ 1177703732. URL https://doi.org/10.1214/aoms/1177703732. Pub- lisher: Institute of Mathematical Statistics

    1964

  10. [18]

    S. Gu, L. Yang, Y . Du, G. Chen, F. Walter, J. Wang, Y . Yang, and A. Knoll. A Review of Safe Reinforcement Learning: Methods, The- ory and Applications, Feb. 2023. URL http://arxiv.org/abs/2205.10330. arXiv:2205.10330 [cs]

    2023 arXiv

  11. [19]

    Haarnoja, A

    T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. In Proceedings of the 35th International Confer- ence on Machine Learning, pages 1861–1870. PMLR, July 2018. URL https://proceedin...

    2018

  12. [20]

    D. Kim, K. Lee, and S. Oh. Trust Region-Based Safe Distribu- tional Reinforcement Learning for Multiple Constraints. Advances in Neural Information Processing Systems , 36:19908–19939, Dec

  13. [21]

    J. Ji, B. Zhang, J. Zhou, X. Pan, W. Huang, R. Sun, Y . Geng, Y . Zhong, J. Dai, and Y . Yang. Safety gymnasium: a unified safe reinforcement learning benchmark. In Thirty-seventh conference on neural inform- ation processing systems datasets and benchmarks track , 2023. URL h...

    2023

  14. [22]

    Keramati, C

    R. Keramati, C. Dann, A. Tamkin, and E. Brunskill. Being Optimistic to Be Conservative: Quickly Learning a CVaR Policy. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04):4436–4443, Apr

  15. [23]

    Z. Liu, Z. Cen, V . Isenbaev, W. Liu, S. Wu, B. Li, and D. Zhao. Constrained Variational Policy Optimization for Safe Reinforcement Learning. In Proceedings of the 39th International Conference on Machine Learning , pages 13644–13668. PMLR, June 2022. URL https://proceedings.m...

    2022

  16. [24]

    X. Ma, L. Xia, Z. Zhou, J. Yang, and Q. Zhao. DSAC: Distribu- tional Soft Actor Critic for Risk-Sensitive Reinforcement Learning, June 2020. URL http://arxiv.org/abs/2004.14547. arXiv:2004.14547 [cs]

    2020 arXiv

  17. [25]

    Moskovitz, J

    T. Moskovitz, J. Parker-Holder, A. Pacchiano, M. Arbel, and M. I. Jordan. Tactical Optimism and Pessimism for Deep Reinforce- ment Learning, Apr. 2022. URL http://arxiv.org/abs/2102.03765. arXiv:2102.03765

    2022 arXiv

  18. [26]

    D. Kim, T. Cho, S. Han, H. Chung, K. Lee, and S. Oh. Spectral-risk safe reinforcement learning with convergence guarantees. InThe thirty- eighth annual conference on neural information processing systems ,

  19. [27]

    R. T. Rockafellar and S. Uryasev. Optimization of conditional value- at-risk. The Journal of Risk , 2(3):21–41, 2000. ISSN 14651211. doi: 10.21314/JOR.2000.038. URL http://www.risk.net/journal-of-risk/ technical-paper/2161159/optimization-conditional-value-risk

    2000

  20. [28]

    J. Liu, Z. Wang, Y . Zheng, J. Hao, C. Bai, J. Ye, Z. Wang, H. Piao, and Y . Sun. OVD-Explorer: Optimism Should Not Be the Sole Pur- suit of Exploration in Noisy Environments. Proceedings of the AAAI Conference on Artificial Intelligence, 38(12):13954–13962, Mar. 2024. ISSN 23...

    2024 doi

  21. [29]

    N. A. Urpí, S. Curi, and A. Krause. Risk-Averse Offline Reinforcement Learning. 2021. URL https://openreview.net/forum?id=TBIzh9b5eaz

    2021

  22. [30]

    K. R. Varshney and H. Alemzadeh. On the Safety of Machine Learning: Cyber-Physical Systems, Decision Sciences, and Data Products. Big Data, 5(3):246–255, Sept. 2017. ISSN 2167-6461, 2167-647X. doi: 10.1089/big.2016.0051. URL http://www.liebertpub.com/doi/10.1089/ big.2016.0051

    2017

  23. [31]

    J. R. Vázquez-Canteli, J. Kämpf, G. Henze, and Z. Nagy. CityLearn v1.0: An OpenAI gym environment for demand response with deep reinforcement learning. In Proceedings of the 6th ACM international conference on systems for energy-efficient buildings, cities, and trans- portatio...

    2019

  24. [32]

    A. Ray, J. Achiam, and D. Amodei. Benchmarking Safe Exploration in Deep Reinforcement Learning. page 25, 2019

    2019

  25. [33]

    Z. Wu, B. Tang, Q. Lin, C. Yu, S. Mao, Q. Xie, X. Wang, and D. Wang. Off-policy primal-dual safe reinforcement learning. In The twelfth in- ternational conference on learning representations , 2024. URL https: //openreview.net/forum?id=vy42bYs1Wo

    2024

  26. [34]

    Tessler, D

    C. Tessler, D. J. Mankowitz, and S. Mannor. Reward Constrained Policy Optimization. 2019

    2019

  27. [35]

    Z. Yang, H. Jin, Y . Tang, and G. Fan. Risk-aware constrained reinforce- ment learning with non-stationary policies. In Proceedings of the 23rd international conference on autonomous agents and multiagent sys- tems, Aamas ’24, pages 2029–2037, Richland, SC, 2024. International...

    2024

  28. [36]

    Zhang, L

    L. Zhang, L. Shen, L. Yang, S. Chen, X. Wang, B. Yuan, and D. Tao. Penalized Proximal Policy Optimization for Safe Reinforcement Learn- ing. In Proceedings of the Thirty-First International Joint Confer- ence on Artificial Intelligence, pages 3744–3750, Vienna, Austria, July

  29. [38]

    Wachi, W

    A. Wachi, W. Hashimoto, X. Shen, and K. Hashimoto. Safe Ex- ploration in Reinforcement Learning: A Generalized Formulation and Algorithms, Oct. 2023. URL http://arxiv.org/abs/2310.03225. arXiv:2310.03225 [cs]

    2023 arXiv

  30. [40]

    Q. Yang, T. D. Simão, S. H. Tindemans, and M. T. J. Spaan. Safety- constrained reinforcement learning with a distributional safety critic. Machine Learning , 112(3):859–887, Mar. 2023. ISSN 0885-6125, 1573-0565. doi: 10.1007/s10994-022-06187-8. URL https://link. springer.com/1...

    2023 doi

  31. [2019]

    URL https://proceedings.neurips.cc/paper_files/paper/2019/hash/ a34bacf839b923770b2c360eefa26748-Abstract.html

    2019

  32. [2020]

    doi: 10.1609/aaai.v34i04.5870

    ISSN 2374-3468, 2159-5399. doi: 10.1609/aaai.v34i04.5870. URL https://ojs.aaai.org/index.php/AAAI/article/view/5870

    doi

  33. [2022]

    ISBN 978-1-956792-00-3

    International Joint Conferences on Artificial Intelligence Organ- ization. ISBN 978-1-956792-00-3. doi: 10.24963/ijcai.2022/520. URL https://www.ijcai.org/proceedings/2022/520. A Hyper-Parameters In this section we outline the hyper-parmeters used across environments. Firstly ...

    2022 doi

  34. [2023]

    URL https://proceedings.neurips.cc/paper_files/paper/2023/hash/ 3f20f2b0315c72201e23512fdbd1ee91-Abstract-Conference.html

    2023

  35. [2024]

    URL https://openreview.net/forum?id=9JFSJitKC0

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.