Pith. sign in

REVIEW 3 major objections 5 minor 36 references

FAWAC: Feasibility Informed Advantage Weighted Regression for Persistent Safety in Offline Reinforcement Learning

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that offline safe RL can be reduced to advantage-weighted regression with a feasibility-informed cost term, and proves a bound on persistent cost violation.

desk verdict FAWAC is a simple and plausible AWR-style baseline for safe offline RL, but the theoretical safety guarantee rests on a false equality and should be rewritten or withdrawn. read the letter →

arxiv 2412.08880 v1 pith:6IWQFXZZ submitted 2024-12-12 cs.LG

Prajwal Koirala , Zhanhong Jiang , Soumik Sarkar , Cody Fleming This is my paper · ORCID
classification cs.LG
keywords safeofflinereinforcementlearningconstrainedMarkovdecisionprocessadvantageweightedregressionfeasibilitypersistentsafetytemptingdatasetimplicitQ-learning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FAWAC reworks Advantage Weighted Regression (AWR) for safe offline reinforcement learning: instead of weighting dataset actions only by reward advantage, it weights them by $A^{\pi_k}(s,a) - \nu A_c^{\pi_k}(s,a)$, a feasibility-informed advantage that penalizes actions with high cost advantage. The paper argues that this single change lets a constrained Markov decision process be solved almost like supervised regression, first in a non-parametric policy space and then by projecting back into a parameterized policy. It supports persistent safety, the requirement $V_c^\pi(s) \le \kappa$, with a worst-case bound, and it reports that the resulting variants outperform existing safe offline RL baselines on the DSRL benchmark, including on 'tempting' datasets filled with high-reward but unsafe trajectories. The point of the paper is that persistent safety need not come from elaborate constraint machinery; a properly informed advantage plus behavior regularization can carry it.

What carries the argument

The load-bearing object is the feasibility-informed advantage $A^{\pi_k}(s,a) - \nu A_c^{\pi_k}(s,a)$ inside the AWR exponential weight. It converts the cost constraint $V_c^{\pi_k}(s) + \frac{1}{1-\gamma}\mathbb{E}_{a\sim\pi}[A_c^{\pi_k}(s,a)] \le \kappa$ into a per-action reweighting of behavior-cloning data, so the same machinery that makes AWR stable in offline RL also enforces safety. The second piece is the projection step, which minimizes $\mathbb{E}_{s\sim d_{\pi_\beta}(s)}[D_{\mathrm{KL}}(\pi^*(\cdot|s)\|\pi_\theta(\cdot|s))]$ and reduces to a weighted maximum-likelihood objective against the dataset. Together they turn a CMDP with a non-parametric search into a supervised-style actor update, with the Lagrangian multiplier $\nu$ (or a fixed penalty $\hat\nu$) doing the safety work.

What would settle it

Take a dataset collected by a behavior policy whose stationary state distribution is measurably different from the stationary distribution of the policy FAWAC learns, compute $\mathbb{E}_{s\sim D}[V_c^\pi(s)]$ from the learned critic, and compare it to the average cost actually incurred in rollouts; if Eq. 26 fails under shift, the rollout cost should exceed the value predicted by the bound, contradicting Proposition 3's guarantee.

Watch

Extended reading notes

Core claim

The paper's central claim is that the constrained policy update can be written, under two simplifying assumptions, as a convex optimization whose closed-form solution is $\pi^*(a|s) \propto \pi_\beta(a|s)\exp((A^{\pi_k}(s,a)-\nu A_c^{\pi_k}(s,a))/\lambda)$, and that projecting this back into parametric space gives a behavior-cloning update in which each dataset action is weighted by the feasibility-informed advantage. This update is the whole algorithm. The paper further claims that if the current policy is feasible, the cost value of the new policy satisfies $\mathbb{E}_{s\in S_0}[V_c^{\pi^*}(s)] \le \kappa + \frac{2\sqrt{2\delta}\gamma \epsilon_c^{\pi^*}}{(1-\gamma)^2}$, so safety violations stay bounded by a term controlled by the behavior-regularization budget $\delta$ and the maximum cost-advantage error. Two practical variants follow: FAWAC-M learns a statewise Lagrange multiplier, and FAWAC-P replaces it with a fixed penalty switched on by the indicator $\mathbf{1}_{\kappa - V_c^{\pi_k}(s) \le 0}$; a third variant, FAWAC-T, drops the reward term entirely on tempting datasets because high reward there is a proxy for unsafety.

Load-bearing premise

The proof assumes that the expected return of a policy equals the average value function over states sampled from the offline dataset; that equality only holds when the dataset's state distribution is the policy's own stationary distribution, which distribution shift in offline RL generally destroys.

Editorial extensions

If this is right

  • FAWAC's actor update is a weighted supervised regression, so it inherits AWR's stability and avoids maximizing over actions directly.
  • The worst-case cost bound in Proposition 3 ties persistent safety to the behavior-regularization budget $\delta$ and the cost-advantage error $\epsilon_c^{\pi^*}$, giving two explicit knobs for safety.
  • The fixed-penalty variant FAWAC-P removes the need to optimize the Lagrange multiplier and, in the paper's experiments, is the variant that most reliably keeps normalized cost below the threshold.
  • On tempting datasets, dropping the reward term from the weight and keeping only the cost-advantage term yields cost compliance without explicit reward maximization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A reader should treat Eq. 26 as the hinge: the guarantee $J(\pi)=\mathbb{E}_{s\sim D}[V^\pi(s)]$ is only true under no distribution shift, so a direct rollout test comparing predicted versus realized cost would stress-test the persistent-safety claim.
  • The tempting-dataset heuristic assumes high reward signals unsafety; on datasets where that correlation is weak, a soft interpolation between reward and cost weights may be safer than the binary switch.
  • The fixed-penalty formulation could be lifted into online fine-tuning: it needs only an indicator on the current cost value, so it can wrap any AWR-style actor without retraining the critic.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces FAWAC, an offline safe RL algorithm based on advantage-weighted regression (AWR). The method augments the AWR advantage with a cost-advantage term, weighted by a Lagrange multiplier or a fixed penalty, and extends the framework to a 'tempting dataset' variant (FAWAC-T). The authors claim that this feasibility-informed advantage yields policies that maximize reward while persistently satisfying a cost constraint V_c^π(s) ≤ κ, and they provide a worst-case bound in Proposition 3. Experiments on the DSRL benchmark (Bullet Safety Gym tasks) report that FAWAC variants outperform several safe offline RL baselines on full and tempting datasets.

Significance. If the theoretical claims and empirical results hold, FAWAC would be a simple, scalable alternative to more complex safe offline RL methods, with a theoretical guarantee that is absent from many heuristics. The paper does provide a substantial empirical evaluation across six tasks and three cost limits, plus an ablation studying the two key hyperparameters. However, the central theoretical contribution rests on an invalid distributional identity (Eq. 26 in the appendix), and the resulting worst-case guarantee is not established. Without the theory, the method reduces to an AWR-style update with a cost penalty, which is a much more incremental contribution. The reproducibility of the experiments is limited by the absence of standard deviations in the reported tables.

major comments (3)
  1. [Appendix, Eq. (26)] The identity J(π) = E_{s∼D}[V_r^π(s)] (and similarly for cost) is not valid in offline RL. The dataset D is sampled from the behavior policy πβ, so the state marginal in the expectation is approximately d^{πβ}, while J(π) is defined by the initial-state distribution ρ0. Even when the expectation is taken under the stationary distribution of π, E_{s∼d^π}[V^π(s)] does not generally equal J(π); for a deterministic MDP with s1→s2 (r=0), s2→s2 (r=1), γ=0.9, ρ0(s1)=1, J(π)=9 while E_{s∼d^π}[V^π(s)]=10. Since this identity is used to pass from Eqs. (10)–(12) to Eq. (27) and then to the surrogate Eqs. (13)–(16) and the closed-form policy in Eq. (18), the derivation of the method and the persistent-safety guarantee are unsupported.
  2. [Appendix, Proof of Proposition 3] The proof of Proposition 3 again invokes Eq. (26) to identify Jc(πk) and Jc(π*) with expectations over D, and it applies Corollary 2 of Achiam et al. (2017), which is stated for the on-policy distribution dπ, while the subsequent inequalities bound quantities under d^{πβ}. The claim that π* satisfies DKL(π*∥πβ) ≤ δ is also not guaranteed by Eq. (18) when λ is fixed rather than chosen as the optimal dual multiplier. Consequently the advertised worst-case bound Es∈S0[V_c^π*(s)] ≤ κ + 2√(2δ)γϵ_c^{π*}/(1−γ)^2 is not established.
  3. [Section 3.3] For the tempting-dataset variant FAWAC-T, the update rule θ_{k+1} = arg min E_{s∼dπβ,a∼πβ}[−exp(−A_c^{πθk}(s,a)/λ) log πθ] is asserted without derivation from the preceding optimization, and the premise V_r^{πβ} ≥ V_r^{π*} is not proven; it is also unclear why discarding the reward advantage entirely is optimal under the stated assumptions. The tempting-dataset contribution therefore lacks formal justification.
minor comments (5)
  1. [Section 4.2, Tables 1 and 2] The tables report only mean values; with three seeds and 20 evaluations per seed, standard deviations should be reported to substantiate the claimed outperformance, especially in cases where differences are small, such as FAWAC-P versus FISOR on several Circle tasks.
  2. [Section 4.3] The mapping from the undiscounted threshold κ′ to the discounted threshold κ is stated without derivation; the sentence 'we set κ = 7.32, corresponding to planning steps of approximately 30, 100, and 140 for κ′ values of 10, 20, and 30' is unclear because three κ′ values map to three different planning-step counts but a single κ.
  3. [Section 3.1, Eq. (22)] The approximation E_{a∼πθ_{k+1}}[A_c^{πθk}(s,a)] ≈ E_{a∼πθk}[A_c^{πθk}(s,a)] = 0 is used to replace the indicator function Ic and to simplify the multiplier update, but no bound or justification is given for the difference between consecutive policies; this is an additional heuristic that affects the FAWAC-P update in Eq. (25).
  4. [Section 3.3] The text says 'assuming that such s ∉ Sf', but the derivation then drops the reward term without explaining whether the indicator in Eq. (10) is zero or the constraint is inactive; please clarify the regime in which FAWAC-T is intended to apply.
  5. [Throughout] There are several typographical issues, including 'FA W AC' with spaces in the abstract and Section 3.2, 'we useδ' on page 4 lacking a space, and an inconsistent reference to 'section 6' for the appendix.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: FAWAC's policy form and safety bound follow from stated assumptions and standard constrained-policy lemmas, with no fitted input renamed as a prediction and no load-bearing self-citation.

full rationale

I found no step in which a predicted quantity is equivalent by construction to a fitted input, nor any load-bearing self-citation. The paper derives the closed-form policy (Eq. 18) by solving the Lagrangian of Eqs. 13-16, and the parametric update (Eqs. 19 and 25) is the standard AWR projection onto the parametric policy class. Proposition 3 is conditional on the stated feasibility assumption pi_k in Pi_f and uses the performance-difference lemma from Achiam et al. (2017); its bound E_{s in S0}[V_c^{pi*}(s)] <= kappa + ... is an inequality derived from that lemma and the constraint, not a restatement of the algorithm's own objective. The transformation of the original CMDP does rely on Eq. 26, J(pi) = E_{s~D}[V_r^pi(s)], and on the assumptions S_{f0}=S_0 and pi_0 in Pi_f; these are explicitly introduced as simplifying assumptions rather than as definitions of the target quantity. The offline dataset marginal d^{pi_beta} does not generally equal the initial-state distribution rho_0, so Eq. 26 is a soundness gap in the persistent-safety proof, but this is a correctness issue rather than circular reasoning. The experimental evaluation uses the external DSRL benchmark and the tempting-dataset construction of Yao et al. (2024), and the reference list contains no self-citations by the authors of this paper. Therefore the derivation is self-contained given its stated assumptions, and the circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on four explicit or implicit assumptions, as listed above. No invented physical or mathematical entities are introduced. The feasibility-informed advantage is a weighted combination of reward and cost advantages, not an entity.

free parameters (4)
  • temperature λ = 2.0
    Controls the sharpness of the advantage-weighted exponential in Eq. 18; manually tuned and ablated in Fig. 3.
  • finite penalty ν̂ = 20.0
    Penalty factor in FAWAC-P (Eq. 25); set by manual tuning, ablated in Fig. 3.
  • multiplier clamp νmax = 20.0
    Upper bound for the statewise multiplier in FAWAC-M; manually set.
  • discounted cost threshold κ = 7.32
    Maps empirical undiscounted thresholds κ' ∈ {10,20,40} to the discounted cost threshold in the CMDP; based on an approximate planning-step conversion following Zhang et al. (2023).
assumptions (4)
  • ad hoc to paper All dataset states are feasible (Sf0 = S0)
    Assumed in Section 3.1 to simplify the optimization problem; the paper acknowledges it may not hold in practice, especially for tempting datasets.
  • ad hoc to paper Initial policy is feasible (π0 ∈ Πf)
    Required for Proposition 3 and for the sequential update to maintain feasibility.
  • domain assumption The static dataset's state distribution approximates the policy's stationary distribution (Eq. 26)
    Used to convert the expected return J(π) into Es~D[V^π(s)], which is generally false in offline RL with distribution shift; a central element of the derivation.
  • standard math Advantage of the current policy equals zero under its own action distribution (Eq. 22)
    By definition of advantage, E_{a~π}[A^π(s,a)] = 0; the paper also approximates the next policy's expectation with the current policy's, justified by the KL constraint.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FAWAC: Feasibility Informed Advantage Weighted Regression for Persistent Safety in Offline Reinforcement Learning." pith.science (2026). https://pith.science/paper/6IWQFXZZ

@misc{pith2026241208880,
  author       = {Pith},
  title        = {Pith review of: FAWAC: Feasibility Informed Advantage Weighted Regression for Persistent Safety in Offline Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6IWQFXZZ}},
  note         = {Machine review of arXiv:2412.08880}
}
read the original abstract

Safe offline reinforcement learning aims to learn policies that maximize cumulative rewards while adhering to safety constraints, using only offline data for training. A key challenge is balancing safety and performance, particularly when the policy encounters out-of-distribution (OOD) states and actions, which can lead to safety violations or overly conservative behavior during deployment. To address these challenges, we introduce Feasibility Informed Advantage Weighted Actor-Critic (FAWAC), a method that prioritizes persistent safety in constrained Markov decision processes (CMDPs). FAWAC formulates policy optimization with feasibility conditions derived specifically for offline datasets, enabling safe policy updates in non-parametric policy space, followed by projection into parametric space for constrained actor training. By incorporating a cost-advantage term into Advantage Weighted Regression (AWR), FAWAC ensures that the safety constraints are respected while maximizing performance. Additionally, we propose a strategy to address a more challenging class of problems that involves tempting datasets where trajectories are predominantly high-rewarded but unsafe. Empirical evaluations on standard benchmarks demonstrate that FAWAC achieves strong results, effectively balancing safety and performance in learning policies from the static datasets.

Figures

Figures reproduced from arXiv: 2412.08880 by the authors.

Figure 1
Figure 1. (a) Block diagram illustrating the FAWAC training with a static dataset. Transitions [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Training curves depicting reward and cost returns for our method and baselines. [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Ablation on the FAWAC hyperparameters νˆ and λ. 4.3. Implementation Details and Ablation Our method employs a Lagrange-based formulation, enforcing a soft constraint to ensure that expected safety violation costs remain below predetermined thresholds. However, the discounted cost return threshold (κ) used in our formulation cannot be directly related to the undiscounted threshold (κ ′ ) used in empirical evaluations… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 14 canonical work pages

  1. [1]

    Constrained policy optimization

    Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning, pages 22--31. PMLR, 2017

    2017

  2. [2]

    Maximum entropy inverse reinforcement learning in continuous state spaces with path integrals

    Navid Aghasadeghi and Timothy Bretl. Maximum entropy inverse reinforcement learning in continuous state spaces with path integrals. In 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 1561--1566. IEEE, 2011

    work page 2011

  3. [3]

    Constrained markov decision processes with total cost criteria: Lagrangian approach and dual linear program

    Eitan Altman. Constrained markov decision processes with total cost criteria: Lagrangian approach and dual linear program. Mathematical methods of operations research, 48: 0 387--417, 1998

    work page 1998

  4. [4]

    Constrained Markov decision processes

    Eitan Altman. Constrained Markov decision processes. Routledge, 2021

    2021

  5. [5]

    Convex optimization

    Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004

    2004

  6. [6]

    Decision transformer: Reinforcement learning via sequence modeling

    Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34: 0 15084--15097, 2021

    2021

  7. [7]

    Pybullet, a python module for physics simulation for games, robotics and machine learning, 2016

    Erwin Coumans and Yunfei Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning, 2016

    2016

  8. [8]

    Directional differentiability of optimal solutions under slater's condition

    Stephan Dempe. Directional differentiability of optimal solutions under slater's condition. Mathematical programming, 59: 0 49--69, 1993

    work page 1993

Show all 36 references
  1. [9]

    Parenting: Safe reinforcement learning from human input

    Christopher Frye and Ilya Feige. Parenting: Safe reinforcement learning from human input. arXiv preprint arXiv:1902.06766, 2019

    1902 arXiv

  2. [10]

    A comprehensive survey on safe reinforcement learning

    Javier Garc a and Fernando Fern \'a ndez. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16 0 (1): 0 1437--1480, 2015

    2015

  3. [11]

    A primal-dual augmented lagrangian

    Philip E Gill and Daniel P Robinson. A primal-dual augmented lagrangian. Computational Optimization and Applications, 51 0 (1): 0 1--25, 2012

    2012

  4. [12]

    Bullet-safety-gym: A framework for constrained reinforcement learning

    Sven Gronauer. Bullet-safety-gym: A framework for constrained reinforcement learning. 2022

    2022

  5. [13]

    A review of safe reinforcement learning: Methods, theory and applications

    Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, Yaodong Yang, and Alois Knoll. A review of safe reinforcement learning: Methods, theory and applications. arXiv preprint arXiv:2205.10330, 2022

    2022 arXiv

  6. [14]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861--1870. PMLR, 2018

    2018

  7. [15]

    Offline reinforcement learning with implicit q-learning

    Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169, 2021

    2021 arXiv

  8. [16]

    Conservative q-learning for offline reinforcement learning

    Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33: 0 1179--1191, 2020

    2020

  9. [17]

    Optidice: Offline policy optimization via stationary distribution correction estimation

    Jongmin Lee, Wonseok Jeon, Byungjun Lee, Joelle Pineau, and Kee-Eung Kim. Optidice: Offline policy optimization via stationary distribution correction estimation. In International Conference on Machine Learning, pages 6120--6130. PMLR, 2021

    2021

  10. [18]

    Coptidice: Offline constrained reinforcement learning via stationary distribution correction estimation

    Jongmin Lee, Cosmin Paduraru, Daniel J Mankowitz, Nicolas Heess, Doina Precup, Kee-Eung Kim, and Arthur Guez. Coptidice: Offline constrained reinforcement learning via stationary distribution correction estimation. arXiv preprint arXiv:2204.08957, 2022

    2022 arXiv

  11. [19]

    Offline reinforcement learning: Tutorial, review, and perspectives on open problems

    Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020

    2005 arXiv

  12. [20]

    Datasets and benchmarks for offline safe reinforcement learning

    Zuxin Liu, Zijian Guo, Haohong Lin, Yihang Yao, Jiacheng Zhu, Zhepeng Cen, Hanjiang Hu, Wenhao Yu, Tingnan Zhang, Jie Tan, et al. Datasets and benchmarks for offline safe reinforcement learning. arXiv preprint arXiv:2306.09303, 2023 a

    2023 arXiv

  13. [21]

    Constrained decision transformer for offline safe reinforcement learning

    Zuxin Liu, Zijian Guo, Yihang Yao, Zhepeng Cen, Wenhao Yu, Tingnan Zhang, and Ding Zhao. Constrained decision transformer for offline safe reinforcement learning. In International Conference on Machine Learning, pages 21611--21630. PMLR, 2023 b

    2023

  14. [22]

    Feasible actor-critic: Constrained reinforcement learning for ensuring statewise safety

    Haitong Ma, Yang Guan, Shegnbo Eben Li, Xiangteng Zhang, Sifa Zheng, and Jianyu Chen. Feasible actor-critic: Constrained reinforcement learning for ensuring statewise safety. arXiv preprint arXiv:2105.10682, 2021

    2021 arXiv

  15. [23]

    Awac: Accelerating online reinforcement learning with offline datasets

    Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020

    2006 arXiv

  16. [24]

    Advantage-weighted regression: Simple and scalable off-policy reinforcement learning

    Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019

    1910 arXiv

  17. [25]

    Trust region policy optimization

    John Schulman. Trust region policy optimization. arXiv preprint arXiv:1502.05477, 2015

    2015 arXiv

  18. [26]

    Equivalence between policy gradients and soft q-learning

    John Schulman, Xi Chen, and Pieter Abbeel. Equivalence between policy gradients and soft q-learning. arXiv preprint arXiv:1704.06440, 2017

    2017 arXiv

  19. [27]

    Responsive safety in reinforcement learning by pid lagrangian methods

    Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods. In International Conference on Machine Learning, pages 9133--9143. PMLR, 2020

    2020

  20. [28]

    Constraints penalized q-learning for safe offline reinforcement learning

    Haoran Xu, Xianyuan Zhan, and Xiangyu Zhu. Constraints penalized q-learning for safe offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8753--8760, 2022

    2022

  21. [29]

    Primal-dual stochastic gradient method for convex programs with many functional constraints

    Yangyang Xu. Primal-dual stochastic gradient method for convex programs with many functional constraints. SIAM Journal on Optimization, 30 0 (2): 0 1664--1692, 2020

    2020

  22. [30]

    Oasis: Conditional distribution shaping for offline safe reinforcement learning

    Yihang Yao, Zhepeng Cen, Wenhao Ding, Haohong Lin, Shiqi Liu, Tingnan Zhang, Wenhao Yu, and Ding Zhao. Oasis: Conditional distribution shaping for offline safe reinforcement learning. arXiv preprint arXiv:2407.14653, 2024

    2024 arXiv

  23. [31]

    Reachability constrained reinforcement learning

    Dongjie Yu, Haitong Ma, Shengbo Li, and Jianyu Chen. Reachability constrained reinforcement learning. In International conference on machine learning, pages 25636--25655. PMLR, 2022

    2022

  24. [32]

    Penalized proximal policy optimization for safe reinforcement learning

    Linrui Zhang, Li Shen, Long Yang, Shixiang Chen, Bo Yuan, Xueqian Wang, and Dacheng Tao. Penalized proximal policy optimization for safe reinforcement learning. arXiv preprint arXiv:2205.11814, 2022

    2022 arXiv

  25. [33]

    Evaluating model-free reinforcement learning toward safety-critical tasks

    Linrui Zhang, Qin Zhang, Li Shen, Bo Yuan, Xueqian Wang, and Dacheng Tao. Evaluating model-free reinforcement learning toward safety-critical tasks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 15313--15321, 2023

    2023

  26. [34]

    First order constrained optimization in policy space

    Yiming Zhang, Quan Vuong, and Keith Ross. First order constrained optimization in policy space. Advances in Neural Information Processing Systems, 33: 0 15338--15349, 2020

    2020

  27. [35]

    Safe offline reinforcement learning with feasibility-guided diffusion model

    Yinan Zheng, Jianxiong Li, Dongjie Yu, Yujie Yang, Shengbo Eben Li, Xianyuan Zhan, and Jingjing Liu. Safe offline reinforcement learning with feasibility-guided diffusion model. arXiv preprint arXiv:2401.10700, 2024

    2024 arXiv

  28. [36]

    Maximum entropy inverse reinforcement learning

    Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pages 1433--1438. Chicago, IL, USA, 2008

    2008

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.