{"id":"8f6401d1-8320-46dd-bf89-810f7b3ac5ec","arxiv_id":"2412.08880","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"FAWAC adds a cost-advantage penalty to advantage weighted regression to keep offline-trained policies within a safety budget, with variants for standard and high-reward-but-unsafe datasets.","lead":"FAWAC modifies advantage weighted regression with a safety cost term to learn policies from static datasets in offline reinforcement learning. It reports strong safety-reward tradeoffs on Bullet Safety Gym benchmarks, but the theoretical derivation contains a questionable distributional equality and the experiments omit error bars.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The safety theorem hinges on Eq. 26, which equates the expected value over the offline dataset with the policy's true return; this equality is false under distribution shift, so the persistent-safety guarantee is unsupported.","rationale":"The reader's weakest_assumption correctly identifies Eq. 26 as the load-bearing flaw. My reading of the Appendix confirms: Eq. 26 is introduced in 'Optimization problem approximation' and immediately used to replace the indicator-weighted objective by J(pi); the same equality is invoked at the end of the Proposition 3 proof. I found no independent argument, such as importance sampling, a stationary-distribution correction, or an initial-state matching condition, that repairs the equality. The paper's own stated assumptions (S_f0 = S0 and pi_0 in Pi_f) are also acknowledged simplifications, but they do not fix Eq. 26. The empirical section reports only three seeds and no error bars or code, so the practical claims cannot be independently assessed, but I do not rest the rejection on that. Since the central theoretical claim of persistent safety is unsupported, the reader's REJECT verdict stands; no verdict adjustment is needed.","tokens_in":14196,"tokens_out":8046,"duration_ms":83847,"concrete_test":"Construct the two-state deterministic MDP with gamma=0.9, initial state s1, single action a, transitions s1--a-->s2 (r=0,c=0) and s2--a-->s2 (r=1,c=0). Let pi be the only policy and let D be the offline dataset generated by pi_beta=pi with its natural state marginal (empirically almost all s2). Compute J(pi) and E_{s~D}[V^pi(s)]: they are 9 and 10, so Eq. 26 is false. Then independently re-derive Proposition 3 without Eq. 26, using the correct identity J_c(pi)=E_{s~rho_0}[V_c^pi(s)] and keeping the d^{pi_beta} expectations explicit. If the conclusion E_{s in S0}[V_c^{pi*}(s)] <= kappa + 2 sqrt(2 delta) gamma epsilon_c^{pi*} / (1-gamma)^2 cannot be recovered, the persistent-safety guarantee is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Equation 26 is the load-bearing step: J(pi) = E_{s~D}[V_r^pi(s)] and J_c(pi) = E_{s~D}[V_c^pi(s)]. It is used twice: (i) in the Appendix to rewrite the original CMDP Eqs. 10-12 as Eq. 27, from which the surrogate Eqs. 13-16 and the closed form Eq. 18 are derived; and (ii) in the proof of Proposition 3, to assert J_c(pi_k) <= kappa from V_c^{pi_k}(s) <= kappa and to identify E_{s in S0}[V_c^{pi*}(s)] with J_c(pi*). In offline RL, D is collected by a behavior policy pi_beta, so the marginal of s in D is approximately d^{pi_beta}; neither d^{pi_beta} nor d^pi equals the initial-state distribution rho_0 that defines J(pi). Even at the stationary distribution of pi, E_{s~d^pi}[V^pi(s)] is not generally J(pi). Example: MDP with deterministic policy, s1 -> s2 (r=0), s2 -> s2 (r=1), gamma=0.9, initial state s1. Then J(pi)=9, V^pi(s2)=10, and d^pi(s2)=1, so E_{s~d^pi}[V^pi]=10, not 9. Hence the transformation of the constrained problem and the Proposition 3 bound do not follow. The algorithm may still behave well empirically, but the paper's theoretical guarantee of persistent safety V_c^pi(s) <= kappa is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces FAWAC, an offline safe RL algorithm based on advantage-weighted regression (AWR). The method augments the AWR advantage with a cost-advantage term, weighted by a Lagrange multiplier or a fixed penalty, and extends the framework to a 'tempting dataset' variant (FAWAC-T). The authors claim that this feasibility-informed advantage yields policies that maximize reward while persistently satisfying a cost constraint V_c^π(s) ≤ κ, and they provide a worst-case bound in Proposition 3. Experiments on the DSRL benchmark (Bullet Safety Gym tasks) report that FAWAC variants outperform several safe offline RL baselines on full and tempting datasets.","tokens_in":14599,"tokens_out":7141,"duration_ms":71164,"significance":"If the theoretical claims and empirical results hold, FAWAC would be a simple, scalable alternative to more complex safe offline RL methods, with a theoretical guarantee that is absent from many heuristics. The paper does provide a substantial empirical evaluation across six tasks and three cost limits, plus an ablation studying the two key hyperparameters. However, the central theoretical contribution rests on an invalid distributional identity (Eq. 26 in the appendix), and the resulting worst-case guarantee is not established. Without the theory, the method reduces to an AWR-style update with a cost penalty, which is a much more incremental contribution. The reproducibility of the experiments is limited by the absence of standard deviations in the reported tables.","major_comments":[{"comment":"The identity J(π) = E_{s∼D}[V_r^π(s)] (and similarly for cost) is not valid in offline RL. The dataset D is sampled from the behavior policy πβ, so the state marginal in the expectation is approximately d^{πβ}, while J(π) is defined by the initial-state distribution ρ0. Even when the expectation is taken under the stationary distribution of π, E_{s∼d^π}[V^π(s)] does not generally equal J(π); for a deterministic MDP with s1→s2 (r=0), s2→s2 (r=1), γ=0.9, ρ0(s1)=1, J(π)=9 while E_{s∼d^π}[V^π(s)]=10. Since this identity is used to pass from Eqs. (10)–(12) to Eq. (27) and then to the surrogate Eqs. (13)–(16) and the closed-form policy in Eq. (18), the derivation of the method and the persistent-safety guarantee are unsupported.","section":"Appendix, Eq. (26)"},{"comment":"The proof of Proposition 3 again invokes Eq. (26) to identify Jc(πk) and Jc(π*) with expectations over D, and it applies Corollary 2 of Achiam et al. (2017), which is stated for the on-policy distribution dπ, while the subsequent inequalities bound quantities under d^{πβ}. The claim that π* satisfies DKL(π*∥πβ) ≤ δ is also not guaranteed by Eq. (18) when λ is fixed rather than chosen as the optimal dual multiplier. Consequently the advertised worst-case bound Es∈S0[V_c^π*(s)] ≤ κ + 2√(2δ)γϵ_c^{π*}/(1−γ)^2 is not established.","section":"Appendix, Proof of Proposition 3"},{"comment":"For the tempting-dataset variant FAWAC-T, the update rule θ_{k+1} = arg min E_{s∼dπβ,a∼πβ}[−exp(−A_c^{πθk}(s,a)/λ) log πθ] is asserted without derivation from the preceding optimization, and the premise V_r^{πβ} ≥ V_r^{π*} is not proven; it is also unclear why discarding the reward advantage entirely is optimal under the stated assumptions. The tempting-dataset contribution therefore lacks formal justification.","section":"Section 3.3"}],"minor_comments":[{"comment":"The tables report only mean values; with three seeds and 20 evaluations per seed, standard deviations should be reported to substantiate the claimed outperformance, especially in cases where differences are small, such as FAWAC-P versus FISOR on several Circle tasks.","section":"Section 4.2, Tables 1 and 2"},{"comment":"The mapping from the undiscounted threshold κ′ to the discounted threshold κ is stated without derivation; the sentence 'we set κ = 7.32, corresponding to planning steps of approximately 30, 100, and 140 for κ′ values of 10, 20, and 30' is unclear because three κ′ values map to three different planning-step counts but a single κ.","section":"Section 4.3"},{"comment":"The approximation E_{a∼πθ_{k+1}}[A_c^{πθk}(s,a)] ≈ E_{a∼πθk}[A_c^{πθk}(s,a)] = 0 is used to replace the indicator function Ic and to simplify the multiplier update, but no bound or justification is given for the difference between consecutive policies; this is an additional heuristic that affects the FAWAC-P update in Eq. (25).","section":"Section 3.1, Eq. (22)"},{"comment":"The text says 'assuming that such s ∉ Sf', but the derivation then drops the reward term without explaining whether the indicator in Eq. (10) is zero or the constraint is inactive; please clarify the regime in which FAWAC-T is intended to apply.","section":"Section 3.3"},{"comment":"There are several typographical issues, including 'FA W AC' with spaces in the abstract and Section 3.2, 'we useδ' on page 4 lacking a space, and an inconsistent reference to 'section 6' for the appendix.","section":"Throughout"}],"recommendation":"reject","confidential_remarks":"I concur with the reader's assessment that Eq. (26) is a load-bearing error. The central theoretical guarantee is unsupported, and the derivation of the surrogate optimization problem inherits the flaw. The paper would require a substantial re-derivation of its theoretical claims, or a retraction of the persistent-safety guarantee, before it could be considered for publication; in its current form the contribution is not sufficient for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: FAWAC is a simple, sensible AWR-style method for safe offline RL, and the empirical results are worth a look. But the theoretical guarantee for persistent safety does not hold as written. Equation 26 in the appendix is false under distribution shift, and it is load-bearing.\n\nEq. 26 says J(pi)=E_{s~D}[V_r^pi(s)] and J_c(pi)=E_{s~D}[V_c^pi(s)]. The dataset marginal is induced by the behavior policy, not by pi and not by the initial-state distribution used in the return. The stress-test counterexample is correct. This equality is used to convert Eqs. 10-12 into Eq. 27 and then into the surrogate Eqs. 13-16, and it is used again in the proof of Proposition 3. So the safety bound on E_{s in S0}[V_c^pi*(s)] does not follow. The step from Eq. 27 to Eqs. 13-16 is also asserted via citations to Achiam et al.; the offline, state-wise constrained setting has a reward objective that should be summed over d^pi_k, not optimized per state as written. That is a second place where the derivation is not actually shown.\n\nWhat is new: Eq. 19/25 is a direct AWR/FOCOPS-style closed form with the KL constraint moved to the behavior policy and a cost-advantage term. The paper itself acknowledges the template. The combination for offline CMDPs is not in the cited literature, so it is a fair new instantiation. FAWAC-T for tempting datasets is a nice practical addition, though the update is stated without derivation.\n\nEmpirically, the method is evaluated on the DSRL benchmark across two task families and three robots, with three seeds and 20 episodes per seed. The results show lower costs than most baselines. But the tables have no error bars, so \"robust statistical significance\" is not supported, and no code is available. Those are fixable issues.\n\nWho is this for: someone building baselines for safe offline RL will find the algorithm simple and possibly useful. Someone reading for the safety guarantee should not rely on the current theory. The core idea is plausible; the problem is that the proof of persistent safety is unsupported.\n\nRecommendation: send it to peer review, but with the clear expectation of major revision. Either a correct bound under a defensible assumption (e.g., distribution correction or explicit conservatism) or a weakened empirical-only claim would make the paper acceptable. As it stands, Proposition 3 and the persistent-safety claim should not be in the paper.","headline":"FAWAC is a simple and plausible AWR-style baseline for safe offline RL, but the theoretical safety guarantee rests on a false equality and should be rewritten or withdrawn.","tokens_in":15119,"tokens_out":3824,"would_cite":false,"duration_ms":39476,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that offline safe RL can be reduced to advantage-weighted regression with a feasibility-informed cost term, and proves a bound on persistent cost violation.","keywords":["safe offline reinforcement learning","constrained Markov decision process","advantage weighted regression","feasibility","persistent safety","tempting dataset","implicit Q-learning"],"falsifier":"Take a dataset collected by a behavior policy whose stationary state distribution is measurably different from the stationary distribution of the policy FAWAC learns, compute $\\mathbb{E}_{s\\sim D}[V_c^\\pi(s)]$ from the learned critic, and compare it to the average cost actually incurred in rollouts; if Eq. 26 fails under shift, the rollout cost should exceed the value predicted by the bound, contradicting Proposition 3's guarantee.","tokens_in":14001,"feed_emoji":"🛡️","tokens_out":7587,"duration_ms":71852,"temperature":0.7,"pith_summary":"FAWAC reworks Advantage Weighted Regression (AWR) for safe offline reinforcement learning: instead of weighting dataset actions only by reward advantage, it weights them by $A^{\\pi_k}(s,a) - \\nu A_c^{\\pi_k}(s,a)$, a feasibility-informed advantage that penalizes actions with high cost advantage. The paper argues that this single change lets a constrained Markov decision process be solved almost like supervised regression, first in a non-parametric policy space and then by projecting back into a parameterized policy. It supports persistent safety, the requirement $V_c^\\pi(s) \\le \\kappa$, with a worst-case bound, and it reports that the resulting variants outperform existing safe offline RL baselines on the DSRL benchmark, including on 'tempting' datasets filled with high-reward but unsafe trajectories. The point of the paper is that persistent safety need not come from elaborate constraint machinery; a properly informed advantage plus behavior regularization can carry it.","feed_headline":"Cost-aware advantage regression keeps offline RL policies safe","feed_subtitle":"Persistent safety comes from one weighted regression objective, no complex constraint machinery.","key_machinery":"The load-bearing object is the feasibility-informed advantage $A^{\\pi_k}(s,a) - \\nu A_c^{\\pi_k}(s,a)$ inside the AWR exponential weight. It converts the cost constraint $V_c^{\\pi_k}(s) + \\frac{1}{1-\\gamma}\\mathbb{E}_{a\\sim\\pi}[A_c^{\\pi_k}(s,a)] \\le \\kappa$ into a per-action reweighting of behavior-cloning data, so the same machinery that makes AWR stable in offline RL also enforces safety. The second piece is the projection step, which minimizes $\\mathbb{E}_{s\\sim d_{\\pi_\\beta}(s)}[D_{\\mathrm{KL}}(\\pi^*(\\cdot|s)\\|\\pi_\\theta(\\cdot|s))]$ and reduces to a weighted maximum-likelihood objective against the dataset. Together they turn a CMDP with a non-parametric search into a supervised-style actor update, with the Lagrangian multiplier $\\nu$ (or a fixed penalty $\\hat\\nu$) doing the safety work.","core_discovery":"The paper's central claim is that the constrained policy update can be written, under two simplifying assumptions, as a convex optimization whose closed-form solution is $\\pi^*(a|s) \\propto \\pi_\\beta(a|s)\\exp((A^{\\pi_k}(s,a)-\\nu A_c^{\\pi_k}(s,a))/\\lambda)$, and that projecting this back into parametric space gives a behavior-cloning update in which each dataset action is weighted by the feasibility-informed advantage. This update is the whole algorithm. The paper further claims that if the current policy is feasible, the cost value of the new policy satisfies $\\mathbb{E}_{s\\in S_0}[V_c^{\\pi^*}(s)] \\le \\kappa + \\frac{2\\sqrt{2\\delta}\\gamma \\epsilon_c^{\\pi^*}}{(1-\\gamma)^2}$, so safety violations stay bounded by a term controlled by the behavior-regularization budget $\\delta$ and the maximum cost-advantage error. Two practical variants follow: FAWAC-M learns a statewise Lagrange multiplier, and FAWAC-P replaces it with a fixed penalty switched on by the indicator $\\mathbf{1}_{\\kappa - V_c^{\\pi_k}(s) \\le 0}$; a third variant, FAWAC-T, drops the reward term entirely on tempting datasets because high reward there is a proxy for unsafety.","pith_inferences":["A reader should treat Eq. 26 as the hinge: the guarantee $J(\\pi)=\\mathbb{E}_{s\\sim D}[V^\\pi(s)]$ is only true under no distribution shift, so a direct rollout test comparing predicted versus realized cost would stress-test the persistent-safety claim.","The tempting-dataset heuristic assumes high reward signals unsafety; on datasets where that correlation is weak, a soft interpolation between reward and cost weights may be safer than the binary switch.","The fixed-penalty formulation could be lifted into online fine-tuning: it needs only an indicator on the current cost value, so it can wrap any AWR-style actor without retraining the critic."],"forward_implications":["FAWAC's actor update is a weighted supervised regression, so it inherits AWR's stability and avoids maximizing over actions directly.","The worst-case cost bound in Proposition 3 ties persistent safety to the behavior-regularization budget $\\delta$ and the cost-advantage error $\\epsilon_c^{\\pi^*}$, giving two explicit knobs for safety.","The fixed-penalty variant FAWAC-P removes the need to optimize the Lagrange multiplier and, in the paper's experiments, is the variant that most reliably keeps normalized cost below the threshold.","On tempting datasets, dropping the reward term from the weight and keeping only the cost-advantage term yields cost compliance without explicit reward maximization."],"supporting_citations":[{"why":"Supplies the constrained policy optimization surrogate and the performance-difference bound used in Eq. 34 that Proposition 3 builds on.","marker":"Achiam et al., 2017"},{"why":"Gives the non-parametric policy-space formulation and strong duality used to derive the closed-form optimal policy.","marker":"Zhang et al., 2020"},{"why":"Provides the Advantage Weighted Regression weighted behavior-cloning objective that FAWAC extends with a cost-advantage term.","marker":"Peng et al., 2019"},{"why":"Supplies the Implicit Q-Learning critic with asymmetric expectile loss that yields the reward and cost advantages.","marker":"Kostrikov et al., 2021"},{"why":"Provides the DSRL benchmark and normalized reward/cost evaluation protocol used in all experiments.","marker":"Liu et al., 2023a"},{"why":"Defines tempting datasets and the OASIS baseline that FAWAC-T is evaluated against.","marker":"Yao et al., 2024"},{"why":"Supplies the FISOR hard-constraint baseline that FAWAC compares against in the safety-performance trade-off.","marker":"Zheng et al., 2024"},{"why":"Provides the CPQ constrained Q-learning baseline that FAWAC outperforms in the reported cost-reward comparisons.","marker":"Xu et al., 2022"}],"fun_headline_variants":["Feasibility-aware advantage regression for safe offline RL","Closed-form weighted update keeps offline RL policies safe","Cost-aware advantage weights: persistent safety in offline RL","Safe offline RL via feasibility-informed policy updates"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proof assumes that the expected return of a policy equals the average value function over states sampled from the offline dataset; that equality only holds when the dataset's state distribution is the policy's own stationary distribution, which distribution shift in offline RL generally destroys.","fun_headline_variants_meta":{"raw":{"variants":["Feasibility-aware advantage regression for safe offline RL","Closed-form weighted update keeps offline RL policies safe","Cost-aware advantage weights: persistent safety in offline RL","Safe offline RL via feasibility-informed policy updates"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000245,"raw_usage":{"total_tokens":1572,"prompt_tokens":1017,"completion_tokens":555,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":633,"completion_tokens_details":{"reasoning_tokens":495}},"tokens_in":633,"tokens_out":555,"duration_ms":6811,"temperature":1.0,"reasoning_tokens":495,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:28:14.814919+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a dataset collected by a behavior policy whose stationary state distribution is measurably different from the stationary distribution of the policy FAWAC learns, compute $\\mathbb{E}_{s\\sim D}[V_c^\\pi(s)]$ from the learned critic, and compare it to the average cost actually incurred in rollouts; if Eq. 26 fails under shift, the rollout cost should exceed the value predicted by the bound, contradicting Proposition 3's guarantee.","supporting_citations":[{"cited_title":"First order constrained optimization in policy space","cited_arxiv_id":null,"evidence_quote":"Gives the non-parametric policy-space formulation and strong duality used to derive the closed-form optimal policy."},{"cited_title":"OASIS: Conditional Distribution Shaping for Offline Safe Reinforcement Learning","cited_arxiv_id":"2407.14653","evidence_quote":"Defines tempting datasets and the OASIS baseline that FAWAC-T is evaluated against."}],"review_version":1}