{"id":"8b3d810f-d93b-4a7a-9b31-15fd61495afd","arxiv_id":"2508.01883","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"A constrained policy optimization method that uses a preemptive log-barrier penalty and a constraint-aware intrinsic reward to reduce safety violations in reinforcement learning.","lead":"This paper introduces PCPO, a safe reinforcement learning method that adds a barrier-style penalty before constraint violations and a boundary-gated intrinsic reward to guide exploration. The authors report lower constraint violations and stable training on Safety-Gymnasium tasks, and provide theoretical bounds on the duality gap and update performance.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Proposition 4.4's proof is a non-sequitur: a larger lower bound on the augmented objective G does not imply a per-iteration ordering of cumulative costs, so the central guarantee of provably fewer violations than Lagrangian methods is unsupported.","rationale":"The paper's central claim is not just that PCPO works empirically; it is that PCPO 'provably reduces constraint violations' and has lower cumulative violations than Lagrangian methods, with the gap tending to infinity. The formal statement of that claim is Proposition 4.4. I examined its proof and found the load-bearing inference invalid: the proof compares lower bounds on ΔG, an augmented objective including reward, barrier, and intrinsic reward, and then directly asserts a cost/constraint-violation ordering. Even if LPCPO > LLagrange, one can have JC(πP) > JC(πL), because the lower bound can be larger due to reward or intrinsic terms while costs behave oppositely. The claimed divergence Δ(T) → ∞ also depends on Eq. (80), an unproved assertion about persistent Lagrangian violations. This is not a matter of outside consensus; it is internal to the argument. I agree with the reader's overall rejection, but I would place the weakest point earlier than the implementation-approximation issue: even if the algorithm solved Eq. (9) exactly, Proposition 4.4 would still not follow from the supplied proof. The approximation gap in Section 5 is a second, independent problem: Theorems 4.2 and 4.3 apply to the exact argmax of the barrier objective (Eq. 9), while Algorithm 1 executes a single linear-quadratic natural-gradient step (Eqs. 17–21), and no error bound connects them. The empirical section reports results over 6 seeds with per-environment tuning and no released code, so the experiments do not compensate for the missing theoretical support. The cleanest corrective action is to re-derive Proposition 4.4: either supply the missing per-iteration cost-gap argument or state the proposition as a conjecture.","tokens_in":23653,"tokens_out":5700,"duration_ms":66646,"concrete_test":"Independently re-derive the step from Eq. (74) to Eq. (76) in Appendix E. Construct a one-state, two-action tabular CMDP and compute, for both PCPO and a Lagrangian method, the exact per-iteration updates and the resulting JC values under the paper's KL trust-region constraint. If there exists any parameter setting satisfying the paper's assumptions where LPCPO > LLagrange but JC(πP) > JC(πL), then the 'Consequently' inference is invalid and Proposition 4.4 cannot stand. As a separate numerical check, run this tabular CMDP for T = 100 iterations and test whether the claimed gap Δ(T) grows to infinity, or whether it eventually stabilizes; Eq. (80) should be verified rather than assumed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim that PCPO provably reduces constraint violations rests on Proposition 4.4 (Eq. 14). The proof in Appendix E does not establish it. After defining LPCPO and LLagrange as lower bounds on the change of the augmented objective G(π) = f(π) − Σ φτ(gCi(π)) + η Σ I, the proof shows LLagrange < LPCPO (Eq. 74) and then asserts 'Consequently' JC(πP_{k+1}) ≤ JC(πL_{k+1}) − δ_k (Eq. 76) and vP_{k+1} ≤ vL_{k+1} − δ_k. This is not a consequence: G includes reward, barrier penalty, and intrinsic reward, so a larger lower bound on ΔG says nothing about the difference in raw cumulative cost JC between the two algorithms. The step also requires a per-iteration bound on δ_k that is never derived. The claim Δ(T) → ∞ additionally relies on Eq. (80), liminf_{T→∞} (1/T) VL(T) > 0, which is asserted rather than proven for the compared Lagrangian baselines. Because Proposition 4.4 is the formal statement of the headline contribution, the theoretical support for 'provably fewer violations' is missing.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Proactive Constrained Policy Optimization (PCPO), a safe RL method that augments the constrained objective with an extended log-barrier term and a constraint-aware intrinsic reward, and updates the policy by a natural-gradient step under a KL trust-region constraint. The authors claim three theoretical results: an upper bound on the duality gap (Theorem 4.2), a lower bound on per-iteration improvement of the augmented objective (Theorem 4.3), and a guarantee that PCPO has asymptotically smaller cumulative constraint violations than Lagrangian safe RL methods (Proposition 4.4). Experiments on Safety Gymnasium velocity and navigation tasks compare PCPO with CUP, EPO, FOCOPS, and TRPOLag, reporting higher reward and more stable cost behavior, along with ablations, sensitivity analyses, and generalization tests.","tokens_in":24031,"tokens_out":4794,"duration_ms":58217,"significance":"If the theoretical claims were valid, the paper would make a useful contribution: the preemptive barrier formulation is a reasonable alternative to Lagrangian methods, and the constraint-aware intrinsic reward is a sensible mechanism for boundary-aware exploration. The experimental study is fairly extensive, including multiple environments, six seeds with bootstrap confidence intervals, ablation studies, sensitivity analyses, and a generalization test. However, the central stated contribution is the theoretical framework, and the proofs contain load-bearing gaps: Proposition 4.1 is circular, Theorem 4.2 bounds a quantity that is not the true duality gap, Theorem 4.3 has algebraic inconsistencies, and Proposition 4.4 does not follow from the preceding lower-bound comparison. Because the headline claim of provably fewer constraint violations is unsupported, the paper's significance as a theoretical contribution is not established, and the empirical results alone are not sufficient to justify publication in its current form.","major_comments":[{"comment":"Proposition 4.4, the formal statement of the paper's main claim that PCPO provably incurs fewer cumulative constraint violations than Lagrangian methods, is not proved. The proof defines LPCPO and LLagrange as lower bounds on the change of the augmented objective G(π)=f(π)−Σφτ(gCi(π))+ηΣI, then shows LLagrange<LPCPO in Eq. (74), and asserts in Eq. (76) that JC(πP_{k+1})≤JC(πL_{k+1})−δ_k. This implication is a non-sequitur: a comparison of lower bounds on ΔG says nothing about the ordering of raw cumulative costs JC between two different algorithms, since G also contains reward and intrinsic-reward terms. The quantity δ_k is defined as the difference of lower bounds of ΔG, not as a per-step cost gap, and no argument is given that δ_k>0 at every iteration. The asymptotic statement Δ(T)→∞ additionally relies on Eq. (80), liminf_{T→∞}(1/T)VL(T)>0, which is asserted without proof for the compared Lagrangian baselines. Thus the central guarantee of provably fewer violations rests on an unproved, and currently false-looking, chain of inference.","section":"§4.3, Appendix E, Eq. (14) and Eqs. (74)–(80)"},{"comment":"Proposition 4.1 is circular. The proof needs to establish that the policy updated with the intrinsic reward has better constraint satisfaction than the policy updated without it, but it assumes this conclusion directly: 'Additionally, the intrinsic reward pushes the policy inward, meaning gCi(πk+1)≤gCi(π̄k+1).' The earlier assumption f(πk+1)≈f(π̄k+1) is also asserted without a bound. These are exactly the facts that would need to be derived from the definitions of the two updates, rather than stipulated. Consequently Eq. (10) of Proposition 4.1 is not proven.","section":"§4.2, Appendix B, Eq. (34)"},{"comment":"Theorem 4.2 does not establish an upper bound on the duality gap of the original problem. The proof defines G(λ*) in Eq. (38) using π*, the optimal policy of the primal problem, and sets λ*i=φ'τ(gCi(π*)) in Eq. (36) without showing that this λ* is a feasible dual solution or that G(λ*) is the value of the Lagrangian dual of problem (1). The quantity −Σλ*i gCi(π*)+ηΣE[Iπ*] is then bounded by m/τ+ηmImax, but this is a bound on a specially constructed expression, not on the true Lagrangian dual gap. Moreover, since η=ω·Gmax/Imax+ε, the term ηmImax is essentially fixed by the definition of η, so the claimed dependence of the gap on τ is not the only parametric dependence. The result as stated is therefore not a duality-gap theorem for the PCPO update.","section":"§4.3, Appendix C, Eqs. (36)–(40)"},{"comment":"The theoretical results analyze the exact argmax of G(πθ) under the KL constraint in Eq. (9), but the implemented algorithm updates θ by a single natural-gradient step on a linear objective with a quadratic KL approximation, Eq. (21). No bound is given on the approximation error between the exact solution of Eq. (9) and the one-step update, nor on the effect of the Fisher-matrix regularization Ĥ=H+λI. Theorems 4.2 and 4.3 and Proposition 4.4 therefore do not apply to the evaluated algorithm unless an additional approximation-error bound is supplied. This is a load-bearing gap between theory and implementation, not merely a presentation issue.","section":"§5.1, Eqs. (9), (17)–(21)"},{"comment":"Theorem 4.3's proof contains algebraic inconsistencies that affect the stated lower bound. In Eq. (61), the second case is −τmdi − (√(2δγ)/(1−γ)²) Σ(επk+1 + τ επk_Ci + τ επk+1_Ci), while Eq. (62) and the theorem's second case claim −mdi + η(πk+1) + Σ(ηCi(πk)+ηCi(πk+1)) − ηΣImax. The factor τ on mdi and on the ε-terms disappears without explanation, and the signs of the ηCi terms are inconsistent with their definition as negative quantities. In the first case, Eq. (54) has a log argument of the form 1+..., while the theorem uses log((2−di)/(2ηCi(πk+1))), a different expression. Because the proof does not connect these expressions, the lower bound in Eq. (13) is not established as stated.","section":"Appendix D, Eqs. (61)–(62) and Theorem 4.3, Eq. (13)"}],"minor_comments":[{"comment":"The text after Proposition 4.4 refers to 'Proposition 4' instead of 'Proposition 4.4'; the appendix heading has the same mismatch.","section":"§4.3"},{"comment":"In the CarCircle1-v0 row, the EPO return is written as '16 , 43 ± 0.58', which appears to be a typo for '16.43 ± 0.58'.","section":"Table A3"},{"comment":"The gating function σ(α(δ+gCi(πθ))) and the subsequent text 'Change the triggering condition ... to δ+gCi(πθ)≥0' are unclear: it is not stated how the gating condition in the equation is reconciled with the textual activation condition.","section":"§4.2, Eq. (8)"},{"comment":"Algorithm 2 includes a backtracking line search to enforce the KL constraint, while Algorithm 1 and Section 5.1 describe the update as the direct closed-form natural-gradient step in Eq. (21). The relationship between these two update procedures should be clarified.","section":"Algorithm 2, line 10"},{"comment":"The Holder-inequality step is written imprecisely: the expectation E[A^{πk+1}_{Ci}] is bounded as if it were a norm of a vector, and the intermediate notation dπk+1−dπk is not defined as a vector here. The intended appeal to Lemma/Corollary 3 of [14] should be stated explicitly.","section":"Appendix D, Eq. (53)"},{"comment":"The sensitivity analysis for ω reports that performance is robust, but the figure axis labels appear to show only the values 0.1, 0.3, 0.5; the caption and text should state the full range of tested values.","section":"Appendix F.4 and Figure A5"}],"recommendation":"reject","confidential_remarks":"The paper's theoretical core—the claimed guarantees of a small duality gap and of asymptotically lower cumulative constraint violations—contains non-sequiturs and circular assumptions that cannot be repaired by local rewording. The strongest claim, Proposition 4.4, would require a substantially different proof strategy and additional assumptions on the compared Lagrangian baseline. Because the central contribution is not sound as stated and the theory is not connected to the implemented algorithm, I recommend rejection. A resubmission that either supplies a correct proof of a weaker guarantee or reframes the contribution as purely empirical could be reconsidered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the algorithmic recipe is a reasonable and timely idea for safe RL, and the experiments show it is competitive and often more stable than Lagrangian baselines. But the theory is not sound as written. Proposition 4.4, the formal statement of the headline claim that PCPO provably has fewer cumulative violations than Lagrangian methods, is a non-sequitur. As-is, reject; with the theory cut down or fixed, there is a decent empirical paper here.\n\nWhat's genuinely good: the motivation is real. Lagrangian methods react post-violation and can oscillate; a preemptive barrier is a sensible fix. The extended log-barrier (Eq. 5) is clean, the boundary-gated intrinsic reward is a plausible way to focus exploration near unsafe boundaries, and the ablation and sensitivity studies are more thorough than typical for this area. The Safety Gymnasium results, though based on only 6 seeds with no released code, show a consistent stability advantage.\n\nWhere it falls apart: Proposition 4.1's proof assumes g_Ci(pi_{k+1}) <= g_Ci(bar pi_{k+1}) — that is exactly what the intrinsic reward is supposed to deliver. Theorem 4.2 defines lambda* as the barrier derivative and then bounds -lambda_i g_Ci; that gives a bound on the barrier objective's gap, not on the duality gap of the original CMDP. Theorem 4.3's appendix derivation has algebra errors and unproven intermediate bounds. Proposition 4.4 is the worst: the proof shows only that a lower bound on the augmented-objective improvement for PCPO exceeds a lower bound for Lagrangian methods. A larger lower bound on G = reward - barrier + intrinsic reward says nothing about ordering raw cumulative costs J_C. The step from Eq. (74) to Eq. (76) is asserted, not derived, and Eq. (80), that Lagrangian methods have non-vanishing average violation in the limit, is also asserted. So the central 'provably fewer violations, gap growing to infinity' claim is unsupported.\n\nThere is also an approximation mismatch: the theory analyzes the exact argmax of G under a KL constraint, while the implementation takes one natural-gradient step on a linear-quadratic approximation. No approximation error bound is given, so Theorems 4.2, 4.3 and Proposition 4.4 do not strictly apply to the algorithm that was run.\n\nRecommendation: send it to reviewers. The idea is worth refereeing and the empirical study gives a concrete basis for assessment. The revision required is major: either actually prove Proposition 4.4 from the real update, or drop the 'provably fewer violations' language and present PCPO as a well-motivated heuristic with strong empirical stability. A serious reviewer should also push for released code and more than 6 seeds.","headline":"Promising algorithm, broken theory: the empirical stability of the barrier-plus-intrinsic-reward recipe deserves a look, but the 'provably fewer violations' claim does not follow from the proofs.","tokens_in":24511,"tokens_out":4037,"would_cite":false,"duration_ms":44064,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PCPO claims that embedding a log-barrier penalty into the safe-RL objective, instead of penalizing violations after they happen, provably reduces constraint violations and stabilizes training, with a duality-gap bound that shrinks as the…","keywords":["safe reinforcement learning","constrained policy optimization","preemptive penalty","log-barrier function","duality gap","intrinsic reward","constrained Markov decision process"],"falsifier":"Run PCPO on a small constrained MDP where the exact argmax of Eq. (9) under the KL constraint can be computed, and compare the implemented update of Eq. (21) with that argmax; if on any such instance the empirical cumulative constraint violation over a long horizon fails to stay below a Lagrangian baseline, then Proposition 4.4's unbounded gap does not hold for the implemented algorithm.","tokens_in":23434,"feed_emoji":"🛡️","tokens_out":9565,"duration_ms":102184,"temperature":0.7,"pith_summary":"Safe reinforcement learning usually enforces constraints by punishing violations after they occur, which can make the policy oscillate and overshoot. This paper proposes PCPO, which instead embeds an extended log-barrier term into the optimization objective so the policy feels a penalty before it reaches the constraint boundary, and adds a constraint-aware intrinsic reward that activates only near the boundary. The paper proves that the barrier formulation has a duality gap bounded by $m/\\tau + \\eta \\sum_i I_i^{\\max}$, that each PCPO update improves the objective by at least an explicit lower bound, and that cumulative constraint violations stay below those of Lagrangian methods by a gap that grows without bound as training continues. In the paper's experiments on velocity and navigation tasks, PCPO reports higher reward with lower and more stable cost than the compared safe-RL baselines.","feed_headline":"Penalty before the violation beats Lagrangian safe RL","feed_subtitle":"PCPO embeds a log-barrier cost that activates near the safety threshold, with duality-gap and improvement bounds to back it.","key_machinery":"The load-bearing object is the extended log-barrier function $\\varphi_\\tau(g) = -\\frac{1}{\\tau}\\log(-g)$ for $g \\leq -1/\\tau^2$, continued linearly as $\\tau g - \\frac{1}{\\tau}\\log(1/\\tau^2)+\\frac{1}{\\tau}$ for $g > -1/\\tau^2$. Its derivative is strictly positive as the constraint slack approaches zero from below, so the policy is pushed away from the boundary before violating; the same derivative yields the implicit dual variable $\\lambda^*_i=\\varphi'_\\tau(g_{C_i}(\\pi^*))$. The constraint-aware intrinsic reward $I^{\\pi}_{C_i}$ gates a normalized cost advantage by a sigmoid activated near the boundary, and the practical update is a natural-gradient step under a KL trust region using a regularized Fisher information matrix.","core_discovery":"The central claim is that a preemptive log-barrier penalty can enforce safety in constrained policy optimization without sacrificing performance, where Lagrangian methods cannot. Concretely, PCPO maximizes $G(\\pi_\\theta)=f(\\pi_\\theta)-\\sum_i \\varphi_\\tau(g_{C_i}(\\pi_\\theta))+\\eta \\sum_i I^{\\pi_\\theta}_{C_i}$, with $\\varphi_\\tau$ an extended log-barrier that is active inside the feasible region as $g_{C_i}$ approaches zero, and $I^{\\pi_\\theta}_{C_i}$ a gated intrinsic reward. Because the barrier derivative is strictly positive near the boundary, the update feels a repulsive force before violation; that derivative acts as an implicit dual variable, and Theorem 4.2 bounds the duality gap by $m/\\tau + \\eta \\sum_i I_i^{\\max}$. Theorem 4.3 gives a lower bound on the per-iteration improvement of $G$, and Proposition 4.4 asserts the cumulative-violation gap over Lagrangian methods grows to infinity with the number of iterations.","pith_inferences":["The missing bridge between theory and practice is an approximation-error bound for the natural-gradient step; adding one would turn the exact-argmax theorems into guarantees for the executed algorithm.","Reading the barrier derivative as an implicit dual variable makes PCPO a primal-dual method whose multiplier is updated by the geometry of the constraint violation, a perspective that could simplify or replace standard multiplier schedules.","The gated intrinsic reward is defined per constraint, so it could be applied to multi-constraint problems by letting each constraint drive exploration only when it is the nearest active boundary; the paper does not test this.","Proposition 4.4 predicts that Lagrangian methods' average violation does not vanish in the tail of training; a direct test is to run the Lagrangian baseline and PCPO well beyond the paper's 10-30 million sample budgets and compare violation rates."],"forward_implications":["The nonzero barrier gradient inside the feasible region means PCPO does not stall at the constraint boundary, so training should show fewer oscillations than Lagrangian or post-hoc penalty methods.","Larger $\\tau$ shrinks the duality-gap upper bound toward $m/\\tau + \\eta \\sum_i I_i^{\\max}$, giving a principled trade-off between constraint tightness and closeness to the true constrained optimum.","Theorem 4.3's explicit lower bound on per-iteration improvement provides an update acceptance test: an observed improvement below the bound signals that the trust-region or barrier parameter needs adjustment.","Proposition 4.4 implies that the cumulative-violation advantage over Lagrangian methods grows with training length, so long-horizon runs should show PCPO costs stabilizing below the threshold while Lagrangian costs keep oscillating."],"supporting_citations":[{"why":"Supplies the performance-difference and constraint-violation bounds that Theorem 4.3 and its proof rely on.","marker":"[14]"},{"why":"Provides the extended log-barrier formulation that PCPO's preemptive penalty mechanism is built on.","marker":"[27]"},{"why":"A Lagrangian and penalty baseline that PCPO compares against and Proposition 4.4 contrasts with.","marker":"[4]"},{"why":"One of the Lagrangian-based safe RL methods that Proposition 4.4 and the motivation compare against.","marker":"[8]"},{"why":"Exemplifies the post-violation hysteresis of Lagrange multiplier updates that motivates the preemptive penalty.","marker":"[9]"},{"why":"The benchmark suite used for the experiments in Section 6.","marker":"[29]"},{"why":"Constrained policy update baseline compared in the experiments.","marker":"[30]"},{"why":"First-order constrained policy optimization baseline compared in the experiments.","marker":"[31]"},{"why":"Trust-region policy optimization baseline, used as TRPOLag, compared in the experiments.","marker":"[32]"},{"why":"Exterior penalty policy optimization baseline compared in the experiments.","marker":"[33]"}],"fun_headline_variants":["Preemptive penalty stops RL violations early","Log-barrier rewards ease safe RL constraints","PCPO: stable policy optimization without overshoot","Boundary-aware RL avoids constraint breaches","Proactive barriers beat reactive Lagrangian safety"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proofs analyze the exact argmax of the barrier objective under the KL constraint, while the implemented algorithm takes a single natural-gradient step on a linear-quadratic approximation; the paper does not bound the error of that approximation, so the theorems apply to the deployed method only if the approximation is accurate.","fun_headline_variants_meta":{"raw":{"variants":["Preemptive penalty stops RL violations early","Log-barrier rewards ease safe RL constraints","PCPO: stable policy optimization without overshoot","Boundary-aware RL avoids constraint breaches","Proactive barriers beat reactive Lagrangian safety"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000157,"raw_usage":{"total_tokens":1233,"prompt_tokens":967,"completion_tokens":266,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":583,"completion_tokens_details":{"reasoning_tokens":201}},"tokens_in":583,"tokens_out":266,"duration_ms":3492,"temperature":1.0,"reasoning_tokens":201,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T05:20:18.999998+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run PCPO on a small constrained MDP where the exact argmax of Eq. (9) under the KL constraint can be computed, and compare the implemented update of Eq. (21) with that argmax; if on any such instance the empirical cumulative constraint violation over a long horizon fails to stay below a Lagrangian baseline, then Proposition 4.4's unbounded gap does not hold for the implemented algorithm.","supporting_citations":[{"cited_title":"Lyapunov-based safe policy optimization for continuous control, 2019","cited_arxiv_id":null,"evidence_quote":"One of the Lagrangian-based safe RL methods that Proposition 4.4 and the motivation compare against."},{"cited_title":"Safety gymnasium: A unified safe reinforcement learning benchmark","cited_arxiv_id":null,"evidence_quote":"The benchmark suite used for the experiments in Section 6."},{"cited_title":"Constrained update projection approach to safe policy optimization","cited_arxiv_id":null,"evidence_quote":"Constrained policy update baseline compared in the experiments."},{"cited_title":"First order constrained optimization in policy space","cited_arxiv_id":null,"evidence_quote":"First-order constrained policy optimization baseline compared in the experiments."},{"cited_title":"repulsive force","cited_arxiv_id":null,"evidence_quote":"Exterior penalty policy optimization baseline compared in the experiments."}],"review_version":1}