{"id":"492ffca2-ddb0-4d89-a280-dd6b63b8b7aa","arxiv_id":"2507.00485","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"PNAct trains a safe RL agent to take unsafe actions only when a hidden trigger is present, while keeping normal safe behavior and reward when the trigger is absent.","lead":"This paper presents PNAct, a way to secretly train safe-reinforcement-learning agents so that a specific marker in the environment makes them deliberately violate safety rules while they behave normally otherwise. It matters because it shows that adding cost limits to reinforcement learning does not by itself protect an agent from hidden backdoors, and it gives the community a concrete attack to design defenses against.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The effectiveness and stealthiness indicators in Table 1 are unreliable point estimates with very large variance and no significance testing; several configurations already violate the indicators, so the central claim is not statistically established.","rationale":"The reader's weakest assumption was that the vase-position trigger violates the B-CMDP redundancy condition P(s_{t+1}|x̃_t,a_t)=P(s_{t+1}|x_t,a_t). I agree this invalidates the theoretical model as stated, but it attacks the framing rather than the empirical result. The central claim—that the trained policy is effective and stealthy—rests on Table 1's point estimates. With variances as large as 101.86 and no significance tests, the binary indicators can flip with modest sampling variation; e.g., the AntGoal n=5 backdoor mean cost 45.53 ± 101.86 has a 95% CI that brushes the threshold κ=25. The reader did note the missing significance testing in the rationale, but identified the trigger redundancy as the single weakest assumption. I consider the statistical reliability of the indicators to be more load-bearing because it directly determines whether the attack works as claimed; if the confidence intervals reveal no significant difference, the central claim is unsupported even if the trigger were perfectly redundant. The reward degradation in Table 2 further weakens the 'does not affect rewards' statement. The proposed re-run with seeds and confidence intervals would settle the issue. Since the reader's verdict already conditions acceptance on improved statistical evidence, my concern does not change the verdict category, so I keep UNCHANGED while partially disagreeing about which assumption is weakest.","tokens_in":14196,"tokens_out":9314,"duration_ms":115153,"concrete_test":"Using the provided code, re-run the evaluation for every configuration in Table 1 (3 environments × 3 safe policies × 5 values of n) with at least 10 independent seeds and 100 episodes per seed. For each configuration, compute 95% bootstrap confidence intervals for (i) the mean cost in backdoor states, (ii) the backdoor–normal cost difference, and (iii) the normal-state reward compared with the corresponding base safe policy in Table 2. The central claim holds for a configuration only if the CI for the backdoor mean cost lies entirely above κ and the CI for the cost difference excludes 0; otherwise the indicators I^E and I^S are not statistically established and the headline claim must be qualified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 1 is the sole evidence for the central claim that PNAct satisfies the effectiveness and stealthiness indicators I^E and I^S defined in Section 4.2. These indicators are evaluated as point estimates of the mean cost and reward over 100 episodes, with no confidence intervals, hypothesis tests, or multiple-seed aggregation. Reported standard deviations are enormous: for AntGoal with n=5 and PPO-Lag, the backdoor cost is 45.53 ± 101.86, while the cost threshold is κ≈25; CarGoal n=5 PPO-Lag shows 47.8 ± 43.47. The standard error of the mean is roughly 4–10, so the 95% CI for the backdoor mean cost typically spans or touches κ, and the backdoor–normal cost difference can easily include 0. Moreover, several rows in Table 1 already have I^E=0 or I^S=0 (e.g., n=25 PointGoal PPO-Lag, n=20 AntGoal RCPO), so the method does not universally meet its own criteria. A secondary issue is that Table 2 shows PNAct's normal-state reward is sometimes lower than the corresponding clean safe policy (e.g., PointGoal PPO-Lag n=5: 8.09 vs 10.33), contradicting the abstract's claim that the attack 'does not affect the rewards obtained by the agent.' Thus the central claim is not statistically supported as reported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PNAct, a backdoor attack framework for Safe Reinforcement Learning. It defines a Backdoored CMDP (B-CMDP) model, formal properties of backdoor policies, and effectiveness/stealthiness metrics, then presents an algorithm that trains a policy to imitate an unsafe policy in trigger states and a safe policy in normal states using positive and negative action samples. Experiments on Safety-Gymnasium (CarGoal, PointGoal, AntGoal) with PPO-Lag, TRPO-Lag, and RCPO as safe base policies and PPO as the unsafe base policy report attack effectiveness and stealthiness indicators for attack intervals n = 5, 10, 15, 20, 25.","tokens_in":14438,"tokens_out":4922,"duration_ms":54521,"significance":"If the results hold, the paper would be a useful contribution to Safe RL security: it proposes evaluation metrics, a general attack framework that does not require per-scenario logic specifications, and an open-source implementation. The attack targets safety constraints rather than merely degrading rewards, which is a meaningful direction for future defenses. The experimental matrix across three environments and three safe-RL algorithms is a strength, as is the availability of code. However, the current evidence does not establish the central claims: the theoretical model is inconsistent with the demonstrated trigger design, and the empirical indicators are point estimates with large variance and multiple configurations already violating the claimed criteria.","major_comments":[{"comment":"The B-CMDP model assumes that triggers are redundant and do not change state-transition probabilities, as stated in Section 4.1: P(s_{t+1}|x̃_t,a_t) = P(s_{t+1}|x_t,a_t). The experiments, however, use the absolute positions of five vases as triggers, and vases are physical obstacles that 'block its view and movement' (Section 5.1). Changing vase positions changes the environment layout and therefore the dynamics. Consequently, the disjointness property X ∩ X̃ = ∅ and the subsequent theoretical properties derived from the redundant-trigger assumption do not formally apply to the evaluated attack. This is a load-bearing mismatch because the paper claims theoretical grounding for PNAct. The authors should either use a trigger that is provably observation-only (e.g., a color patch that does not alter dynamics) or generalize the B-CMDP framework to allow P(s_{t+1}|x̃_t,a_t) ≠ P(s_{t+1}|x_t,a_t) and re-derive the properties.","section":"Section 4.1 vs. Section 5.1"},{"comment":"The effectiveness and stealthiness indicators I^E and I^S are point estimates computed from 100 episodes with no confidence intervals, hypothesis tests, or per-seed aggregation. The reported standard deviations are enormous; for example, CarGoal n=5 PPO-Lag shows normal cost 23.2±31.35 and backdoor cost 47.8±43.47, and AntGoal n=5 PPO-Lag shows backdoor cost 45.53±101.86 with κ≈25. The standard error of the mean is roughly 2–10, so 95% confidence intervals often overlap the cost threshold κ, and the backdoor–normal cost differences can include zero. Moreover, the table already contains configurations where the indicators are 0, such as PointGoal n=25 PPO-Lag (I^E=0, I^S=0), AntGoal n=20 RCPO (I^E=0), and AntGoal n=25 TRPO-Lag and RCPO (I^E=0). The claim that the attack 'generally' meets effectiveness and stealthiness is therefore not statistically supported. The authors should report per-seed results, significance tests, or bootstrap confidence intervals, and explicitly quantify how many configurations satisfy the indicators.","section":"Table 1, Section 5.2"},{"comment":"The paper asserts that minimizing the action-sample loss L_act satisfies the KL-divergence objective in Eq. (9), but no proof or derivation is provided. The surrogate loss is not equivalent to matching π*_u in backdoor states: the negative-sample term maximizes J(π(s_t), a^-_t), which pushes the policy away from π*_s but does not by itself guarantee convergence to π*_u, and the weighting λ in Eq. (16) introduces a trade-off that is absent from Eq. (9). Because the theoretical properties of PNAct are a stated contribution, the authors should either prove the relationship formally or state precisely the assumptions under which the surrogate is an approximate solution to Eq. (9).","section":"Section 4.3, Eqs. (13)–(14) and Remark 1"},{"comment":"The abstract and Section 1 claim that the attack 'does not affect the rewards obtained by the agent.' Table 2 shows that PNAct's normal-state reward is often lower than the corresponding clean safe policy, for example PointGoal PPO-Lag n=5 normal reward 8.09 vs. the base PPO-Lag normal reward 10.33, and similarly for n=10 (8.34 vs. 10.33) and n=15 (7.38 vs. 10.33). Definition 2's stealthiness indicator only requires V_r(backdoor) ≥ V_r(normal), which does not test reward preservation relative to a clean policy. The claim should be either withdrawn or evaluated against clean baselines with appropriate statistical tests.","section":"Abstract, Section 1, and Table 2"}],"minor_comments":[{"comment":"The pseudocode for the if-then-else construct is malformed: lines 4–9 place 'end' before the 'else', which is not valid pseudocode. The intended conditional should be written as 'if ψ_t then ... else ... end'.","section":"Algorithm 1"},{"comment":"The distribution analysis states that cumulative rewards and costs are sampled from '1000 rounds' of the PNAct model, while Table 1 says each experiment averages over 100 complete episodes. These numbers should be reconciled.","section":"Section 5.2"},{"comment":"The cost threshold κ is never explicitly defined in the training configuration. Figure 5 uses a dashed line at y = 25, and Table 1's indicators presumably use κ = 25, but the value should be stated in the text before the results are presented.","section":"Section 5.1"},{"comment":"There are several typographical issues: 'Yanget al.' in Section 1, the grammatically incomplete sentence in the abstract ('It is the first attack framework ... is to implant backdoors'), and inconsistent use of 'as,t' vs. 'a_s,t' in Algorithm 1.","section":"Global"},{"comment":"The table header 'n Safe RLPolicy' is ambiguous; clarifying that n is the attack interval and that the second column lists the safe RL algorithm used for the safe base policy would improve readability.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a relevant security question for Safe RL and the code availability is a plus, but the reviewer's assessment is that the central empirical and theoretical claims require substantial revision. The trigger-redundancy mismatch and the lack of statistical testing are the primary blockers; both are potentially fixable within the scope of the manuscript. I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper introduces a real new attack idea for Safe RL backdoors, but the empirical support is weaker than the paper claims, and the theory section is more declarative than proven.\n\nThe genuinely new piece is PNAct's use of positive and negative action samples from pre-trained safe and unsafe policies, switched by a state-conditioned signal. That is a clean departure from Jiang et al.'s STL-based method, and the claim that it applies across environments without scenario-specific logic parsers checks out in the Safety-Gymnasium setups. The B-CMDP framing, even if informal, is a useful way to state the attack objective. Credit where due: code and supplementary material are provided, the attack is evaluated on three agents and three safe-RL base algorithms, and the authors explicitly note the trade-off between attack frequency and effectiveness.\n\nThe soft spots are real and load-bearing. Table 1 is the only systematic evidence, and the point estimates come with enormous standard deviations (e.g., cost 45.53 ± 101.86 with threshold 25). No confidence intervals, no significance tests, no multi-seed aggregation. Several rows already have I^E=0 or I^S=0, so the claim that PNAct 'generally' meets the metrics is not established by the data. The stress-test note is correct that the backdoor–normal difference is often within noise. Also the abstract's claim that the attack 'does not affect rewards' is contradicted by Table 2 in at least one configuration (PointGoal PPO-Lag n=5: 8.09 vs 10.33 normal reward). That is a minor overstatement but should be fixed.\n\nThe theory section is the other soft spot. Properties 1 and 2 are stated without proof; they read as definitions or desiderata rather than theorems. The surrogate loss in Eq. (13) is not shown to implement the KL objective in Remark 1. And the B-CMDP assumption that triggers are redundant conflicts with the actual trigger, which is the absolute positions of five vases—changing vase positions changes the transition dynamics. So the formalism does not apply to the demonstrated attack as written. This is fixable by either redefining the trigger to something truly observation-only or by relaxing the redundancy assumption.\n\nWho is this for: RL security researchers and anyone working on safe RL deployment. The idea deserves discussion even if the current evidence is thin. I'd send it to a serious referee, but the referee should require multi-seed runs with confidence intervals, a proof or clear statement of what is assumed rather than proven, and reconciliation of the trigger model with the experiments. Right now it is a promising proof-of-concept, not a fully supported claim.","headline":"PNAct is a genuine new attack idea for Safe RL, but the statistical evidence is too weak to support the paper's central claims, and the theory overreaches.","tokens_in":15028,"tokens_out":2045,"would_cite":false,"duration_ms":21437,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a backdoor attack framework called PNAct can train Safe RL agents to take unsafe actions only when specific trigger states appear, while preserving rewards and safe behavior otherwise.","keywords":["backdoor attacks","safe reinforcement learning","constrained Markov decision process","policy poisoning","trigger states","effectiveness and stealthiness metrics","positive and negative action samples"],"falsifier":"Measure the next-state distribution under the same action with and without the vase-position trigger; if $P(s_{t+1} \\mid \\tilde{x}_t, a_t)$ differs from $P(s_{t+1} \\mid x_t, a_t)$, the B-CMDP redundancy assumption fails. Alternatively, retrain PNAct with a trigger that only overlays a pattern in the observation without changing the environment layout; if the backdoor no longer takes effect, the demonstrated attack relies on dynamics changes the model rules out.","tokens_in":13918,"feed_emoji":"🚩","tokens_out":5272,"duration_ms":57953,"temperature":0.7,"pith_summary":"The paper claims that Safe Reinforcement Learning is vulnerable to a new kind of backdoor attack: an agent trained by an attacker will behave safely and earn normal rewards in ordinary states, but will take unsafe, constraint-violating actions whenever the environment contains a specific trigger. To make this precise, the authors introduce the Backdoored CMDP (B-CMDP) model and two evaluation metrics, effectiveness and stealthiness, which measure whether the backdoored policy violates cost constraints under the trigger and whether it preserves reward and safety otherwise. The proposed framework, PNAct, trains the victim policy using positive action samples that pull it toward the desired safe or unsafe action depending on the state, and negative action samples that push it away from the opposite behavior. Experiments in safety-constrained continuous-control tasks are reported as supporting the claim that PNAct meets both metrics across most tested configurations. If true, this matters because deployed safe RL systems could be silently compromised without any drop in reward that would alert a monitoring user.","feed_headline":"Safe RL agents can be backdoored to take risky actions","feed_subtitle":"A new attack trains safe-RL agents to obey constraints normally, then violate them when a trigger appears in the environment.","key_machinery":"The load-bearing mechanism is the pair of positive and negative action samples: when the trigger is present, a positive sample is drawn from the optimal unsafe policy and a negative sample from the optimal safe policy, and the roles are reversed in normal states. The policy is trained to minimize $\\lambda J(\\mathring{\\pi}(s_t), a_t^+) - (1-\\lambda) J(\\mathring{\\pi}(s_t), a_t^-)$, which pulls it toward the desired action and away from the undesired one. This is combined with reward and cost value critics, updated with a Lagrangian-style squared temporal-difference loss, and an attack signal generator $u(f,k,t)$ that controls how often and for how long trigger states appear during training. The B-CMDP model supplies the formal setting by assuming triggers are redundant in state transitions and that normal and backdoor states are disjoint.","core_discovery":"The paper's central claim is that a backdoor policy can be trained to behave as the optimal safe policy in normal states and as the optimal unsafe policy in backdoor states, so that it maximizes reward while violating safety constraints exactly when a trigger is present. The authors formalize this as a property of the policy's action distribution and value functions, and they propose PNAct as a concrete training procedure. PNAct switches positive and negative action samples according to an attack signal, and its loss combines an action-sample loss with reward and cost critic losses in a Lagrangian-style actor-critic update. In the paper's experiments, the trigger is the positioning of objects in the environment, and the trained policy is reported to keep costs below the safety threshold in normal episodes while pushing costs above it in triggered episodes, with rewards at least as high when triggered.","pith_inferences":["The paper's experiments use the absolute positions of vases as the trigger, which changes the physical layout the agent must navigate; if that layout change alters transition probabilities, the formal redundancy assumption of B-CMDP is not satisfied by the demonstrated attack.","A direct implication the paper does not develop is that defending against this attack requires monitoring cost conditioned on environmental configurations, not just aggregate reward or cost statistics.","The proposed positive and negative action-sample loss could be adapted to other safety specifications, such as risk-sensitive or temporal-logic constraints, by redefining which actions count as safe and unsafe.","A natural testable extension is to check whether the attack remains effective when the trigger is observation-only and provably does not change the environment dynamics, which would separate the framework's theory from its demonstration."],"forward_implications":["A poisoned Safe RL policy can violate safety constraints at attacker-chosen moments while appearing normal the rest of the time.","Because the attack does not lower rewards, a defender who monitors only task performance will likely miss it.","The framework is stated to generalize across environments without scenario-specific trigger specifications, unlike prior STL-based Safe RL backdoors.","Existing backdoor defenses designed for reward-decreasing attacks are said not to apply, since this attack increases cost rather than decreasing reward."],"supporting_citations":[{"why":"defines the Constrained Markov Decision Process that B-CMDP extends with normal and backdoor states.","marker":"[Altman, 1998]"},{"why":"supplies the Safety-Gymnasium benchmark environments used for all experiments.","marker":"[Ji et al., 2023]"},{"why":"RCPO is one of the safe RL algorithms used to train the optimal safe policy that provides positive and negative action samples.","marker":"[Tessler et al., 2019]"},{"why":"TrojDRL establishes the general backdoor-attack setting in deep RL that PNAct adapts to Safe RL.","marker":"[Kiourti et al., 2020]"},{"why":"presents the prior STL-based backdoor attack in Safe RL that this paper contrasts as less general.","marker":"[Jiang et al., 2024]"}],"fun_headline_variants":["Backdoor attacks make safe RL agents unsafe on trigger","Safe RL vulnerable to backdoor attacks that flout safety","PNAct: Hidden trigger turns safe RL policies risky","Backdoor in safe RL: normal safe, triggered unsafe","Safe RL agents can be hijacked to violate constraints"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The theory assumes the trigger adds redundant information that leaves the environment's transition dynamics untouched, but in the experiments the trigger is moving vases to fixed positions, which alters the physical layout the agent must navigate.","fun_headline_variants_meta":{"raw":{"variants":["Backdoor attacks make safe RL agents unsafe on trigger","Safe RL vulnerable to backdoor attacks that flout safety","PNAct: Hidden trigger turns safe RL policies risky","Backdoor in safe RL: normal safe, triggered unsafe","Safe RL agents can be hijacked to violate constraints"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000351,"raw_usage":{"total_tokens":1895,"prompt_tokens":905,"completion_tokens":990,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":911}},"tokens_in":521,"tokens_out":990,"duration_ms":7757,"temperature":1.0,"reasoning_tokens":911,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:14:36.631511+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the next-state distribution under the same action with and without the vase-position trigger; if $P(s_{t+1} \\mid \\tilde{x}_t, a_t)$ differs from $P(s_{t+1} \\mid x_t, a_t)$, the B-CMDP redundancy assumption fails. Alternatively, retrain PNAct with a trigger that only overlays a pattern in the observation without changing the environment layout; if the backdoor no longer takes effect, the demonstrated attack relies on dynamics changes the model rules out.","supporting_citations":[{"cited_title":"Constrained markov decision processes with total cost criteria: Lagrangian approach and dual linear program","cited_arxiv_id":null,"evidence_quote":"defines the Constrained Markov Decision Process that B-CMDP extends with normal and backdoor states."},{"cited_title":"Mankowitz, and Shie Mannor","cited_arxiv_id":null,"evidence_quote":"RCPO is one of the safe RL algorithms used to train the optimal safe policy that provides positive and negative action samples."},{"cited_title":"Trojdrl: Evaluation of back- door attacks on deep reinforcement learning","cited_arxiv_id":null,"evidence_quote":"TrojDRL establishes the general backdoor-attack setting in deep RL that PNAct adapts to Safe RL."},{"cited_title":"Backdoor attacks on safe reinforcement learning- enabled cyber–physical systems","cited_arxiv_id":null,"evidence_quote":"presents the prior STL-based backdoor attack in Safe RL that this paper contrasts as less general."}],"review_version":1}