{"id":"ed31338e-a05f-41c7-9a6f-4def5b772f72","arxiv_id":"2501.15529","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":12,"one_line_summary":"UNIDOOR adaptively adjusts the backdoor reward using monitored benign performance and attack success rate, enabling action-level backdoor attacks across discrete and continuous DRL tasks without task-specific reward tuning.","lead":"UNIDOOR is a framework that adaptively tunes the backdoor reward during training so poisoned deep reinforcement learning agents learn trigger-to-action bindings across many different tasks. It removes the need to hand-pick a fixed reward value, and adds action tampering so continuous-action agents can be backdoored reliably.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Performance Monitoring sums rewards from the same replay buffer that Transition Poisoning rewrites with ±r†, so the BTP control signal may track backdoor-reward artifacts rather than true benign-task performance; the 'adaptive via performance monitoring' claim needs this signal to be valid.","rationale":"The paper's strongest claim is \"universal\" action-level backdoor attacks achieved by \"adaptive exploration of backdoor reward functions through performance monitoring,\" eliminating expert knowledge and grid search. For that claim to hold, the BTP and ASR signals must be faithful estimates of benign and backdoor performance. The ASR probe is online and faithful. The BTP estimator is not obviously faithful: it is read from the same replay buffer whose contents Algorithm 4 changes. If it reads only benign trajectories, the paper should say so and the algorithm should filter poisoned entries; it does not. This is load-bearing because every reward update in Adaptive Exploration is a function of Pt. The ablation (Table 4) shows Adaptive Exploration matters, so a corrupted Pt would mean the mechanism may be effective for reasons other than the stated performance-monitoring rationale. The reader's t_n/t_b concern is real but softened by the 20% robustness ablation; the contamination concern is not addressed anywhere in the manuscript. I would keep the CONDITIONAL verdict: the empirical evaluation is substantial, but acceptance should require demonstrating that monitored BTP is computed from unpoisoned trajectories or that poisoning-induced bias does not change the adaptation decisions.","tokens_in":28081,"tokens_out":8775,"duration_ms":85194,"concrete_test":"Inspect the public repository (github.com/maoubo/UNIDOOR) to see how Algorithm 1 samples trajectories; specifically whether BTP sums rewards from transitions already altered by Algorithm 4. Then run the paper's Gym/MuJoCo experiments under two monitoring variants: (a) BTP as currently implemented from the poisoned replay buffer; (b) BTP computed only from unpoisoned trajectories (e.g., a separate benign replay buffer or pre-poisoning rollouts). Compare final CP and the sequence of r† updates. If variant (b) produces materially different CP or different reward-adjustment decisions (e.g., r† moves opposite), the monitoring signal is contaminated and the adaptive-exploration claim needs to be re-scoped. Also compute the correlation between monitored BTP and the true evaluation BTP across a grid of r† to test the Fig. 4 premise.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"UNIDOOR's central claim is that Performance Monitoring supplies a task-agnostic BTP signal on which Adaptive Exploration can rely (Sec. 4.2-4.5). But Algorithm 4 replaces transitions in the victim's replay buffer with (s̃, ã, r̃), r̃ = ±r†, and Algorithm 1 computes episode-wise BTP as Σ r_i from \"trajectories and transitions stored in the victim's replay buffer.\" Once Transition Poisoning is active, the monitored BTP therefore contains the injected backdoor rewards, whose magnitude and sign depend on r† and on the action-tampering schedule. The observed \"BTP decreases as backdoor reward increases\" (Fig. 4) can then be an artifact of summing large ±r† terms, not evidence that the policy's true benign performance is degrading. Since Adaptive Exploration increases r† when BTP is above expectation and decreases it when BTP falls (Eqs. 11-12, Algorithm 5), an artifact in BTP directly drives the reward adjustment. The final reported BTP (Eq. 14) is evaluated separately on the benign task, so good end-to-end results do not by themselves validate the internal monitoring signal. Additionally, the reader's concern about t_n/t_b remains, but it is secondary: even with perfect schedules, a contaminated BTP would break the controller's premise.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes UNIDOOR, a framework for action-level backdoor attacks in deep reinforcement learning. The framework combines four modules: performance monitoring (estimating benign-task performance and attack success rate from trajectories in the victim's replay buffer), initial freezing (delaying poisoning to avoid backdoor-task dominance), transition poisoning (state/action tampering plus a hackable backdoor reward), and adaptive exploration (a feedback controller that adjusts the backdoor reward based on the monitored benign-task performance and attack success rate). The authors claim that this adaptive mechanism removes the need for hand-tuned backdoor rewards or grid search, and they evaluate the framework on 11 Gym/MPE/PyBullet tasks, 53 backdoor designs, and PPO/DDPG/MADDPG, comparing against four fixed-reward baselines. They also report ablations, post-training attacks, activation-strategy demonstrations, and visualizations of stealthiness.","tokens_in":28457,"tokens_out":5385,"duration_ms":51017,"significance":"If substantiated, the adaptive-reward mechanism would be a useful advance over fixed- or hand-designed backdoor rewards, and the observation that action tampering is important in continuous action spaces is a concrete and evidently load-bearing insight supported by the ablation study. The evaluation is broad in task coverage and design diversity, and the public code release is a further strength. However, the central claim depends on the validity of the internal performance-monitoring signal: the monitored BTP is computed from the same replay buffer that transition poisoning rewrites with synthetic rewards, so the feedback controller may be reacting to artifacts rather than to true benign-task performance. Until that signal is validated, the paper's main explanatory mechanism and the 'no expert knowledge' claim remain unproven. The paper is well within the scope of a security-oriented ML venue and addresses a topic of genuine concern.","major_comments":[{"comment":"The core feedback loop appears to be contaminated by the attack itself. Performance Monitoring (Eq. 5) computes episode-wise BTP by summing rewards in trajectories from the victim's replay buffer, but Transition Poisoning (Algorithm 4) rewrites transitions in that same buffer, replacing rt with ±r†. Once poisoning is active, the monitored Pt is therefore a mixture of true benign rewards and synthetic backdoor rewards, so the claim that Adaptive Exploration tracks benign-task performance is not established. The paper never distinguishes the monitored BTP used during training from the unbiased BTP defined in Eq. (14) and used in the evaluations, and Figure 4's empirical relation is likely computed on true performance, not on the contaminated buffer signal. This is load-bearing for the paper's central mechanism. Please provide a direct validation experiment: run UNIDOOR, and plot the monitored Pt against the truly evaluated BTP (e.g., Eq. 14) over the course of training, both with and without excluding poisoned transitions from the monitoring trajectory. If the contaminated signal does not track true BTP, either modify the monitoring to use unpoisoned trajectories or revise the claim that the controller is adapting to benign-task performance.","section":"Sec. 4.2 (Algorithm 1), Sec. 4.4 (Algorithm 4), and Sec. 4.5 (Eqs. 11–12)"},{"comment":"The claim that UNIDOOR 'eliminates the reliance on expert knowledge and grid search' is stronger than the method supports. The adversary must specify the expected convergence times tn and tb for the benign and backdoor tasks, the thresholds φt and φp, the smoothing factor β, the lower/upper reward bounds rl and ru, and the exploration step ω, plus the poisoning interval I_p. Although the ablation in Fig. 9(c) shows robustness to ±20% errors in tn and tb, the need for schedule estimates is a form of task-specific prior knowledge. Please weaken the claim to 'reduces reliance on expert knowledge and avoids grid search' or provide evidence that UNIDOOR succeeds when tn and tb are not provided at all (e.g., fully adaptive schedule-free variants).","section":"Sec. 4.5, Eqs. (11)–(12), and Abstract"},{"comment":"The primary results are averages over only three random seeds, and the tables report no standard deviations or confidence intervals for BTP or ASR (only the CP standard deviation is given in the text). Given the inherent variance of DRL training, the statements that UNIDOOR achieves 'top-1 CP in 84.6% of scenarios' or improves ASR/CP by specific percentages need per-metric error bars or a significance test to be reliable. Please add seed-wise standard deviations (or bootstrap intervals) for BTP, ASR, and CP, or justify why three seeds are sufficient for the claimed effect sizes.","section":"Sec. 6, Tables 1–3"}],"minor_comments":[{"comment":"The action tampering frequency f is an important parameter in Algorithm 4, but no value is given in the implementation details in Sec. 5. Please specify it.","section":"Sec. 4.4, Algorithm 4, and Sec. 5"},{"comment":"Eq. (7) normalizes BTP using Pl and Pu, but the clipping to [0,1] is only mentioned in Appendix B; consider stating this explicitly in Sec. 4.2 to avoid the implication that Pt always lies in [0,1] by construction.","section":"Sec. 4.2 and Appendix B"},{"comment":"The y-axis label 'Performance' in Figure 4 is ambiguous; clarify that the curves represent the true evaluated BTP and ASR from the empirical study, not the monitored values from the replay buffer.","section":"Figure 4 and Sec. 4.5"},{"comment":"The notation BTP and ASR is used both for the monitored estimates in Sec. 4.2 and for the unbiased evaluation metrics in Sec. 5, which is confusing. Consider distinguishing them, e.g., with hats (\\(\\widehat{\\mathrm{BTP}}\\)) for the monitored quantities.","section":"Sec. 4.2 vs. Sec. 5"},{"comment":"The paper mentions that Chen et al. [11] proposed a dynamic backdoor reward mechanism, but that method (MARNet) is not included in the comparison tables. Please state why it is excluded, or add a comparison, since it is the most closely related adaptive-reward work.","section":"Sec. 2.2 and Sec. 5"},{"comment":"Typo: 'szie' should be 'size'.","section":"Algorithm 5, line 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is suitable for the journal's security/ML scope, and the code release is a strong point. The main risk is that the adaptive controller's monitoring signal is confounded by the poisoning itself; this is a scientific issue that a targeted experiment can resolve, but it must be resolved before the central claim is accepted. The 'eliminating expert knowledge' phrasing should also be toned down or better evidenced."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"UNIDOOR is a real step forward for action-level backdoor attacks in DRL. The adaptive exploration of the backdoor reward, driven by monitored BTP and ASR, is new, and so is the action-tampering component for continuous action spaces. The evaluation is broad: 11 tasks, 53 backdoor designs, 3 algorithms, with ablations showing that removing adaptive exploration or action tampering substantially hurts CP. The paper also releases code and states its limitations honestly. That is worth credit.\n\nThe main soft spot is the stress-test concern, and it holds up on reading. Performance Monitoring (Algorithm 1) pulls trajectories from the victim's replay buffer, and Transition Poisoning (Algorithm 4) rewrites rewards in that same buffer to ±r†. So the monitored BTP contains the injected backdoor rewards. The controller's premise—that BTP inversely correlates with r†—may therefore be partly an artifact of the monitoring procedure rather than a reflection of true benign-task performance. The paper evaluates final BTP separately (Eq. 14), so the end-to-end numbers are credible, but the internal signal validity matters for the 'no expert knowledge / universal' claim. If the controller is reacting to artifacts, its generalization across tasks is less established than the paper suggests.\n\nThe reader's separate concern about needing t_n and t_b (expected convergence times) is real but secondary. The ablation shows some robustness to ±20% error, but the adversary still needs task-specific schedule knowledge. Three seeds with no per-cell error bars is also thin, though the cross-scenario standard deviations for CP help a little.\n\nOverall, this deserves serious peer review. The empirical core is strong and the adaptive mechanism is a useful new building block. A referee should ask for an analysis or experiment that separates monitored BTP from true benign performance, and for error bars. I would cite the framework and the action-tampering finding, but I would not treat the 'universal' claim as settled.","headline":"Solid empirical paper with a genuinely new adaptive reward mechanism; the monitoring loop may be chasing its own poisoned rewards, which tempers the 'universal' claim but not the value of the work.","tokens_in":29009,"tokens_out":3574,"would_cite":true,"duration_ms":33739,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"UNIDOOR adaptively tunes the backdoor reward during training, making action-level backdoor attacks work across DRL tasks without hand-set values.","keywords":["backdoor attack","deep reinforcement learning","action-level backdoor","reward poisoning","transition poisoning","adaptive exploration","multi-task learning","attack universality"],"falsifier":"Run UNIDOOR on a task where the benign-performance-versus-backdoor-reward curve is non-monotone, for instance a benchmark in which a medium backdoor reward selectively destroys a sub-behavior required for the benign task while a larger reward does not, and check whether the contraction phase still converges to a high attack success rate without collapsing benign performance.","tokens_in":27898,"feed_emoji":"🎯","tokens_out":7436,"duration_ms":62567,"temperature":0.7,"pith_summary":"Deep reinforcement learning policies can be hijacked by action-level backdoors: a trigger pattern in the observation is bound to a target action by rewriting the reward in poisoned training transitions. Prior attacks fix that backdoor reward to a constant or flip it conditionally, and this paper shows empirically that no static choice works across tasks, as performance fluctuates or the attack fails outright. UNIDOOR instead treats the attack as a multi-task learning problem and adapts the backdoor reward online, guided by monitored benign-task performance and attack success rate. The paper's claim is that this removes the need for expert knowledge or grid search and makes action-level backdoors universal across discrete and continuous actions, single and multi-agent settings, and sparse and dense rewards.","feed_headline":"A self-tuning backdoor reward makes DRL attacks work across tasks","feed_subtitle":"Fixed backdoor rewards often fail; monitoring benign performance and attack success and adapting the reward online fixes that.","key_machinery":"The load-bearing mechanism is the Adaptive Exploration module, a one-dimensional interval search over the backdoor reward $r^\\dagger$. The framework maintains an interval $[r_l, r_u]$ and a current reward $r^\\dagger$; in the expansion phase it raises the upper bound when benign performance is ahead of expectation but attack success lags, and in the contraction phase it narrows the interval by moving $r_u$ or $r_l$ to the current reward depending on which signal falls below its conservative expectation. Those expectations, $E_t$ and $E_t^\\dagger$, are time-dependent ramps keyed to an estimated convergence time for each task. Everything else in UNIDOOR supports this search: Performance Monitoring produces the normalized BTP and ASR inputs, Initial Freezing delays poisoning, and Transition Poisoning tampers with states, actions, and rewards in the replay buffer.","core_discovery":"The central claim is that the backdoor reward can be discovered adaptively rather than hand-set, and that this one change universalizes action-level backdoor attacks. UNIDOOR monitors the victim through two normalized signals: benign task performance (BTP) and attack success rate (ASR), both smoothed by exponential weighted averaging. A short Initial Freezing phase delays poisoning so the backdoor task does not dominate early training, and Transition Poisoning then rewrites states, actions, and rewards in the victim's replay buffer, adding noise to target actions in continuous spaces. The key relationship the framework exploits is that BTP falls and ASR rises as the backdoor reward increases. Adaptive Exploration uses that relationship to run a bracketed search over an interval of backdoor rewards, expanding the interval while the backdoor task converges and then contracting it until a working reward is isolated. The paper reports that this yields higher comprehensive performance than fixed-reward baselines across 11 tasks, 53 backdoor designs, and three DRL algorithms, and that inactive backdoors are stealthy under state-distribution and neuron-activation comparisons.","pith_inferences":["The 'no expert knowledge' claim is bounded by the schedule estimates $t_n$ and $t_b$; a defender who randomizes training duration would make the conservative expectation harder to set, so the search would likely need more adjustments or fail. This is an inference about the attack's practical boundary, not a claim in the paper.","The monotone relationship between backdoor reward and the two performance signals may break at extreme rewards, where the backdoor task dominates and collapses the shared policy; a useful stress test is to run UNIDOOR on tasks with clipped rewards or bounded episode returns.","The same adaptive reward search could be run defensively as a cheap probe to estimate how vulnerable a given DRL policy is to action-level backdooring.","Framing injection as a multi-task trade-off suggests that treating benign performance and attack success as competing objectives, rather than via a one-dimensional interval search, could improve sample efficiency and stability."],"forward_implications":["An adversary with only replay-buffer write access can inject a working action-level backdoor into a DRL policy without knowing the task's reward scale in advance.","Continuous-action policies are not safe by default: action tampering with noise lets the attack succeed even when the target action is rarely sampled.","The same attack recipe transfers across discrete and continuous action spaces, single and multiple agents, and sparse or dense rewards; the paper's evaluations report this across 53 backdoor designs.","Post-training injection into a well-trained policy is possible but degraded, which the paper attributes to loss of plasticity in the trained network.","Once injected, persistent activation of the backdoor can collapse benign performance catastrophically, as shown by the paper's activation demonstrations on control tasks."],"supporting_citations":[{"why":"TrojDRL defines the transition-poisoning baseline with a fixed backdoor reward and motivates the empirical study showing fixed rewards fail across tasks.","marker":"[25]"},{"why":"IDT is the in-distribution trigger baseline with conditional reward flipping that UNIDOOR compares against.","marker":"[2]"},{"why":"BadRL is the sparse targeted backdoor baseline that sets the backdoor reward to the minimum positive reward.","marker":"[13]"},{"why":"TW is the temporal watermark baseline that adds 10 to matched-action rewards, used as a comparison method.","marker":"[8]"},{"why":"MARNet provides a prior dynamic backdoor reward mechanism that remains task-specific, which UNIDOOR extends toward universality.","marker":"[11]"},{"why":"SleeperNets supplies the Monte Carlo target-optimality approach and the inner-loop/outer-loop poisoning paradigm categorization used in the appendix.","marker":"[46]"},{"why":"PPO is one of the three victim DRL algorithms on which UNIDOOR is evaluated.","marker":"[47]"},{"why":"The multi-agent environments and MADDPG algorithm come from this work and are used for the multi-agent evaluations.","marker":"[35]"},{"why":"DDPG is the continuous-control victim algorithm used in the multi-agent and continuous-action evaluations.","marker":"[42]"}],"fun_headline_variants":["Self-tuning backdoor rewards universalize DRL attacks","UNIDOOR adapts backdoor rewards on the fly for DRL","Adaptive backdoor rewards make DRL attacks universal","No hand-tuning: UNIDOOR finds backdoor rewards for DRL","UNIDOOR's adaptive search cracks DRL backdoor rewards"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The controller assumes the adversary can estimate how long the benign task and the backdoor task each take to converge, and that benign performance falls while attack success rises whenever the backdoor reward is increased; if the estimates are wrong or the monotonicity fails, the reward is adjusted in the wrong direction.","fun_headline_variants_meta":{"raw":{"variants":["Self-tuning backdoor rewards universalize DRL attacks","UNIDOOR adapts backdoor rewards on the fly for DRL","Adaptive backdoor rewards make DRL attacks universal","No hand-tuning: UNIDOOR finds backdoor rewards for DRL","UNIDOOR's adaptive search cracks DRL backdoor rewards"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000784,"raw_usage":{"total_tokens":3507,"prompt_tokens":1040,"completion_tokens":2467,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":656,"completion_tokens_details":{"reasoning_tokens":2378}},"tokens_in":656,"tokens_out":2467,"duration_ms":16683,"temperature":1.0,"reasoning_tokens":2378,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:11:30.984977+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run UNIDOOR on a task where the benign-performance-versus-backdoor-reward curve is non-monotone, for instance a benchmark in which a medium backdoor reward selectively destroys a sub-behavior required for the benign task while a larger reward does not, and check whether the contraction phase still converges to a high attack success rate without collapsing benign performance.","supporting_citations":[{"cited_title":"TrojDRL: Evaluation of Backdoor Attacks on Deep Reinforcement Learning","cited_arxiv_id":null,"evidence_quote":"TrojDRL defines the transition-poisoning baseline with a fixed backdoor reward and motivates the empirical study showing fixed rewards fail across tasks."},{"cited_title":"Poisoning Deep Re- inforcement Learning Agents with In-Distribution Trig- gers","cited_arxiv_id":null,"evidence_quote":"IDT is the in-distribution trigger baseline with conditional reward flipping that UNIDOOR compares against."},{"cited_title":"BadRL: Sparse Targeted Backdoor Attack against Reinforcement Learning","cited_arxiv_id":null,"evidence_quote":"BadRL is the sparse targeted backdoor baseline that sets the backdoor reward to the minimum positive reward."},{"cited_title":"Temporal Watermarks for Deep Rein- forcement Learning Models","cited_arxiv_id":null,"evidence_quote":"TW is the temporal watermark baseline that adds 10 to matched-action rewards, used as a comparison method."},{"cited_title":"MARNet: Backdoor Attacks Against Cooperative Multi- Agent Reinforcement Learning","cited_arxiv_id":null,"evidence_quote":"MARNet provides a prior dynamic backdoor reward mechanism that remains task-specific, which UNIDOOR extends toward universality."},{"cited_title":"SleeperNets: Universal Backdoor Poisoning Attacks Against Reinforcement Learning Agents","cited_arxiv_id":null,"evidence_quote":"SleeperNets supplies the Monte Carlo target-optimality approach and the inner-loop/outer-loop poisoning paradigm categorization used in the appendix."},{"cited_title":"Proximal Policy Optimiza- tion Algorithms","cited_arxiv_id":null,"evidence_quote":"PPO is one of the three victim DRL algorithms on which UNIDOOR is evaluated."},{"cited_title":"Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments","cited_arxiv_id":null,"evidence_quote":"The multi-agent environments and MADDPG algorithm come from this work and are used for the multi-agent evaluations."},{"cited_title":"Continuous Control with Deep Reinforce- ment Learning","cited_arxiv_id":null,"evidence_quote":"DDPG is the continuous-control victim algorithm used in the multi-agent and continuous-action evaluations."}],"review_version":1}