Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

UNIDOOR: A Universal Framework for Action-Level Backdoor Attacks in Deep Reinforcement Learning

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read UNIDOOR adaptively tunes the backdoor reward during training, making action-level backdoor attacks work across DRL tasks without hand-set values.

desk verdict Solid empirical paper with a genuinely new adaptive reward mechanism; the monitoring loop may be chasing its own poisoned rewards, which tempers the 'universal' claim but not the value of the work. read the letter →

arxiv 2501.15529 v1 pith:FVOAPZCG submitted 2025-01-26 cs.LG cs.AIcs.CR

Oubo Ma , Linkang Du , Yang Dai , Chunyi Zhou , Qingming Li , Yuwen Pu , Shouling Ji This is my paper · ORCID
classification cs.LGcs.AIcs.CR
keywords backdoorattackdeepreinforcementlearningaction-levelrewardpoisoningtransitionadaptiveexplorationmulti-taskuniversality
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Deep reinforcement learning policies can be hijacked by action-level backdoors: a trigger pattern in the observation is bound to a target action by rewriting the reward in poisoned training transitions. Prior attacks fix that backdoor reward to a constant or flip it conditionally, and this paper shows empirically that no static choice works across tasks, as performance fluctuates or the attack fails outright. UNIDOOR instead treats the attack as a multi-task learning problem and adapts the backdoor reward online, guided by monitored benign-task performance and attack success rate. The paper's claim is that this removes the need for expert knowledge or grid search and makes action-level backdoors universal across discrete and continuous actions, single and multi-agent settings, and sparse and dense rewards.

What carries the argument

The load-bearing mechanism is the Adaptive Exploration module, a one-dimensional interval search over the backdoor reward $r^\dagger$. The framework maintains an interval $[r_l, r_u]$ and a current reward $r^\dagger$; in the expansion phase it raises the upper bound when benign performance is ahead of expectation but attack success lags, and in the contraction phase it narrows the interval by moving $r_u$ or $r_l$ to the current reward depending on which signal falls below its conservative expectation. Those expectations, $E_t$ and $E_t^\dagger$, are time-dependent ramps keyed to an estimated convergence time for each task. Everything else in UNIDOOR supports this search: Performance Monitoring produces the normalized BTP and ASR inputs, Initial Freezing delays poisoning, and Transition Poisoning tampers with states, actions, and rewards in the replay buffer.

What would settle it

Run UNIDOOR on a task where the benign-performance-versus-backdoor-reward curve is non-monotone, for instance a benchmark in which a medium backdoor reward selectively destroys a sub-behavior required for the benign task while a larger reward does not, and check whether the contraction phase still converges to a high attack success rate without collapsing benign performance.

Watch

Extended reading notes

Core claim

The central claim is that the backdoor reward can be discovered adaptively rather than hand-set, and that this one change universalizes action-level backdoor attacks. UNIDOOR monitors the victim through two normalized signals: benign task performance (BTP) and attack success rate (ASR), both smoothed by exponential weighted averaging. A short Initial Freezing phase delays poisoning so the backdoor task does not dominate early training, and Transition Poisoning then rewrites states, actions, and rewards in the victim's replay buffer, adding noise to target actions in continuous spaces. The key relationship the framework exploits is that BTP falls and ASR rises as the backdoor reward increases. Adaptive Exploration uses that relationship to run a bracketed search over an interval of backdoor rewards, expanding the interval while the backdoor task converges and then contracting it until a working reward is isolated. The paper reports that this yields higher comprehensive performance than fixed-reward baselines across 11 tasks, 53 backdoor designs, and three DRL algorithms, and that inactive backdoors are stealthy under state-distribution and neuron-activation comparisons.

Load-bearing premise

The controller assumes the adversary can estimate how long the benign task and the backdoor task each take to converge, and that benign performance falls while attack success rises whenever the backdoor reward is increased; if the estimates are wrong or the monotonicity fails, the reward is adjusted in the wrong direction.

Editorial extensions

If this is right

  • An adversary with only replay-buffer write access can inject a working action-level backdoor into a DRL policy without knowing the task's reward scale in advance.
  • Continuous-action policies are not safe by default: action tampering with noise lets the attack succeed even when the target action is rarely sampled.
  • The same attack recipe transfers across discrete and continuous action spaces, single and multiple agents, and sparse or dense rewards; the paper's evaluations report this across 53 backdoor designs.
  • Post-training injection into a well-trained policy is possible but degraded, which the paper attributes to loss of plasticity in the trained network.
  • Once injected, persistent activation of the backdoor can collapse benign performance catastrophically, as shown by the paper's activation demonstrations on control tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 'no expert knowledge' claim is bounded by the schedule estimates $t_n$ and $t_b$; a defender who randomizes training duration would make the conservative expectation harder to set, so the search would likely need more adjustments or fail. This is an inference about the attack's practical boundary, not a claim in the paper.
  • The monotone relationship between backdoor reward and the two performance signals may break at extreme rewards, where the backdoor task dominates and collapses the shared policy; a useful stress test is to run UNIDOOR on tasks with clipped rewards or bounded episode returns.
  • The same adaptive reward search could be run defensively as a cheap probe to estimate how vulnerable a given DRL policy is to action-level backdooring.
  • Framing injection as a multi-task trade-off suggests that treating benign performance and attack success as competing objectives, rather than via a one-dimensional interval search, could improve sample efficiency and stability.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes UNIDOOR, a framework for action-level backdoor attacks in deep reinforcement learning. The framework combines four modules: performance monitoring (estimating benign-task performance and attack success rate from trajectories in the victim's replay buffer), initial freezing (delaying poisoning to avoid backdoor-task dominance), transition poisoning (state/action tampering plus a hackable backdoor reward), and adaptive exploration (a feedback controller that adjusts the backdoor reward based on the monitored benign-task performance and attack success rate). The authors claim that this adaptive mechanism removes the need for hand-tuned backdoor rewards or grid search, and they evaluate the framework on 11 Gym/MPE/PyBullet tasks, 53 backdoor designs, and PPO/DDPG/MADDPG, comparing against four fixed-reward baselines. They also report ablations, post-training attacks, activation-strategy demonstrations, and visualizations of stealthiness.

Significance. If substantiated, the adaptive-reward mechanism would be a useful advance over fixed- or hand-designed backdoor rewards, and the observation that action tampering is important in continuous action spaces is a concrete and evidently load-bearing insight supported by the ablation study. The evaluation is broad in task coverage and design diversity, and the public code release is a further strength. However, the central claim depends on the validity of the internal performance-monitoring signal: the monitored BTP is computed from the same replay buffer that transition poisoning rewrites with synthetic rewards, so the feedback controller may be reacting to artifacts rather than to true benign-task performance. Until that signal is validated, the paper's main explanatory mechanism and the 'no expert knowledge' claim remain unproven. The paper is well within the scope of a security-oriented ML venue and addresses a topic of genuine concern.

major comments (3)
  1. [Sec. 4.2 (Algorithm 1), Sec. 4.4 (Algorithm 4), and Sec. 4.5 (Eqs. 11–12)] The core feedback loop appears to be contaminated by the attack itself. Performance Monitoring (Eq. 5) computes episode-wise BTP by summing rewards in trajectories from the victim's replay buffer, but Transition Poisoning (Algorithm 4) rewrites transitions in that same buffer, replacing rt with ±r†. Once poisoning is active, the monitored Pt is therefore a mixture of true benign rewards and synthetic backdoor rewards, so the claim that Adaptive Exploration tracks benign-task performance is not established. The paper never distinguishes the monitored BTP used during training from the unbiased BTP defined in Eq. (14) and used in the evaluations, and Figure 4's empirical relation is likely computed on true performance, not on the contaminated buffer signal. This is load-bearing for the paper's central mechanism. Please provide a direct validation experiment: run UNIDOOR, and plot the monitored Pt against the truly evaluated BTP (e.g., Eq. 14) over the course of training, both with and without excluding poisoned transitions from the monitoring trajectory. If the contaminated signal does not track true BTP, either modify the monitoring to use unpoisoned trajectories or revise the claim that the controller is adapting to benign-task performance.
  2. [Sec. 4.5, Eqs. (11)–(12), and Abstract] The claim that UNIDOOR 'eliminates the reliance on expert knowledge and grid search' is stronger than the method supports. The adversary must specify the expected convergence times tn and tb for the benign and backdoor tasks, the thresholds φt and φp, the smoothing factor β, the lower/upper reward bounds rl and ru, and the exploration step ω, plus the poisoning interval I_p. Although the ablation in Fig. 9(c) shows robustness to ±20% errors in tn and tb, the need for schedule estimates is a form of task-specific prior knowledge. Please weaken the claim to 'reduces reliance on expert knowledge and avoids grid search' or provide evidence that UNIDOOR succeeds when tn and tb are not provided at all (e.g., fully adaptive schedule-free variants).
  3. [Sec. 6, Tables 1–3] The primary results are averages over only three random seeds, and the tables report no standard deviations or confidence intervals for BTP or ASR (only the CP standard deviation is given in the text). Given the inherent variance of DRL training, the statements that UNIDOOR achieves 'top-1 CP in 84.6% of scenarios' or improves ASR/CP by specific percentages need per-metric error bars or a significance test to be reliable. Please add seed-wise standard deviations (or bootstrap intervals) for BTP, ASR, and CP, or justify why three seeds are sufficient for the claimed effect sizes.
minor comments (6)
  1. [Sec. 4.4, Algorithm 4, and Sec. 5] The action tampering frequency f is an important parameter in Algorithm 4, but no value is given in the implementation details in Sec. 5. Please specify it.
  2. [Sec. 4.2 and Appendix B] Eq. (7) normalizes BTP using Pl and Pu, but the clipping to [0,1] is only mentioned in Appendix B; consider stating this explicitly in Sec. 4.2 to avoid the implication that Pt always lies in [0,1] by construction.
  3. [Figure 4 and Sec. 4.5] The y-axis label 'Performance' in Figure 4 is ambiguous; clarify that the curves represent the true evaluated BTP and ASR from the empirical study, not the monitored values from the replay buffer.
  4. [Sec. 4.2 vs. Sec. 5] The notation BTP and ASR is used both for the monitored estimates in Sec. 4.2 and for the unbiased evaluation metrics in Sec. 5, which is confusing. Consider distinguishing them, e.g., with hats (\(\widehat{\mathrm{BTP}}\)) for the monitored quantities.
  5. [Sec. 2.2 and Sec. 5] The paper mentions that Chen et al. [11] proposed a dynamic backdoor reward mechanism, but that method (MARNet) is not included in the comparison tables. Please state why it is excluded, or add a comparison, since it is the most closely related adaptive-reward work.
  6. [Algorithm 5, line 1] Typo: 'szie' should be 'size'.

Circularity Check

1 steps flagged · score 4.0 of 10

Performance Monitoring's BTP is computed from the same replay-buffer transitions that Transition Poisoning rewrites with ±r†, making the adaptive controller's input partially self-referential; the end-to-end benchmark claims remain externally evaluated.

  1. self definitional [Sec. 4.2 (Eq. 5), Sec. 4.4 (Algorithm 4), Sec. 4.5 (Algorithm 5)]
    ""The adversary leverages the trajectories and transitions stored in the victim agent’s replay buffer to monitor the performance of both benign and backdoor tasks." ... "˙P = Σ ri" ... "˜rt ← ( r†t if ˜at = Fa(δ); −r†t otherwise ).""

    Once Transition Poisoning runs, the transitions in the replay buffer used by Performance Monitoring have had their rewards replaced with ±r†. Hence the monitored episode-wise BTP of Eq. 5 is a function of the backdoor reward r†, which is exactly the variable that Adaptive Exploration (Algorithm 5) adjusts when Pt deviates from its conservative expectation. The correlation between 'BTP' and r† that motivates the controller is therefore partly a definitional artifact of summing poisoned rewards, not an independent measurement of benign-task performance. The final evaluation (Eq. 14) uses a clean, separately computed BTP, so the benchmark results are not themselves circular, but the paper's claimed task-agnostic 'performance monitoring' basis for adaptation is self-referential.

full rationale

The paper's headline claims are supported by external comparisons against TrojDRL, IDT, BadRL, and TW using independently computed BTP, ASR, and CP (Eqs. 14-16), so the empirical superiority of UNIDOOR is not fitted or forced by definition. The one structural circularity is internal: the monitored BTP that drives Adaptive Exploration is computed (Eq. 5) from trajectories whose rewards Transition Poisoning has overwritten with ±r† (Algorithm 4), so the control signal is contaminated by the control variable. This undermines the 'task-agnostic performance monitoring' premise and makes the Fig. 4 negative correlation partially self-fulfilling, but it does not invalidate the external benchmark comparisons. The dependence of the conservative expectation on tn and tb is a real threat-model assumption but a robustness concern, not circularity. No load-bearing self-citation or renamed known result was found.

Assumptions & free parameters 12 free parameters · 7 assumptions · 0 invented entities

The central claim rests on the adversary's capability, on an empirical monotonicity relation between reward and performance, and on task-specific schedule knowledge (t_n, t_b, training bounds). None of these are derived within the paper, so the contribution is an empirical framework rather than a first-principles result.

free parameters (12)
  • EWA smoothing factor beta = 0.99
    Controls smoothing of monitored BTP and ASR in Eqs. 6 and 10; chosen once for all tasks.
  • Norm constraint epsilon = 0.05
    Defines action equivalence for ASR and reward hacking in continuous action spaces (Eq. 9, Algorithm 4).
  • Trajectory threshold phi_t = 10
    Lift condition for Initial Freezing in low-complexity tasks (Algorithm 3).
  • Performance threshold phi_p = 0.05
    Lift condition for Initial Freezing in cold-start tasks (Algorithm 3).
  • Perturbation radius rho = 0.025
    Uniform noise scale for action tampering in continuous action spaces (Section 4.4).
  • Expected near-convergence BTP E_n = 0.97
    Target for conservative BTP expectation in Eq. 11; hand-set.
  • Expected near-convergence ASR E_b = 0.97
    Target for conservative ASR expectation in Eq. 12; hand-set.
  • Benign task convergence time t_n = 0.75 of training progress
    Expected convergence time of the benign task in Eq. 11; task-specific expert knowledge.
  • Backdoor task convergence time t_b = 0.50 of training progress
    Expected convergence time of the backdoor task in Eq. 12; task-specific expert knowledge.
  • Exploration step size omega = not fixed globally, recommended 1-3 times initial r_l
    Step size for increasing backdoor reward in expansion phase (Algorithm 5, Section 7.2).
  • Initial backdoor reward bounds r_l, r_u = min/max of rewards collected during Initial Freezing
    Initializes the reward exploration space (Appendix E); data-derived but task-dependent.
  • Poisoning interval I_p and action tampering frequency f = interval in {16,32,48,64}, frequency not fixed
    Control how often transitions are poisoned and how often actions are replaced (Algorithm 4, Section 7.2).
assumptions (7)
  • domain assumption The adversary can perturb the victim's observations and access or modify transitions in the replay buffer.
    Stated in Section 3 as the adversary's capability; all attack modules depend on this access.
  • domain assumption Benign task performance (BTP) is negatively correlated with backdoor reward while attack success rate (ASR) is positively correlated.
    Used in Section 4.5 to decide when to increase or decrease r_t; supported only by the empirical Figure 4, not by a proof.
  • domain assumption Action-level backdoor tasks are less complex than benign tasks and thus can dominate early training.
    Justifies Initial Freezing in Section 4.3; argued informally from goal, state, and action space sizes.
  • domain assumption The backdoor task converges faster than the benign task (t_b < t_n).
    Used in Eq. 12 to define the conservative ASR expectation; if false, the expansion phase timing is wrong.
  • domain assumption The adversary knows or can estimate the benign task's upper and lower performance bounds P_u and P_l.
    Normalization in Eq. 7 and Appendix B relies on these bounds; biased bounds degrade the controller as shown in Section 7.2.
  • domain assumption EWA-smoothed, normalized BTP and ASR faithfully represent the victim's training status.
    Performance Monitoring (Section 4.2) assumes the smoothed indicators are reliable enough for reward adaptation despite DRL non-stationarity.
  • domain assumption Training performance follows a cold-start, rapid-growth, steady three-stage curve with a known convergence time.
    The conservative expectations in Eqs. 11-12 are piecewise linear based on this assumed shape.

how reviews work

0 comments
Cite this review

Pith. "Pith review of UNIDOOR: A Universal Framework for Action-Level Backdoor Attacks in Deep Reinforcement Learning." pith.science (2026). https://pith.science/paper/FVOAPZCG

@misc{pith2026250115529,
  author       = {Pith},
  title        = {Pith review of: UNIDOOR: A Universal Framework for Action-Level Backdoor Attacks in Deep Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FVOAPZCG}},
  note         = {Machine review of arXiv:2501.15529}
}
read the original abstract

Deep reinforcement learning (DRL) is widely applied to safety-critical decision-making scenarios. However, DRL is vulnerable to backdoor attacks, especially action-level backdoors, which pose significant threats through precise manipulation and flexible activation, risking outcomes like vehicle collisions or drone crashes. The key distinction of action-level backdoors lies in the utilization of the backdoor reward function to associate triggers with target actions. Nevertheless, existing studies typically rely on backdoor reward functions with fixed values or conditional flipping, which lack universality across diverse DRL tasks and backdoor designs, resulting in fluctuations or even failure in practice. This paper proposes the first universal action-level backdoor attack framework, called UNIDOOR, which enables adaptive exploration of backdoor reward functions through performance monitoring, eliminating the reliance on expert knowledge and grid search. We highlight that action tampering serves as a crucial component of action-level backdoor attacks in continuous action scenarios, as it addresses attack failures caused by low-frequency target actions. Extensive evaluations demonstrate that UNIDOOR significantly enhances the attack performance of action-level backdoors, showcasing its universality across diverse attack scenarios, including single/multiple agents, single/multiple backdoors, discrete/continuous action spaces, and sparse/dense reward signals. Furthermore, visualization results encompassing state distribution, neuron activation, and animations demonstrate the stealthiness of UNIDOOR. The source code of UNIDOOR can be found at https://github.com/maoubo/UNIDOOR.

Figures

Figures reproduced from arXiv: 2501.15529 by the authors.

Figure 1
Figure 1. The impact of backdoor reward and task varia [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The schematic diagram of DRL. specific action or a probability distribution over actions in A. As shown in [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The framework of UNIDOOR. ① is the monitored BTP, ② is the lift status of Initial Freezing, ③ is the latest backdoor reward, ④ is the monitored BTP and ASR, and ⑤ is the updated backdoor reward. introduces fluctuations in the unprocessed BTP curve, mak￾ing it unreliable for accurately reflecting the victim agent’s true performance. We address this problem by applying an exponentially weighted average, expressed as P… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: The impact trend of backdoor rewards on the per [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: In the expansion and contraction phases, the conser [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Action-level backdoor activation devastates the vic [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: The comparison of state distributions. MountainCar. The task aims to control the car to swing back and forth, building potential energy to reach the top of the rightmost hill. The adversary can continuously accelerate the car to the right, preventing it from swinging …
Figure 9
Figure 9. Figure 9: The impact of factors such as estimate bias, poison [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 10
Figure 10. Figure 10: The state distributions of benign and backdoored [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: The t-SNE activations of the benign and back [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: The effectiveness of fine-tuning and super-fine [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. TrojanTO: Action-Level Backdoor Attacks against Trajectory Optimization Models

    cs.LG 2025-06 conditional novelty 5.0 of 10

    TrojanTO implants action-level backdoors into Decision Transformer style offline RL models using 10 trajectories, alternating trigger optimization and model fine-tuning, reaching average attack success 0.719 while pre...

Reference graph

Works this paper leans on

63 extracted references · 63 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 Technical Report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 Technical Report. arXiv, 2023

    work page 2023

  2. [2]

    Poisoning Deep Re- inforcement Learning Agents with In-Distribution Trig- gers

    Chace Ashcraft and Kiran Karra. Poisoning Deep Re- inforcement Learning Agents with In-Distribution Trig- gers. arXiv, 2021

    work page 2021

  3. [3]

    Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Data

    Tim Baumgärtner, Yang Gao, Dana Alon, and Donald Metzler. Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Data. In COLM, 2024

    work page 2024

  4. [4]

    Vulnerability of Deep Reinforcement Learning to Policy Induction At- tacks

    Vahid Behzadan and Arslan Munir. Vulnerability of Deep Reinforcement Learning to Policy Induction At- tacks. In MLDM, 2017

    work page 2017

  5. [5]

    Machine Un- learning

    Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine Un- learning. In S&P, 2021

    work page 2021

  6. [6]

    Poisoning and Backdooring Contrastive Learning

    Nicholas Carlini and Andreas Terzis. Poisoning and Backdooring Contrastive Learning. In ICLR, 2022

    work page 2022

  7. [7]

    Towards Evaluating the Robustness of Neural Networks

    Nicholas Carlini and David Wagner. Towards Evaluating the Robustness of Neural Networks. In S&P, 2017

    work page 2017

  8. [8]

    Temporal Watermarks for Deep Rein- forcement Learning Models

    Kangjie Chen, Shangwei Guo, Tianwei Zhang, Shuxin Li, and Yang Liu. Temporal Watermarks for Deep Rein- forcement Learning Models. In AAMAS, 2021

    work page 2021

Show all 63 references
  1. [9]

    Decision Transformer: Reinforcement Learning via Sequence Modeling

    Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision Transformer: Reinforcement Learning via Sequence Modeling. In NeurIPS, 2021

    2021

  2. [10]

    BIRD: Generalizable Back- door Detection and Removal for Deep Reinforcement Learning

    Xuan Chen, Wenbo Guo, Guanhong Tao, Xiangyu Zhang, and Dawn Song. BIRD: Generalizable Back- door Detection and Removal for Deep Reinforcement Learning. In NeurIPS, 2024

    2024

  3. [11]

    MARNet: Backdoor Attacks Against Cooperative Multi- Agent Reinforcement Learning

    Yanjiao Chen, Zhicong Zheng, and Xueluan Gong. MARNet: Backdoor Attacks Against Cooperative Multi- Agent Reinforcement Learning. IEEE Transactions on Dependable and Secure Computing, 2022

    2022

  4. [12]

    PyBullet, a Python Module for Physics Simulation for Games, Robotics and Machine Learning

    Erwin Coumans and Yunfei Bai. PyBullet, a Python Module for Physics Simulation for Games, Robotics and Machine Learning. http://pybullet.org, 2021

    2021

  5. [13]

    BadRL: Sparse Targeted Backdoor Attack against Reinforcement Learning

    Jing Cui, Yufei Han, Yuzhe Ma, Jianbin Jiao, and Junge Zhang. BadRL: Sparse Targeted Backdoor Attack against Reinforcement Learning. In AAAI, 2024

    2024

  6. [14]

    Is Mamba Compatible with Trajec- tory Optimization in Offline Reinforcement Learning? In NeurIPS, 2024

    Yang Dai, Oubo Ma, Longfei Zhang, Xingxing Liang, Shengchao Hu, Mengzhu Wang, Shouling Ji, Jincai Huang, and Li Shen. Is Mamba Compatible with Trajec- tory Optimization in Offline Reinforcement Learning? In NeurIPS, 2024

    2024

  7. [15]

    Loss of Plasticity in Deep Con- tinual Learning

    Shibhansh Dohare, J Fernando Hernandez-Garcia, Qingfeng Lan, Parash Rahman, A Rupam Mahmood, and Richard S Sutton. Loss of Plasticity in Deep Con- tinual Learning. Nature, 2024

    2024

  8. [16]

    ORL- AUDITOR: Dataset Auditing in Offline Deep Reinforce- ment Learning

    Linkang Du, Min Chen, Mingyang Sun, Shouling Ji, Peng Cheng, Jiming Chen, and Zhikun Zhang. ORL- AUDITOR: Dataset Auditing in Offline Deep Reinforce- ment Learning. In NDSS, 2024

    2024

  9. [17]

    Discovering Faster Matrix Multiplication Algorithms with Reinforce- ment Learning

    Alhussein Fawzi, Matej Balog, Aja Huang, Thomas Hubert, Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, et al. Discovering Faster Matrix Multiplication Algorithms with Reinforce- ment Learning. Nature, 2022. 14

    2022

  10. [18]

    Adversarial Poli- cies: Attacking Deep Reinforcement Learning

    Adam Gleave, Michael Dennis, Cody Wild, Neel Kant, Sergey Levine, and Stuart Russell. Adversarial Poli- cies: Attacking Deep Reinforcement Learning. In ICLR, 2020

    2020

  11. [19]

    BAFFLE: Backdoor Attack in Offline Reinforcement Learning

    Chen Gong, Zhou Yang, Yunpeng Bai, Junda He, Jieke Shi, Kecen Li, Arunesh Sinha, Bowen Xu, Xinwen Hou, David Lo, et al. BAFFLE: Backdoor Attack in Offline Reinforcement Learning. In S&P, 2024

    2024

  12. [20]

    Adversarial Policy Learning in Two-Player Competitive Games

    Wenbo Guo, Xian Wu, Sui Huang, and Xinyu Xing. Adversarial Policy Learning in Two-Player Competitive Games. In ICML, 2021

    2021

  13. [21]

    SHINE: Shielding Backdoors in Deep Reinforcement Learning

    Wenbo Guo, Zhuowen Yuan, Jinyuan Jia, Bo Li, and Dawn Song. SHINE: Shielding Backdoors in Deep Reinforcement Learning. In ICML, 2024

    2024

  14. [22]

    Adversarial Attacks on Neural Network Policies

    Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial Attacks on Neural Network Policies. arXiv, 2017

    2017

  15. [23]

    The 37 Implementation Details of Proximal Policy Optimiza- tion

    Shengyi Huang, Rousslan Fernand Julien Dossa, An- tonin Raffin, Anssi Kanervisto, and Weixun Wang. The 37 Implementation Details of Proximal Policy Optimiza- tion. In ICLR Blog Track, 2022

    2022

  16. [24]

    Highly Accurate Protein Struc- ture Prediction with AlphaFold

    John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, et al. Highly Accurate Protein Struc- ture Prediction with AlphaFold. Nature, 2021

    2021

  17. [25]

    TrojDRL: Evaluation of Backdoor Attacks on Deep Reinforcement Learning

    Panagiota Kiourti, Kacper Wardega, Susmit Jha, and Wenchao Li. TrojDRL: Evaluation of Backdoor Attacks on Deep Reinforcement Learning. In DAC, 2020

    2020

  18. [26]

    Plasticity Loss in Deep Reinforcement Learning: A Survey

    Timo Klein, Lukas Miklautz, Kevin Sidak, Claudia Plant, and Sebastian Tschiatschek. Plasticity Loss in Deep Reinforcement Learning: A Survey. arXiv, 2024

    2024

  19. [27]

    Combinatorial Optimization

    Bernhard H Korte, Jens Vygen, B Korte, and J Vygen. Combinatorial Optimization. 2011

    2011

  20. [28]

    On Infor- mation and Sufficiency

    Solomon Kullback and Richard A Leibler. On Infor- mation and Sufficiency. The annals of mathematical statistics, 1951

    1951

  21. [29]

    Exploration in Deep Reinforcement Learning: A Survey

    Pawel Ladosz, Lilian Weng, Minwoo Kim, and Hyon- dong Oh. Exploration in Deep Reinforcement Learning: A Survey. Information Fusion, 2022

    2022

  22. [30]

    Spatiotemporally Con- strained Action Space Attacks on Deep Reinforcement Learning Agents

    Xian Yeow Lee, Sambit Ghadai, Kai Liang Tan, Chin- may Hegde, and Soumik Sarkar. Spatiotemporally Con- strained Action Space Attacks on Deep Reinforcement Learning Agents. In AAAI, 2020

    2020

  23. [31]

    Online Poi- soning Attack Against Reinforcement Learning under Black-box Environments

    Jianhui Li, Bokang Zhang, and Junfeng Wu. Online Poi- soning Attack Against Reinforcement Learning under Black-box Environments. arXiv, 2024

    2024

  24. [32]

    Fine-Pruning: Defending against Backdooring Attacks on Deep Neural Networks

    Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Fine-Pruning: Defending against Backdooring Attacks on Deep Neural Networks. In RAID, 2018

    2018

  25. [33]

    Rethinking Adversarial Policies: A Gen- eralized Attack Formulation and Provable Defense in RL

    Xiangyu Liu, Souradip Chakraborty, Yanchao Sun, and Furong Huang. Rethinking Adversarial Policies: A Gen- eralized Attack Formulation and Provable Defense in RL. In ICLR, 2024

    2024

  26. [34]

    HDRS: A Hybrid Reputation System with Dynamic Update Interval for Detecting Malicious Ve- hicles in V ANETs

    Xuejiao Liu, Oubo Ma, Wei Chen, Yingjie Xia, and Yux- uan Zhou. HDRS: A Hybrid Reputation System with Dynamic Update Interval for Detecting Malicious Ve- hicles in V ANETs. IEEE Transactions on Intelligent Transportation Systems, 2021

    2021

  27. [35]

    Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments

    Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, and Pieter Abbeel. Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments. In NIPS, 2017

    2017

  28. [36]

    A Data- free Backdoor Injection Approach in Neural Networks

    Peizhuo Lv, Chang Yue, Ruigang Liang, Yunfei Yang, Shengzhi Zhang, Hualong Ma, and Kai Chen. A Data- free Backdoor Injection Approach in Neural Networks. In USENIX Security, 2023

    2023

  29. [37]

    ABM-V: An Adaptive Backoff Mechanism for Mitigating Broadcast Storm in V ANETs

    Oubo Ma, Xuejiao Liu, and Yingjie Xia. ABM-V: An Adaptive Backoff Mechanism for Mitigating Broadcast Storm in V ANETs. IEEE Transactions on Vehicular Technology, 2023

    2023

  30. [38]

    SUB-PLAY: Adversarial Policies against Partially Observed Multi- Agent Reinforcement Learning Systems

    Oubo Ma, Yuwen Pu, Linkang Du, Yang Dai, Ruo Wang, Xiaolei Liu, Yingcai Wu, and Shouling Ji. SUB-PLAY: Adversarial Policies against Partially Observed Multi- Agent Reinforcement Learning Systems. In CCS, 2024

    2024

  31. [39]

    Targeted At- tack Synthesis for Smart Grid Vulnerability Analysis

    Suman Maiti, Anjana Balabhaskara, Sunandan Ad- hikary, Ipsita Koley, and Soumyajit Dey. Targeted At- tack Synthesis for Smart Grid Vulnerability Analysis. In CCS, 2023

    2023

  32. [40]

    Implicit Poisoning attacks in Two-Agent Reinforcement Learn- ing: Adversarial Policies for Training-Time Attacks

    Mohammad Mohammadi, Jonathan Nöther, Debmalya Mandal, Adish Singla, and Goran Radanovic. Implicit Poisoning attacks in Two-Agent Reinforcement Learn- ing: Adversarial Policies for Training-Time Attacks. In AAMAS, 2023

    2023

  33. [41]

    Gym Documentation

    OpenAI. Gym Documentation. https://www. gymlibrary.dev

  34. [42]

    Continuous Control with Deep Reinforce- ment Learning

    Lillicrap T P. Continuous Control with Deep Reinforce- ment Learning. In ICLR, 2016

    2016

  35. [43]

    Is Poisoning a Real Threat to LLM Alignment? Maybe More so Than You Think

    Pankayaraj Pathmanathan, Souradip Chakraborty, Xi- angyu Liu, Yongyuan Liang, and Furong Huang. Is Poisoning a Real Threat to LLM Alignment? Maybe More so Than You Think. In AAAI, 2025

    2025

  36. [44]

    15 Stable-Baselines3: Reliable Reinforcement Learning Im- plementations

    Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. 15 Stable-Baselines3: Reliable Reinforcement Learning Im- plementations. Journal of Machine Learning Research, 2021

    2021

  37. [45]

    Reward Poisoning in Reinforcement Learning: Attacks against Unknown Learners in Unknown Envi- ronments

    Amin Rakhsha, Xuezhou Zhang, Xiaojin Zhu, and Adish Singla. Reward Poisoning in Reinforcement Learning: Attacks against Unknown Learners in Unknown Envi- ronments. arXiv, 2021

    2021

  38. [46]

    SleeperNets: Universal Backdoor Poisoning Attacks Against Reinforcement Learning Agents

    Ethan Rathbun, Christopher Amato, and Alina Oprea. SleeperNets: Universal Backdoor Poisoning Attacks Against Reinforcement Learning Agents. In NeurIPS, 2024

    2024

  39. [47]

    Proximal Policy Optimiza- tion Algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimiza- tion Algorithms. arXiv, 2017

    2017

  40. [48]

    Fine-Tuning Is All You Need to Mitigate Backdoor Attacks

    Zeyang Sha, Xinlei He, Pascal Berrang, Mathias Hum- bert, and Yang Zhang. Fine-Tuning Is All You Need to Mitigate Backdoor Attacks. arXiv, 2022

    2022

  41. [49]

    Backdoor Pre-trained Models can Transfer to All

    Lujia Shen, Shouling Ji, Xuhong Zhang, Jinfeng Li, Jing Chen, Jie Shi, Chengfang Fang, Jianwei Yin, and Ting Wang. Backdoor Pre-trained Models can Transfer to All. In CCS, 2021

    2021

  42. [50]

    Mastering the Game of Go without Human Knowledge

    David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, et al. Mastering the Game of Go without Human Knowledge. Nature, 2017

    2017

  43. [51]

    Stealthy and Effi- cient Adversarial Attacks against Deep Reinforcement Learning

    Jianwen Sun, Tianwei Zhang, Xiaofei Xie, Lei Ma, Yan Zheng, Kangjie Chen, and Yang Liu. Stealthy and Effi- cient Adversarial Attacks against Deep Reinforcement Learning. In AAAI, 2020

    2020

  44. [52]

    Reinforcement Learning: An Introduction

    Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. 2018

    2018

  45. [53]

    Demon in the Variant: Statistical Analysis of DNNs for Robust Backdoor Contamination Detection

    Di Tang, XiaoFeng Wang, Haixu Tang, and Kehuan Zhang. Demon in the Variant: Statistical Analysis of DNNs for Robust Backdoor Contamination Detection. In USENIX Security, 2021

    2021

  46. [54]

    Distral: Robust Multitask Rein- forcement Learning

    Yee Teh, Victor Bapst, Wojciech M Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust Multitask Rein- forcement Learning. NeruIPS, 2017

    2017

  47. [55]

    Ad- versarial Attacks on Multi-Agent Communication

    James Tu, Tsunhsuan Wang, Jingkang Wang, Sivabalan Manivasagam, Mengye Ren, and Raquel Urtasun. Ad- versarial Attacks on Multi-Agent Communication. In ICCV, 2021

    2021

  48. [56]

    A Survey of Multi-Task Deep Reinforcement Learning

    Nelson Vithayathil Varghese and Qusay H Mahmoud. A Survey of Multi-Task Deep Reinforcement Learning. Electronics, 2020

    2020

  49. [57]

    BACKDOORL: Backdoor At- tack against Competitive Reinforcement Learning

    Lun Wang, Zaynah Javed, Xian Wu, Wenbo Guo, Xinyu Xing, and Dawn Song. BACKDOORL: Backdoor At- tack against Competitive Reinforcement Learning. In IJCAI, 2021

    2021

  50. [58]

    Adversarial Policies Beat Superhuman Go AIs

    Tony Tong Wang, Adam Gleave, Tom Tseng, Kellin Pelrine, Nora Belrose, Joseph Miller, Michael D Dennis, et al. Adversarial Policies Beat Superhuman Go AIs. In ICML, 2023

    2023

  51. [59]

    Ad- versarial Policy Training against Deep Reinforcement Learning

    Xian Wu, Wenbo Guo, Hua Wei, and Xinyu Xing. Ad- versarial Policy Training against Deep Reinforcement Learning. In USENIX Security, 2021

    2021

  52. [60]

    RLID- V: Reinforcement Learning-Based Information Dissem- ination Policy Generation in V ANETs.IEEE Transac- tions on Intelligent Transportation Systems, 2023

    Yingjie Xia, Xuejiao Liu, Jing Ou, and Oubo Ma. RLID- V: Reinforcement Learning-Based Information Dissem- ination Policy Generation in V ANETs.IEEE Transac- tions on Intelligent Transportation Systems, 2023

    2023

  53. [61]

    Design of Intentional Backdoors in Sequen- tial Models

    Zhaoyuan Yang, Naresh Iyer, Johan Reimann, and Nu- rali Virani. Design of Intentional Backdoors in Sequen- tial Models. arXiv, 2019

    2019

  54. [62]

    Reinforcement Unlearning

    Dayong Ye, Tianqing Zhu, Congcong Zhu, Derui Wang, Kun Gao, Zewei Shi, Sheng Shen, Wanlei Zhou, and Minhui Xue. Reinforcement Unlearning. In NDSS, 2025

    2025

  55. [63]

    AIRS: Explanation for Deep Reinforce- ment Learning based Security Applications

    Jiahao Yu, Wenbo Guo, Qi Qin, Gang Wang, Ting Wang, and Xinyu Xing. AIRS: Explanation for Deep Reinforce- ment Learning based Security Applications. In USENIX Security, 2023. A Attack Scenarios As a universal framework for action-level backdoor attacks, UNIDOOR is applicable t...

    2023

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.