Training RL Agents for Multi-Objective Network Defense Tasks

Ahmad Ridley; Akash Trivedi; Andres Molina-Markham; Derek Tsui; Lauren Brandt; Luis Robaina; Nicholas Potteiger; Ransom Winder; Sean Steinle

REVIEW 3 major objections 7 minor 48 references

Training RL Agents for Multi-Objective Network Defense Tasks

T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Training a single reinforcement-learning agent on a broad universe of network-defense tasks produces defenders that generalize to attacks never seen during training, outperforming specialists trained on the attack itself.

desk verdict Real contribution in the PDDL task encoding and dynamic curriculum, but the headline generalization result is not yet statistically supported because the main comparisons are single runs without error bars. read the letter →

arxiv 2505.22531 v2 pith:QN6RQWWC submitted 2025-05-28 cs.LG cs.AIcs.CR

Andres Molina-Markham , Luis Robaina , Sean Steinle , Akash Trivedi , Derek Tsui , Nicholas Potteiger , Lauren Brandt , Ransom Winder

show 1 more author

Ahmad Ridley

This is my paper · ORCID

classification cs.LGcs.AIcs.CR

keywords open-endedlearningreinforcementcybersecuritynetworkdefensecurriculumtaskuniverseactionrepresentationsgeneralization

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that open-ended learning transfers from robotics and games to network defense: training one reinforcement-learning agent on a diverse, procedurally generated universe of defense tasks, rather than on a single attack scenario, yields defenders that learn faster and handle conditions they never met. The key evidence is a held-out generalization result: a policy trained with dynamic task selection, which never faced a denial-of-service attack, still outperformed a policy trained specifically to mitigate DoS when both were tested on a 200-host network under that attack. The paper also reports that a difficulty-driven curriculum reaches higher performance with fewer training steps than fixed-task training, and that task diversity protects performance when the adversary masks its behavior. If the claim holds, it gives the cyber-defense gym community a concrete template: keep action spaces, observation spaces, and reward interfaces consistent across a broad task universe, and let the curriculum expand the task distribution as the agent improves. The authors further recommend that benchmark builders adopt broad task universes rather than a few 'representative' scenarios.

What carries the argument

The load-bearing object is the task pair $(N, G)$: $N$ is a network's dynamics (initial configuration, gray-agent user policies, red-agent attacker policies, and a function $f$ that converts abstract defender actions into executable parametric actions), and $G$ is a boolean goal plus a numeric metric written in PDDL, turned into a bounded sparse reward $r(x)=\tfrac{1}{2}e^{x/2}+0.5$ when the goal is satisfied and $r(x)=\tfrac{1}{2}e^{x/2}$ when it is not. Two design choices make the universe learnable as a single curriculum: fixed action and observation spaces across all tasks, with a small set of non-parametric action representations standing in for thousands of parametric actions; and PDDL goal-metric pairs (43 of them) that keep the reward semantics consistent as goals change. The third piece is the outer training loop, dynamic task selection, which raises cardinality parameters such as host counts, subnet counts, and red-and-gray behavior variability as the agent's mean score crosses a threshold and lowers them when the score falls, producing a difficulty-driven curriculum.

What would settle it

Run the identical dynamic-task-selection curriculum in a second, independent network-defense environment, for example an emulation built from real hosts, real services, and live traffic, or an unrelated public benchmark, and give the resulting policy the same held-out DoS test. If the broadly trained agent no longer matches or beats a DoS-trained specialist on that independent ground, the reported generalization advantage is a property of the simulator's structure rather than of open-ended training.

Watch

Extended reading notes

Core claim

The paper's central claim is that a single RL agent trained by dynamic task selection over a universe of network-defense tasks, each specified as a pair $(N, G)$ of network dynamics and a PDDL-encoded security-and-QoS goal, learns a policy that generalizes to defense tasks it never trained on. In the main experiment, an agent whose curriculum covered exfiltration and ransomware attacks, plus variations in network scale, user behavior, and red-agent behavior, but never a DoS attack, outperformed at DoS mitigation a policy trained exclusively on DoS. The authors read this as evidence that open-ended learning principles carry over to cyber defense: broad capability built across related tasks beats narrow competency tuned to a single attack. They further report that the difficulty-driven curriculum reaches higher reward with fewer training steps than fixed-task training, and that training against red agents that interleave gray-like behavior produces policies that do not fall apart when the adversary deviates from its basic tactics. The paper frames these results as a demonstration that the task universe, not the single environment, should be the unit of design for RL-based network defense.

Load-bearing premise

The load-bearing premise is that the FARLAND simulator, with its gray-agent user model, red-agent tactics, QoS event model, and host-and-subnet dynamics, faithfully represents realistic enterprise networks, because every training run, comparison, and generalization test takes place inside it and nothing validates the simulator against real traffic or independent benchmarks.

Editorial extensions

If this is right

A defender whose curriculum includes exfiltration and ransomware tasks, but never DoS, will beat a DoS-trained specialist at DoS mitigation when both face the same held-out network under attack.
Dynamic, difficulty-driven task selection reaches higher performance with fewer training steps than fixed-task training, and it masters harder tasks than uniform-random or unguided smooth curricula under the same computing budget.
Replacing the full parametric action space with a small set of action representations prevents learning stagnation, so curriculum training scales to networks whose raw action spaces would be thousands of actions.
Training against red agents that interleave gray-like (user-mimicking) behavior yields policies that maintain performance when the adversary deviates from its basic behavior.
Cyber-defense gym and benchmark builders should define broad task universes with consistent goal, reward, and action interfaces, because evaluating on small sets of representative tasks misleads assessments of defender robustness.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The recipe should transfer to other blue action sets and other attack categories: nothing in the argument depends on the five defender actions tested, so a direct testable extension is holding out an entire tactic family, exactly as the paper holds out DoS, and checking that the advantage persists.
The paper's evaluation suggests a concrete reporting standard for cyber-defense RL: report performance on a held-out task from the same universe, not just on held-out seeds of the training task; the authors argue for this but do not state it as a formal evaluation rule.
Because all comparisons run inside one simulator, the strongest open question is external validity: running the identical curriculum in an emulation-based environment with real hosts and traffic would show whether the generalization benefit survives the simulation-to-reality gap, a test the paper does not provide.
The appendix's curriculum findings imply that task-selection heuristics are as consequential as the RL algorithm itself: front-loading the hardest goal-metric pairs, or exposing the agent only to an inactive adversary, derails training, so a principled theory of when to demote tasks would directly improve defender training.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague, here's my read. What's actually new: applying open-ended-learning ideas to network defense with a concrete PDDL-based goal/metric encoding and PDDL-driven action masking, plus a dynamic difficulty curriculum. That is a useful recipe for cyber-defense gyms, and the paper shows that a dynamic-curriculum agent outperforms a fixed-task agent on a held-out DoS task it never saw. That's a genuine result worth taking seriously.

The paper does several things well. The task universe is described concretely with the parameters that vary, and the PDDL goal-metric pairs are listed in full in the appendix. The comparisons include fixed vs dynamic, random vs difficulty vs smooth, and robustness to red-agent deviation. The observation and action normalization across tasks is thoughtful. The writing is honest about what is not solved.

The soft spots are real and they coincide with the stress-test. The main learning curves in Figures 5-8 appear to come from single training runs per condition; the only averaged baseline is the 'smooth changes' one. With PPO and probabilistic red/gray agents, run-to-run variance can easily be as large as the reported gaps. There are no error bars or confidence intervals. Section 4.3 also says the 6 goal-metric pairs were selected post hoc, which is fine for a proof of concept but weakens the representation comparison. The whole evaluation runs inside FARLAND, the authors' own simulator, with no validation against public benchmarks and code 'forthcoming.' None of these are fatal individually, but together they mean the central claim—that dynamic task selection gives more generalizable defenders—is plausible, not yet established.

The paper is still worth engaging. The target audience is researchers building cyber-defense benchmarks; they should read it before designing their next gym. It deserves a serious peer review, with the requirement that the authors release code, seeds, and error bars. If those hold up, the result is an important one.

Referee Report

3 major / 7 minor

Summary. The paper proposes an open-ended-learning-inspired training framework for network defense in the FARLAND simulator. It defines a universe of tasks (N,G) with consistent action/observation interfaces, PDDL-specified goals and metrics, and dynamic difficulty-based task selection. The central empirical claim, developed in Section 4, is that dynamic task selection outperforms fixed-task training and single-task DoS training, reaches higher performance in fewer steps, and generalizes to an unseen DoS attack and to deviations in red-agent behavior. The paper also compares task-selection strategies and goal-metric representations, and reports PPO hyperparameters.

Significance. If the empirical claims held, this would be a useful contribution to cyber-RL: it gives a concrete way to define broad, consistent task universes and shows that curricula with difficulty and diversity can help. The experimental design is thoughtful: comparing dynamic vs fixed, difficulty vs random vs smooth, and robustness to deviations are the right comparisons. The paper is candid about limitations of task-selection research. However, the central empirical support is currently weakened by missing seed variance and error bars on the pivotal figures, the private simulator with no external validation, and the unavailability of code; these are fixable but currently prevent full assessment.

major comments (3)

[§4.1/§4.2, Figs. 5–8] The central comparison between dynamic and fixed task selection, and the generalization comparison against a DoS-trained policy, are reported without error bars, confidence intervals, or multiple training seeds. The only place multiple runs are mentioned is the 'smooth changes' baseline in Section 4.1 ('we train the policy several times and report the average'); the dynamic and fixed policies appear to be single runs. Because PPO is stochastic and the red and gray agent policies are probabilistic, run-to-run variance can easily exceed the reported differences. The headline claims ('dynamic task selection outperforms fixed...', 'dynamic ... outperforms the policy trained to mitigate DoS') therefore are not statistically supported as reported. Please provide at least 3–5 seeds for each condition and report means with variance or confidence intervals, or effect sizes.
[§4 overall; §4.5; Acknowledgments] The manuscript states twice that source code is 'forthcoming' (Section 4.5 and Acknowledgments), and all experiments run inside the private FARLAND simulator (reference [29]) with no external validation. The dynamic task selection procedure in Section 3.2 also leaves unspecified the promotion/demotion thresholds, difficulty-level sampling ranges, and the exact selection criterion for goal-metric pairs. Together these make the central empirical claim impossible to reproduce or independently assess from the manuscript. At minimum, the paper should specify all curriculum parameters, release code or a public benchmark interface, and provide some validation of FARLAND against real network traffic or an existing public cyber-defense gym (e.g., CybORG, CyberBattleSim, or NASimEmu), or explicitly restrict the claims to the simulator.
[§4.3] The six goal-metric pairs used to compare representations were selected post hoc ('we selected 6 that the agent was able to reasonably master'). This selection on the outcome variable makes Figures 10–11 uninterpretable as a comparison of goal representations, because the chosen subset is conditioned on agent performance. If this comparison is meant to support the framework's generality, all 11 evaluated pairs should be reported, or a pre-registered selection criterion should be stated; the text should not conclude that 'the one-hot representation results in higher performance' from this biased subset.

minor comments (7)

[§4, opening paragraph] The evaluation section says 'First, Section 4.1 evaluates... Then, Section 4.1 compares different strategies'; the duplication of Section 4.1 is confusing and one of the two references should likely point to a later subsection or to a different part of Section 4.
[§3.1.3] The sparse reward formula is ambiguous in the rendering '1 2ex2 + 0.5'; please clarify whether the exponent is x, x/2, or x^2, since the stated asymptotic behavior (reward converging to 0.5 for satisfied goals as penalties grow) is only consistent with an exponent that decays as x becomes more negative.
[Appendix 6.1] The appendix contains placeholder citations '[?]' for the claims that optimal behavior may be achieved via multiple rewards and that poorly designed dense rewards can be worse than sparse rewards; these need concrete references.
[Appendix 6.2, PDDL terms] The glossary defines 'declared-victory', but the PDDL listings for goals 42 and 43 use 'red-declared-victory'; align these terms so the appendix is internally consistent.
[§4.2] The sentence 'the agent training via dynamic task selection is ultimately trained against a more sophisticated red TTP' should read 'the agent trained via dynamic task selection...'.
[Table 1] The column header 'hyperparamters' is a typo for 'hyperparameters'.
[Figs. 5–8] Figure 5 is referenced in both Section 4.1 and Section 4.2 for different claims, but the figure does not appear to have labeled panels; add clear panel labels so the reader can tell which curve supports which comparison.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the generalization and robustness claims are empirical comparisons with procedurally distinct training and test tasks, and the FARLAND self-citation only identifies the evaluation environment rather than supplying a derived result.

full rationale

The paper's central results are empirical RL comparisons, not derivations. Section 4.2 explicitly evaluates a policy trained on a curriculum that never includes a DoS attack against a policy trained to mitigate DoS; the test task (200-host network, DoS red agent with deviation levels) is procedurally distinct from the training tasks, so the claimed generalization is not an identity or a fitted parameter renamed as a prediction. Section 4.1 compares dynamic task selection against fixed, random, and smooth-update baselines on the same evaluation protocol; the task-selection rule changes the training distribution, not the evaluation metric, so no equation-level reduction is present. The reward function and PDDL goal-metric pairs define the learning objective rather than the evaluation outcome; the evaluation measures whether the learned policy satisfies those objectives under unseen dynamics. The only self-citation, [29], describes FARLAND as the simulation environment used for all experiments and is acknowledged in the Acknowledgments; it is a tool, not a theorem invoked to force the result, and none of the comparisons are derived from the claims of [29]. Section 4.3's post hoc selection of six goal-metric pairs the agent could master is a curriculum-design choice for comparing goal representations; it does not by construction produce the DoS generalization result reported in Section 4.2. Concerns about single training runs, missing error bars, simulator fidelity, and unreleased source code are evidence-strength and reproducibility limitations, not circularity of the derivation chain. No circular step can be exhibited with a specific equation or definitional equivalence, so the honest finding is no circularity.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on design choices (reward weights, curriculum thresholds, goal subsets) and on the fidelity of the FARLAND simulator. No new physical or mathematical entity is postulated.

free parameters (5)

Reward penalty weights for QoS-aware sparse reward
Section 6.1 states penalties for crown-jewel relocations, host isolations, and bad QoS events contribute 'a number in the interval [-1, 0]' per event, with specific weights not derived from data.
Coefficient in sparse reward function r(x)
Section 3.1.3 defines r(x) = (1/2)e^{x^2} (as printed) plus 0.5 offset; the coefficient and the chosen exponent form are hand-selected to bound rewards between 0 and 1.
Difficulty-level sampling ranges
Section 3.2/Figure 4: the maximum sampling values for hosts, subnets, red/gray variability at each curriculum level are hand-designed; these ranges define what 'harder' means.
Promotion/demotion thresholds for dynamic task selection
Section 3.2: the agent is promoted when 'mean scores reach a level' and demoted when below a level; these score thresholds are not specified or fitted.
Subset of 6 goal-metric pairs for representation comparison
Section 4.3: from 11 evaluated goal-metric pairs, 'we selected 6 that the agent was able to reasonably master' before comparing representations, a post hoc selection.

assumptions (5)

domain assumption FARLAND simulation is a faithful model of realistic enterprise network defense.
All training and evaluation (Section 4) use this simulator; the paper provides no validation against real networks or public benchmarks.
domain assumption A fixed action space with abstract action representations does not limit the agent's ability to solve the tested defense tasks.
Section 3.1.2 fixes A and O across the task universe; the sufficiency of this interface for all goals is assumed.
domain assumption The 43 PDDL goal-metric pairs (Appendix 6.2) span the defender objectives relevant to the paper's conclusions.
Section 3.1.3 states goals are encoded in PDDL; the coverage of the space of security/QoS goals is asserted, not derived.
domain assumption PPO with the hyperparameters in Table 1 is an adequate learner; results would be similar with other RL algorithms.
Section 4.5 fixes PPO hyperparameters 'through a combination of experimentation and recommendations'; no sensitivity analysis is provided.
domain assumption The designed reward functions are invariant across network size, TTP, and traffic, making cross-task comparisons fair.
Section 6.1 asserts invariance; the claim is design intent rather than a verified property for all task instantiations.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Training RL Agents for Multi-Objective Network Defense Tasks." pith.science (2026). https://pith.science/paper/QN6RQWWC

@misc{pith2026250522531,
  author       = {Pith},
  title        = {Pith review of: Training RL Agents for Multi-Objective Network Defense Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QN6RQWWC}},
  note         = {Machine review of arXiv:2505.22531}
}

read the original abstract

Open-ended learning (OEL) -- which emphasizes training agents that achieve broad capability over narrow competency -- is emerging as a paradigm to develop artificial intelligence (AI) agents to achieve robustness and generalization. However, despite promising results that demonstrate the benefits of OEL, applying OEL to develop autonomous agents for real-world cybersecurity applications remains a challenge. We propose a training approach, inspired by OEL, to develop autonomous network defenders. Our results demonstrate that like in other domains, OEL principles can translate into more robust and generalizable agents for cyber defense. To apply OEL to network defense, it is necessary to address several technical challenges. Most importantly, it is critical to provide a task representation approach over a broad universe of tasks that maintains a consistent interface over goals, rewards and action spaces. This way, the learning agent can train with varying network conditions, attacker behaviors, and defender goals while being able to build on previously gained knowledge. With our tools and results, we aim to fundamentally impact research that applies AI to solve cybersecurity problems. Specifically, as researchers develop gyms and benchmarks for cyber defense, it is paramount that they consider diverse tasks with consistent representations, such as those we propose in our work.

Figures

Figures reproduced from arXiv: 2505.22531 by the authors.

**Figure 2.** Reward curve derived from the goal-metric pair shown in Figure 3. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Example of a goal and a metric in PDDL: When an active red agent is detected, the agent should use decoys [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (16 more)

**Figure 4.** Figure 4: Maximum sampling values at different levels for dynamic task selection of a blue agent with no DoS. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

**Figure 5.** Figure 5: Comparing Defender Generalization Ability: Dynamic vs. Fixed Task Training Approaches [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

**Figure 6.** Figure 6: Comparison of two policies: one with the fixed task selection strategy from Section 4.1, and another with [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]

**Figure 7.** Figure 7: Comparison of task selection strategies: one based on increasing difficulty levels, another selecting tasks [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

**Figure 8.** Figure 8: Performance comparison of two policies: one trained with a fixed task to mitigate a traditional exfiltration [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]

**Figure 9.** Figure 9: Mean reward for configurations, each fixing one of 11 goalmetric pairs [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]

**Figure 12.** Figure 12: Comparison of two learning strategies that differ only in the available actions: one uses action representations, [PITH_FULL_IMAGE:figures/full_fig_p013_12.png]

**Figure 13.** Figure 13: Goal-metrics 1-12. • worst-contributor-relocations: An integer that corresponds to the number of actions attempting to relocate a suspicious host to a different subnet • worst-contributor-honeys: An integer that corresponds to the number of actions attempting to reloc…

**Figure 14.** Figure 14: Goal-metrics 13-24. 6.3 Curriculum Design Though the challenge of architecting a good curriculum is a research area in its own right, we have a few practical findings from our experience designing a curriculum. Tier-Based Structuring of Goals. Training independent pol…

**Figure 15.** Figure 15: Goal-metrics 25-36. • Initialization: Goal 1 • Basic Skills: Goals 2–21 • Applied Skills: Goals 22–41 • Advanced Goals: Goals 42–43 In this context, skills refer to goals that focus the agent on using specific actions either for their own sake (basic skills) or to ach…

**Figure 16.** Figure 16: Goal-metrics 37-43. 0.75 and a compromise rate of 10%, significantly outperforming the reverse goal order, which failed to progress through the curriculum. Importance of Ascending Difficulty. Structuring curriculums with ascending difficulty is essential, as front-loa…

**Figure 17.** Figure 17: Performance comparison of three policies. One policy corresponds to training with the fixed task selection [PITH_FULL_IMAGE:figures/full_fig_p022_17.png]

**Figure 18.** Figure 18: Performance comparison of multiple task selection strategies. One strategy uses difficulty levels to drive [PITH_FULL_IMAGE:figures/full_fig_p023_18.png]

**Figure 19.** Figure 19: Performance comparison of two policies. One policy was trained via a fixed task, with the goal of mitigating [PITH_FULL_IMAGE:figures/full_fig_p024_19.png]

**Figure 20.** Figure 20: Comparison in learning performance between two learning strategies that only differ in the actions available [PITH_FULL_IMAGE:figures/full_fig_p025_20.png]

**Figure 21.** Figure 21: Performance comparison of algorithms with the same dynamic task selection strategy. [PITH_FULL_IMAGE:figures/full_fig_p026_21.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 32 canonical work pages

[29]

Winder, and Ahmad Ridley

Andres Molina-Markham, Ransom K. Winder, and Ahmad Ridley. 2021. Network Defense is Not a Game.CoRR abs/2104.10262 (2021).https://arxiv.org/abs/2104.10262 15 Training RL Agents for Multi-Objective Network Defense TasksA PREPRINT

arXiv 2021
[1]

CAGE Challenge 4

2024. CAGE Challenge 4. (2024). https://codalab.lisn.upsaclay.fr/competitions/17672#results

work page 2024
[2]

Alex Andrew, Sam Spillard, Joshua Collyer, and Neil Dhir. 2022. Developing Optimal Causal Cyber-Defence Agents via Cyber Security Simulation. InWorkshop on Machine Learning for Cybersecurity (ML4Cyber)

work page 2022
[3]

Marcin Andrychowicz, Anton Raichuk, Piotr Sta´nczyk, Manu Orsini, Sertan Girgin, Raphaël Marinier, Leonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, Sylvain Gelly, and Olivier Bachem. 2021. What Matters for On-Policy Deep Actor-Critic Methods? A Large-Scale Study. InInternational Conference on Learning Representations.https://openrevie...

work page 2021
[4]

Andy Applebaum, Camron Dennler, Patrick Dwyer, Marina Moskowitz, Harold Nguyen, Nicole Nichols, Nicole Park, Paul Rachwalski, Frank Rau, Adrian Webster, and Melody Wolk. 2022. Bridging Automated to Autonomous Cyber Defense: Foundational Analysis of Tabular Q-Learning. InProceedings of the 15th ACM Workshop on Artificial Intelligence and Security. ACM, Los...

arXiv 2022
[5]

Callum Baillie, Maxwell Standen, Jonathon Schwartz, Michael Docking, David Bowman, and Junae Kim. 2020. Cyborg: An autonomous cyber operations research gym.arXiv preprint arXiv:2002.10667(2020)

arXiv 2020
[6]

David Balduzzi, Marta Garnelo, Yoram Bachrach, Wojciech Czarnecki, Julien Perolat, Max Jaderberg, and Thore Graepel. 2019. Open-ended learning in symmetric zero-sum games. InInternational Conference on Machine Learning. PMLR, 434–443

work page 2019
[7]

Ahmad Hoirul Basori and Sharaf Jameel Malebary. 2020. Deep Reinforcement Learning for Adaptive Cyber Defense and Attacker’s Pattern Identification. InAdvances in Cyber Security Analytics and Decision Systems. Springer, 15–26

work page 2020

Show all 48 references

[8]

Elizabeth Bates, Vasilios Mavroudis, and Chris Hicks. 2023. Reward Shaping for Happier Autonomous Cyber Security Agents. InProceedings of the 16th ACM Workshop on Artificial Intelligence and Security. ACM, Copenhagen Denmark, 221–232.https://doi.org/10.1145/3605764.3623916

2023
[9]

Roman Beltiukov, Wenbo Guo, Arpit Gupta, and Walter Willinger. 2023. In Search of NetUnicorn: A Data- Collection Platform to Develop Generalizable ML Models for Network Security Problems. InProceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security (...

2023
[10]

Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. InICML ’09

2009
[11]

Yash Chandak, Georgios Theocharous, James Kostas, Scott Jordan, and Philip Thomas. 2019. Learning Action Representations for Reinforcement Learning. InProceedings of the 36th International Conference on Machine Learning (Proceedings of Machine Learning Research), Kamalika Chau...

2019
[12]

Stephane Doncieux, David Filliat, Natalia Díaz-Rodríguez, Timothy Hospedales, Richard Duro, Alexandre Coninx, Diederik M Roijers, Benoît Girard, Nicolas Perrin, and Olivier Sigaud. 2018. Open-ended learning: a conceptual framework based on representational redescription.Fronti...

2018
[13]

Bartlett, Ilya Sutskever, and Pieter Abbeel

Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbeel. 2016. RL\^2\: Fast Reinforcement Learning via Slow Reinforcement Learning. (2016). https://doi.org/10.48550/ARXIV. 1611.02779

2016 doi arXiv
[14]

Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. 2018. Diversity is All You Need: Learning Skills without a Reward Function. (2018).https://doi.org/10.48550/ARXIV.1802.06070

2018 doi arXiv
[15]

Ming Feng and Hao Xu. 2017. Deep reinforecement learning based optimal defense for cyber-physical system in presence of unknown cyber-attack. In2017 IEEE Symposium Series on Computational Intelligence (SSCI). IEEE, Honolulu, HI, 1–8.https://doi.org/10.1109/SSCI.2017.8285298

2017
[16]

Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine. 2019. Online Meta-Learning. (2019). https://doi.org/10.48550/ARXIV.1902.08438

2019 doi arXiv
[17]

Myles Foley, Chris Hicks, Kate Highnam, and Vasilios Mavroudis. 2022. Autonomous Network Defence using Reinforcement Learning. InProceedings of the 2022 ACM on Asia Conference on Computer and Communications Security. 1252–1254

2022
[18]

Rohit Gangupantulu, Tyler Cody, Paul Park, Abdul Rahman, Logan Eisenbeiser, Dan Radke, Ryan Clark, and Christopher Redino. 2022. Using cyber terrain in reinforcement learning for penetration testing. In2022 IEEE International Conference on Omni-layer Intelligent Systems (COINS...

2022
[19]

Kim Hammar and Rolf Stadler. 2024. Learning Near-Optimal Intrusion Responses Against Dynamic Attackers. IEEE Transactions on Network and Service Management21, 1 (Feb. 2024), 1158–1177. https://doi.org/10. 1109/TNSM.2023.3293413

2024
[20]

Aaron Havens, Zhanhong Jiang, and Soumik Sarkar. [n. d.]. Online Robust Policy Learning in the Presence of Unknown Adversaries. ([n. d.])
[21]

A. S. Jacobs, R. Beltiukov, W. Willinger, R. A. Ferreira, A. Gupta, and L. Z. Granville. 2022. AI/ML and Network Security: The Emperor has no Clothes. InProceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security (CCS ’22). Association for Computing M...

2022
[22]

Jaromír Janisch, Tomáš Pevný, and Viliam Lisý. 2023. NASimEmu: Network Attack Simulator & Emulator for Training Agents Generalizing to Novel Scenarios. (Aug. 2023). http://arxiv.org/abs/2305.17246 arXiv:2305.17246 [cs]

2023 arXiv
[23]

Dmitry Kalashnikov, Jacob Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, and Karol Hausman. 2021. MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale. (2021).https://doi.org/10.48550/ARXIV.2104.08212

2021 doi arXiv
[24]

Yuji Kanagawa and Tomoyuki Kaneko. 2019. Rogue-gym: A new challenge for generalization in reinforcement learning. In2019 IEEE Conference on Games (CoG). IEEE, 1–8

2019
[25]

El Rami, Adrian Taylor, James Hailing Rao, and Thomas Kunz

Li Li, Jean-Pierre S. El Rami, Adrian Taylor, James Hailing Rao, and Thomas Kunz. 2022. Enabling A Network AI Gym for Autonomous Cyber Agents. In2022 IEEE International Conference on Computational Science and Computational Intelligence (CSCI). IEEE, 172–177

2022
[26]

Li Li, Raed Fayad, and Adrian Taylor. 2021. Cygil: A cyber gym for training autonomous agents over emulated network systems.arXiv preprint arXiv:2109.03331(2021)

2021 arXiv
[27]

Xiangyu Liu, Hangtian Jia, Ying Wen, Yujing Hu, Yingfeng Chen, Changjie Fan, Zhipeng Hu, and Yaodong Yang
[28]

Chris McCarthy. 2023. Primaite. (2023). https://github.com/Autonomous-Resilient-Cyber-Defence/ PrimAITE

2023
[30]

Jakob Nyberg and Pontus Johnson. 2023. Training Automated Defense Strategies Using Graph-based Cyber Attack Simulations. (April 2023).http://arxiv.org/abs/2304.11084arXiv:2304.11084 [cs]

2023 arXiv
[31]

Jakob Nyberg, Pontus Johnson, and Andras Mehes. 2022. Cyber threat response using reinforcement learning in graph-based attack simulations. InNOMS 2022-2022 IEEE/IFIP Network Operations and Management Symposium. IEEE, Budapest, Hungary, 1–4.https://doi.org/10.1109/NOMS54207.20...

2022
[32]

2019.Solving Rubik’s Cube with a Robot Hand

OpenAI, Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojciech Zaremba, and Lei Zh...

2019
[33]

Nicolas Perez-Nieves, Yaodong Yang, Oliver Slumbers, David H Mguni, Ying Wen, and Jun Wang. 2021. Modelling behavioural diversity for learning in open-ended games. InInternational conference on machine learning. PMLR, 8514–8524

2021
[34]

Vieri Giuliano Santucci, Pierre-Yves Oudeyer, Andrew Barto, and Gianluca Baldassarre. 2020. Intrinsically motivated open-ended learning in autonomous robots. (2020)

2020
[35]

Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. 2019. Dynamics-Aware Unsupervised Discovery of Skills. (2019).https://doi.org/10.48550/ARXIV.1907.01657

2019 doi arXiv
[36]

Archit Sharma, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. 2021. Autonomous Rein- forcement Learning via Subgoal Curricula. (2021).https://doi.org/10.48550/ARXIV.2107.12931

2021 doi arXiv
[37]

Maxwell Standen, Martin Lucas, David Bowman, Toby J Richer, Junae Kim, and Damian Marriott. 2021. Cyborg: A gym for the development of autonomous cyber agents.arXiv preprint arXiv:2108.09118(2021)

2021 arXiv
[38]

Strom, Andy Applebaum, Douglas P

Blake E. Strom, Andy Applebaum, Douglas P. Miller, Kathryn C. Nickels, Adam G. Pennington, and Cody B. Thomas. 2018. MITRE ATT&CK™ : Design and Philosophy. (July 2018). https://www.mitre.org/ publications/technical-papers/mitre-attack-design-and-philosophy

2018
[39]

Zhi-Xuan Tan. 2022. PDDL.jl: An Extensible Interpreter and Compiler Interface for Fast and Flexible AI Planning. (2022)

2022
[40]

Microsoft Defender Research Team, Christian Seifert, Michael Betser, William Blum, James Bono, and Kate Farris. 2021. CyberBattleSim. (2021).https://github.com/microsoft/cyberbattlesim

2021
[41]

Open Ended Learning Team, Adam Stooke, Anuj Mahajan, Catarina Barros, Charlie Deck, Jakob Bauer, Jakub Sygnowski, Maja Trebacz, Max Jaderberg, Michaël Mathieu, Nat McAleese, Nathalie Bradley-Schmieg, Nathaniel Wong, Nicolas Porcel, Roberta Raileanu, Steph Hughes-Fitt, Valentin...
[42]

Khuong Tran, Ashlesha Akella, Maxwell Standen, Junae Kim, David Bowman, Toby Richer, and Chin-Teng Lin
[43]

https: //arxiv.org/abs/2107.12808

Open-Ended Learning Leads to Generally Capable Agents.CoRRabs/2107.12808 (2021). https: //arxiv.org/abs/2107.12808

2021 arXiv
[44]

Shicheng Zhou, Jingju Liu, Dongdong Hou, Xiaofeng Zhong, and Yue Zhang. 2021. Autonomous penetration testing based on improved deep q-network.Applied Sciences11, 19 (2021), 8823

2021
[45]

Deep hierarchical reinforcement agents for automated penetration testing.arXiv preprint arXiv:2109.06449 (2021)

2021 arXiv
[46]

Yaodong Yang, Jun Luo, Ying Wen, Oliver Slumbers, Daniel Graves, Haitham Bou Ammar, Jun Wang, and Matthew E Taylor. 2021. Diverse auto-curriculum is critical for successful real-world multiagent learning systems. arXiv preprint arXiv:2102.07659(2021)

2021 arXiv
[48]

i d " : 1 ,

Zhengwei Zhu, Miaojie Chen, Chenyang Zhu, and Yanping Zhu. 2024. Effective defense strategies in network security using improved double dueling deep Q-network.Computers & Security136 (Jan. 2024), 103578. https://doi.org/10.1016/j.cose.2023.103578 16 Training RL Agents for Mult...

2024
[2021]

Towards unifying behavioral and response diversity for open-ended learning in zero-sum games.Advances in Neural Information Processing Systems34 (2021), 941–952

2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [29]

Winder, and Ahmad Ridley

Andres Molina-Markham, Ransom K. Winder, and Ahmad Ridley. 2021. Network Defense is Not a Game.CoRR abs/2104.10262 (2021).https://arxiv.org/abs/2104.10262 15 Training RL Agents for Multi-Objective Network Defense TasksA PREPRINT

arXiv 2021

[2] [1]

CAGE Challenge 4

2024. CAGE Challenge 4. (2024). https://codalab.lisn.upsaclay.fr/competitions/17672#results

work page 2024

[3] [2]

Alex Andrew, Sam Spillard, Joshua Collyer, and Neil Dhir. 2022. Developing Optimal Causal Cyber-Defence Agents via Cyber Security Simulation. InWorkshop on Machine Learning for Cybersecurity (ML4Cyber)

work page 2022

[4] [3]

Marcin Andrychowicz, Anton Raichuk, Piotr Sta´nczyk, Manu Orsini, Sertan Girgin, Raphaël Marinier, Leonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, Sylvain Gelly, and Olivier Bachem. 2021. What Matters for On-Policy Deep Actor-Critic Methods? A Large-Scale Study. InInternational Conference on Learning Representations.https://openrevie...

work page 2021

[5] [4]

Andy Applebaum, Camron Dennler, Patrick Dwyer, Marina Moskowitz, Harold Nguyen, Nicole Nichols, Nicole Park, Paul Rachwalski, Frank Rau, Adrian Webster, and Melody Wolk. 2022. Bridging Automated to Autonomous Cyber Defense: Foundational Analysis of Tabular Q-Learning. InProceedings of the 15th ACM Workshop on Artificial Intelligence and Security. ACM, Los...

arXiv 2022

[6] [5]

Callum Baillie, Maxwell Standen, Jonathon Schwartz, Michael Docking, David Bowman, and Junae Kim. 2020. Cyborg: An autonomous cyber operations research gym.arXiv preprint arXiv:2002.10667(2020)

arXiv 2020

[7] [6]

David Balduzzi, Marta Garnelo, Yoram Bachrach, Wojciech Czarnecki, Julien Perolat, Max Jaderberg, and Thore Graepel. 2019. Open-ended learning in symmetric zero-sum games. InInternational Conference on Machine Learning. PMLR, 434–443

work page 2019

[8] [7]

Ahmad Hoirul Basori and Sharaf Jameel Malebary. 2020. Deep Reinforcement Learning for Adaptive Cyber Defense and Attacker’s Pattern Identification. InAdvances in Cyber Security Analytics and Decision Systems. Springer, 15–26

work page 2020