Pith. sign in

REVIEW 2 major objections 6 minor 145 references

LLM penetration-testing agents advanced through four bottleneck-driven phases, while CTF platforms became dual evaluation-and-training infrastructure, leaving three linked open challenges.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 09:09 UTC pith:E7XJDHAN

load-bearing objection Solid, usable map of LLM pentest agents: the 81-paper inventory and CTF dual-use framing are the real value; the four-phase bottleneck story is a clean narrative, not a proven causal law. the 2 major comments →

arxiv 2607.02605 v1 pith:E7XJDHAN submitted 2026-07-01 cs.SE

A Survey of LLM-Driven Penetration Testing: Taxonomy, Co-Evolution, and Open Challenges

Zheyuan He , Jiaxun Dong , Zihao Li , Ting Chen , Gelei Deng , Feng Luo , Jinkun Ji , Yuanlong Cao
show 1 more author
Xiapu Luo
This is my paper
classification cs.SE
keywords Large Language ModelAgentsPenetration testingCyber securityCTF platformsRLVRAgent4Pentest
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This survey of 81 papers from 2023–2026 argues that autonomous LLM-based penetration-testing systems, called Agent4Pentest, grew without a shared map of the field. The authors organize the literature into six categories and show that agent designs moved through four phases—from text-only planners that still needed humans to run commands, through tool-using and multi-agent systems, to agents trained with reinforcement learning on verifiable outcomes such as flag capture. Each phase shift, they claim, was forced by a specific capability limit the previous generation could not clear. They further show that Capture-the-Flag platforms changed from mere testbeds into the main substrate for both scoring agents and training them with rewards, and that domain-specific tools win efficiency only inside narrow tasks that existing benchmarks make hard to compare. A sympathetic reader cares because the same co-evolution that accelerates progress also produces unreliable scores, weak multi-stage attack performance, and scarce training data—three problems the authors say cannot be fixed separately.

Core claim

The central claim is that Agent4Pentest architectures and their evaluation infrastructure have co-evolved in four bottleneck-driven phases, culminating in Reinforcement Learning with Verifiable Rewards (RLVR), while CTF platforms shifted from evaluation-only testbeds into dual-purpose training substrates; across a six-category taxonomy of 81 papers, the remaining gaps—evaluation reliability, multi-stage attack limits, and training-data scarcity—are structurally linked.

What carries the argument

The four-phase architectural evolution (text-only reasoning → tool-augmented single agents → multi-agent coordination → RLVR) together with the six-category taxonomy; each phase transition is framed as the response to a distinct bottleneck (execution autonomy, context management, training-data scarcity, sample efficiency), and the taxonomy supplies the comparative frame.

Load-bearing premise

The claim that each architectural phase was forced by a single capability bottleneck, rather than by concurrent model improvements, publication trends, or parallel design choices.

What would settle it

If a careful re-dating of the 81 papers shows heavy temporal overlap among the four phases, or if high-performing systems routinely skip intermediate phases without hitting the claimed bottlenecks, the bottleneck-driven co-evolution story would not hold.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • RLVR shifts capability acquisition from imitating expert demonstrations to reward-driven self-improvement, enabling strategies outside human writeups.
  • CTF platforms and writeup archives become central dual infrastructure for both evaluation and RL training, tightening train–test coupling.
  • Domain-specific frameworks gain efficiency via formal state encoding, knowledge injection, constrained actions, and specialized oracles, but remain hard to compare across domains.
  • The field is already expanding from pure offense into adversarial defense and compliance oversight of autonomous agents.
  • Evaluation reliability, multi-stage attack performance, and training-data scarcity cannot be solved independently.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If binary success metrics stay dominant, published gains will increasingly measure format familiarity rather than transferable real-world skill.
  • Standard protocols that cleanly separate CTF writeup training data from evaluation tasks will matter as much as new agent architectures once RLVR is routine.
  • Without shared benchmarks that admit commercial automated pentesting products, research claims of progress cannot be grounded against deployed capability.
  • Specialization patterns that work for fully observable, precisely verifiable surfaces (such as privilege escalation) are likely to spread first to other closed domains before open-ended enterprise networks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. This survey analyzes 81 Agent4Pentest papers (2023–June 2026) and organizes them into a six-category taxonomy (benchmarks, general-purpose systems, domain-specific frameworks, CTF systems, defense, surveys). It traces a four-phase architectural evolution from text-only reasoning agents through tool-augmented and multi-agent systems to RLVR-trained agents, arguing that each transition is driven by a distinct capability bottleneck. Key claims include: CTF platforms now serve dual evaluation/RL-training roles; domain-specific systems share four specialization mechanisms but remain hard to compare; and evaluation reliability, multi-stage attack performance, and training-data scarcity are structurally linked open challenges. The manuscript supplies a full corpus table (Table I), venue/year statistics, coverage comparison with prior surveys (Table III), and detailed category analyses (§IV–§VIII).

Significance. If the inventory and co-evolution framing hold, the paper supplies a usable shared vocabulary and comparison scaffold for a fragmented, preprint-heavy subfield. Strengths include an enumerated 81-paper corpus with architecture/MA/RL/LLM/Env tags (Table I), explicit dual-author open coding and inclusion criteria (§II), concrete coverage gaps versus prior surveys (Table III), and documentation of CTF dual use and domain specialization mechanisms that prior SoKs treat only partially. These contributions are valuable even if the causal bottleneck narrative is read more cautiously as chronological organization rather than strict causal history.

major comments (2)
  1. §III-B and Fig. 5 present the four-phase sequence as bottleneck-driven co-evolution (execution autonomy → context management → training-data scarcity → sample efficiency). The corpus and year tags support chronological accumulation, but the manuscript does not quantify phase overlap, concurrent multi-type publications, or alternative drivers (model scale, benchmark fashion, product choices). The causal gloss is load-bearing for the abstract’s co-evolution claim; either add a short robustness discussion (overlap counts, concurrent systems) or soften the language to chronological organization of capability limits so the inventory remains the primary contribution.
  2. §IX asserts that evaluation reliability, multi-stage performance, and training-data scarcity are “structurally linked” and cannot be addressed independently. The category analyses (§IV–§VII) supply supporting examples (contamination, binary metrics, sparse rewards, purpose-built vs shared benchmark gaps), but the linkage is asserted rather than formalized. A brief dependency sketch or explicit cross-references tying each open challenge to concrete evidence in earlier sections would make this central claim falsifiable rather than rhetorical.
minor comments (6)
  1. Abstract and §I use both “Agents4Pentest” and “Agent4Pentest”; standardize the class name.
  2. Table I header and body: Category II is listed as 34 in one place and 36 in the taxonomy summary (Table II); reconcile the count.
  3. Fig. 5 caption and body contain duplicated sentences about D-CIPHER-style context management in the CTF section narrative; clean residual copy-paste.
  4. §II: state the exact search end date and any dual-coding agreement statistic if available; “June 2026” is given but inter-rater reliability is not.
  5. Several success-rate comparisons across purpose-built vs shared benchmarks (Figs. 14, 16, 19) are carefully caveated in text; ensure figure captions restate non-comparability so they are not read as direct capability rankings.
  6. Category V has only two papers; a one-sentence note on why defense is retained as a full category rather than folded into surveys/position would help readers.

Circularity Check

0 steps flagged

No significant circularity: a literature survey whose taxonomy and four-phase narrative organize an external 81-paper corpus rather than redefine success metrics or force conclusions by construction.

full rationale

This is a systematic survey of Agent4Pentest (81 papers, 2023–2026). Its load-bearing content is an inventory, a six-category taxonomy derived by open coding of primary contributions, chronological architectural phases, and open challenges grounded in cited benchmarks and systems. Claims such as CTF dual-use, RLVR as a shift from imitation to verifiable rewards, domain specialization mechanisms, and linked evaluation/multi-stage/data gaps are supported by external papers (Tables I–IV, Figs. 2–19) and explicit comparisons to prior surveys (Table III). Taxonomy labels and phase boundaries are author constructs, but they classify cited systems rather than fit parameters and re-label them as predictions. Minor self-positioning against prior surveys is normal survey practice and not load-bearing for the inventory. The soft interpretive claim that each architectural transition is bottleneck-driven is a causal gloss on chronology, not a circular reduction of a result to its inputs. No self-definitional equations, fitted-input-as-prediction, uniqueness theorems imported from the authors, or ansatz smuggled via self-citation appear. Score 1 reflects only ordinary survey self-positioning; the derivation chain is self-contained against the external corpus.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 4 invented entities

The paper’s load-bearing content is classificatory and historical, not parametric. It depends on a scoped definition of Agent4Pentest, a literature-search protocol, and two invented organizing schemes (six categories; four phases). No fitted physical constants appear; free choices are coding boundaries and inclusion of two non-LLM baselines.

free parameters (2)
  • Corpus cutoff and size (81 papers through June 2026)
    The inventory and category counts depend on the search window and inclusion threshold; different cutoffs would change N and phase timing.
  • Six-category primary-contribution assignment rule
    Each paper is forced into exactly one category via iterative open coding; alternative primary labels would reallocate counts and dependencies in Fig. 4.
axioms (4)
  • domain assumption An Agent4Pentest paper must propose, evaluate, or benchmark an LLM-based agent for penetration testing or closely related offensive tasks, with a concrete technical contribution.
    §II inclusion criteria define the field boundary; static analysis / IDS-only papers are excluded by construction.
  • ad hoc to paper Architectural history is usefully partitioned into four successive phases, each resolving one bottleneck and exposing the next.
    §III-B and Fig. 5 impose a linear bottleneck narrative on chronological systems; this is an interpretive model, not a theorem.
  • domain assumption Binary task-completion metrics and CTF-style isolation systematically risk overestimating real-world multi-stage capability.
    §IV-C relies on cited contamination and multi-host/WAF results; treated as a standing evaluation premise for open challenges.
  • ad hoc to paper Two non-LLM systems (ChainReactor PDDL planner; Li et al. DQN) are admissible as architectural baselines.
    §II retains them despite the LLM-agent scope because their loops are analogous; this expands comparison but is a survey design choice.
invented entities (4)
  • Agent4Pentest class no independent evidence
    purpose: Name and bound the class of LLM agents that autonomously conduct penetration testing.
    Terminological umbrella introduced in the introduction; useful but not independently measured outside this framing.
  • Six-category taxonomy (Benchmarks, General AutoPT, Domain-specific, CTF systems, Defense, Surveys) no independent evidence
    purpose: Partition the 81-paper corpus by primary contribution and structural role.
    Derived by open coding in §III-A; alternative taxonomies are possible.
  • Four-phase architectural evolution ending in RLVR no independent evidence
    purpose: Explain capability growth as bottleneck-driven transitions culminating in verifiable-reward training.
    Central narrative device of the survey; phases are author-imposed on published systems.
  • Four specialization mechanisms (formal state encoding, domain knowledge injection, constrained action spaces, specialized verification oracles) no independent evidence
    purpose: Abstract recurring design patterns across domain-specific frameworks.
    §VII-B synthesis; mechanisms are descriptive clusters, not separately validated constructs.

pith-pipeline@v1.1.0-grok45 · 43634 in / 3143 out tokens · 35986 ms · 2026-07-12T09:09:21.972369+00:00 · methodology

0 comments
read the original abstract

Agents4Pentest, an emerging class of LLM-based autonomous penetration testing systems, has become a rapidly growing area in security research. Despite this growth, the field still lacks a unified taxonomy, a systematic understanding of how agent architectures and evaluation benchmarks have co-evolved, and a clear characterization of remaining capability and reliability gaps. This survey addresses these gaps through a systematic analysis of 81 papers between 2023 and 2026. We organize the literature into six categories: evaluation benchmarks, general-purpose systems, domain-specific frameworks, CTF-based systems, defense-oriented research, and surveys. We further trace a four-phase architectural evolution from text-only reasoning agents to agents trained with Reinforcement Learning with Verifiable Rewards (RLVR), showing that each transition is driven by a distinct capability bottleneck. Our analysis yields several key findings. First, RLVR marks a shift in capability acquisition from imitation of expert demonstrations to reward-driven self-improvement, enabling agents to discover previously undocumented attack strategies. Second, CTF platforms have evolved from evaluation testbeds into dual-purpose infrastructure for both agent evaluation and RL training. Third, domain-specific frameworks improve efficiency through recurring specialization mechanisms, but their gains remain largely confined to narrow task classes and are difficult to compare across domains because existing evaluations rely on different benchmarks. Fourth, the field is expanding beyond offensive automation toward adversarial defense and security compliance. Across these categories, we identify three structurally linked open challenges: evaluation reliability, limited performance on multi-stage attack scenarios, and scarcity of high-quality training data.

Figures

Figures reproduced from arXiv: 2607.02605 by Feng Luo, Gelei Deng, Jiaxun Dong, Jinkun Ji, Ting Chen, Xiapu Luo, Yuanlong Cao, Zheyuan He, Zihao Li.

Figure 2
Figure 2. Figure 2: Papers by year and six-category taxonomy across the 81 papers. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Distribution by publication venues across the 81 surveyed papers. [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Research landscape of the 81 surveyed papers across six categories. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Four-phase architectural evolution of Agent4Pentest systems. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: Distribution of primary evaluation metrics across the 19 benchmarks [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Contamination effect: removing 71 memorisation events detected by [PITH_FULL_IMAGE:figures/full_fig_p009_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Evaluation variance: repeating an identical attack scenario 100 times [PITH_FULL_IMAGE:figures/full_fig_p009_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Reported results of the four CTF agent systems, grouped by evalua [PITH_FULL_IMAGE:figures/full_fig_p010_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Knowledge–performance gap on CTF tasks. GPT-4o achieves 87.83% [PITH_FULL_IMAGE:figures/full_fig_p011_11.png] view at source ↗
Figure 13
Figure 13. Figure 13: General-purpose Agent4Pentest systems’ four-phase architectural [PITH_FULL_IMAGE:figures/full_fig_p013_13.png] view at source ↗
Figure 15
Figure 15. Figure 15: Four core components of general-purpose Agent4Pentest systems and [PITH_FULL_IMAGE:figures/full_fig_p013_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Performance of representative Agent4Pentest systems across three [PITH_FULL_IMAGE:figures/full_fig_p015_16.png] view at source ↗
Figure 18
Figure 18. Figure 18: Presence of four architectural specialization mechanisms across the [PITH_FULL_IMAGE:figures/full_fig_p016_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Reported success rates across three evaluation settings for [PITH_FULL_IMAGE:figures/full_fig_p017_19.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

145 extracted references · 27 linked inside Pith

  1. [1]

    Service grid fed- eration architecture for heterogeneous domains,

    Y . Murakami, M. Tanaka, D. Lin, and T. Ishida, “Service grid fed- eration architecture for heterogeneous domains,” in2012 IEEE Ninth International Conference on Services Computing. IEEE, 2012, pp. 539–546

    2012

  2. [2]

    Adaptive energy-aware computation offloading for cloud of things systems,

    Y . Nan, W. Li, W. Bao, F. C. Delicato, P. F. Pires, Y . Dou, and A. Y . Zomaya, “Adaptive energy-aware computation offloading for cloud of things systems,”IEEE access, vol. 5, pp. 23 947–23 957, 2017

    2017

  3. [3]

    Dynamic service invocation control in service composition environments,

    D. Lin, Y . Murakami, and M. Tanaka, “Dynamic service invocation control in service composition environments,” in2010 IEEE Interna- tional Conference on Services Computing. IEEE, 2010, pp. 25–32

    2010

  4. [4]

    Fine-grained two- factor access control for web-based cloud computing services,

    J. K. Liu, M. H. Au, X. Huang, R. Lu, and J. Li, “Fine-grained two- factor access control for web-based cloud computing services,”IEEE Transactions on Information Forensics and Security, vol. 11, no. 3, pp. 484–497, 2015

    2015

  5. [5]

    Trust-based access control for secure cloud computing,

    I. Ray and I. Ray, “Trust-based access control for secure cloud computing,” inHigh Performance Cloud Auditing and Applications. Springer, 2013, pp. 189–213

    2013

  6. [6]

    Weidman,Penetration Testing: A Hands-On Introduction to Hack- ing

    G. Weidman,Penetration Testing: A Hands-On Introduction to Hack- ing. No Starch Press, 2014

    2014

  7. [7]

    Security and privacy challenges in cloud computing environments,

    H. Takabi, J. B. Joshi, and G.-J. Ahn, “Security and privacy challenges in cloud computing environments,”IEEE Security & Privacy, vol. 8, no. 6, pp. 24–31, 2010

    2010

  8. [8]

    Dynamic security risk manage- ment using bayesian attack graphs,

    N. Poolsappasit, R. Dewri, and I. Ray, “Dynamic security risk manage- ment using bayesian attack graphs,”IEEE Transactions on Dependable and Secure Computing, vol. 9, no. 1, pp. 61–74, 2011

    2011

  9. [9]

    Two-factor data security protection mechanism for cloud storage system,

    J. K. Liu, K. Liang, W. Susilo, J. Liu, and Y . Xiang, “Two-factor data security protection mechanism for cloud storage system,”IEEE Transactions on Computers, vol. 65, no. 6, pp. 1992–2004, 2015

    1992

  10. [10]

    Engebretson,The basics of hacking and penetration testing: ethical hacking and penetration testing made easy

    P. Engebretson,The basics of hacking and penetration testing: ethical hacking and penetration testing made easy. Elsevier, 2013

    2013

  11. [11]

    Examining penetration tester behavior in the collegiate penetration testing competition,

    B. S. Meyers, S. F. Almassari, B. N. Keller, and A. Meneely, “Examining penetration tester behavior in the collegiate penetration testing competition,”ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 31, no. 3, pp. 1–25, 2022

    2022

  12. [12]

    Penetration testing–reconnaissance with nmap tool,

    N. Kaur, “Penetration testing–reconnaissance with nmap tool,”Inter- national Journal of Advanced Research in Computer Science, vol. 8, no. 3, pp. 844–846, 2017

    2017

  13. [13]

    Toss a fault to your witcher: Applying grey-box coverage-guided mutational fuzzing to detect sql and command injection vulnerabilities,

    E. Trickel, F. Pagani, C. Zhu, L. Dresel, G. Vigna, C. Kruegel, R. Wang, T. Bao, Y . Shoshitaishvili, and A. Doup´e, “Toss a fault to your witcher: Applying grey-box coverage-guided mutational fuzzing to detect sql and command injection vulnerabilities,” in2023 IEEE symposium on security and privacy (SP). IEEE, 2023, pp. 2658–2675

    2023

  14. [14]

    {FUGIO}: Automatic exploit generation for{PHP}object injection vulnerabilities,

    S. Park, D. Kim, S. Jana, and S. Son, “{FUGIO}: Automatic exploit generation for{PHP}object injection vulnerabilities,” in31st USENIX Security Symposium (USENIX Security 22), 2022, pp. 197–214

    2022

  15. [15]

    {ChainReactor}: Automated privilege es- calation chain discovery via{AI}planning,

    G. De Pasquale, I. Grishchenko, R. Iesari, G. Pizarro, L. Cavallaro, C. Kruegel, and G. Vigna, “{ChainReactor}: Automated privilege es- calation chain discovery via{AI}planning,” in33rd USENIX Security Symposium (USENIX Security 24), 2024, pp. 5913–5929

    2024

  16. [16]

    A comprehensive detection method for the lateral movement stage of apt attacks,

    D. He, H. Gu, S. Zhu, S. Chan, and M. Guizani, “A comprehensive detection method for the lateral movement stage of apt attacks,”IEEE Internet of Things Journal, vol. 11, no. 5, pp. 8440–8447, 2023

    2023

  17. [17]

    A comprehensive overview of large language models,

    H. Naveed, A. U. Khan, S. Qiu, M. Saqib, S. Anwar, M. Usman, N. Akhtar, N. Barnes, and A. Mian, “A comprehensive overview of large language models,”ACM Transactions on Intelligent Systems and Technology, vol. 16, no. 5, pp. 1–72, 2025

    2025

  18. [18]

    Large language models for cyber security: A systematic literature review,

    H. Xu, S. Wang, N. Li, K. Wang, Y . Zhao, K. Chen, T. Yu, Y . Liu, and H. Wang, “Large language models for cyber security: A systematic literature review,”ACM Transactions on Software Engineering and Methodology, 2024

    2024

  19. [19]

    {PentestGPT}: Evaluating and harnessing large language models for automated penetration testing,

    G. Deng, Y . Liu, V . Mayoral-Vilches, P. Liu, Y . Li, Y . Xu, T. Zhang, Y . Liu, M. Pinzger, and S. Rass, “{PentestGPT}: Evaluating and harnessing large language models for automated penetration testing,” in33rd USENIX Security Symposium (USENIX Security 24), 2024, pp. 847–864

    2024

  20. [20]

    Expel: Llm agents are experiential learners,

    A. Zhao, D. Huang, Q. Xu, M. Lin, Y .-J. Liu, and G. Huang, “Expel: Llm agents are experiential learners,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 17, 2024, pp. 19 632– 19 642

    2024

  21. [21]

    Autotool: Efficient tool selection for large language model agents,

    J. Jia and Q. Li, “Autotool: Efficient tool selection for large language model agents,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 40, no. 37, 2026, pp. 31 265–31 273

    2026

  22. [22]

    A survey on the feedback mechanism of llm- based ai agents,

    Z. Liu, X. Bai, K. Chen, X. Chen, X. Li, Y . Xiang, J. Liu, H.-D. Li, Y . Wang, L. Nieet al., “A survey on the feedback mechanism of llm- based ai agents,” inProceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence. International Joint Conferences on Artificial Intelligence, 2025, pp. 10 582–10 592

    2025

  23. [23]

    Spaiware: Uncovering a novel artificial intelligence attack vector through persistent memory in llm applications and agents,

    M. Herrador and J. Rehberger, “Spaiware: Uncovering a novel artificial intelligence attack vector through persistent memory in llm applications and agents,”Future Generation Computer Systems, vol. 174, p. 107994, 2026. 20

    2026

  24. [24]

    Exe- cutable code actions elicit better llm agents,

    X. Wang, Y . Chen, L. Yuan, Y . Zhang, Y . Li, H. Peng, and H. Ji, “Exe- cutable code actions elicit better llm agents,” inForty-first International Conference on Machine Learning, 2024

    2024

  25. [25]

    Vulnbot: Autonomous penetration testing for a multi-agent collaborative framework,

    H. Kong, D. Hu, J. Ge, L. Li, T. Li, and B. Wu, “Vulnbot: Autonomous penetration testing for a multi-agent collaborative framework,”arXiv preprint arXiv:2501.13411, 2025

    Pith/arXiv arXiv 2025

  26. [26]

    Automated penetration testing with llm agents and classical planning,

    L. Wang, X. Shi, Z. Li, Y . Jiang, S. Tan, Y . Jiang, J. Cheng, W. Chen, X. Shen, Z. LIet al., “Automated penetration testing with llm agents and classical planning,”arXiv preprint arXiv:2512.11143, 2025

    arXiv 2025

  27. [27]

    Pentest-r1: Towards autonomous penetration testing reasoning optimized via two-stage reinforcement learning,

    H. Kong, D. Hu, J. Ge, L. Li, H. Li, and T. Li, “Pentest-r1: Towards autonomous penetration testing reasoning optimized via two-stage reinforcement learning,”arXiv preprint arXiv:2508.07382, 2025

    arXiv 2025

  28. [28]

    Cyber- zero: Training cybersecurity agents without runtime,

    T. Y . Zhuo, D. Wang, H. Ding, V . Kumar, and Z. Wang, “Cyber- zero: Training cybersecurity agents without runtime,” inNeurIPS 2025 Fourth Workshop on Deep Learning for Code, 2025

    2025

  29. [29]

    Cybench: A framework for evaluating cybersecurity capabilities and risks of language models,

    A. K. Zhang, N. Perry, R. Dulepet, J. Ji, C. Menders, J. Lin, E. Jones, G. Hussein, S. Liu, D. Jasperet al., “Cybench: A framework for evaluating cybersecurity capabilities and risks of language models,” inInternational Conference on Learning Representations, vol. 2025, 2025, pp. 25 094–25 243

    2025

  30. [30]

    Autopenbench: A vulnerability testing benchmark for generative agents,

    L. Gioacchini, A. Delsanto, I. Drago, M. Mellia, G. Siracusano, and R. Bifulco, “Autopenbench: A vulnerability testing benchmark for generative agents,” inProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track, 2025, pp. 1615–1624

    2025

  31. [31]

    Nyu ctf bench: A scalable open-source benchmark dataset for evaluating llms in offensive security,

    M. Shao, S. Jancheska, M. Udeshi, B. Dolan-Gavitt, H. Xi, K. Milner, B. Chen, M. Yin, S. Garg, P. Krishnamurthyet al., “Nyu ctf bench: A scalable open-source benchmark dataset for evaluating llms in offensive security,”Advances in Neural Information Processing Systems, vol. 37, pp. 57 472–57 498, 2024

    2024

  32. [32]

    Towards effective offensive security llm agents: Hyperparameter tuning, llm as a judge, and a lightweight ctf benchmark,

    M. Shao, N. Rani, K. Milner, H. Xi, M. Udeshi, S. Aggarwal, V . S. C. Putrevu, S. K. Shukla, P. Krishnamurthy, F. Khorramiet al., “Towards effective offensive security llm agents: Hyperparameter tuning, llm as a judge, and a lightweight ctf benchmark,” inProceedings of the AAAI Conference on Artificial Intelligence, 2026

    2026

  33. [33]

    Enigma: Interactive tools substantially assist lm agents in finding security vulnerabilities,

    T. Abramovich, M. Udeshi, M. Shao, K. Lieret, H. Xi, K. Milner, S. Jancheska, J. Yang, C. E. Jimenez, F. Khorramiet al., “Enigma: Interactive tools substantially assist lm agents in finding security vulnerabilities,” inForty-second International Conference on Machine Learning, 2024

    2024

  34. [34]

    Chimera: Harnessing multi- agent llms for automatic insider threat simulation,

    J. Yu, X. Xie, Q. Hu, Y . Ma, and Z. Zhao, “Chimera: Harnessing multi- agent llms for automatic insider threat simulation,” inProceedings of the Network and Distributed System Security Symposium (NDSS), 2026

    2026

  35. [35]

    Awe: Adaptive agents for dynamic web penetration testing,

    A. S. Jaswal and A. Baghel, “Awe: Adaptive agents for dynamic web penetration testing,”Proceedings of the Network and Distributed System Security Symposium (NDSS) Workshop (LAST-X), 2026

    2026

  36. [36]

    Hackers or hallucinators? a comprehensive analysis of llm-based automated penetration testing,

    J. Peng, Z. Li, C. You, Y . Wang, H. Sun, X. Tian, S. Zhang, J. Liu, J. Zhao, R. Liuet al., “Hackers or hallucinators? a comprehensive analysis of llm-based automated penetration testing,”arXiv preprint arXiv:2604.05719, 2026

    Pith/arXiv arXiv 2026

  37. [37]

    Benchmarking practices in llm-driven offensive security: Testbeds, metrics, and experiment design,

    A. Happe and J. Cito, “Benchmarking practices in llm-driven offensive security: Testbeds, metrics, and experiment design,”arXiv preprint arXiv:2504.10112, 2025

    Pith/arXiv arXiv 2025

  38. [38]

    On the surprising efficacy of llms for penetration-testing,

    ——, “On the surprising efficacy of llms for penetration-testing,”arXiv preprint arXiv:2507.00829, 2025

    Pith/arXiv arXiv 2025

  39. [39]

    A unified modeling framework for automated penetration testing,

    Y . Wang, S. Liu, W. Wang, C. Zhou, C. Zhang, J. Jin, and C. Zhu, “A unified modeling framework for automated penetration testing,” Computers & Security, p. 104787, 2025

    2025

  40. [40]

    Automated penetration testing: Formal- ization and realization,

    C. Skandylas and M. Asplund, “Automated penetration testing: Formal- ization and realization,”Computers & Security, vol. 155, p. 104454, 2025

    2025

  41. [41]

    {CTF}:{State-of-the-Art}and building the next generation,

    C. Taylor, P. Arias, J. Klopchic, C. Matarazzo, and E. Dube, “{CTF}:{State-of-the-Art}and building the next generation,” in2017 USENIX Workshop on Advances in Security Education (ASE 17), 2017

    2017

  42. [42]

    Cve-bench: A benchmark for ai agents’ ability to exploit real-worldweb application vulnerabilities,

    Y . Zhu, A. Kellermann, D. Bowman, P. Li, A. Gupta, A. Danda, R. Fang, C. Jensen, E. Ihli, J. Bennet al., “Cve-bench: A benchmark for ai agents’ ability to exploit real-worldweb application vulnerabilities,” Proceedings of Machine Learning Research, vol. 267, pp. 79 850– 79 867, 2025

    2025

  43. [43]

    Guidelines for performing systematic literature reviews in software engineering,

    S. Keeleet al., “Guidelines for performing systematic literature reviews in software engineering,” 2007

    2007

  44. [44]

    Intelligent penetration testing through integrated knowledge graph and historical decision enhancement,

    Q. Li, L. Wen, A. Chattopadhyay, C. Tu, M. Hu, F. Shi, M. Zhang, R. Wang, and Z. Pan, “Intelligent penetration testing through integrated knowledge graph and historical decision enhancement,”IEEE Trans- actions on Dependable and Secure Computing, 2026

    2026

  45. [45]

    Intercode: Stan- dardizing and benchmarking interactive coding with execution feed- back,

    J. Yang, A. Prabhakar, K. Narasimhan, and S. Yao, “Intercode: Stan- dardizing and benchmarking interactive coding with execution feed- back,”Advances in Neural Information Processing Systems, vol. 36, pp. 23 826–23 854, 2023

    2023

  46. [46]

    Llm agents can autonomously exploit one-day vulnerabilities,

    R. Fang, R. Bindu, A. Gupta, and D. Kang, “Llm agents can autonomously exploit one-day vulnerabilities,”arXiv preprint arXiv:2404.08144, 2024

    Pith/arXiv arXiv 2024

  47. [47]

    Got root? a linux priv-esc benchmark,

    A. Happe and J. Cito, “Got root? a linux priv-esc benchmark,”arXiv preprint arXiv:2405.02106, 2024

    Pith/arXiv arXiv 2024

  48. [48]

    Hacksynth: Llm agent and eval- uation framework for autonomous penetration testing,

    L. Muzsai, D. Imolai, and A. Luk ´acs, “Hacksynth: Llm agent and eval- uation framework for autonomous penetration testing,”arXiv preprint arXiv:2412.01778, 2024

    Pith/arXiv arXiv 2024

  49. [49]

    An empirical evaluation of llms for solving offensive security challenges,

    M. Shao, B. Chen, S. Jancheska, B. Dolan-Gavitt, S. Garg, R. Karri, and M. Shafique, “An empirical evaluation of llms for solving offensive security challenges,”arXiv preprint arXiv:2402.11814, 2024

    Pith/arXiv arXiv 2024

  50. [50]

    Catastrophic cyber capabilities benchmark (3cb): Robustly evaluating llm agent cyber offense capabilities,

    A. Anurin, J. Ng, J. Hoelscher-Obermaier, and E. Kran, “Catastrophic cyber capabilities benchmark (3cb): Robustly evaluating llm agent cyber offense capabilities,” inWorkshop on Datasets and Evaluators of AI Safety

  51. [51]

    Towards automated penetration testing: Introducing llm benchmark, analysis, and improve- ments,

    I. Isozaki, M. Shrestha, R. Console, and E. Kim, “Towards automated penetration testing: Introducing llm benchmark, analysis, and improve- ments,” inAdjunct Proceedings of the 33rd ACM Conference on User Modeling, Adaptation and Personalization, 2025, pp. 404–419

    2025

  52. [52]

    Pen- testeval: Benchmarking llm-based penetration testing with modular and stage-level design,

    R. Yang, M. Cheng, G. Deng, T. Zhang, J. Wang, and X. Xie, “Pen- testeval: Benchmarking llm-based penetration testing with modular and stage-level design,”arXiv preprint arXiv:2512.14233, 2025

    arXiv 2025

  53. [53]

    Cybergym: Evaluating ai agents’ real-world cybersecurity capabilities at scale,

    Z. Wang, T. Shi, J. He, M. Cai, J. Zhang, and D. Song, “Cybergym: Evaluating ai agents’ real-world cybersecurity capabilities at scale,” arXiv preprint arXiv:2506.02548, 2025

    arXiv 2025

  54. [54]

    Hackworld: Evaluating computer-use agents on exploit- ing web application vulnerabilities,

    X. Ren, P. Jiang, K. Li, Z. Huang, X. Du, J. Jiang, Z. Xing, J. Sun, and T. Y . Zhuo, “Hackworld: Evaluating computer-use agents on exploit- ing web application vulnerabilities,”arXiv preprint arXiv:2510.12200, 2025

    arXiv 2025

  55. [55]

    Pacebench: A framework for evaluating practical ai cyber-exploitation capabilities,

    Z. Liu, L. Huang, J. Zhang, D. Liu, Y . Tian, and J. Shao, “Pacebench: A framework for evaluating practical ai cyber-exploitation capabilities,” arXiv preprint arXiv:2510.11688, 2025

    arXiv 2025

  56. [56]

    Ctfusion: A ctf-based benchmark for llm agent evaluation,

    D. Lee, G.-e. Bae, and I. Yun, “Ctfusion: A ctf-based benchmark for llm agent evaluation,”arXiv preprint arXiv:2605.11504, 2026

    Pith/arXiv arXiv 2026

  57. [57]

    How reliable are ai attackers against a fixed vulnerable target? a 400-run empirical study of llm penetration testing consis- tency,

    G. T. Erdem, “How reliable are ai attackers against a fixed vulnerable target? a 400-run empirical study of llm penetration testing consis- tency,”arXiv preprint arXiv:2605.30096, 2026

    Pith/arXiv arXiv 2026

  58. [58]

    Cybergym-e2e: Scalable real- world benchmark for ai agents’ end-to-end cybersecurity capabilities,

    T. Shi, R. Rheem, D. Jiang, M. Wang, F. De La Riega, Z. Wang, J. Jiang, A. Cheung, S. Tai, J. Chaet al., “Cybergym-e2e: Scalable real- world benchmark for ai agents’ end-to-end cybersecurity capabilities,” arXiv preprint arXiv:2606.04460, 2026

    Pith/arXiv arXiv 2026

  59. [59]

    Exploitgym: Can ai agents turn security vulnerabilities into real attacks?

    Z. Wang, N. Schiller, H. Li, S. S. Narayana, M. Nasr, N. Carlini, X. Qi, E. Wallace, E. Bursztein, L. Invernizziet al., “Exploitgym: Can ai agents turn security vulnerabilities into real attacks?”arXiv preprint arXiv:2605.11086, 2026

    Pith/arXiv arXiv 2026

  60. [60]

    Penheal: A two-stage llm framework for automated pentesting and optimal remediation,

    J. Huang and Q. Zhu, “Penheal: A two-stage llm framework for automated pentesting and optimal remediation,” inProceedings of the workshop on autonomous cybersecurity, 2023, pp. 11–22

    2023

  61. [61]

    Autoattacker: A large language model guided system to implement automatic cyber-attacks,

    J. Xu, J. W. Stokes, G. McDonald, X. Bai, D. Marshall, S. Wang, A. Swaminathan, and Z. Li, “Autoattacker: A large language model guided system to implement automatic cyber-attacks,”arXiv preprint arXiv:2403.01038, 2024

    Pith/arXiv arXiv 2024

  62. [62]

    Pentest-ai, an llm-powered multi- agents framework for penetration testing automation leveraging mitre attack,

    S. G. Bianou and R. G. Batogna, “Pentest-ai, an llm-powered multi- agents framework for penetration testing automation leveraging mitre attack,” in2024 IEEE International Conference on Cyber Security and Resilience (CSR). IEEE, 2024, pp. 763–770

    2024

  63. [63]

    Pentestagent: Incorporating llm agents to automated penetration testing,

    X. Shen, L. Wang, Z. Li, Y . Chen, W. Zhao, D. Sun, J. Wang, and W. Ruan, “Pentestagent: Incorporating llm agents to automated penetration testing,” inASIA CCS ’25, 2025

    2025

  64. [64]

    Autopentester: An llm agent-based framework for automated pentesting,

    Y . Ginige, A. Niroshan, S. Jain, and S. Seneviratne, “Autopentester: An llm agent-based framework for automated pentesting,” in2025 IEEE 24th International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom). IEEE, 2025, pp. 163–174

    2025

  65. [65]

    Pentest++: Elevating ethical hacking with ai and automation,

    H. S. Al-Sinani and C. J. Mitchell, “Pentest++: Elevating ethical hacking with ai and automation,”arXiv preprint arXiv:2502.09484, 2025

    Pith/arXiv arXiv 2025

  66. [66]

    Rapidpen: Fully automated ip-to-shell penetration testing with llm-based agents,

    S. Nakatani, “Rapidpen: Fully automated ip-to-shell penetration testing with llm-based agents,”arXiv preprint arXiv:2502.16730, 2025

    arXiv 2025

  67. [67]

    Guided reasoning in llm-driven penetration testing using structured attack trees,

    K. Nakano, R. Fayyazi, S. Yang, and M. Zuzak, “Guided reasoning in llm-driven penetration testing using structured attack trees,” inSecond Conference on Language Modeling, 2025. 21

    2025

  68. [68]

    Breachseek: A multi-agent automated penetration tester,

    I. Alshehri, A. Alshehri, A. Almalki, M. Bamardouf, and A. Ak- bar, “Breachseek: A multi-agent automated penetration tester,”arXiv preprint arXiv:2409.03789, 2024

    Pith/arXiv arXiv 2024

  69. [69]

    Controller makes pentesting better: An improved multi-agent auto- mated penetration testing framework,

    X. Geng, N. An, B. Xu, X. Yang, B. Jiang, B. Liu, and J. Liu, “Controller makes pentesting better: An improved multi-agent auto- mated penetration testing framework,” in2025 IEEE 24th International Conference on Trust, Security and Privacy in Computing and Commu- nications (TrustCom). IEEE, 2025, pp. 845–852

    2025

  70. [70]

    Shell or nothing: Real-world benchmarks and memory- activated agents for automated penetration testing,

    W. Mai, G. Hong, Q. Liu, J. Chen, J. Dai, X. Pan, Y . Zhang, and M. Yang, “Shell or nothing: Real-world benchmarks and memory- activated agents for automated penetration testing,”arXiv preprint arXiv:2509.09207, 2025

    arXiv 2025

  71. [71]

    Refpentester: A knowledge- informed self-reflective penetration testing framework based on large language models,

    H. Dai, Y . Li, J. Yan, and Z. Zhang, “Refpentester: A knowledge- informed self-reflective penetration testing framework based on large language models,” in2025 22nd Annual International Conference on Privacy, Security, and Trust (PST). IEEE, 2025, pp. 1–8

    2025

  72. [72]

    Ptfusion: Llm- driven context-aware knowledge fusion for web penetration testing,

    W. Wang, H. Gu, Z. Wu, H. Chen, X. Chen, and F. Shi, “Ptfusion: Llm- driven context-aware knowledge fusion for web penetration testing,” Information Fusion, p. 103731, 2025

    2025

  73. [73]

    Pentestmcp: Llm and mcp based multi-agent framework for automated penetration testing,

    J. Zhai, X. Zhou, H. Miao, Z. Li, Z. Li, and H. Yang, “Pentestmcp: Llm and mcp based multi-agent framework for automated penetration testing,” 2025

    2025

  74. [74]

    xoffense: An ai-driven autonomous penetration testing framework with offensive knowledge-enhanced llms and multi agent systems,

    P. D. Luong, T. G. Le Bao, N. V . Khai Tam, D. H. Nguyen Khoa, N. H. Quyen, V .-H. Phamet al., “xoffense: An ai-driven autonomous penetration testing framework with offensive knowledge-enhanced llms and multi agent systems,”arXiv e-prints, pp. arXiv–2509, 2025

    2025

  75. [75]

    Cai: An open, bug bounty-ready cybersecurity ai,

    V . Mayoral-Vilches, L. J. Navarrete-Lozano, M. Sanz-G ´omez, L. S. Espejo, M. Crespo- ´Alvarez, F. Oca-Gonzalez, F. Balassone, A. Glera- Pic´on, U. Ayucar-Carbajo, J. A. Ruiz-Alcaldeet al., “Cai: An open, bug bounty-ready cybersecurity ai,”arXiv preprint arXiv:2504.06017, 2025

    Pith/arXiv arXiv 2025

  76. [76]

    Redteamllm: an agentic ai framework for offensive security,

    B. Challita and P. Parrend, “Redteamllm: an agentic ai framework for offensive security,” inIFIP International Workshop on Artificial Intelligence for Knowledge Management. Springer, 2025, pp. 337– 354

    2025

  77. [77]

    Red-mirror: Agentic llm-based autonomous penetration testing with reflective verification and knowledge-augmented interac- tion,

    T. V . Khang, N. D. N. Khang, N. H. Khoa, D. T. T. Hien, V .-H. Pham, and P. T. Duy, “Red-mirror: Agentic llm-based autonomous penetration testing with reflective verification and knowledge-augmented interac- tion,”arXiv preprint arXiv:2603.27127, 2026

    arXiv 2026

  78. [78]

    What makes a good llm agent for real-world penetration testing?

    G. Deng, Y . Liu, Y . Li, R. Yang, X. Xie, J. Zhang, H. Qiu, and T. Zhang, “What makes a good llm agent for real-world penetration testing?” arXiv preprint arXiv:2602.17622, 2026

    arXiv 2026

  79. [79]

    Incalmo: An autonomous llm-assisted system for red teaming multi- host networks,

    B. Singer, K. Lucas, L. Adiga, M. Jain, L. Bauer, and V . Sekar, “Incalmo: An autonomous llm-assisted system for red teaming multi- host networks,” inProceedings of the 47th IEEE Symposium on Security and Privacy, 2026

    2026

  80. [80]

    Cipher: Cybersecurity intelligent penetration- testing helper for ethical researcher,

    D. Pratama, N. Suryanto, A. A. Adiputra, T.-T.-H. Le, A. Y . Kadiptya, M. Iqbal, and H. Kim, “Cipher: Cybersecurity intelligent penetration- testing helper for ethical researcher,”Sensors, vol. 24, no. 21, p. 6878, 2024

    2024

Showing first 80 references.