pith. sign in

arxiv: 2606.25332 · v1 · pith:TV4BA2V6new · submitted 2026-06-24 · 💻 cs.CR · cs.AI

Decoupling Reconnaissance and Exploitation: Measuring the Capability Boundaries of LLM-Based Web Penetration Testing

Liwei Yu , Shuo Li , Ming Zhou , Ge Chu , Yan Guo This is my paper

Pith reviewed 2026-06-25 21:22 UTC · model grok-4.3

classification 💻 cs.CR cs.AI
keywords LLM agentspenetration testingweb vulnerabilitiesreconnaissanceexploitationevaluation frameworkautonomous agents
0
0 comments X

The pith

LLM penetration-testing agents reach 90% exploitation success with accurate vulnerability context but only 50% recall when performing reconnaissance autonomously.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces a two-stage evaluation that supplies ground-truth vulnerability details to isolate exploitation performance from reconnaissance failures in LLM agents. Across five agents and 50 aligned vulnerabilities in 70 web testbeds, exploitation succeeds at up to 90% while autonomous reconnaissance plateaus near 50% due to failures parsing unstructured telemetry. Different agent architectures show distinct strengths: multi-agent designs handle long sequences such as deserialization better, while monolithic and graph-driven designs excel on short-chain injections and cross-session access control, respectively. This separation reveals that current bottlenecks lie primarily in reconnaissance rather than in the ability to craft exploits once the target is known.

Core claim

With accurate vulnerability context supplied, agents achieve functional success rates up to 90.0%; autonomous reconnaissance, measured by targeted vulnerability recall, reaches only approximately 50.0% and is limited chiefly by failures to parse unstructured telemetry. The decoupled framework using ground-truth injection and knowledge-driven ablation across 70 high-fidelity testbeds isolates these capabilities on a 50-vulnerability subset.

What carries the argument

Two-stage decoupled evaluation framework that separates reconnaissance from exploit execution via ground-truth injection and knowledge-driven ablation.

Load-bearing premise

The 70 testbeds and 50-vulnerability subset, together with the injection and ablation procedures, isolate exploitation capability without testbed-specific artifacts or agent-implementation confounds.

What would settle it

Run the same agents on a fresh collection of web testbeds whose vulnerabilities are unknown in advance and measure whether autonomous recall remains near 50% or rises substantially when telemetry parsing is improved.

read the original abstract

Large Language Models (LLMs) have shown promise for automated penetration testing, yet existing end-to-end black-box evaluations are highly susceptible to error cascading: failures in early reconnaissance can mask an agent's actual ability to exploit vulnerabilities. To more accurately characterize these capabilities, we propose a two-stage decoupled evaluation framework that separates exploit execution from reconnaissance. Using ground-truth injection and knowledge-driven ablation across 70 high-fidelity web vulnerability testbeds, our framework isolates exploitation performance from reconnaissance noise. We empirically evaluate five open-source penetration-testing agents, covering multiagent, monolithic, and graph-driven architectures, on a strictly aligned subset of 50 representative vulnerabilities. The results reveal a substantial capability gap. With accurate vulnerability context, agents achieve a functional success rate of up to 90.0%, whereas autonomous reconnaissance, measured by targeted vulnerability recall, plateaus at approximately 50.0%, primarily due to failures in parsing unstructured telemetry. Cross-architectural analysis further reveals distinct capability niches: multi-agent isolation is more effective for long-sequence interactions such as de-serialization, while monolithic and graph-driven designs perform better on short-chain injections and cross-session access-control vulnerabilities, respectively. This decoupled evaluation work provides a fine-grained benchmarking protocol and an empirical basis for designing next-generation automated offensive security agents.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper proposes a two-stage decoupled evaluation framework for LLM-based web penetration testing that isolates exploitation from reconnaissance via ground-truth injection and knowledge-driven ablation. Across 70 high-fidelity testbeds and a strictly aligned subset of 50 vulnerabilities, five agents (multi-agent, monolithic, graph-driven) are evaluated. With accurate vulnerability context the agents reach up to 90% functional success; autonomous reconnaissance yields only ~50% targeted vulnerability recall, mainly from telemetry parsing failures. The work also reports architecture-specific niches (multi-agent for long-sequence interactions, monolithic for short-chain injections, graph-driven for cross-session access control).

Significance. If the isolation holds, the framework supplies a practical benchmarking protocol that can separate reconnaissance noise from exploitation capability and thereby inform next-generation agent design. The evaluation of multiple distinct architectures on the same testbed set is a clear strength; the high-fidelity testbeds and reproducible ablation procedure further increase the utility of the reported numbers for the community.

major comments (2)
  1. [§4.2] §4.2 (ground-truth injection): the description must explicitly enumerate every field supplied in the injected context (vulnerability identity alone, or also parameter names, expected payload shapes, session-state cues, etc.). Without this, it is impossible to verify that the 90% success rate reflects pure exploitation capability rather than scaffolding unavailable after realistic reconnaissance.
  2. [Experimental setup] Experimental setup (selection of the 50-vulnerability subset and statistical reporting): the criteria used to choose the 50 vulnerabilities from the 70 testbeds are not stated, nor are error bars, confidence intervals, or any statistical test for the headline 90.0% and 50.0% figures. These omissions directly affect the load-bearing claim of a substantial capability gap.
minor comments (1)
  1. [Abstract / §5] The abstract and §5 should clarify whether the 50-vulnerability subset was chosen before or after seeing agent performance, to rule out selection bias.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments, which highlight areas where additional clarity will strengthen the manuscript. We address each major comment below and will revise the paper accordingly.

read point-by-point responses

  1. Referee: [§4.2] §4.2 (ground-truth injection): the description must explicitly enumerate every field supplied in the injected context (vulnerability identity alone, or also parameter names, expected payload shapes, session-state cues, etc.). Without this, it is impossible to verify that the 90% success rate reflects pure exploitation capability rather than scaffolding unavailable after realistic reconnaissance.

    Authors: We agree that §4.2 does not currently enumerate the precise fields supplied during ground-truth injection. In the revised manuscript we will add an explicit enumeration of all injected fields (vulnerability identity, parameter names, expected payload shapes, session-state cues, and any other context elements). This will allow readers to assess exactly what information is provided and confirm that the 90% figure measures exploitation given that context. revision: yes

  2. Referee: [Experimental setup] Experimental setup (selection of the 50-vulnerability subset and statistical reporting): the criteria used to choose the 50 vulnerabilities from the 70 testbeds are not stated, nor are error bars, confidence intervals, or any statistical test for the headline 90.0% and 50.0% figures. These omissions directly affect the load-bearing claim of a substantial capability gap.

    Authors: We acknowledge that the manuscript does not state the selection criteria for the 50-vulnerability subset nor provide error bars, confidence intervals, or statistical tests. The 50 vulnerabilities were selected as a strictly aligned subset that preserves the distribution of vulnerability types and interaction lengths present in the full 70 testbeds; we will document these criteria explicitly. We will also add error bars, 95% confidence intervals, and appropriate statistical comparisons (e.g., paired tests) for the 90.0% and 50.0% headline figures in the revised version. revision: yes

Circularity Check

0 steps flagged

Empirical measurement study with no derivation chain or self-referential predictions

full rationale

The paper is a purely empirical evaluation of LLM agents on fixed web vulnerability testbeds using ground-truth injection and ablation. No equations, fitted parameters, or derivations are present; reported success rates (90% exploitation, 50% recon recall) are direct experimental measurements against external testbeds rather than quantities defined or predicted from the paper's own inputs. No self-citation load-bearing steps, uniqueness theorems, or ansatzes appear in the provided text. The evaluation framework is self-contained against the chosen benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The central claims rest on domain assumptions about the representativeness of the chosen testbeds and the validity of ground-truth injection for isolating capabilities; no free parameters or invented entities are introduced.

axioms (2)
  • domain assumption The 70 high-fidelity web vulnerability testbeds and the strictly aligned 50-vulnerability subset are representative of real-world web security scenarios.
    All reported success rates and architectural comparisons are measured against these testbeds.
  • domain assumption Ground-truth injection and knowledge-driven ablation cleanly isolate exploitation performance from reconnaissance noise without introducing new confounding factors.
    This premise underpins the claim that the observed 90% versus 50% gap reflects true capability boundaries.

pith-pipeline@v0.9.1-grok · 5764 in / 1436 out tokens · 30201 ms · 2026-06-25T21:22:25.000551+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

21 extracted references · 9 canonical work pages · 1 internal anchor

  1. [1]

    In: 33rd USENIX Security Symposium (USENIX Security 24)

    Deng, G., Liu, Y., Mayoral-Vilches, V., Liu, P., Li, Y., Xu, Y., Zhang, T., Liu, Y., Pinzger, M., Rass, S.: {PentestGPT}: Evaluating and harnessing large language models for auto- mated penetration testing. In: 33rd USENIX Security Symposium (USENIX Security 24). pp. 847–864 (2024)

    2024

  2. [2]

    LLM Agents can Autonomously Exploit One-day Vulnerabilities

    Fang, R., Bindu, R., Gupta, A., Kang, D.: Llm agents can autonomously exploit one -day vulnerabilities. arXiv preprint arXiv:2404.08144 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  3. [3]

    arXiv preprint arXiv:2402.06664 (2024)

    Fang, R., Bindu, R., Gupta, A., Zhan, Q., Kang, D.: Llm agents can autonomously hack websites. arXiv preprint arXiv:2402.06664 (2024)

    work page arXiv 2024

  4. [4]

    In: Findings of the Association for Computational Linguistics: NAACL 2024

    Gong, R., Huang, Q., Ma, X., Noda, Y., Durante, Z., Zheng, Z., Terzopoulos, D., Fei -Fei, L., Gao, J., Vo, H.: Mindagent: Emergent gaming interaction. In: Findings of the Association for Computational Linguistics: NAACL 2024. pp. 3154–3183 (2024)

    2024

  5. [5]

    arXiv preprint arXiv:2504.10112 (2025)

    Happe, A., Cito, J.: Benchmarking practices in llm-driven offensive security: Testbeds, met- rics, and experiment design. arXiv preprint arXiv:2504.10112 (2025)

    work page arXiv 2025

  6. [6]

    arXiv preprint arXiv:2505.10321 (2025)

    Henke, J.: Autopentest: Enhancing vulnerability management with autonomous llm agents. arXiv preprint arXiv:2505.10321 (2025)

    work page arXiv 2025

  7. [7]

    arXiv preprint arXiv:2601.19106 (2026)

    Khati, D., Rodriguez-Cardenas, D., Pantzer, P., Poshyvanyk, D.: Detecting and correcting hallucinations in llm -generated code via deterministic ast analysis. arXiv preprint arXiv:2601.19106 (2026)

    work page arXiv 2026

  8. [8]

    The University of Western Ontario (Canada) (2024)

    Khule, T.: STAF: Leveraging LLMs for automated attack tree-based security test generation. The University of Western Ontario (Canada) (2024)

    2024

  9. [9]

    In: International Conference on Learning Rep- resentations

    Liu, X., Yu, H., Zhang, H., Xu, Y., Lei, X., Lai, H., Gu, Y., Ding, H., Men, K., Yang, K., et al.: Agentbench: Evaluating llms as agents. In: International Conference on Learning Rep- resentations. vol. 2024, pp. 52989–53046 (2024)

    2024

  10. [10]

    In: International Conference on Learning Representations

    Qin, Y., Liang, S., Ye, Y., Zhu, K., Yan, L., Lu, Y., Lin, Y., Cong, X., Tang, X., Qian, B., et al.: Toolllm: Facilitating large language models to master 16000+ real -world apis. In: International Conference on Learning Representations. vol. 2024, pp. 9695–9717 (2024)

    2024

  11. [11]

    arXiv preprint arXiv:2505.17107 (2025) 12 L

    Shao, M., Xi, H., Rani, N., Udeshi, M., Putrevu, V.S.C., Milner, K., Dolan -Gavitt, B., Shukla, S.K., Krishnamurthy, P., Khorrami, F., et al.: Craken: Cybersecurity llm agent with knowledge-based execution. arXiv preprint arXiv:2505.17107 (2025) 12 L. Yu Author et al

    work page arXiv 2025

  12. [12]

    Advances in neural information processing systems 36, 8634–8652 (2023)

    Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., Yao, S.: Reflexion: Language agents with verbal reinforcement learning. Advances in neural information processing systems 36, 8634–8652 (2023)

    2023

  13. [13]

    arXiv preprint arXiv:2501.16466 (2025)

    Singer, B., Lucas, K., Adiga, L., Jain, M., Bauer, L., Sekar, V.: Incalmo: An autonomous llm-assisted system for red teaming multi -host networks. arXiv preprint arXiv:2501.16466 (2025)

    work page arXiv 2025

  14. [14]

    arXiv preprint arXiv:2601.22720 (2026)

    Tung, I.K., Shi, Y.X., Chien, A., Liu, W., Zheng, L.: Aegis: White -box attack path genera- tion using llms and training effectiveness evaluation for large-scale cyber defence exercises. arXiv preprint arXiv:2601.22720 (2026)

    work page arXiv 2026

  15. [15]

    Vangeli, M., Brynielsson, J., Cohen, M., Kamrani, F.: Context relay for long-running pene- tration-testing agents

  16. [16]

    Journal of Cybersecurity 3(3), 38– 56 (2025)

    Wang, H., Wang, Q., Guo, Y., Zhang, Q., Wang, C., Zhou, M.: A survey on automatic ex- ploitation for offensive and defensive cyber operations. Journal of Cybersecurity 3(3), 38– 56 (2025)

    2025

  17. [17]

    Applied Sciences 15(16), 9096 (2025)

    Wu, X., Tian, Y., Chen, Y., Ye, P., Cui, X., Jia, J., Li, S., Liu, J., Niu, W.: Curriculumpt: Llm-based multi-agent autonomous penetration testing with curriculum-guided task sched- uling. Applied Sciences 15(16), 9096 (2025)

    2025

  18. [18]

    arXiv preprint arXiv:2403.01038 (2024)

    Xu, J., Stokes, J.W., McDonald, G., Bai, X., Marshall, D., Wang, S., Swaminathan, A., Li, Z.: Autoattacker: A large language model guided system to implement automatic cyber - attacks. arXiv preprint arXiv:2403.01038 (2024)

    work page arXiv 2024

  19. [19]

    In: International Con- ference on Networking and Network Applications (NaNA)

    Zhou, M., Ma, Y., Hu, X., Lin, R., Wang, Q., Mao, W., Si, C.: Characterizing network threats against industrial control systems using honeypot technology. In: International Con- ference on Networking and Network Applications (NaNA). pp. 451–457 (2025)

    2025

  20. [20]

    In: IEEE 23rd International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom)

    Zhou, M., Zheng, Z., Zhang, P., Lu, S., Xie, Y., Jin, Z.: Securenet-awmi: Safeguarding net- work with optimal feature selection algorithm. In: IEEE 23rd International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom). pp. 1506–1511 (2024)

    2024

  21. [21]

    In: In- ternational Conference on Learning Representations

    Zhou, S., Xu, F.F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Ou, T., Bisk, Y., Fried, D., et al.: Webarena: A realistic web environment for building autonomous agents. In: In- ternational Conference on Learning Representations. vol. 2024, pp. 15585–15606 (2024)

    2024