pith. sign in

arxiv: 2606.04460 · v1 · pith:DVLEMRVNnew · submitted 2026-06-03 · 💻 cs.CR · cs.AI· cs.LG

CyberGym-E2E: Scalable Real-World Benchmark for AI Agents' End-to-End Cybersecurity Capabilities

Tianneng Shi , Robin Rheem , Dongwei Jiang , Mona Wang , Francisco De La Riega , Zhun Wang , Jingzhi Jiang , Alexander Cheung
show 8 more authors
Sean Tai Jonah Cha Jianhong Tu Gabriel Han Chenguang Wang Jingxuan He Wenbo Guo Dawn Song
This is my paper

Pith reviewed 2026-06-28 06:20 UTC · model grok-4.3

classification 💻 cs.CR cs.AIcs.LG
keywords cybersecurity benchmarkAI agentsvulnerability discoveryproof-of-concept generationpatch generationend-to-end evaluationreal-world vulnerabilitiesopen-source projects
0
0 comments X

The pith

CyberGym-E2E supplies 920 real-world vulnerabilities across 139 projects as a benchmark for AI agents to perform vulnerability discovery, proof-of-concept generation, and patch generation.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Existing cybersecurity evaluations of AI systems remain limited in scale or fail to cover the complete sequence from finding a flaw through creating an exploit and producing a fix. The paper addresses this by constructing CyberGym-E2E, a benchmark assembled through an automated pipeline that converts public vulnerability reports into executable test environments. A sympathetic reader would care because the benchmark lets researchers measure whether current or future agents can operate autonomously across the full remediation lifecycle rather than on isolated subtasks. If the benchmark functions as intended, it supplies a concrete yardstick for tracking progress toward AI systems that handle real software security work end to end.

Core claim

The central claim is that an automated, agent-enhanced pipeline can convert open-source vulnerability records into a large collection of realistic evaluation environments, yielding a benchmark of 920 vulnerabilities drawn from 139 distinct projects that jointly tests AI agents on discovery, proof-of-concept creation, and patch generation.

What carries the argument

The automated, agent-enhanced pipeline that ingests open-source vulnerability data and produces ready-to-use evaluation environments for the full discovery-to-remediation cycle.

If this is right

  • AI agents can now be scored on the complete sequence of vulnerability handling rather than on isolated subtasks.
  • Performance comparisons become possible across many projects and hundreds of distinct flaws under standardized conditions.
  • The construction method itself can be reused to enlarge the benchmark as new vulnerabilities are disclosed.
  • Results on the benchmark directly indicate whether an agent can both locate and remediate issues without manual setup.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Widespread use of the benchmark would likely shift research focus from narrow detection tasks toward integrated agent pipelines that also produce patches.
  • The same pipeline approach could be adapted to evaluate agents on vulnerabilities that affect closed-source or proprietary code once suitable data sources become available.
  • If agents improve on this benchmark, downstream systems could begin to automate routine security maintenance at the scale of entire open-source ecosystems.

Load-bearing premise

An automated pipeline can reliably turn public vulnerability data into test environments whose behavior matches real-world conditions.

What would settle it

A controlled comparison in which the same AI agents achieve markedly different success rates when run on the benchmark environments versus on unmodified live instances of the same vulnerable software.

Figures

Figures reproduced from arXiv: 2606.04460 by Alexander Cheung, Chenguang Wang, Dawn Song, Dongwei Jiang, Francisco De La Riega, Gabriel Han, Jianhong Tu, Jingxuan He, Jingzhi Jiang, Jonah Cha, Mona Wang, Robin Rheem, Sean Tai, Tianneng Shi, Wenbo Guo, Zhun Wang.

Figure 1
Figure 1. Figure 1: Overview of our automated agent-enhanced pipeline to construct tasks and environments from open-source vulnerability data. ronments (e.g., Ubuntu 16.04), where today’s agents cannot run out of the box. To handle vulnerabilities tied to these legacy systems, we first identify the vulnerable revision and reconstruct the ground-truth PoC using the automated pipeline described in steps 1–2 in Section 3.3. We t… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of benchmark task settings and agent evaluation. For the end-to-end setting, the PoC is generated by the agent. An intermediate evaluation then checks if the PoC triggers a vulnerability, and if so, the agent-generated patch and associated crash logs are used for the subsequent patch generation subtask. For the patch-only setting, the ground-truth PoC and crash logs are provided. For both settings… view at source ↗
Figure 3
Figure 3. Figure 3: Example agent trajectory for end-to-end vulnerability discovery and patching. The agent analyzes a GraphicsMagick codebase to find a heap-buffer-overflow vulnerability in ReadMNGImage(). The trajectory shows systematic exploration: browsing the codebase, locating the vulnerable function via grep, examining the code structure, constructing a minimal MNG file as a proof-of-concept, and iteratively refining a… view at source ↗
Figure 4
Figure 4. Figure 4: Agent prompt for end-to-end vulnerability discovery and patching. Keywords indicate section headers, paths show file locations, commands show executable scripts, and warnings highlight critical notes. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Agent prompt for patch-only vulnerability patching. Agents receive the ground-truth PoC and crash log, isolating the task to root cause analysis and patch generation. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
read the original abstract

AI has the potential to transform cybersecurity by enabling systems that can autonomously detect, analyze, and remediate software vulnerabilities. However, existing cybersecurity evaluations of AI systems are limited in scale or scope, and fail to capture the end-to-end lifecycle of real-world software vulnerability discovery and remediation. To address this gap, we propose CyberGym-E2E, a large-scale and realistic end-to-end cybersecurity benchmark that comprehensively evaluates AI agents' abilities across the full lifecycle of vulnerability discovery, PoC generation, and patch generation. CyberGym-E2E is comprehensive and scalable, as we build an automated, agent-enhanced pipeline for transforming open-source vulnerability data into realistic evaluation environments. Currently, the benchmark consists of 920 real-world vulnerabilities across 139 different open-source projects.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 0 minor

Summary. The paper proposes CyberGym-E2E, a benchmark of 920 real-world vulnerabilities across 139 open-source projects, constructed via an automated agent-enhanced pipeline to evaluate AI agents on the full end-to-end cybersecurity lifecycle of vulnerability discovery, PoC generation, and patch generation.

Significance. If the pipeline produces environments whose exploitability and remediation properties match the originals, the resource would be a meaningful contribution by supplying a large-scale, realistic testbed that addresses documented limitations in prior AI cybersecurity evaluations.

major comments (1)
  1. [Abstract and §3] Abstract and §3: the claim that the automated pipeline yields realistic evaluation environments is unsupported by any quantitative fidelity metrics (e.g., PoC reproduction rate on generated vs. original setups, or audit of configuration/dependency drift). This is load-bearing for the central claim that downstream agent scores measure real-world end-to-end capability.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the constructive comment on the need for quantitative validation of environment fidelity. We address the major comment below.

read point-by-point responses

  1. Referee: [Abstract and §3] Abstract and §3: the claim that the automated pipeline yields realistic evaluation environments is unsupported by any quantitative fidelity metrics (e.g., PoC reproduction rate on generated vs. original setups, or audit of configuration/dependency drift). This is load-bearing for the central claim that downstream agent scores measure real-world end-to-end capability.

    Authors: We agree that the manuscript currently lacks quantitative fidelity metrics to support the realism claim for the generated environments. While the pipeline is constructed to replicate original vulnerability conditions from real-world reports and code, explicit measurements (such as PoC reproduction rates between generated and original setups or audits for configuration/dependency drift) are not reported. In the revised version we will add these metrics, including PoC reproduction rates evaluated on a representative sample and a dependency-drift analysis, to §3 and update the abstract accordingly. revision: yes

Circularity Check

0 steps flagged

No circularity: benchmark construction is a new resource, not a derived claim

full rationale

The paper proposes CyberGym-E2E as a new benchmark resource built via an automated pipeline from open-source vulnerability data. No equations, fitted parameters, predictions, or uniqueness theorems appear in the provided text. The central claim is the existence and scale of the benchmark itself (920 vulnerabilities across 139 projects), which is not reduced to prior inputs by construction. Self-citation is absent from the load-bearing steps, and the pipeline description is presented as a methodological contribution rather than a self-referential derivation. This matches the default case of a self-contained resource proposal with no circular reduction.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

The proposal rests on the domain assumption that open-source vulnerabilities can be automatically turned into realistic test environments and that 920 cases suffice for comprehensive evaluation.

axioms (1)
  • domain assumption Open-source vulnerability data can be transformed into realistic evaluation environments via an automated, agent-enhanced pipeline
    Invoked in the abstract as the method for building the benchmark at scale.
invented entities (1)
  • CyberGym-E2E benchmark no independent evidence
    purpose: To evaluate AI agents across the full lifecycle of vulnerability discovery, PoC generation, and patch generation
    Newly proposed resource introduced in this paper.

pith-pipeline@v0.9.1-grok · 5716 in / 1299 out tokens · 27713 ms · 2026-06-28T06:20:45.809130+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Red-Teaming the Agentic Red-Team

    cs.CR 2026-06 unverdicted novelty 6.0

    Agentic offensive security tools share design flaws enabling API key exfiltration, persistence, and sandbox escape, addressed via a new cyber kill chain and robust architecture principles.

Reference graph

Works this paper leans on

20 extracted references · 6 canonical work pages · cited by 1 Pith paper · 1 internal anchor

  1. [1]

    Disrupting the first reported AI -orchestrated cyber espionage campaign

    Anthropic . Disrupting the first reported AI -orchestrated cyber espionage campaign. https://www.anthropic.com/news/disrupting-AI-espionage, November 2025

    2025

  2. [2]

    Secureagentbench: Benchmarking secure code generation under realistic vulnerability scenarios, 2025

    Chen, J., Huang, H., Lyu, Y., An, J., Shi, J., Yang, C., Zhang, T., Tian, H., Li, Y., Li, Z., Zhou, X., Hu, X., and Lo, D. Secureagentbench: Benchmarking secure code generation under realistic vulnerability scenarios, 2025. URL https://openreview.net/forum?id=8uDFRItIoe

    2025

  3. [3]

    Memory safety — Chromium Security

    Chromium . Memory safety — Chromium Security . https://www.chromium.org/Home/chromium-security/memory-safety/. Accessed: 2025-11-27

    2025

  4. [4]

    Vulnerability detection with code language models: How far are we?arXiv preprint arXiv:2403.18624, 2024

    Ding, Y., Fu, Y., Ibrahim, O., Sitawarin, C., Chen, X., Alomair, B., Wagner, D., Ray, B., and Chen, Y. Vulnerability detection with code language models: How far are we? arXiv preprint arXiv:2403.18624, 2024

    work page arXiv 2024

  5. [5]

    OSS-Fuzz: Continuous Fuzzing for Open Source Software

    Google. OSS-Fuzz: Continuous Fuzzing for Open Source Software . https://github.com/google/oss-fuzz. Accessed: 2025-05-10

    2025

  6. [6]

    Frontier ai's impact on the cybersecurity landscape, 2025

    Guo, W., Potter, Y., Shi, T., Wang, Z., Zhang, A., and Song, D. Frontier ai's impact on the cybersecurity landscape, 2025. URL https://arxiv.org/abs/2504.05408

    work page arXiv 2025

  7. [7]

    Implications of Rewriting a Browser Component in Rust

    Hosfelt, D. Implications of Rewriting a Browser Component in Rust . https://hacks.mozilla.org/2019/02/rewriting-a-browser-component-in-rust/, February 2019. Mozilla Hacks blog. Accessed: 2025-11-27

    2019

  8. [8]

    Sec-bench: Automated benchmarking of llm agents on real-world software security tasks

    Lee, H., Zhang, Z., Lu, H., and Zhang, L. Sec-bench: Automated benchmarking of llm agents on real-world software security tasks. In Neural Information Processing Systems (NeurIPS), 2025

    2025

  9. [9]

    ARVO: Atlas of Reproducible Vulnerabilities for Open-Source Software

    Mei, X., Singaria, P. S., Del Castillo, J., Xi, H., Bao, T., Wang, R., Shoshitaishvili, Y., Doup \'e , A., Pearce, H., Dolan-Gavitt, B., et al. Arvo: Atlas of reproducible vulnerabilities for open source software. arXiv preprint arXiv:2408.02153, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  10. [10]

    Cyberseceval 4: Advancing the evaluation of cybersecurity risks and capabilities in large language models

    Meta AI . Cyberseceval 4: Advancing the evaluation of cybersecurity risks and capabilities in large language models. https://meta-llama.github.io/PurpleLlama/CyberSecEval/, 2025

    2025

  11. [11]

    A proactive approach to more secure code

    MSRC. A proactive approach to more secure code . https://www.microsoft.com/en-us/msrc/blog/2019/07/a-proactive-approach-to-more-secure-code, July 2019. Microsoft MSRC blog. Accessed: 2025-11-27

    2019

  12. [12]

    SECODEPLT : A unified benchmark for evaluating the security risks and capabilities of code gen AI

    Nie, Y., Wang, Z., Yang, Y., Jiang, R., Tang, Y., Davies, X., Gal, Y., Li, B., Guo, W., and Song, D. SECODEPLT : A unified benchmark for evaluating the security risks and capabilities of code gen AI . In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025. URL https://openreview.net/forum?id=vbr63iQsbv

    2025

  13. [13]

    CWEval: Outcome- driven evaluation on functionality and security of LLM code genera- tion,

    Peng, J., Cui, L., Huang, K., Yang, J., and Ray, B. CWEval : Outcome-driven evaluation on functionality and security of LLM code generation, 2025. URL https://arxiv.org/abs/2501.08200

    work page arXiv 2025

  14. [14]

    Nyu ctf bench: A scalable open-source benchmark dataset for evaluating llms in offensive security

    Shao, M., Jancheska, S., Udeshi, M., Dolan-Gavitt, B., xi, h., Milner, K., Chen, B., Yin, M., Garg, S., Krishnamurthy, P., Khorrami, F., Karri, R., and Shafique, M. Nyu ctf bench: A scalable open-source benchmark dataset for evaluating llms in offensive security. In Advances in Neural Information Processing Systems, volume 37, pp.\ 57472--57498, 2024. URL...

    2024

  15. [15]

    Secrepobench: Benchmarking code agents for secure code completion in real-world repositories

    Shen, C., Dilgren, C., Chiniya, P., Griffith, L., Ding, Y., and Chen, Y. Secrepobench: Benchmarking code agents for secure code completion in real-world repositories. arXiv preprint arXiv:2504.21205, 2025

    work page arXiv 2025

  16. [16]

    Cy- berGym: Evaluating AI agents’ real-world cybersecurity capabilities at scale.arXiv preprint arXiv:2506.02548, 2025

    Wang, Z., Shi, T., He, J., Cai, M., Zhang, J., and Song, D. Cybergym: Evaluating ai agents' real-world cybersecurity capabilities at scale, 2025. URL https://arxiv.org/abs/2506.02548

    work page arXiv 2025

  17. [17]

    PATCHAGENT: a practical program repair agent mimicking human expertise

    Yu, Z., Guo, Z., Wu, Y., Yu, J., Xu, M., Mu, D., Chen, Y., and Xing, X. PATCHAGENT: a practical program repair agent mimicking human expertise. USENIX Association, USA, 2025. ISBN 978-1-939133-52-6

    2025

  18. [18]

    K., Ji, J., Menders, C., Dulepet, R., Qin, T., Wang, R

    Zhang, A. K., Ji, J., Menders, C., Dulepet, R., Qin, T., Wang, R. Y., Wu, J., Liao, K., Li, J., Hu, J., et al. Bountybench: Dollar impact of ai agent attackers and defenders on real-world cybersecurity systems. In Neural Information Processing Systems (NeurIPS), 2025 a

    2025

  19. [19]

    K., Perry, N., Dulepet, R., Ji, J., Menders, C., Lin, J

    Zhang, A. K., Perry, N., Dulepet, R., Ji, J., Menders, C., Lin, J. W., Jones, E., Hussein, G., Liu, S., Jasper, D. J., Peetathawatchai, P., Glenn, A., Sivashankar, V., Zamoshchin, D., Glikbarg, L., Askaryar, D., Yang, H., Zhang, A., Alluri, R., Tran, N., Sangpisit, R., Oseleononmen, K. O., Boneh, D., Ho, D. E., and Liang, P. Cybench: A framework for evalu...

    2025

  20. [20]

    Cve-bench: A benchmark for ai agents' ability to exploit real-world web application vulnerabilities

    Zhu, Y., Kellermann, A., Bowman, D., Li, P., Gupta, A., Danda, A., Fang, R., Jensen, C., Ihli, E., Benn, J., et al. Cve-bench: A benchmark for ai agents' ability to exploit real-world web application vulnerabilities. In International Conference on Machine Learning (ICML), 2025

    2025