REVIEW 11 cited by
CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilities
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilities
read the original abstract
Large language model (LLM) agents are increasingly capable of autonomously conducting cyberattacks, posing significant threats to existing applications. This growing risk highlights the urgent need for a real-world benchmark to evaluate the ability of LLM agents to exploit web application vulnerabilities. However, existing benchmarks fall short as they are limited to abstracted Capture the Flag competitions or lack comprehensive coverage. Building a benchmark for real-world vulnerabilities involves both specialized expertise to reproduce exploits and a systematic approach to evaluating unpredictable threats. To address this challenge, we introduce CVE-Bench, a real-world cybersecurity benchmark based on critical-severity Common Vulnerabilities and Exposures. In CVE-Bench, we design a sandbox framework that enables LLM agents to exploit vulnerable web applications in scenarios that mimic real-world conditions, while also providing effective evaluation of their exploits. Our evaluation shows that the state-of-the-art agent framework can resolve up to 13% of vulnerabilities.
Forward citations
Cited by 11 Pith papers
-
AgentCyberRange: Benchmarking Frontier AI Systems in Realistic Cyber Ranges
Introduces the first open multi-host cyber range benchmark AgentCyberRange with Cage toolchain and evaluates six frontier AI systems on web exploitation and post-exploitation tasks across 110 vulnerabilities.
-
Certifying Ghosts: How Cybersecurity AI Agents Break the EU Cyber Resilience Act
Cybersecurity AI agents invalidate the CRA's premises about vulnerability discovery, exploitation, and remediation speed, making static conformity certificates false and requiring continuous agent-operated defense.
-
ZERO-APT: A Closed-Loop Adversarial Framework for LLM-Driven Automated Penetration Testing under Intelligent Defense
ZERO-APT is a closed-loop framework that integrates an LLM attacker, configurable LLM defender, and judge agent to achieve 79% attack success rate, 0.860 causal consistency, and full decision auditability in penetrati...
-
CyberEvolver: Structured Self-Evolution for Cybersecurity Agents On the Fly
CyberEvolver introduces a four-layer self-evolving agent architecture with trace-to-diagnosis and population beam search that raises seed agent success rates by 13.6% on CTF, exploitation, and penetration tasks across...
-
CyBiasBench: Benchmarking Bias in LLM Agents for Cyber-Attack Scenarios
LLM agents exhibit persistent attack-selection biases as fixed traits independent of success rates, with a bias momentum effect that resists steering and yields no performance gain.
-
RealVuln: Benchmarking Rule-Based, General-Purpose LLM, and Security-Specialized Scanners on Real-World Code
RealVuln benchmark finds security-specialized scanners outperform general-purpose LLMs and rule-based SAST tools on hand-labeled vulnerable Python code under F3 scoring, with all artifacts released.
-
The Emergence of Autonomous Penetration Capabilities in Large Language Model-Powered AI Systems
New evaluation shows LLM-powered agents achieve 10.7% to 69.3% success in autonomous penetration on servers with varying numbers of secure services.
-
From Controlled to the Wild: Evaluation of Pentesting Agents for the Real-World
A practical evaluation protocol for AI pentesting agents that uses validated vulnerability discovery, LLM semantic matching, and bipartite scoring to assess performance in realistic, complex targets.
-
The Emergence of Autonomous Penetration Capabilities in Large Language Model-Powered AI Systems
A tiered server benchmark with 300 targets shows current LLMs achieve autonomous penetration success rates of 10.7-69.3% using only general cybersecurity tools and no target-specific knowledge.
-
FORGE: Multi-Agent Graduated Exploitation and Detection Engineering
FORGE deploys a fixed five-agent pipeline on 603 CVEs to achieve 67.8% L1+ exploitation success at $1.50 per CVE while generating detection rules whose grounding improves with deeper exploitation traces.
-
xOffense: An Autonomous Multi-Agent Framework for Penetration Testing with Domain-Adapted Large Language Models
xOffense automates penetration testing via a fine-tuned Qwen3-32B LLM in a multi-agent setup with specialized agents for reconnaissance, vulnerability scanning, and exploitation, reporting 79.17% sub-task completion o...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.