CTFusion: A CTF-based Benchmark for LLM Agent Evaluation

Dongjun Lee; Ga-eun Bae; Insu Yun

REVIEW 3 major objections 3 minor 1 cited by

Live CTF streaming evaluation avoids contamination and cheating that make static CTF benchmarks unreliable for LLM cybersecurity agents.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 19:03 UTC pith:X6JDETBD

load-bearing objection Useful live-CTF harness for agent eval; central contamination claim is plausible but uncheckable from the abstract alone. the 3 major comments →

arxiv 2605.11504 v2 pith:X6JDETBD submitted 2026-05-12 cs.LG cs.CR

CTFusion: A CTF-based Benchmark for LLM Agent Evaluation

Dongjun Lee , Ga-eun Bae , Insu Yun This is my paper

classification cs.LG cs.CR

keywords LLM agentscybersecurity evaluationCapture The Flagdata contaminationLive CTFModel Context ProtocolCTFdagent benchmarking

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that current Capture The Flag (CTF) benchmarks for LLM-based cybersecurity agents are unreliable because they reuse published challenges, so models can draw on training data or web search instead of genuine multi-step reasoning. The authors confirm the problem by adding a web-search tool to an existing agent and observing that it can solve challenges via leakage rather than skill. They introduce CTFusion, a streaming evaluation framework that runs against live, ongoing CTF events. CTFusion keeps agents independent under a single shared team account and forwards only the first correct flag for each challenge so that competition dynamics do not distort scores. Implemented as a Model Context Protocol server on the popular CTFd platform, the system is designed to work with many CTF events and agent types. Experiments with three LLMs, two agents, and five live CTFs show that static benchmarks can mis-rank agents while CTFusion yields a more trustworthy measure of real capability. The framework is released as open source.

Core claim

Existing CTF benchmarks that reuse published challenges are contaminated and cheat-prone for LLM cybersecurity agents, whereas CTFusion—a streaming live-CTF framework that preserves per-agent independence under one team account and forwards only the first correct flag—provides a robust evaluation alternative, demonstrated across three LLMs, two agents, and five live CTFs.

What carries the argument

CTFusion: a Model Context Protocol (MCP) server on CTFd that streams live CTF challenges, isolates agents under a single team account, and scores only the first correct flag per challenge so that competition effects and prior leakage are minimized.

Load-bearing premise

Live CTF challenges stay free of public leakage for the whole evaluation window, and the single-team-plus-first-flag design truly isolates agents without introducing new scoring bias.

What would settle it

Run the same three LLMs and two agents on a fresh live CTF after its challenges have already appeared in public write-ups or training corpora; if CTFusion scores then collapse to the same inflated pattern as static benchmarks, the contamination-resistance claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Static CTF leaderboards can no longer be trusted as evidence of agent capability once web search or memorization is available.
Agent evaluation can be repeated continuously against successive live events instead of a fixed challenge set.
Any agent that speaks the Model Context Protocol can plug into CTFd-based live CTFs without custom platform work.
Open release of the MCP server lets other labs adopt the same isolation and first-flag rules for comparable scores.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If live CTFs themselves begin to leak mid-event, CTFusion may need an automatic challenge-freshness filter or short-lived private instances.
The same first-correct-forwarding pattern could be reused for other multi-agent live contests (bug bounties, red-team exercises) where shared accounts are operationally convenient.
Longitudinal CTFusion scores across successive live events could become a public, contamination-resistant capability index for cybersecurity agents.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Useful live-CTF harness for agent eval; central contamination claim is plausible but uncheckable from the abstract alone.

read the letter

The one thing worth knowing: this is a systems/benchmark paper that packages Live-CTF evaluation for LLM agents as an MCP server on CTFd, with a couple of operational tricks (per-agent independence under one team account, first-correct-flag forwarding) aimed at reducing contamination and competition side-effects. That combination is the actual contribution, not a new theory of agents.

What they do well is name a real problem. Static CTF leaderboards that reuse published challenges are easy to game once web search or training data leaks in; their web-search integration anecdote is a concrete illustration, not just hand-waving. Building on CTFd and shipping as MCP is practical engineering that other groups can actually plug into. If the open-source claim holds and the five Live CTFs are documented cleanly, this is the kind of artifact people will run rather than re-implement.

Soft spots, in proportion: we only have the abstract. The load-bearing claim—that existing benchmarks are unreliable while CTFusion is robust—rests on three LLMs, two agents, and five Live CTFs whose protocols, numbers, baselines, and leakage checks we cannot see. The premise that those live events stayed clean for the evaluation window, and that single-team + first-flag forwarding does not itself bias scores or hide multi-agent behavior, is stated but not independently verifiable here. That is an information gap, not an internal contradiction. Circularity risk is low; this is empirical harness work, not fitted predictions dressed up as theory. Novelty is incremental (live contests and CTFd already exist); significance is real for the agent-eval and cyber-eval subfields but not field-redefining.

Who it is for: people building or ranking LLM cybersecurity agents who currently trust static CTF suites. A serious referee should see the full paper, code, and event logs. I would send it to peer review rather than desk-reject; the problem is real and the engineering response is coherent enough to deserve scrutiny. I would not cite it yet from abstract alone, and I would only bring it to reading group if someone is actively building agent eval harnesses this quarter.

Referee Report

3 major / 3 minor

Summary. The paper claims that existing Capture-The-Flag (CTF) benchmarks for LLM-based cybersecurity agents are unreliable because they reuse published challenges, enabling data contamination and cheating (illustrated by integrating web search into an agent). It introduces CTFusion, a streaming Live-CTF evaluation framework implemented as an MCP server on CTFd, designed to preserve per-agent independence under a single team account and to reduce competition effects by forwarding only the first correct flag per challenge. Experiments with three LLMs, two agents, and five Live CTFs are said to show that static CTF benchmarks can be unreliable while CTFusion is a robust alternative; the system is released as open source.

Significance. If substantiated, CTFusion would address a timely and practical problem: contamination and tool-enabled cheating undermine static CTF benchmarks just as LLM agents are being applied to security tasks. Strengths claimed in the abstract include an open-source MCP/CTFd implementation with broad event and agent applicability, an explicit practical demonstration of contamination via web search, and an operational design aimed at Live-CTF isolation. A validated, platform-compatible streaming evaluation method would be useful to the agent-evaluation and cybersecurity communities. Significance remains conditional on experimental evidence that is not inspectable from the abstract alone.

major comments (3)

[Abstract] The central empirical claim—that existing CTF benchmarks are unreliable while CTFusion is robust—rests on experiments with three LLMs, two agents, and five Live CTFs. From the abstract alone, protocols, quantitative results, baselines, contamination measurements, statistical significance, and failure cases are unreported. Without these load-bearing details, the claimed contrast cannot be verified or falsified, so the paper’s main contribution cannot yet be assessed.
[Abstract] The robustness claim depends on the premise that Live CTF challenges remain free of public leakage during evaluation and that single-team-account operation with first-correct-flag forwarding isolates agents without introducing scoring bias or incomplete multi-agent coverage. The abstract only sketches these operational choices; they are load-bearing axioms for preferring CTFusion over static benchmarks and require explicit justification, threat analysis, and empirical checks in the full manuscript.
[Abstract] The practical confirmation of contamination via web-search integration is cited as evidence that existing benchmarks are unreliable. For this to support the central claim, the manuscript must specify the agent and benchmark used, success rates with versus without search, and why gains reflect contamination rather than legitimate tool use. Those details are not available here and are necessary to ground the unreliability argument.

minor comments (3)

[Abstract] The phrase “preserves per-agent independence under a single team account” assumes familiarity with CTFd team mechanics; a brief clarification would help general readers.
[Abstract] The open-source release is asserted but not linkable from the abstract; the full paper should provide a repository URL and a minimal reproducibility package (configs, agent wrappers, evaluation logs).
[Abstract] The five Live CTFs should be characterized (categories, difficulty, duration, participant scale) so readers can judge generalizability of the robustness claim.

Circularity Check

1 steps flagged

No significant circularity: empirical systems paper; mild residual only from design choices partly defining 'robust' evaluation.

specific steps

self definitional [Abstract (design + claim of robustness)]
"CTFusion preserves per-agent independence under a single team account and reduces competition impact by forwarding only the first correct flag per challenge. ... we demonstrate that existing CTF benchmarks can be unreliable in assessing LLM-based agents, while CTFusion can serve as a robust solution for evaluating cybersecurity agents."

Mild only: the same operational choices that constitute CTFusion's design (isolation under one team account; first-correct-flag forwarding) are also part of what the authors treat as making the evaluation 'robust.' This is ordinary systems framing, not a prediction forced by construction from fitted inputs or a uniqueness theorem. No reduction of an empirical claim to a pure definition is exhibited beyond this overlap.

full rationale

This is an abstract-only review of an empirical systems/benchmark paper, not a first-principles derivation. There is no fitted parameter renamed as a prediction, no uniqueness theorem imported from the same authors, no ansatz smuggled via self-citation, and no renaming of a known empirical pattern as a novel result. The central contrast (static CTF benchmarks are contamination-prone; Live-CTF streaming with per-agent independence and first-correct-flag forwarding is more robust) is meant to be checked against external Live CTF events and agent runs, not forced by construction from the paper's own inputs. The only mild residual is that the operational rules (single team account with per-agent independence; forward only the first correct flag) are both the design of CTFusion and part of what the authors mean by a 'robust' evaluation protocol—normal for systems papers and not a load-bearing circular derivation. Score 1 reflects that minor definitional overlap without elevating it to fitted-input or self-definitional circularity of the kinds that warrant 6+. Full text unavailable; no equations or self-citation chain to inspect.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 1 invented entities

Abstract-only audit. No fitted scientific constants. The work rests on domain assumptions about CTF validity as a proxy for cybersecurity agent skill, on operational isolation rules that define fair multi-agent scoring under one team account, and on the entity CTFusion itself as the proposed harness. No new physical particles or forces; the invented entity is the software framework and its scoring policy.

axioms (4)

domain assumption Reuse of published CTF challenges exposes LLM-agent evaluation to training-data contamination and web-search cheating.
Stated as the motivating problem and supported only by a brief practical confirmation (web search tool on an existing agent); treated as given for the need for Live CTFs.
domain assumption Live CTF challenges during an event are sufficiently novel that agents cannot trivially retrieve solutions from prior public sources.
Required for CTFusion’s robustness claim; not proven in the abstract and may fail if challenges leak mid-event or reuse known patterns.
ad hoc to paper Per-agent independence under a single team account plus forwarding only the first correct flag adequately isolates agents and reduces competition impact without distorting evaluation.
Design choice introduced by CTFusion; load-bearing for fair multi-agent live scoring; correctness not independently evidenced in the abstract.
domain assumption CTF solve rates under the described harness are a meaningful measure of cybersecurity agent capability.
Standard in the CTF-for-agents literature the abstract builds on; not re-derived here.

invented entities (1)

CTFusion (Live-CTF streaming evaluation framework / MCP server on CTFd) no independent evidence
purpose: Provide contamination-resistant, multi-agent-compatible evaluation of LLM cybersecurity agents on live CTF events.
The paper’s primary artifact; independent evidence would be public code, third-party re-use, and external live-event results, which the abstract claims via open source but does not ship here.

pith-pipeline@v1.1.0-grok45 · 6106 in / 2881 out tokens · 31617 ms · 2026-07-14T19:03:57.516878+00:00 · methodology

0 comments

read the original abstract

Recent advances in Large Language Models (LLMs) have enabled agentic systems for complex, multi-step tasks; cybersecurity is emerging as a prominent application. To evaluate such agents, researchers widely adopt Capture The Flag (CTF) benchmarks. However, current CTF benchmarks reuse existing challenges, which exposes them to data contamination and potential cheating. Notably, we confirmed these issues in practice by integrating web search tools into an existing agent. To address these limitations, we present CTFusion, a streaming evaluation framework built on Live CTFs. To achieve this, CTFusion preserves per-agent independence under a single team account and reduces competition impact by forwarding only the first correct flag per challenge. Moreover, we implement CTFusion as a Model Context Protocol (MCP) server on the widely used CTFd platform, which offers broad applicability to diverse CTF events and agent types. Through experiments with three LLMs, two agents, and five Live CTFs, we demonstrate that existing CTF benchmarks can be unreliable in assessing LLM-based agents, while CTFusion can serve as a robust solution for evaluating cybersecurity agents. We release CTFusion as open source to foster future research in this area.

Figures

Figures reproduced from arXiv: 2605.11504 by Dongjun Lee, Ga-eun Bae, Insu Yun.

**Figure 1.** Figure 1: Monthly distribution of CTF competitions (2025). 2.2. Related Work LLM Agents for Vulnerability Discovery. A variety of LLM agents have been developed to automate vulnerability discovery in CTFs and real-world systems. ENIGMA (Abramovich et al., 2025) first demonstrated this capability, and D-CIPHER (Udeshi et al., 2025) further improved upon it by incorporating an auto-prompter agent to guide exploitati… view at source ↗

**Figure 2.** Figure 2: Success rates of D-CIPHER-WEB and D-CIPHER on NYU CTF BENCH [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Figure 3: Evidence of direct flag retrieval by D-CIPHER-WEB for the “1nsayne” challenge. 3.2. Cheating Evidence in Evaluations We evaluated D-CIPHER-WEB on NYU CTF BENCH and found that it achieved a substantially higher success rate than the original D-CIPHER. We conducted these evaluations using the same conditions and metrics as in §5.1. Specifically, D-CIPHER-WEB solved 24.07% of problems compared to D-CIPHER’s … view at source ↗

**Figure 4.** Figure 4: Overview of the CTFUSION framework architecture. 4. CTFUSION 4.1. Overview CTFUSION enables fair evaluation of multiple agents on LIVE CTFS while keeping the live competition intact. In a live CTF, each account corresponds to a team on the public scoreboard. Creating one account per agent would inflate team counts and distort rankings. Sharing a single account avoids this, but it breaks independence: once … view at source ↗

**Figure 5.** Figure 5: Performance Comparison: Live CTFs vs NYU CTF Bench 15.0% (×2.4), and CLAUDE 3.5-SONNET from 5.1% to 11.4% (×2.2). At the agent level, ENIGMA increased from 7.2% to 16.8% (×2.3), and D-CIPHER from 5.3% to 12.6% (×2.4). Two primary factors explain the observed performance gap: • Task difficulty: Although both LIVE CTFS and NYU CTF BENCH evaluations use identical environments and interaction, problem difficu… view at source ↗

**Figure 6.** Figure 6: Default prompt for D-CIPHER-WEB. Default prompt [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗

**Figure 7.** Figure 7: Specialized prompt for pwn challenges in D-CIPHER-WEB. prompt defines a two-role workflow—Planner and Executor. The Planner produces an iterative, step-by-step plan for solving the designated CTF challenge, and the Executor performs the delegated tasks within the containerized evaluation. The agent pair is provided with a Linux environment, access to network interfaces, and web search access. The prompt re… view at source ↗

**Figure 8.** Figure 8: Success rates across five LIVE CTFS and NYU CTF BENCH. CubeCTF. We participated in CUBECTF, which ran from July 4, 2025, 22:16 UTC to July 7, 2025, 00:25 UTC. The competition hosted 1,059 teams and included 16 problems, of which 375 teams solved one or more. GPT-4.1 with ENIGMA ranked 163rd, while GPT-4.1 with D-CIPHER ranked 180th. CLAUDE 3.5-SONNET with ENIGMA did not achieve a rank, but CLAUDE 3.5-SONNE… view at source ↗

**Figure 10.** Figure 10: Problem-solving rates of all model-agent combinations on UIUCTF. GPT-4.1 Claude 3.5-Sonnet Gemini 2.5-Flash 1.82% 0.00% 0.00% 0.00% 1.82% 1.82% EnIGMA D-cipher [PITH_FULL_IMAGE:figures/full_fig_p018_10.png] view at source ↗

**Figure 11.** Figure 11: Problem-solving rates of all model-agent combinations on WWCTF. GPT-4.1 Claude 3.5-Sonnet Gemini 2.5-Flash 2.70% 0.00% 0.00% 0.00% 0.00% 0.00% EnIGMA D-cipher [PITH_FULL_IMAGE:figures/full_fig_p018_11.png] view at source ↗

**Figure 13.** Figure 13: Problem-solving rates of all model-agent combinations on SCRIPTCTF. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_13.png] view at source ↗

**Figure 14.** Figure 14: Problem-solving rates for all model-agent pairs on 2023-Quals. GPT-4.1 Claude 3.5-Sonnet Gemini 2.5-Flash 0.00% 18.75% 0.00% 18.75% 6.25% 12.50% EnIGMA D-cipher [PITH_FULL_IMAGE:figures/full_fig_p019_14.png] view at source ↗

**Figure 16.** Figure 16: Problem-solving rates for all model-agent pairs on 2022-Quals. GPT-4.1 Claude 3.5-Sonnet Gemini 2.5-Flash 9.09% 9.09% 0.00% 0.00% 9.09% 0.00% EnIGMA D-cipher [PITH_FULL_IMAGE:figures/full_fig_p019_16.png] view at source ↗

**Figure 18.** Figure 18: Problem-solving rates for all model-agent pairs on 2021-Quals. GPT-4.1 Claude 3.5-Sonnet Gemini 2.5-Flash 11.11% 11.11% 11.11% 0.00% 0.00% 0.00% EnIGMA D-cipher [PITH_FULL_IMAGE:figures/full_fig_p020_18.png] view at source ↗

**Figure 20.** Figure 20: Problem-solving rates for all model-agent pairs on 2020-Quals. GPT-4.1 Claude 3.5-Sonnet Gemini 2.5-Flash 40.00% 10.00% 10.00% 20.00% 30.00% 20.00% EnIGMA D-cipher [PITH_FULL_IMAGE:figures/full_fig_p020_20.png] view at source ↗

**Figure 22.** Figure 22: Problem-solving rates for all model-agent pairs on 2019-Quals. GPT-4.1 Claude 3.5-Sonnet Gemini 2.5-Flash 14.29% 14.29% 14.29% 14.29% 28.57% 14.29% EnIGMA D-cipher [PITH_FULL_IMAGE:figures/full_fig_p020_22.png] view at source ↗

**Figure 24.** Figure 24: Problem-solving rates for all model-agent pairs on 2018-Quals. GPT-4.1 Claude 3.5-Sonnet Gemini 2.5-Flash 11.11% 11.11% 11.11% 11.11% 11.11% 11.11% EnIGMA D-cipher [PITH_FULL_IMAGE:figures/full_fig_p021_24.png] view at source ↗

**Figure 26.** Figure 26: Problem-solving rates for all model-agent pairs on 2017-Quals. GPT-4.1 Claude 3.5-Sonnet Gemini 2.5-Flash 0.00% 0.00% 0.00% 14.29% 0.00% 0.00% EnIGMA D-cipher [PITH_FULL_IMAGE:figures/full_fig_p021_26.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

A Survey of LLM-Driven Penetration Testing: Taxonomy, Co-Evolution, and Open Challenges
cs.SE 2026-07 accept novelty 5.5

LLM pentest agents co-evolved through four bottleneck-driven phases into RLVR systems, while CTF platforms became dual evaluation/training infrastructure and three linked reliability gaps remain.