Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

A compact 597-line reference harness aims to make LLM-driven penetration-testing experiments comparable and reproducible.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 05:09 UTC pith:HAURFE5Y

load-bearing objection Cochise is a genuinely useful minimal reference harness for LLM-pentesting research, but the evaluation is thin and the unpinned environment means the numbers and corpus are point-in-time observations. the 3 major comments →

arxiv 2605.11671 v2 pith:HAURFE5Y submitted 2026-05-12 cs.CR cs.AIcs.SE

Cochise: A Reference Harness for Autonomous Penetration Testing

Andreas Happe , J\"urgen Cito This is my paper
classification cs.CR cs.AIcs.SE
keywords autonomous penetration testingLLM agentsreference harnessreproducibilitytrajectory loggingPlanner-Executor architectureActive Directorycost analysis
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that progress in LLM-driven autonomous penetration testing is hard to measure because each prototype bundles architecture, prompting, and tooling choices. It introduces Cochise, a 597-line Python harness that fixes the execution interface, model abstraction, state handling, and trajectory log format, so agents can be ablated and compared under a common protocol. The paper also releases replay and analysis tools plus a corpus of JSON trajectory logs from a live Active Directory testbed, so researchers can study agent behavior without provisioning the heavy 48-64 GB RAM / 190 GB testbed. The reported capability numbers are illustrative rather than the point: the central claim is that the harness is a stable, reusable scaffold for future experiments.

Core claim

The paper's central claim is that a deliberately minimal, non-state-of-the-art harness can serve as a reference point for the field of autonomous penetration testing. Cochise separates a long-term planner from a short-lived ReAct executor, so cross-task knowledge accumulates in a structured Pentest-Task-Tree outside the LLM context, while per-task executor state is discarded to bound cost and isolate failures. The harness connects an LLM to a Linux jump host over SSH and logs every interaction in a structured JSON format. The released trace corpus from GOAD runs is offered as a high-fidelity record of agent behavior in an adversarial, multi-step environment, enabling offline study without re

What carries the argument

The load-bearing mechanism is the separation of a planner, which maintains long-term state in a Pentest-Task-Tree and a knowledge component, from per-task ReAct executors that issue SSH commands and self-correct based on command outputs. This design bounds the executor's context window and makes the planner the integration point for cross-task findings. The JSON trajectory log, with typed events for LLM calls, commands, outputs, and token/cost breakdowns, is what turns raw runs into analyzable, replayable empirical data.

Load-bearing premise

The evaluation and the released corpus treat the GOAD testbed and the queried LLM API versions as stable, fixed experimental objects; if either changes, the published numbers and logs become historical artifacts rather than reproducible benchmarks.

What would settle it

Run the same Cochise harness against a pinned GOAD version with a dated model API snapshot and fixed sampling settings, then compare the resulting trajectories and per-account costs to the released corpus; if the five-run averages for accounts per hour (15.77 for Gemini-3-Flash, 37.75 for Claude-4.7-Opus) cannot be reproduced within sampling noise, the quantitative evaluation is not stable.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Researchers can compare LLMs, agent architectures, and prompting strategies under a common protocol instead of reimplementing harnesses from scratch.
  • The released trajectory corpus lowers the barrier to studying agent behavior for researchers who lack the 48-64 GB RAM / 190 GB storage needed to run the GOAD testbed.
  • The per-task executor reset isolates context-level failures, making it easier to attribute errors to the architecture versus the underlying model.
  • The cost and token accounting in the logs exposes cost-capability trade-offs across models, as demonstrated by the reported five-run averages for two frontier models.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the harness gains adoption, the field could shift from reporting isolated success rates to controlled ablations of memory, prompting, and tool integration; that shift is the implicit promise of a reference harness.
  • The released JSON corpus could be reused for offline behavior analysis, such as studying failure-recovery patterns, tool-selection heuristics, or cost predictors, without running new live attacks.
  • A natural extension is to apply the same harness to non-security agentic software engineering tasks, since the paper frames autonomous pentesting as an instance of agentic software engineering.
  • The 12x cost difference between the two evaluated models suggests that cost-aware model selection could become a standard experimental axis in this area, but that is an inference beyond the paper's own claims.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents Cochise, a deliberately small (597 LOC per Table 1) Python harness for running LLM-driven autonomous penetration-testing experiments in controlled target environments. The system separates a long-horizon Planner, which maintains an external task tree (PTT), from per-task ReAct Executors with ephemeral context, and connects to a Linux jump host over SSH. The paper's contribution is framed as a 'reference harness' rather than a high-performance agent: it ships replay and analysis tools, a JSON trajectory format, and a released corpus of GOAD traces. The evaluation reports a LOC comparison against cAI, incalmo, and pentestGPT, and a capability run on the GOAD testbed using five runs each of Gemini-3-Flash and Claude-4.7-Opus, with cost and compromise statistics.

Significance. If the reproducibility issues can be addressed, Cochise is a genuinely useful community asset: it is compact, open-sourced, archived on Zenodo, and includes replay/analysis tooling plus a real trajectory corpus from a heavyweight testbed that many researchers cannot provision. The authors appropriately avoid claiming state-of-the-art attack performance and instead position the artifact as a stable, inspectable scaffold for ablation and comparison. This directly addresses the bundling problem identified in related systems. The central claim is credible in substance, but the evidence, especially Table 2 and the released corpus, needs environment/model version pinning before it can serve as a reproducible reference. No circularity issue exists: the paper makes empirical measurements against an external testbed and does not fit parameters to derive its claims.

major comments (3)
  1. [Section 4, Table 2; Section 3.3] The capability evaluation is not independently reproducible as reported. The GOAD testbed is described only as 'Game of Active Directory' with no version/commit, VM snapshot hashes, or provisioning state. The LLMs are given as 'Gemini-3-Flash (preview)' and 'Claude-4.7-Opus' without API release dates, model snapshots, or sampling settings, and the executor's 'maximum number of interaction rounds' is not stated. Because the central claim is that Cochise is a reference harness for fair comparison, and the released JSON corpus inherits this under-specification, a later researcher cannot tell whether an outcome change is due to the harness, the target, the LLM, or the prompt. Please report these parameters in the paper and, more importantly, make each released JSON log self-describing (harness commit, GOAD version/snapshot, model/API metadata, sampling configuration, and executor limits). Th
  2. [Section 4, Table 2] The capability evidence consists of five runs per model with no error bars on the success-rate statements ('3/5 runs...') and no baseline or ablation. The paper motivates Cochise by the difficulty of 'determining what is gained over a simple agent and harness', yet the evaluation compares two models only, not architectures, and does not include a planner-ablated ReAct-only, mini-SWE-agent, or other baseline condition using the same logging/execution interface. For a reference harness, the 'meaningful long-horizon trajectories' claim should be supported by at least one comparative condition; otherwise the observed behavior cannot be attributed to the harness design. Please add a minimal baseline condition or explicitly temper the Section 4 capability claims.
  3. [Section 4, Table 1] The compactness comparison uses LOC counts for cAI, incalmo, and pentestGPT without citing the commit versions counted. Since these repositories evolve, the comparison is not reproducible. The 'Core LoC' definition also differs across projects and excludes pentestGPT's claude-code dependency. Please pin versions, state the date of retrieval, and document how the core subsets were selected. This affects the paper's first evaluation dimension.
minor comments (5)
  1. [Abstract; Section 1; Table 1] The abstract states '630 LOC' while Section 1 and Table 1 state '597 LOC'. Reconcile the numbers.
  2. [Section 4; Section 3.1] Typographical and grammatical issues: 'preformed' should be 'performed'; 'whichs\'' should be 'whose'; 'The overall displayed in Figure 1' is missing a noun such as 'architecture'.
  3. [Section 3.3] The JSON log format is described only in prose. Provide an example entry or a short schema appendix so third-party parsers can rely on a stable format.
  4. [Figure 2] The screenshot caption is informative but the figure is not referenced in the main text; add an in-text reference or remove it.
  5. [Section 2] The exclusion of pentestGPT is relegated to a footnote. If it is excluded because it is single-host, state that explicitly in the main text when comparing related harnesses.

Circularity Check

0 steps flagged

No circular derivation: Cochise reports empirical measurements against an external testbed; the self-citations are motivational, not load-bearing.

full rationale

Cochise's central claim is that it provides a compact, reusable reference harness for autonomous pentesting experiments. This is supported by the released 597-LOC implementation, the replay/analysis tools, and the captured JSON trajectory corpus, not by a derived quantity. Section 4's capability evidence is empirical measurement: five runs each of Gemini-3-Flash (preview) and Claude-4.7-Opus against the third-party GOAD testbed (Table 2), with no fitted parameter and no quantity predicted from a fitted input. The only self-references to [2] appear as background ('A prior version of our research [2] was used by PentestGPT as a point of comparison') and as context for the new evaluation ('In addition to the results presented in [2]'); the current capability numbers are new runs, and the reusable-infrastructure claim does not depend on the truth of [2]. The 'adoption of an earlier cochise version as positive precedent [1]' is explicitly attributed to third-party work [1], not to the authors' own prior paper. No uniqueness theorem, imported ansatz, or renamed known result is used. The unpinned GOAD/LLM environment is a genuine reproducibility limitation, but it is an external-validity concern, not a circularity of derivation. Therefore score 0.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 0 invented entities

No invented physical or conceptual entities are introduced; the paper is a software artifact plus empirical evaluation. The main unstated premises are environmental stability of GOAD and LLM APIs, and the design assumption that the Planner-Executor split is beneficial without direct ablation.

free parameters (1)
  • maximum executor interaction rounds
    The executor stops after a maximum number of interaction rounds (Section 3.2), but the default value is not reported; it bounds runtime and cost and can affect task completion.
axioms (4)
  • domain assumption Assumed-breach setting: agent starts with SSH access to a trusted Linux execution host with network reachability to the target and required tools.
    Section 3 states this assumption; if it fails, the harness's safety model and applicability change.
  • domain assumption GOAD is a valid, reproducible third-party Active Directory testbed for evaluating autonomous pentesting.
    Section 4 evaluates against GOAD without pinning its version or configuration; if GOAD changes, the numbers and released trajectories may not transfer.
  • domain assumption LLM API behavior, pricing, and versions are sufficiently stable to support cost/capability comparisons.
    Section 4 compares costs across Gemini-3-Flash preview and Claude-4.7-Opus; providers change models and pricing over time, making comparisons snapshot-like.
  • ad hoc to paper Planner-Executor with ephemeral executor state is an effective design for long-horizon multi-step compromises.
    Section 3.2 motivates this design, but no ablation in the paper demonstrates that this architecture is necessary or superior to alternatives.

pith-pipeline@v1.3.0-alltime-deepseek · 5270 in / 11324 out tokens · 118322 ms · 2026-08-04T05:09:43.556781+00:00 · methodology

0 comments
read the original abstract

Recent work on LLM-driven autonomous penetration testing reports promising results, but existing systems often bundle architectural, prompting, and tool-integration choices together. This makes it difficult to determine what is gained over a simple agent and harness. We present Cochise, a 630 LOC Python reference implementation for autonomous penetration-testing experiments. Cochise connects to a Linux execution host over SSH and supports attacking controlled target environments reachable from that jump host. The prototype implements a Planner--Executor architecture in which long-term state is maintained by the planner, while a ReAct-style executor issues commands over SSH and self-corrects based on command outputs. The scenario prompt can be adapted to different target environments. We evaluate the harness against a live third-party testbed, Game of Active Directory (GOAD). Cochise is intended not as a state-of-the-art penetration-testing agent, but as a reusable experimental infrastructure for comparing models, agent architectures, and penetration-testing traces. Alongside the prototype, we release replay and analysis tools: (i) cochise-replay for offline visualization of captured runs, (ii) cochise-analyze-logs and cochise-analyze-graphs for cost, token, duration, and compromise analysis, and (iii) a corpus of JSON trajectory logs from GOAD runs, so that researchers can study agent behavior without provisioning the 48--64 GB RAM / 190 GB storage testbed themselves. Tool demo video available at https://youtu.be/2mQimB1ufyI.

Figures

Figures reproduced from arXiv: 2605.11671 by Andreas Happe, J\"urgen Cito.

Figure 1
Figure 1. Figure 1: System and Agent Architecture. Orange components [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Screenshot of an active cochise run, showing how cochise used certipy-ad to execute an ADCS ESC1 attack to gain [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. The Ethics of Autonomous AI Agents for Offensive Security

    cs.CR 2026-07 conditional novelty 6.0

    Autonomous AI hacking tools combine three kinds of indeterminacy—action, impact, and users—making moral responsibility diffuse and giving attackers a short-term advantage under current cost asymmetries.

Reference graph

Works this paper leans on

7 extracted references · 3 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Gelei Deng, Yi Liu, Yuekang Li, Ruozhao Yang, Xiaofei Xie, Jie Zhang, Han Qiu, and Tianwei Zhang. 2026. What Makes a Good LLM Agent for Real-world Penetration Testing? arXiv:2602.17622 [cs.CR] https://arxiv.org/abs/2602.17622

    arXiv 2026

  2. [2]

    Andreas Happe and Jürgen Cito. 2025. Can LLMs Hack Enterprise Networks? Autonomous Assumed Breach Penetration-Testing Active Directory Networks. ACM Trans. Softw. Eng. Methodol.(Sept. 2025). doi:10.1145/3766895 Just Accepted

    doi:10.1145/3766895 2025

  3. [3]

    Víctor Mayoral-Vilches, Luis Javier Navarrete-Lozano, María Sanz-Gómez, Lidia Salas Espejo, Martiño Crespo-Álvarez, Francisco Oca-Gonzalez, Francesco Balassone, Alfonso Glera-Picón, Unai Ayucar-Carbajo, Jon Ander Ruiz-Alcalde, Stefan Rass, Martin Pinzger, and Endika Gil-Uriarte. 2025. CAI: An Open, Bug Bounty-Ready Cybersecurity AI. arXiv:2504.06017 [cs.C...

    Pith/arXiv arXiv 2025

  4. [4]

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. Advances in neural information processing systems36 (2023), 8634–8652

    2023

  5. [5]

    Brian Singer, Keane Lucas, Lakshmi Adiga, Meghna Jain, Lujo Bauer, and Vyas Sekar. 2025. Incalmo: An Autonomous LLM-assisted System for Red Teaming Multi-Host Networks. arXiv:2501.16466 [cs.CR] https://arxiv.org/abs/2501.16466

    arXiv 2025

  6. [6]

    John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R Narasimhan, and Ofir Press. 2024. SWE-agent: Agent-Computer Inter- faces Enable Automated Software Engineering. InThe Thirty-eighth Annual Con- ference on Neural Information Processing Systems. https://arxiv.org/abs/2405.15793

    Pith/arXiv arXiv 2024

  7. [7]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. arXiv:2210.03629 [cs.CL] https://arxiv.org/abs/2210.03629 Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009

    Pith/arXiv arXiv 2023