Pith. sign in

REVIEW 3 major objections 6 minor 25 references

For a fixed LLM, the client that drives a security-tool orchestrator can swing CTF solve rates by a factor of two, and leftover failures are reasoning-bound rather than missing tools.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 06:27 UTC pith:57FCLJVJ

load-bearing objection Clean fixed-model client gap (already visible under free-solve); improvement numbers are weaker than the methodological point, and the authors mostly say so. the 3 major comments →

arxiv 2607.02873 v1 pith:57FCLJVJ submitted 2026-07-03 cs.SE cs.AIcs.CR

Determinants and Limits of LLM Security-Tool Orchestration: A Study with HexStrike-AI

Romain Gerard , Assmaa Zeghaider , Yan Guo This is my paper
classification cs.SE cs.AIcs.CR
keywords LLM agentspenetration testingcapture the flagtool orchestrationModel Context Protocolagent harnessempirical evaluationsecurity tools
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper treats an open-source security-tool orchestrator (150+ tools over the Model Context Protocol) as a testbed and asks what actually bounds LLM agents that drive it. Across 86 picoCTF challenges and 774 trials, the same DeepSeek model solves 59.7% of challenges with one client and 27.9% with another—a 2.1× gap—so a model’s score is only meaningful when the client is named. A bundled intervention (tool corrections, behavior changes, eleven new capability tools) lifts overall success from 55.4% to 72.0%, with more recoveries from default-behavior fixes than from new specialized tools. What still fails is classified as reasoning- or environment-bound, not a missing-tool gap. Anyone who builds or benchmarks such agents should treat the harness as a first-order factor and measure constraint leakage rather than assume it.

Core claim

Holding the DeepSeek model fixed, two clients produce a 2.1× baseline solve-rate gap (59.7% vs 27.9%). After tool corrections, agent-behavior changes, and eleven new capability tools, overall success rises from 55.4% to 72.0%, every configuration improves significantly, and residual failures are reasoning- or environment-bound rather than missing-tool. A model’s solve rate is therefore meaningful only when the client that produced it is named.

What carries the argument

The evaluate–diagnose–intervene workflow: a 774-trial baseline (86 challenges × three tool-access regimes × three model/client configurations), failure-mode characterization, then re-run of only prior non-successes under a bundled fix, with each recovery attributed to specialized tools (49) or to behavioral and default fixes (79).

Load-bearing premise

The fixes, prompts, and new tools were designed and tested on the same 86 challenges used to score them, so the measured lift and the “not missing-tool” residual taxonomy may not hold on held-out or non-CTF targets.

What would settle it

Re-run the same three model/client configurations on a held-out CTF suite or authorized real targets without re-tuning tools or prompts; if the client gap collapses, the post-fix lift disappears, or residual failures become missing-tool again, the central claims fail.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A model’s CTF or pentest solve rate is only interpretable when the client (harness) that produced it is named.
  • Benchmarks that vary models while fixing an unnamed harness may be measuring the harness as much as the model.
  • Instruction-only tool constraints leak native calls and must be measured, not assumed, especially for weaker clients.
  • Once core tooling is filled in, further gains come more from client orchestration and reasoning than from catalog breadth.
  • Mid-tier challenges absorb the largest intervention gains; hard multi-step and environment-bound tasks remain the floor.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Offensive-agent leaderboards will keep flipping until harness and client are treated as first-class experimental factors alongside the model.
  • Soft instruction-only tool gates systematically favor stronger clients; fair “orchestrator-only” comparisons may need hard technical disablement of native tools.
  • The finding that behavioral redirects recovered more trials than new specialized tools may generalize to other tool-heavy agent domains beyond CTF.
  • Real engagements will likely enlarge the environment-bound failure class that flag-centric CTF settings understate.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper evaluates HexStrike-AI, an MCP-based security-tool orchestrator with 150+ tools, as a testbed for what bounds LLM agent success on offensive tasks. Across 86 picoCTF challenges, three tool-access regimes, and three model/client configurations (774 trials), it reports a large client effect for a fixed DeepSeek model (RooCode 59.7% vs 5ire 27.9%, ~2.1×), a monotonic difficulty gradient, and a bundled intervention (tool fixes, prompt/behavior changes, eleven new capability tools) that raises overall solve rate from 55.4% to 72.0% with significant paired gains. Residual failures after intervention are characterized as reasoning- or environment-bound rather than missing-tool. A 60-run stability sub-study supports single-run reproducibility. The authors are explicit that the study uses one benchmark, that fixes were tuned on the same challenges, and that multi-model generality of the client effect remains a hypothesis.

Significance. If the results hold, the paper makes a concrete methodological contribution to LLM-agent evaluation in security: a model’s CTF/orchestrator solve rate is not interpretable without naming the driving client/harness, and soft tool-access constraints must be measured rather than assumed. The 774-trial matrix, Wilson intervals, McNemar tests, native-tool leakage logging, recovery attribution (79 behavioral vs 49 specialized-tool), and stability sub-study are strengths relative to single-rate CTF reports. The evaluate–diagnose–intervene design and public fork further support reuse. The work is timely as MCP-style security orchestrators proliferate, even though external validity beyond picoCTF and one fixed-model client pair is limited by design.

major comments (3)
  1. [§3.4–3.5, Table 1, §4.3, §6.1] RQ3 residual claim and same-set intervention (§3.4–3.5, Table 1, §4.3, §6.1): The conclusion that residual failures are “reasoning- or environment-bound rather than missing-tool” is load-bearing for RQ3, but the eleven capability tools, redirects, and prompt rules were motivated by and tuned on the same 86 challenges later re-scored, with 56 trials pre-judged “out of reach” and excluded from re-run while still counted as failures. On this design, remaining failures are partly residual by construction. The manuscript already flags same-set tuning; it should either (a) reframe the residual taxonomy as exploratory on this benchmark rather than a general bound, or (b) add a hold-out / leave-some-challenges-untuned analysis, or (c) report how many residual failures would still be tool-addressable under a pre-registered tool budget. Without one of these, the “where capability runs out” claim o
  2. [§4.1, Tables 2–3, §4.6, §5.1] Client-effect isolation vs. constraint leakage and sampling (§4.1 Tables 2–3, §4.6, §5.1): The 2.1× DeepSeek client gap is the paper’s strongest isolation and is already visible under Exp 1 free-solve (59.3% vs 30.2%), which is important because it is not solely an artifact of soft HexStrike-only instructions. However, §5.1 attributes the gap to agent loop, tool-call repair, context/retry, and constraint enforcement without decomposing how much of the overall 2.1× is carried by Exp 1 versus Exp 2/3, where 2,908 native execute_command and 717 Bash calls are concentrated on the weaker clients. Temperature/top-p are also unpinned client defaults. For the central “name the client” claim, please report the client contrast with 95% CIs and tests separately by regime (especially Exp 1), and state more carefully what fraction of the gap could still be soft-constraint non-adherence or default sam
  3. [§3.4–3.5, §4.2, §6.1] RQ2 attribution under a bundled, one-directional intervention (§3.4–3.5, §4.2, §6.1): Recoveries are attributed 79 to behavioral/default fixes vs 49 to specialized tools, but the intervention is a single bundle (minimal baseline prompt → hypothesis-first rules + decomposition + tool nudges + new tools), and only prior non-successes were re-run (b=0 in McNemar by construction). The 79/49 split therefore cannot cleanly separate prompt upgrade from tooling, and post-fix rates cannot detect regressions among baseline successes. The paper notes both points; still, for RQ2 (“does recovery come more from specialized tools or default behavior?”) the claim should be stated as an upper/lower bound under a one-way re-run of a non-ablated bundle, or an ablation (prompt-only vs tools-only on a subset) should be added. As written, “behavior > tools” is suggestive but not identified.
minor comments (6)
  1. [Figure 1] Figure 1 caption packs Phase I/II numbers densely; a small table of trial counts (baseline 774, excluded 56, re-run 289, recoveries 128) would help readers verify the 72.0% post-fix arithmetic.
  2. [§3.3] PARTIAL is defined and then folded into non-success for all rates; consider reporting PARTIAL rates in an appendix so partial-credit progress is not discarded for secondary analysis.
  3. [§3.6] Equation (1) Wilson interval and (2)–(3) McNemar/z-tests are standard; stating software/package used for the exact binomial reduction when b=0 would aid reproducibility.
  4. [§6.2] DeepSeek is a rolling deepseek-chat alias whose snapshot may have shifted (§6.2); if any run dates or API version pins are available, list them; otherwise keep the caveat more visible near Table 2.
  5. [Abstract, §3.6, Tables] Minor typos/spacing: “HexStrikeAI” vs “HexStrike-AI” in the abstract; “McNemarp” missing space in abstract; “T able” line breaks in table titles; “wetesttheclienteffect” spacing glitch in §3.6.
  6. [§2] Related work is strong on harness/ACI literature; a one-sentence contrast with Yu et al. (ref. 23), who found HexStrike strongest among five frameworks, would clarify why this paper treats it as testbed rather than competitor.

Circularity Check

0 steps flagged

No significant circularity: empirical client-gap and intervention measurements do not reduce by construction to their inputs.

full rationale

This is an empirical evaluation-and-intervention study, not a first-principles derivation. The load-bearing client claim (DeepSeek fixed; RooCode 59.7% vs 5ire 27.9%, 2.1×, two-proportion z) is a measured contrast on a fixed challenge matrix, not a quantity defined from or fitted to the outcome it reports. The 55.4%→72.0% lift and the 79/49 behavior-vs-tool attribution are before/after measurements of an engineered bundle on the same 86 picoCTF challenges; the paper itself states that the fixes were tuned on that set and that multi-model generality of the client effect is only a hypothesis. Same-set evaluation is a generalization / internal-validity threat, not a circular reduction (no parameter is fitted and then re-labeled a prediction; no equation equates an output to its defining input). Residual “reasoning- or environment-bound rather than missing-tool” is an interpretive taxonomy of observed failures after intervention, not a uniqueness theorem or self-definitional identity. There is no load-bearing self-citation uniqueness claim, no ansatz smuggled via author prior work, and no renaming of a known closed-form result. Per the analyzer default, honest non-finding applies.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

This is an empirical systems paper, not a derivation. Load-bearing premises are domain assumptions about CTF as proxy, instruction-based tool constraints, and author-judged reachability of failures—not free physical constants or invented particles. The main “parameters” are design choices (which 56 trials are out of reach; which tools/prompts to add) fitted to the evaluation set itself.

free parameters (3)
  • out_of_reach_exclusion_set
    56 previously-unsuccessful trials excluded from re-run by author judgment as needing substantial new tooling or infrastructure; choice affects which recoveries are attempted though excluded trials remain failures in totals.
  • intervention_bundle_contents
    Hypothesis-first preamble, phased decomposition, per-category nudges, eleven capability tools, and tool redirects were selected after baseline failure analysis on the same 86 challenges—hand-chosen engineering knobs, not held-out.
  • PARTIAL_vs_FAILED_manual_boundary
    Author-assigned PARTIAL vs FAILED from logs; folded into non-success for rates, but still a free labeling threshold for recovery attribution narratives.
axioms (4)
  • domain assumption picoCTF flag capture under sanctioned instances is a valid proxy for measuring LLM security-tool orchestration capability.
    Stated as construct-validity threat in §6.1 and related work; entire evaluation rests on this proxy.
  • domain assumption Forbidding native tools by prompt instruction (without technical disablement) is a meaningful tool-access regime for comparing Free vs HexStrike-ranked/strict.
    §3.2 Exp2/3 design; later §4.6 shows heavy leakage, so the axiom is strained but still load-bearing for regime comparisons.
  • ad hoc to paper A client effect measured on DeepSeek alone supports treating “name the client” as general evaluation guidance (generality left as hypothesis).
    §5.1 and Abstract explicitly limit the strong claim to DeepSeek while still drawing methodological implications for the field.
  • domain assumption Single-run SUCCESS/FAIL verdicts are stable enough for configuration-level conclusions outside the 60-run sub-study.
    Stability sub-study (§3.7, §4.5) supports this mainly for stronger clients; weaker client shows borderline variance.

pith-pipeline@v1.1.0-grok45 · 16540 in / 3553 out tokens · 32789 ms · 2026-07-12T06:27:13.116350+00:00 · methodology

0 comments
read the original abstract

Large language model agents driving security tool suites over the Model Context Protocol are increasingly common. Yet the factors that bound their capability remain poorly characterized: how much depends on the model versus the client that drives it, whether constraining the agent to the orchestrator's own tools helps, and where capability is limited by reasoning rather than by missing tools. Using HexStrikeAI, an open-source orchestrator that exposes 150+ tools, as a testbed, we follow a methodology that evaluates the system, diagnoses its failures, and applies targeted improvements. We run 86 picoCTF challenges across seven categories and three difficulty tiers, under three tool-access regimes and three model/client configurations (774 trials). We then apply corrections to existing tools, agent-behavior changes, and eleven new capability tools, and re-run the previously-unsuccessful trials. The diagnosis isolates the driving client as a first-order factor for a fixed model (a 2.1 * gap between two DeepSeek clients) and a monotonic difficulty gradient, with the largest gains in the mid tier. The overall solve rate rises from 55.4% to 72.0%, and every configuration improves significantly (paired McNemar p < 0.001, non-overlapping 95% confidence intervals). The residual failures are reasoning- or environment-bound rather than missing-tool. A 60-run stability sub-study finds single-run verdicts reproducible (17/20 unanimous). We discuss what the results imply for how such orchestrators should be evaluated, and we are explicit about the limits: the study uses a single benchmark, the fixes were tuned on the same challenges they were evaluated on, and the client effect is demonstrated for one model only, so its generality to other models remains a hypothesis.

Figures

Figures reproduced from arXiv: 2607.02873 by Assmaa Zeghaider, Romain Gerard, Yan Guo.

Figure 1
Figure 1. Figure 1: HexStrike-AI architecture and our evaluation-and-intervention workflow. (A) The upstream pipeline routes an LLM client through a FastMCP layer (169 MCP tools) and a Flask server (176 routes) out to 150+ external Kali utilities, over MCP, HTTP, and shell; our contributions (shaded) add the global modifiers and prompt design on the client, the logging and measurement layer, and the eleven capability tools, f… view at source ↗
Figure 2
Figure 2. Figure 2: The two-step run prompt. Step 1 is a fixed setup (identity, metadata, timer), identical every run and in both phases. Step 2 is the task step: at baseline a one-line role instruction that varies only by regime, plus the challenge text; the intervention later adds the hypothesis-first block (a–c), the always-on rules, and a phased-decomposition call (the per-category tool nudges and the strengthened Exp 3 w… view at source ↗
Figure 3
Figure 3. Figure 3: Baseline solve rate (%) across the seven categories and three difficulty tiers. Capability falls off sharply with difficulty in almost every category, with the weakest cells in the Hard column where the baseline tooling is largely absent. a one-line prompt nudge rather than new code. This redirection was the single largest lever, and on its own it drove more recoveries than all the tools we built combined.… view at source ↗
Figure 4
Figure 4. Figure 4: Solve rate before and after the fixes for each configuration, with 95% Wilson confidence intervals. All three improve, and each configuration’s before and after inter￾vals do not overlap: Claude 78.7% to 89.9%, RooCode 59.7% to 76.4%, and 5ire 27.9% to 49.6%. time (a live race or multi-round protocol), multi-step stateful chains (every step must land), and one-off toolchains. What remains is reasoning- or … view at source ↗
Figure 5
Figure 5. Figure 5: Post-fix solve rate (%) across the seven categories and three difficulty tiers, the after to the baseline map in [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

25 extracted references · 11 linked inside Pith

  1. [1]

    arXiv preprint arXiv:2507.21113 (2025)

    Abdulzada, F., et al.: Vulnerability mitigation system (VMS): LLM agent and evaluation framework for autonomous penetration testing. arXiv preprint arXiv:2507.21113 (2025)

    Pith/arXiv arXiv 2025

  2. [2]

    In: Proceedings of the 42nd International Con- ference on Machine Learning (ICML)

    Abramovich, T., et al.: EnIGMA: Interactive tools substantially assist LM agents in finding security vulnerabilities. In: Proceedings of the 42nd International Con- ference on Machine Learning (ICML). pp. 246–355 (2025)

    2025

  3. [3]

    arXiv preprint arXiv:2604.19354 (2026)

    Al-Kaswan, A., Vizner, R., Plotnikov, M., van Deursen, A., Hájek, M.: Do agents dream of root shells? partial-credit evaluation of LLM agents in capture the flag challenges. arXiv preprint arXiv:2604.19354 (2026)

    Pith/arXiv arXiv 2026

  4. [4]

    com/news/model-context-protocol(2024)

    Anthropic: Introducing the Model Context Protocol.https://www.anthropic. com/news/model-context-protocol(2024)

    2024

  5. [5]

    arXiv preprint arXiv:2605.20023 (2026)

    Chacko, S.J., Islam, C.M., Huggleston, J., Liu, X.: When skills don’t help: A neg- ative result on procedural knowledge for tool-grounded agents in offensive cyber- security. arXiv preprint arXiv:2605.20023 (2026)

    Pith/arXiv arXiv 2026

  6. [6]

    In: Proceedings of the USENIX Security Symposium (2024) 16 R

    Deng, G., Liu, Y., Mayoral-Vilches, V., Liu, P., Li, Y., Xu, Y., Zhang, T., Liu, Y., Pinzger, M., Rass, S.: PentestGPT: Evaluating and harnessing large language models for automated penetration testing. In: Proceedings of the USENIX Security Symposium (2024) 16 R. Gerard et al

    2024

  7. [7]

    Deng, G., et al.: What makes a good LLM agent for real-world penetration testing? arXiv preprint arXiv:2602.17622 (2026)

    arXiv 2026

  8. [8]

    arXiv preprint arXiv:2605.30096 (2026)

    Erdem, G.T.: How reliable are AI attackers against a fixed vulnerable target? a 400-run empirical study of LLM penetration testing consistency. arXiv preprint arXiv:2605.30096 (2026)

    Pith/arXiv arXiv 2026

  9. [9]

    GitHub repository,https://github.com/BreadyBred/ hexstrike-eval(2026)

    Gerard, R., Zeghaider, A., Guo, Y.: HexStrike-AI: Evaluation and tar- geted improvements. GitHub repository,https://github.com/BreadyBred/ hexstrike-eval(2026)

    2026

  10. [10]

    arXiv preprint arXiv:2504.10112 (2025)

    Happe, A., Cito, J.: Benchmarking practices in LLM-driven offensive security: Testbeds, metrics, and experiment design. arXiv preprint arXiv:2504.10112 (2025)

    Pith/arXiv arXiv 2025

  11. [11]

    In: Proceedings of the 33rd ACM Conference on User Modeling, Adaptation and Personalization (UMAP) (2025)

    Isozaki, I., Shrestha, M., Console, R., Kim, E.: Towards automated penetration testing: Introducing LLM benchmark, analysis, and improvements. In: Proceedings of the 33rd ACM Conference on User Modeling, Adaptation and Personalization (UMAP) (2025)

    2025

  12. [12]

    Mayoral-Vilches, V., et al.: Towards cybersecurity superintelligence (CSI): What’s the best harness for cybersecurity? arXiv preprint arXiv:2605.28334 (2026)

    Pith/arXiv arXiv 2026

  13. [13]

    GitHub repository,https://github.com/0x4m4/ hexstrike-ai(2025)

    Osama, M.: HexStrike-AI. GitHub repository,https://github.com/0x4m4/ hexstrike-ai(2025)

    2025

  14. [14]

    In: Advances in Neural Information Processing Sys- tems (NeurIPS) (2024)

    Patil, S.G., Zhang, T., Wang, X., Gonzalez, J.E.: Gorilla: Large language model connected with massive APIs. In: Advances in Neural Information Processing Sys- tems (NeurIPS) (2024)

    2024

  15. [15]

    In: Advances in Neural Information Processing Systems (NeurIPS) (2023)

    Schick, T., et al.: Toolformer: Language models can teach themselves to use tools. In: Advances in Neural Information Processing Systems (NeurIPS) (2023)

    2023

  16. [16]

    In: Advances in Neural Information Process- ing Systems (NeurIPS) (2024)

    Shao, M., et al.: NYU CTF Bench: A scalable open-source benchmark dataset for evaluating LLMs in offensive security. In: Advances in Neural Information Process- ing Systems (NeurIPS) (2024)

    2024

  17. [17]

    arXiv preprint arXiv:2505.17107 (2025)

    Shao, M., et al.: CRAKEN: Cybersecurity LLM agent with knowledge-based exe- cution. arXiv preprint arXiv:2505.17107 (2025)

    Pith/arXiv arXiv 2025

  18. [18]

    In: Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) (2026)

    Shao, M., et al.: Towards effective offensive security LLM agents: Hyperparameter tuning, LLM as a judge, and a lightweight CTF benchmark. In: Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) (2026)

    2026

  19. [19]

    arXiv preprint arXiv:2605.28840 (2026)

    Yagubyan, A.: How consistent are LLM agents? measuring behavioral reproducibil- ity in multi-step tool-calling pipelines. arXiv preprint arXiv:2605.28840 (2026)

    Pith/arXiv arXiv 2026

  20. [20]

    In: Advances in Neural Information Processing Systems (NeurIPS) (2024)

    Yang, J., Jimenez, C.E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., Press, O.: SWE-agent: Agent-computer interfaces enable automated software engineering. In: Advances in Neural Information Processing Systems (NeurIPS) (2024)

    2024

  21. [21]

    In: Proceedings of the Inter- national Conference on Learning Representations (ICLR) (2023)

    Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., Cao, Y.: ReAct: Synergizing reasoning and acting in language models. In: Proceedings of the Inter- national Conference on Learning Representations (ICLR) (2023)

    2023

  22. [22]

    arXiv preprint arXiv:2605.27922 (2026)

    Yao, Y., et al.: Harness-Bench: Measuring harness effects across models in realistic agent workflows. arXiv preprint arXiv:2605.27922 (2026)

    Pith/arXiv arXiv 2026

  23. [23]

    arXiv preprint arXiv:2606.25332 (2026)

    Yu, L., Li, S., Zhou, M., Chu, G., Guo, Y.: Decoupling reconnaissance and exploita- tion: Measuring the capability boundaries of LLM-based web penetration testing. arXiv preprint arXiv:2606.25332 (2026)

    Pith/arXiv arXiv 2026

  24. [24]

    In: Proceedings of the International Conference on Learning Representations (ICLR) (2025)

    Zhang, A.K., et al.: Cybench: A framework for evaluating cybersecurity capabilities and risks of language models. In: Proceedings of the International Conference on Learning Representations (ICLR) (2025)

    2025

  25. [25]

    arXiv preprint arXiv:2605.23950 (2026)

    Zhang,Y.,Xu,W.,Wang,J.,Hamm,J.,Ge,Y.,Reddy,C.K.:StopcomparingLLM agents without disclosing the harness. arXiv preprint arXiv:2605.23950 (2026)

    Pith/arXiv arXiv 2026