Pith. sign in

REVIEW 6 major objections 4 minor 3 cited by

Multi-Agent Penetration Testing AI for the Web

T0 review · 6 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An open-source multi-agent LLM system solves 80 of 104 web pentest challenges, matching commercial performance at a median $0.12 per target.

desk verdict Solid engineering and a useful cost model, but the headline 76.9% is not trustworthy until the 43 repaired XBOW images are shown to preserve the original challenges. read the letter →

arxiv 2508.20816 v1 pith:PKWRG7AW submitted 2025-08-28 cs.CR cs.AI

Isaac David , Arthur Gervais This is my paper · ORCID
classification cs.CRcs.AI
keywords multi-agentsystemsLLMagentswebpenetrationtestingXSSSQLinjectionSSRFcostanalysisXBOWbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MAPTA is an open-source multi-agent system that combines an LLM coordinator, sandboxed tool-execution agents, and a separate validation agent to autonomously find and prove web vulnerabilities. On the 104-challenge XBOW benchmark it solves 80 challenges (76.9%), close to the commercial system's reported 84.6%, at a median cost of $0.117 per challenge. It achieves perfect scores on SSRF and misconfiguration, strong scores on injection and authorization, and weak scores on XSS and blind SQL injection. The paper argues this shows tool-grounded, verify-by-execution multi-agent design is a viable path to continuous, cost-predictable security assessment.

What carries the argument

The load-bearing mechanism is the three-role agent architecture with a single-pass controller: a Coordinator agent reasons about attack paths and delegates through tools, one or more Sandbox agents execute commands and Python in a shared per-job Docker container, and a Validation agent consumes each candidate finding and verifies it by concrete execution, returning pass/fail evidence (a captured flag in CTF mode, side-effect evidence in real-world mode). The per-job container gives stateful reuse of artifacts across subtasks while keeping LLM contexts isolated, and the Validation step is what converts theoretical findings into verified exploits.

What would settle it

Compare MAPTA's success rate on the 43 repaired challenges against the same challenges reproduced from pristine originals (or archived source) with identical flags and exploit paths; any mismatch in success or flag value would show the fixes altered the benchmark. Alternatively, a manual audit of pre- and post-fix differential behavior (route responses, error messages, flag location) that finds a behavioral change in even a handful of the 43 images would invalidate the headline comparison.

Watch

Extended reading notes

Core claim

The central claim is that a multi-agent LLM architecture with mandatory end-to-end exploit validation can perform autonomous black-box web penetration testing at near-commercial success rates while remaining fully open and reproducible. MAPTA separates strategic planning (Coordinator), tactical execution (Sandbox agents sharing one Docker container per job), and verification (Validation agent that executes candidate proofs-of-concept and returns pass/fail evidence). This design turns vulnerability assessment from hypothesis generation into empirical validation, eliminating most false positives. The paper supports the claim with an 80/104 success rate on the 104-challenge benchmark, a $21.38

Load-bearing premise

The 43 repaired Docker images must preserve the original challenges' behavior and difficulty; if the fixes changed how a vulnerability can be exploited, the 76.9% score and the comparison to the commercial baseline no longer measure the same benchmark.

Editorial extensions

If this is right

  • Early stopping at roughly 40 tool calls, $0.30, or 300 seconds would prune most failing attempts, cutting wasted spend on targets the agent cannot crack.
  • Operating cost of $0.073 per successful benchmark challenge and $3.67 per open-source app assessment makes continuous, scheduled pentesting economically plausible at scale.
  • Perfect SSRF and misconfiguration scores plus 83% authorization success suggest the strongest near-term use is access-control and server-side request forgery review, not XSS.
  • Findings are ready-to-use proofs-of-concept with evidence, not pattern-match alerts, so organizations can feed them straight into triage or CVE processes.
  • Because 43 of the 104 challenge images had to be repaired, the benchmark's longevity depends on community maintenance of those fixes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The negative resource-success correlation suggests the agent is not 'persisting' but thrashing: adding a second, differently configured agent (e.g., a payload-generation specialist for XSS) might convert some of the 24 failures without raising median cost.
  • The 43 repaired Docker images are unvalidated; a reader wanting to trust the 76.9% comparison should require before/after equivalence checks (same flag, same exploit path, same intended difficulty) on those images.
  • If the real-world results hold up under CVE review, open-source projects with modest budgets could obtain near-commercial-grade continuous security assessment, shifting vulnerability discovery from scheduled audits toward always-on monitoring.
  • Blind SQL injection (0/3) is likely a timing-analysis gap, not a reasoning gap; existing blind-injection techniques (conditional delays) could be added as a specialist tool inside sandbox agents.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 4 minor

Summary. The paper describes MAPTA, an LLM-based multi-agent system for autonomous web penetration testing. Three roles (Coordinator, Sandbox, Validation) share a per-job Docker container, and a bounded loop performs hypothesis synthesis, tool execution, PoC assembly, and validation. Evaluation is in two modes: blackbox CTF on the 104-challenge XBOW benchmark, where MAPTA reportedly solves 80 challenges (76.9%) with total LLM cost $21.38 and median $0.117 per challenge; and whitebox assessment of ten open-source repositories, where it reports 19 vulnerabilities with 10 pending CVEs. The paper also reports negative correlations between resource usage and success and proposes early-stopping thresholds of about 40 tool calls, $0.30, and 300 seconds. Code and a modified XBOW framework are released; 43 of the 104 XBOW Docker images were manually repaired to fix deprecated software versions.

Significance. The paper has real strengths: open-source release, a binary flag-based success metric that avoids false-positive ambiguity, detailed token-level cost accounting, and a Validation agent that executes candidate PoCs. If the empirical claims are supported, MAPTA would be a useful open baseline for autonomous web pentesting and a valuable data point on cost-performance tradeoffs. The reported 76.9% on XBOW is appealing, but it currently rests on an unverified modified benchmark and a single run per challenge; the cost-threshold analysis is in-sample; and the real-world findings are anonymized and unverifiable. The potential contribution is significant, but the evidence as presented does not yet justify the headline comparison or the actionable early-stopping claims.

major comments (6)
  1. [Section 3 (Benchmark modification)] The central claim of 76.9% (80/104) is computed on a benchmark in which 43 of the 104 XBOW Docker images were manually repaired. No diff-level equivalence analysis is provided. Without evidence that the fixes preserve vulnerability semantics, difficulty, flags, and exploit paths, the comparison with XBOW's 84.6% and all downstream per-category and cost conclusions are not measuring the same object. Please provide (a) a per-fix diff or classification (dependency-only vs application-logic changes), (b) success rates separately for the untouched and repaired subsets, and (c) a validation protocol showing preserved exploitability on the repaired images.
  2. [Section 3.3 (Early-stopping thresholds)] The thresholds of about 40 tool calls, $0.30, and 300 seconds are derived from the 95th percentile of successful challenges on the same 104-challenge data and then presented as practical early-stopping thresholds. This is in-sample fitting rather than prediction. Please re-label these as descriptive statistics or validate them out-of-sample (e.g., held-out challenges, repeated runs, or a separate benchmark); as written, the actionable contribution is circular.
  3. [Section 3.2, Table 2 (Run variance)] The evaluation appears to consist of a single run per challenge. Given the stochastic LLM controller, the 76.9% point estimate and category-level rates (e.g., 57% XSS, 0% blind SQLi) carry no uncertainty. Please report multiple runs for at least a random subset, or provide confidence intervals, and clarify whether the reported cost/time values come from the same single run. The p<0.001 correlations also assume independent observations, which a single-run design does not establish.
  4. [Section 4.1, Table 3 (Real-world validation)] The real-world findings are anonymized (OSN-XX), with no CVE identifiers, patch status, or disclosure artifacts for the 10 findings said to be under CVE review. This makes the practical-impact claims unverifiable. Please provide a reproducible disclosure package (e.g., redacted PoCs after maintainer response) or clearly separate verified and pending findings, and state the verification status of each reported vulnerability.
  5. [Section 3.4 (XBOW comparison)] The competitive-with-commercial claim rests on XBOW's self-reported 84.6% from a blog post with no published protocol, and on the modified 104-image benchmark. Even if the image fixes are shown to be semantically neutral, the evaluation protocols may differ. Please add a direct comparison run under a documented protocol on the same images, or qualify the claim as comparable to XBOW's reported result under a partially updated benchmark.
  6. [Section 3 (Memorization)] The claim that MAPTA's solutions are genuine discovery because no online CTF solutions were found is an argument from absence, with no search methodology or contamination probe. Since GPT-5 training data may include public benchmark repositories, please provide a contamination check (e.g., query the model with challenge descriptions and tools disabled, or test for exact flag/solution memorization) and disclose the search procedure; otherwise remove or weaken the claim.
minor comments (4)
  1. [Throughout] Typo: 'OW ASP' appears in Section 2.3, Section 5.1, and reference [17]; should be 'OWASP'.
  2. [Section 2.4] CTF mode uses a single agent, while the abstract and introduction emphasize multi-agent coordination and a Validation agent. Clarify how end-to-end PoC validation applies to the CTF results, since validation reduces to flag extraction in that mode.
  3. [Section 1.1 / Related Work] The claim of being the first open-source multi-agent penetration testing AI should be carefully scoped relative to open-source projects such as PentestGPT; clarify the precise novelty (multi-agent vs multi-stage, web-focused, validation-oriented).
  4. [Figure 7] The Sankey diagram is visually dense and difficult to read. Provide a companion table with exact counts per vulnerability category and outcome.

Circularity Check

1 steps flagged · score 4.0 of 10

Mild post-hoc fitting in early-stopping thresholds; central 76.9% result is grounded in the external XBOW benchmark and is not circular.

  1. fitted input called prediction [Abstract; §3.3 (Resources and Success Correlations); §1.1 Key Insights]
    "Specifically, production deployments can implement early stopping when tool usage exceeds 40+ calls (95th percentile of successful challenges), cost surpasses $0.30 per target (indicating likely failure), or execution time reaches 300+ seconds without significant progress."

    The early-stopping thresholds are fit to the same 104-challenge outcomes they are then recommended for: 40 tool calls, $0.30, and 300 seconds are quantiles of the successful-challenge distribution in that evaluation. Choosing the 95th percentile of successful challenges makes it true by construction that roughly 95% of successful runs fall below the threshold; presenting this as an actionable production rule ('production deployments can implement early stopping') is a post-hoc summary statistic, not an independent predictive finding. The paper itself labels the values as fitted ('95th percentile of successful challenges'), so the 'practical early-stopping thresholds' contribution reduces to a restatement of the benchmark distribution. This is peripheral to the central 76.9% success-rate cl

full rationale

The central claim—76.9% success on 104 XBOW challenges—is an empirical evaluation against an external benchmark, not a derivation from the paper's own inputs. Success is determined by flag extraction, and the reported 80/104 score is a measured outcome, not a fitted parameter. No load-bearing self-citations appear: references to XBOW and other systems are external, and the authors do not invoke a prior 'uniqueness theorem' or ansatz from their own work. The only identifiable circularity is the early-stopping guidance in §3.3, where thresholds (40 tool calls, $0.30, 300 seconds) are computed from the same dataset and then presented as actionable. This is a mild form of fitted-input-called-prediction and slightly inflates the contribution, but it does not affect the primary benchmark result. The 43 manually repaired XBOW Docker images are a potential external-validity threat (no pre/post equivalence analysis is provided), but that is an evidence gap about benchmark integrity, not a circular derivation. Overall, the central empirical claim is self-contained and externally grounded; the circularity score reflects only the peripheral fitted thresholds.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central performance claims rest on benchmark integrity (modified 43/104 Docker images), single-run statistics, and the realism of local whitebox findings. The early-stopping thresholds are post-hoc data-derived numbers presented as contributions.

free parameters (4)
  • Early-stopping tool-call threshold = ~40 tool calls
    95th percentile of successful challenges in Section 3.3, presented as an actionable stopping rule without held-out validation.
  • Early-stopping cost threshold = $0.30
    Derived from the same 104-challenge cost distribution (Section 3.3).
  • Early-stopping time threshold = 300 seconds
    Derived from solve-time distribution (Section 3.3).
  • Agent budget caps (max time, tool calls, cost per job)
    Section 2.4 mentions bounded loops and budget caps, but exact values are not reported; these caps influence which challenges are counted as failures (e.g., max time 1428.7s suggests a cap).
assumptions (5)
  • domain assumption XBOW ground-truth flags and the 43 manually fixed Docker images preserve original challenge semantics and difficulty.
    Section 3 states 43/104 images required fixes due to deprecated software; no equivalence analysis is provided, and fixes are not yet merged upstream.
  • domain assumption A single run per challenge adequately estimates the system's success rate.
    Section 3.2 reports one pass over 104 challenges; LLM agents are stochastic, and no repeated runs or confidence intervals are given.
  • domain assumption Vulnerabilities found in local whitebox deployments reflect exploitable issues in the real open-source applications.
    Section 4.1 anonymizes the vulnerable targets (OSN-XX) and provides no external confirmation of exploitability beyond the authors' own PoCs.
  • domain assumption GPT-5's pricing and behavior at evaluation time are stable and correctly applied.
    Section 3.1 uses GPT-5 pricing ($1.25/1M input, $10/1M output, $0.125/1M cached); the cost analysis depends on these external values.
  • ad hoc to paper The absence of online CTF solutions implies MAPTA's solutions are genuine discoveries rather than memorized outputs.
    Section 3: 'We further have not found any online CTF solutions for this benchmark, and hence believe...' Absence of search results is weak evidence and does not test for model memorization.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multi-Agent Penetration Testing AI for the Web." pith.science (2026). https://pith.science/paper/PKWRG7AW

@misc{pith2026250820816,
  author       = {Pith},
  title        = {Pith review of: Multi-Agent Penetration Testing AI for the Web},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PKWRG7AW}},
  note         = {Machine review of arXiv:2508.20816}
}
abstract

AI-powered development platforms are making software creation accessible to a broader audience, but this democratization has triggered a scalability crisis in security auditing. With studies showing that up to 40% of AI-generated code contains vulnerabilities, the pace of development now vastly outstrips the capacity for thorough security assessment. We present MAPTA, a multi-agent system for autonomous web application security assessment that combines large language model orchestration with tool-grounded execution and end-to-end exploit validation. On the 104-challenge XBOW benchmark, MAPTA achieves 76.9% overall success with perfect performance on SSRF and misconfiguration vulnerabilities, 83% success on broken authorization, and strong results on injection attacks including server-side template injection (85%) and SQL injection (83%). Cross-site scripting (57%) and blind SQL injection (0%) remain challenging. Our comprehensive cost analysis across all challenges totals $21.38 with a median cost of $0.073 for successful attempts versus $0.357 for failures. Success correlates strongly with resource efficiency, enabling practical early-stopping thresholds at approximately 40 tool calls or $0.30 per challenge. MAPTA's real-world findings are impactful given both the popularity of the respective scanned GitHub repositories (8K-70K stars) and MAPTA's low average operating cost of $3.67 per open-source assessment: MAPTA discovered critical vulnerabilities including RCEs, command injections, secret exposure, and arbitrary file write vulnerabilities. Findings are responsibly disclosed, 10 findings are under CVE review.

Figures

Figures reproduced from arXiv: 2508.20816 by the authors.

Figure 1
Figure 1. MAPTA multi-agent architecture with single￾pass controller with evidence-gated branching. Three roles: a Coordinator (strategy and orchestration), one or more Sand￾box agents (tactical execution in an isolated per-job Docker environment), and a Validation agent (concrete PoC exe￾cution and pass/fail evidence). The Coordinator dynami￾cally decides whether to delegate to sandbox agents via the sandbox_agent tool or to… view at source ↗
Figure 2
Figure 2. Cumulative distribution of challenge completion [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. CDF of total costs (left) and per-challenge cost by token type (right). Solved challenges maintain lower median costs [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Cumulative distribution of token usage across token [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Tool usage patterns across challenges showing the distribution of command execution calls versus Python runtime calls [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Command usage heatmap across challenges showing the frequency of specific commands used. The [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 9
Figure 9. Figure 9: Vulnerability distribution and assessment costs [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]
Figure 7
Figure 7. Figure 7: Vulnerability category distribution across 104 [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Correlation analysis between challenge success and [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 10
Figure 10. Figure 10: Assessment time versus vulnerability discovery [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Baselines Before Architecture: Evaluating Coding Agents for Autonomous Penetration Testing

    cs.CR 2026-07 conditional novelty 6.0 of 10

    Plain coding CLI agents, especially Codex with newer GPT models, solve 70-96 of 104 XBOW benchmark tasks, matching or exceeding some published security-harness scores under model-matched comparisons.

  2. From Controlled to the Wild: Evaluation of Pentesting Agents for the Real-World

    cs.AI 2026-05 unverdicted novelty 6.0 of 10

    A practical evaluation protocol for AI pentesting agents that uses validated vulnerability discovery, LLM semantic matching, and bipartite scoring to assess performance in realistic, complex targets.

  3. A Survey of LLM-Driven Penetration Testing: Taxonomy, Co-Evolution, and Open Challenges

    cs.SE 2026-07 accept novelty 5.5 of 10

    LLM pentest agents co-evolved through four bottleneck-driven phases into RLVR systems, while CTF platforms became dual evaluation/training infrastructure and three linked reliability gaps remain.

Reference graph

Works this paper leans on

29 extracted references · 27 canonical work pages · cited by 3 Pith papers

  1. [1]

    A survey of business logic vulnerabilities in web applications

    Waleed Alasmary, Feras Khan, Ghada Almashaqbeh, et al. A survey of business logic vulnerabilities in web applications. Information, 16(7):585, 2025

    work page 2025

  2. [2]

    Pythia: Grammar-based fuzzing of rest apis with coverage- guided feedback and learning-based mutations

    Vaggelis Atlidakis, Roxana Geambasu, Patrice Gode- froid, Marina Polishchuk, and Baishakhi Ray. Pythia: Grammar-based fuzzing of rest apis with coverage- guided feedback and learning-based mutations. In ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), 2020. 14

    work page 2020

  3. [3]

    Restler: Stateful rest api fuzzing

    Vaggelis Atlidakis, Patrice Godefroid, and Marina Pol- ishchuk. Restler: Stateful rest api fuzzing. In Inter- national Conference on Software Engineering (ICSE), 2019

    work page 2019

  4. [4]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Ad- vances in neural information processing systems, vol- ume 33, pages 1877–1901, 2020

    work page 1901

  5. [5]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. In arXiv preprint arXiv:2107.03374, 2021

    arXiv 2021

  6. [6]

    Large language models for cyber security: A systematic litera- ture review

    Xiaozhu Chen, Yuhang Zhou, Zihan Wang, et al. Large language models for cyber security: A systematic litera- ture review. arXiv preprint arXiv:2405.04760, 2024

    arXiv 2024

  7. [7]

    Refpentester: A knowledge-informed self-reflective pen- etration testing framework based on llms, 2025

    Hanzheng Dai, Yuanliang Li, Zhibo Zhang, and Jun Yan. Refpentester: A knowledge-informed self-reflective pen- etration testing framework based on llms, 2025

    work page 2025

  8. [8]

    Pentestgpt: Evaluating and harnessing large language models for automated pene- tration testing

    Gelei Deng, Ziniu Hu, Yueqi Chen, Haoyu Wang, Bangjie Yin, Yinzhi Cao, Gang Wang, Yan Chen, Xinyu Xing, and Zhiqiang Lin. Pentestgpt: Evaluating and harnessing large language models for automated pene- tration testing. In USENIX Security, 2024

    work page 2024

Show all 29 references
  1. [9]

    Damn vulnerable web application (dvwa), 2025

    Ryan Dewhurst. Damn vulnerable web application (dvwa), 2025

    2025

  2. [10]

    Ai agents for offsec with zero false positives, 2025

    Brendan Dolan-Gavitt. Ai agents for offsec with zero false positives, 2025

    2025

  3. [11]

    Our big sleep agent makes a big leap

    Google Cloud CISO Office. Our big sleep agent makes a big leap. Google Cloud Blog, 2025

    2025

  4. [12]

    From naptime to big sleep: Using large language models to find real-world vulnerabilities

    Google Project Zero. From naptime to big sleep: Using large language models to find real-world vulnerabilities. Project Zero Blog, 2024

    2024

  5. [13]

    Penheal: A two-stage llm framework for automated pentesting and optimal remediation

    Junjie Huang and Quanyan Zhu. Penheal: A two-stage llm framework for automated pentesting and optimal remediation. In Proceedings of the ACM Conference Companion on Computer and Communications Security (ACM CCS Companion), AutonomousCyber ’24: Pro- ceedings of the Workshop on ...

    2024

  6. [14]

    Business logic attacks: Why traditional tools fall short

    Imperva. Business logic attacks: Why traditional tools fall short. https://www.imperva.com/blog/ business-logic-attacks-traditional-tools-shortcomings/ ,

  7. [15]

    Kalopisis

    N. Kalopisis. Browser-empowered llm agents for web penetration testing. Master’s thesis, University of Twente, 2025

    2025

  8. [16]

    Com- parison and evaluation on static application security testing (sast) tools for java

    Kaixuan Li, Sen Chen, Lingling Fan, Ruitao Feng, Han Liu, Chengwei Liu, Yang Liu, and Yixiang Chen. Com- parison and evaluation on static application security testing (sast) tools for java. In ESEC/FSE, 2023

    2023

  9. [17]

    Owasp api security top 10: 2023, 2023

    OW ASP Foundation. Owasp api security top 10: 2023, 2023

    2023

  10. [18]

    Owasp juice shop, 2025

    OW ASP Foundation. Owasp juice shop, 2025

    2025

  11. [19]

    Owasp webgoat, 2025

    OW ASP Foundation. Owasp webgoat, 2025

    2025

  12. [20]

    Zed attack proxy (zap) documen- tation, 2025

    OW ASP ZAP Project. Zed attack proxy (zap) documen- tation, 2025

    2025

  13. [21]

    Asleep at the key- board? assessing the security of github copilot’s code contributions

    Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Bren- dan Dolan-Gavitt, and Ramesh Karri. Asleep at the key- board? assessing the security of github copilot’s code contributions. In 2022 IEEE Symposium on Security and Privacy (SP), pages 754–768. IEEE, 2022

    2022

  14. [22]

    Burp suite documentation, 2025

    PortSwigger Ltd. Burp suite documentation, 2025

    2025

  15. [23]

    Web application vul- nerabilities in 2020–2021

    Positive Technologies. Web application vul- nerabilities in 2020–2021. https://global. ptsecurity.com/en/research/analytics/ web-vulnerabilities-2020-2021/ , 2021. Ac- cessed: 2025-08-21

    2020

  16. [24]

    Toolformer: Language models can teach themselves to use tools, 2023

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Can- cedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools, 2023

    2023

  17. [25]

    Xbow validation bench- marks

    XBOW Engineering. Xbow validation bench- marks. https://github.com/xbow-engineering/ validation-benchmarks, 2024. Accessed: 2024-12- 01

    2024

  18. [26]

    Gpt-5 performance analysis for autonomous penetration testing

    XBOW Engineering. Gpt-5 performance analysis for autonomous penetration testing. XBOW Blog, 2025. Accessed: 2025-01-26

    2025

  19. [27]

    Jim ’enez, Ofir Press, and Karthik Narasimhan

    John Yang, Carlos E. Jim ’enez, Ofir Press, and Karthik Narasimhan. Swe-agent: Agent-computer interfaces enable automated software engineering. In Advances in Neural Information Pro- cessing Systems (NeurIPS), 2024

    2024

  20. [28]

    React: Synergizing reasoning and acting in language models, 2022

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, 2022. 15

    2022

  21. [2023]

    Accessed: 2025-08-21

    2025

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.