Pith. sign in

REVIEW 3 major objections 4 minor 35 references

Benchmarking LLM-Assisted Blue Teaming via Standardized Threat Hunting

T0 review · 3 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Structured, modular threat-hunting workflows substantially outperform open-ended LLM reasoning in blue-team defense.

desk verdict CyberTeam is a substantial new benchmark, but the headline performance claim is confounded by retrieval leakage and needs a cleaner control before the numbers can be trusted. read the letter →

arxiv 2509.23571 v3 pith:BMPDGQJA submitted 2025-09-28 cs.CR cs.AI

Yuqiao Meng , Luoxi Tang , Feiyang Yu , Xi Li , Guanhua Yan , Ping Yang , Zhaohan Xi This is my paper
classification cs.CRcs.AI
keywords blueteamthreathuntingLLMbenchmarkoperationalmodulesstandardizedworkflowretrieval-augmentedgenerationcyberintelligencedependencychain
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CyberTeam is a benchmark that asks whether large language models can be steered through realistic blue-team threat hunting by decomposing the job into a standardized dependency chain of analytical tasks. The paper's central claim is that this modular, operation-guided structure is not just an organizational convenience; it materially improves LLM outputs relative to in-context learning, chain-of-thought, and tree-of-thought prompting. On response-and-mitigation tasks, standardized modules push leading models above 90% Hit@10 for playbook recommendation and above 92% for advisory correlation, with the largest gains on multi-hop tasks that require evidence to flow from attribution through behavior analysis to action. The paper also reports that semantic noise in threat logs degrades performance more than token-level noise, and that module traces make LLM outputs auditable. The stakes are practical: if the claim holds, defensive AI systems should embed LLMs in structured analyst pipelines rather than treating them as open-ended reasoners.

What carries the argument

The load-bearing mechanism is the operational-module dependency chain: a fixed, four-stage workflow (attribution → behavior analysis → prioritization → response & mitigation) in which each analytical task is associated with a small set of discrete LLM operations, executed in sequence and passed downstream. The nine modules—NER, REX, SUM, SIM, MAP, RAG, SPA, CLS, MATH—act as structure-forcing wrappers: some (NER, REX, SPA, CLS, MATH) constrain outputs to schema or computation, while others (SUM, RAG, MAP) leave room for open-ended reasoning, creating the standardization-plus-flexibility balance the benchmark is designed to test.

What would settle it

Run CyberTeam's response-and-mitigation tasks in two conditions: standardized modules with retrieval restricted to a held-out source that was excluded from ground-truth construction, and open-ended CoT with the same retrieval source available. If the standardized pipeline's gains over CoT shrink to near zero, the claimed advantage of modular structure is actually an artifact of answer access; if the gap persists, the dependency-chain structure is doing the work.

Watch

Extended reading notes

Core claim

On its own terms, CyberTeam claims that standard operating procedures can be encoded for LLMs as a two-stage structure: threat hunting is modeled as a dependency chain across four stages—threat attribution, behavior analysis, prioritization, and response & mitigation—and each of the 30 analytical tasks in the chain is executed through a small set of operational modules (such as named entity recognition, regex parsing, span localization, retrieval-augmented generation, classification, summarization, and math). When LLMs are guided through this structure, the paper reports consistent gains over ICL, CoT, and ToT across all task categories, with the largest improvements in tasks requiring stric

Load-bearing premise

The evaluation assumes that the RAG module does not retrieve from the same corpora used to create the ground truth, even though the benchmark is built from NVD, MITRE, and vendor-threat-intelligence sources and the RAG component queries external threat-intelligence databases and APIs of the same families.

Editorial extensions

If this is right

  • Deploying LLMs in blue-team tools should follow standardized, modular workflows rather than open-ended prompt-based agents.
  • Structured decomposition yields largest gains in multi-hop tasks such as event-sequence reconstruction and response generation, where an upstream output must be preserved and passed to downstream decisions.
  • Module-traced outputs are auditable: analysts can attribute each conclusion to a specific operation (e.g., RAG retrieval, NER extraction) rather than an opaque reasoning chain.
  • Semantic paraphrase noise is a greater threat to LLM reliability than token-level corruption, implying that imprecise or shifted threat-report wording can mislead even module-guided models.
  • A moderate latency overhead—between CoT and ToT—buys higher reliability and repeatability, making the pipeline a practical middle ground for operational security use.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the standardized-vs-open-ended gain survives a leakage-controlled evaluation (see weakest assumption), the same modular decomposition pattern should transfer to other analyst workflows—SOC triage, digital forensics, incident-report drafting—where procedural knowledge and auditability matter.
  • The RAG module's reliance on the same external intelligence families used to construct ground truth means the benchmark's headline advantage could be partly a retrieval-access effect; a controlled test would separate access-to-knowledge from reasoning structure.
  • The noise-robustness result suggests field scores on clean curated reports will overstate operational performance on messy, paraphrased, or adversarial incident text; a natural extension is to adversarially paraphrase the input logs and report score degradation curves as a standard benchmark feature.
  • The paper's latency numbers imply a deployment policy: use fast ICL for triage filtering and reserve the modular pipeline for high-stakes response decisions—an operational read that the paper's trade-off discussion gestures toward but does not formalize.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents CYBERTEAM, a benchmark and standardized reasoning workflow for LLM-assisted blue-team threat hunting. It organizes threat hunting into 30 tasks across four stages (Threat Attribution, Behavior Analysis, Prioritization, Response & Mitigation) and maps each task to sequences of 9 operational modules (NER, REX, SUM, SIM, MAP, RAG, SPA, CLS, MATH). Data are aggregated from 23 vulnerability databases and threat-intelligence platforms. The experiments compare the standardized pipeline ('Ours') against open-ended ICL, CoT, and ToT on 11 LLMs and cybersecurity agents, reporting that the standardized approach improves performance on tasks such as playbook recommendation and advisory correlation (e.g., >90% Hit@10 for GPT-o4). The paper also evaluates individual-task performance and robustness to token-level and semantic-level noise.

Significance. If the headline comparison were clean, this would be a valuable contribution: CYBERTEAM covers a broad, realistic task space; it provides a modular, interpretable workflow; the authors release code; and the benchmark is substantially larger than several existing cybersecurity LLM benchmarks. The central claim—that decomposing blue-team threat hunting into standardized operational modules improves LLM performance relative to open-ended reasoning—is practically important and plausible. However, as reported, the claim is not established because the RAG module's retrieval sources overlap the sources used to build ground truth, and no ablation isolates the effect of modular standardization from the effect of giving only the standardized arm access to external information. The benchmark resource remains useful, but the comparative evaluation needs reworking before the paper's main conclusion can be accepted.

major comments (3)
  1. [§3.2, §4.1, Appendix B.6, Figure 2] The headline comparison in Table 3 is confounded by asymmetric information access. Section 3.2 states that ground-truth data are built from NVD, MITRE CVE/ATT&CK/CWE/CAPEC/D3FEND, vendor advisories, VirusTotal, AlienVault OTX, MISP, and industry threat reports. Appendix B.6 defines RAG as combining LLM output with 'real-time retrieval from external threat intelligence APIs or databases,' and Figure 2 explicitly shows Playbook Recommendation invoking RAG to 'retrieve and rank playbooks from threat databases, e.g., MITRE D3FEND.' For tasks such as Playbook Recommendation, Advisory Correlation, and Actor Identification, the retrieval index can therefore contain the very records used to define ground truth, while the ICL/CoT/ToT baselines are closed-book prompts. The large gains in Table 3 (e.g., GPT-o4: 91.4 vs 89.2 CoT for Playbook Recommendation; 92.3 vs 84.2 ToT for Advisory Correlation)
  2. [§4.1, Table 3] All reported results are single-point estimates with no error bars, confidence intervals, or significance tests. Several cells show small gaps (e.g., Gemini-2.5 Playbook Recommendation: 91.8 vs 80.1 CoT is large, but Security Control Adjust for Gemini-2.5 is 88.5 vs 79.2 CoT; Llama-4 Patch Tool Suggestion is 82.1 vs 74.2 ToT), and without repeated sampling or bootstrap intervals the reader cannot judge whether these differences are reliable. Because the paper's central claim rests on consistent superiority across models and tasks, the evaluation should report variance (e.g., multiple temperature samples or seeds) at least for the headline response-and-mitigation tasks in Table 3.
  3. [§4.1, RQ1] The interpretation of the RQ1 results in the text goes beyond what the experiment supports. The paper attributes the gains to 'standard operating procedures,' reduced hallucination, and 'correct order and information flow' (Section 4.1), but the experimental design does not manipulate these factors independently. The RAG-vs-no-RAG confound means that the observed differences could be entirely due to retrieval, not to task decomposition, dependency ordering, or modular structure. Please provide an ablation that holds information access constant; without it, the mechanisms named in Section 4.1 are not tested.
minor comments (4)
  1. [Figure 1] Minor typos and inconsistencies: 'comprimised' should be 'compromised'; the C2 domain is 'dns-update.evilcorp.cn' but the text later says 'Block connections to *.evilcrop.net'; the IP '192.169.10.21' appears in the response action while the incident description uses '192.168.10.21'.
  2. [Appendix C.4] The Hit@k definition is clear, but the inclusion of 'Prompt 6. Hit Evaluation Prompt' is confusing: it appears to be an LLM prompt for evaluating retrieval outputs, not part of the threat-hunting pipeline. Please clarify whether Hit@k is computed automatically from ranked outputs or via this prompt, and if the latter, how the prompt's own errors are controlled.
  3. [Appendix C.1 / References] The BERTScore citation appears twice (as 'Zhang* et al., 2020' in Section 4 metrics and again in the bibliography), and the reference list contains both 'Zhang* et al.' and 'Zhang et al.' entries with overlapping content. Please unify. Also, 'UNITEST' should be 'unittest'.
  4. [Section 3.3] The 'Highlight.' callout boxes interrupt the prose and appear to be formatting artifacts. Consider integrating these points into the main text or removing the callouts.

Circularity Check

1 steps flagged · score 6.0 of 10

RAG retrieval sources overlap benchmark ground truth and are exclusive to the standardized arm, confounding the main comparison.

  1. other [Section 3.2; Appendix B.6; Figure 2; Section 4.1 (RQ1)]
    "CYBERTEAM collects threat metadata from two primary sources: (1) vulnerability databases ... and (2) threat intelligence platforms ... RAG combines LLM output with real-time retrieval from external threat intelligence APIs or databases. ... Retrieve and rank playbooks from threat databases, e.g., MITRE D3FEND. ... From Table 3, we observe that using operational modules (Ours) outperforms typical open-ended reasoning methods."

    The benchmark's ground truth for Response & Mitigation tasks is built from the same sources RAG queries. Sec. 3.2 lists NVD, MITRE CVE/ATT&CK/CWE/CAPEC/D3FEND, vendor advisories, VirusTotal, OTX, MISP as construction sources; App. B.6 defines RAG as retrieval from 'external threat intelligence APIs or databases'; Fig. 2 shows RAG 'retrieve and rank playbooks from threat databases, e.g., MITRE D3FEND' and 'retrieve advisories using malware name.' The ICL/CoT/ToT baselines are closed-book prompts with no retrieval. Hence >90% Hit@10 Playbook Recommendation and >92% Advisory Correlation are, by construction, compatible with retrieving the exact source that defines ground truth. No ablation removes RAG from the standardized arm, and no open-ended baseline is given retrieval, so information acc

full rationale

The core empirical claim (Section 4.1) compares the modularized CYBERTEAM pipeline against open-ended ICL/CoT/ToT prompting. However, the modularized arm is the only one with a retrieval component, and that retrieval component draws on the same vulnerability databases and threat-intel platforms (NVD, MITRE, D3FEND, vendor advisories, VirusTotal, OTX, MISP) from which the benchmark's ground-truth labels for playbook recommendation, advisory correlation, patch tool suggestion, and patch code generation are derived. As a result, the observed advantage may reflect answer access rather than standardized reasoning structure; the paper provides no ablation to separate the two. I therefore score this as a partial circularity/confound (6) rather than full circularity, since the modular workflow itself may still add value and the benchmark has substantial independent content. No load-bearing self-citation or imported-uniqueness pattern is present.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the task taxonomy, the correctness of external threat-intelligence sources as ground truth, and the assumption that prompt-based modules implement the named operations. No numerical parameters are fitted in a statistical sense, but several evaluation thresholds (Hit@10, noise ratios, shot counts) are hand-chosen without sensitivity analysis.

free parameters (3)
  • Hit@k threshold k = 10
    Default retrieval threshold used for Playbook Recommendation, Patch Tool Suggestion, and Advisory Correlation; no sensitivity analysis across k values is reported.
  • ICL example counts = 5 and 10
    In-context learning baselines use 5 or 10 illustrative examples; the choice is hand-picked and no scaling analysis is given.
  • LLM sampling hyperparameters = temperature 0.7, top-p 0.95, max tokens 2048
    Used for all LLM queries (Appendix D); these settings are chosen by hand and can affect variance and output length.
assumptions (5)
  • ad hoc to paper Threat hunting can be modeled as a dependency chain of 30 discrete tasks organized into four stages and addressed by 9 modular operations.
    Defined in Table 2 and Section 3.1; no external validation that this taxonomy covers real-world threat hunting.
  • domain assumption The 23 public threat-intelligence sources provide complete and correct ground truth for all tasks.
    Section 3.2 and Appendix A build the benchmark on NVD, MITRE, vendor advisories, and threat platforms; erroneous entries would propagate into every metric.
  • ad hoc to paper Each named operation (NER, REX, SUM, SIM, MAP, RAG, SPA, CLS, MATH) is faithfully executed by the prompted LLM.
    Appendix B implements all operations as prompts or regex instructions; the LLM may not perform true retrieval (RAG) or reliable entity extraction.
  • domain assumption The evaluation metrics reflect threat-hunting quality.
    F1, accuracy, BERTScore, Hit@10, pass rate, and Dist are used without human validation or correlation with expert judgments.
  • domain assumption The input 'Cyber Threat Logs' resemble real incident-report data.
    Figure 1 and examples show idealized logs; no distribution analysis is provided to establish representativeness.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Benchmarking LLM-Assisted Blue Teaming via Standardized Threat Hunting." pith.science (2026). https://pith.science/paper/BMPDGQJA

@misc{pith2026250923571,
  author       = {Pith},
  title        = {Pith review of: Benchmarking LLM-Assisted Blue Teaming via Standardized Threat Hunting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BMPDGQJA}},
  note         = {Machine review of arXiv:2509.23571}
}
read the original abstract

As cyber threats continue to grow in scale and sophistication, blue team defenders increasingly require advanced tools to proactively detect and mitigate risks. Large Language Models (LLMs) offer promising capabilities for enhancing threat analysis. However, their effectiveness in real-world blue team threat-hunting scenarios remains insufficiently explored. This paper presents CyberTeam, a benchmark designed to guide LLMs in blue teaming practice. CyberTeam constructs a standardized workflow in two stages. First, it models realistic threat-hunting workflows by capturing the dependencies among analytical tasks from threat attribution to incident response. Next, each task is addressed through a set of operational modules tailored to its specific analytical requirements. This transforms threat hunting into a structured sequence of reasoning steps, with each step grounded in a discrete operation and ordered according to task-specific dependencies. Guided by this framework, LLMs are directed to perform threat-hunting tasks through modularized steps. Overall, CyberTeam integrates 30 tasks and 9 operational modules to guide LLMs through standardized threat analysis. We evaluate both leading LLMs and state-of-the-art cybersecurity agents, comparing CyberTeam against open-ended reasoning strategies. Our results highlight the improvements enabled by standardized design, while also revealing the limitations of open-ended reasoning in real-world threat hunting.

Figures

Figures reproduced from arXiv: 2509.23571 by the authors.

Figure 1
Figure 1. A CYBERTEAM threat hunting example equipped with operational modules. Module names: NER–named entity recognition, REX–regex parsing, MAP–text mapping, RAG–retrieval-augmented generation, CLS–classification, SUM–summarization. Broader Coverage. CYBERTEAM is constructed from a diverse and large-scale repository of threat intelligence data sourced from 23 vulnerability databases, including MITRE (MITRE Corporation, 202… view at source ↗
Figure 2
Figure 2. A threat hunting example demonstrating a dependency chain of analytical tasks, where [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Threat-hunting performance (scaled to 100%) on individual tasks, evaluating under GPT [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: LLM performance (metrics corresponding to Table 2) when input threat logs are perturbed [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Threat-hunting performance on individual tasks, evaluating under GPT-4o. [PITH_FULL_IMAGE:figures/full_fig_p026_5.png]
Figure 6
Figure 6. Figure 6: Threat-hunting performance on individual tasks, evaluating under SevenLLM-7B. [PITH_FULL_IMAGE:figures/full_fig_p027_6.png]
Figure 7
Figure 7. Figure 7: Threat-hunting performance on individual tasks, evaluating under Gemini-pro. [PITH_FULL_IMAGE:figures/full_fig_p027_7.png]
Figure 8
Figure 8. Figure 8: Threat-hunting performance on individual tasks, evaluating under Llama-405B. [PITH_FULL_IMAGE:figures/full_fig_p028_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

35 extracted references · 25 linked inside Pith

  1. [2]

    Exploring llms for mal- ware detection: Review, framework design, and countermeasure approaches.arXiv preprint arXiv:2409.07587,

    Jamal Al-Karaki, Muhammad Al-Zafar Khan, and Marwan Omar. Exploring llms for mal- ware detection: Review, framework design, and countermeasure approaches.arXiv preprint arXiv:2409.07587,

    arXiv

  2. [4]

    Recent activity by APT29 involving phishing attacks

    Structured Query for Retrieval System Prompt:You are a cybersecurity assistant. Formulate a concise search query to retrieve current information about the topic specified below. Instructions:Based on the topic“Recent activity by APT29 involving phishing attacks”, generate a query such as: “APT29 phishing campaign 2024 indicators, tools, and targets site:m...

    2024

  3. [7]

    Pentestgpt: An llm-empowered automatic penetration testing tool.arXiv preprint arXiv:2308.06782,

    Gelei Deng, Yi Liu, Víctor Mayoral-Vilches, Peng Liu, Yuekang Li, Yuan Xu, Tianwei Zhang, Yang Liu, Martin Pinzger, and Stefan Rass. Pentestgpt: An llm-empowered automatic penetration testing tool.arXiv preprint arXiv:2308.06782,

    arXiv

  4. [9]

    Given a vulnerability description and metric values (Confidentiality, Integrity, Availability, Scope, Attack Vector, etc.), compute the CVSS v3.1 Base Score

    CVSS Score Computation Prompt System Prompt:You are a cybersecurity scoring assistant. Given a vulnerability description and metric values (Confidentiality, Integrity, Availability, Scope, Attack Vector, etc.), compute the CVSS v3.1 Base Score. Instructions:Use the official CVSS equations and apply the rounding rules specified in the standard. Return both...

    2020

  5. [10]

    Lawbench: Benchmarking legal knowledge of large language models.arXiv preprint arXiv:2309.16289,

    Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Songyang Zhang, Kai Chen, Zongwen Shen, and Jidong Ge. Lawbench: Benchmarking legal knowledge of large language models.arXiv preprint arXiv:2309.16289,

    arXiv

  6. [11]

    Getting pwn’d by ai: Penetration testing with large language models

    Andreas Happe and Jürgen Cito. Getting pwn’d by ai: Penetration testing with large language models. InProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 2082–2086,

  7. [12]

    Linking threat tactics, techniques, and patterns with defensive weaknesses, vulnerabilities and affected platform configurations for cyber hunting.arXiv preprint arXiv:2010.00533,

    Erik Hemberg, Jonathan Kelly, Michal Shlapentokh-Rothman, Bryn Reinstadler, Katherine Xu, Nick Rutar, and Una-May O’Reilly. Linking threat tactics, techniques, and patterns with defensive weaknesses, vulnerabilities and affected platform configurations for cyber hunting.arXiv preprint arXiv:2010.00533,

    arXiv 2010

  8. [14]

    Assessing cybersecurity vulnerabilities in code large language models.arXiv preprint arXiv:2404.18567,

    Md Imran Hossen, Jianyi Zhang, Yinzhi Cao, and Xiali Hei. Assessing cybersecurity vulnerabilities in code large language models.arXiv preprint arXiv:2404.18567,

    arXiv

Show all 35 references
  1. [15]

    Agentgen: Enhancing planning abilities for large language model based agent via environment and task generation.arXiv preprint arXiv:2408.00764,

    Mengkang Hu, Pu Zhao, Can Xu, Qingfeng Sun, Jianguang Lou, Qingwei Lin, Ping Luo, and Saravan Rajmohan. Agentgen: Enhancing planning abilities for large language model based agent via environment and task generation.arXiv preprint arXiv:2408.00764,

    arXiv

  2. [16]

    Sevenllm: Benchmarking, eliciting, and enhancing abilities of large language models in cyber threat intelligence.arXiv preprint arXiv:2405.03446,

    Hangyuan Ji, Jian Yang, Linzheng Chai, Chaoren Wei, Liqun Yang, Yunlong Duan, Yunli Wang, Tianzhen Sun, Hongcheng Guo, Tongliang Li, et al. Sevenllm: Benchmarking, eliciting, and enhancing abilities of large language models in cyber threat intelligence.arXiv preprint arXiv:2405.03446,

    arXiv

  3. [17]

    Swe-bench: Can language models resolve real-world github issues?arXiv preprint arXiv:2310.06770,

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues?arXiv preprint arXiv:2310.06770,

    arXiv

  4. [18]

    From ml to llm: Evaluating the robustness of phishing webpage detection models against adversarial attacks.arXiv preprint arXiv:2407.20361,

    Aditya Kulkarni, Vivek Balachandran, Dinil Mon Divakaran, and Tamal Das. From ml to llm: Evaluating the robustness of phishing webpage detection models against adversarial attacks.arXiv preprint arXiv:2407.20361,

    arXiv

  5. [19]

    Mary M Lucas, Justin Yang, Jon K Pomeroy, and Christopher C Yang

    12 Preprint. Mary M Lucas, Justin Yang, Jon K Pomeroy, and Christopher C Yang. Reasoning with large language models for medical question answering.Journal of the American Medical Informatics Association, 31(9):1964–1975,

    1964

  6. [20]

    John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi

    URLhttps://attack.mitre.org/. John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp. InProceedings of the 2020 Conference on Empirical Methods in Natural ...

    2020

  7. [21]

    Self-contradictory hallucinations of large language models: Evaluation, detection and mitigation.arXiv preprint arXiv:2305.15852,

    Niels Mündler, Jingxuan He, Slobodan Jenko, and Martin Vechev. Self-contradictory hallucinations of large language models: Evaluation, detection and mitigation.arXiv preprint arXiv:2305.15852,

    arXiv

  8. [22]

    Hacksynth: Llm agent and evaluation framework for autonomous penetration testing.arXiv preprint arXiv:2412.01778,

    Lajos Muzsai, David Imolai, and András Lukács. Hacksynth: Llm agent and evaluation framework for autonomous penetration testing.arXiv preprint arXiv:2412.01778,

    arXiv

  9. [23]

    Lamd: Context-driven android malware detection and classification with llms.arXiv preprint arXiv:2502.13055,

    Xingzhi Qian, Xinran Zheng, Yiling He, Shuo Yang, and Lorenzo Cavallaro. Lamd: Context-driven android malware detection and classification with llms.arXiv preprint arXiv:2502.13055,

    arXiv

  10. [25]

    Pentestagent: Incorporating llm agents to automated penetration testing.arXiv preprint arXiv:2411.05185,

    Xiangmin Shen, Lingzhi Wang, Zhenyuan Li, Yan Chen, Wencheng Zhao, Dawei Sun, Jiashui Wang, and Wei Ruan. Pentestagent: Incorporating llm agents to automated penetration testing.arXiv preprint arXiv:2411.05185,

    arXiv

  11. [26]

    Lprotector: An llm-driven vulnerability detection system.arXiv preprint arXiv:2411.06493,

    Ze Sheng, Fenghua Wu, Xiangwu Zuo, Chao Li, Yuxin Qiao, and Lei Hang. Lprotector: An llm-driven vulnerability detection system.arXiv preprint arXiv:2411.06493,

    arXiv

  12. [27]

    Response inconsistency of large language models in high-stakes military decision making

    Aryan Shrivastava. Response inconsistency of large language models in high-stakes military decision making. Adi Simhi, Itay Itzhak, Fazl Barez, Gabriel Stanovsky, and Yonatan Belinkov. Trust me, i’m wrong: High-certainty hallucinations in llms.arXiv preprint arXiv:2502.12964,

    arXiv

  13. [28]

    Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun

    URLhttps://vuldb.com/. Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. Towards understanding chain-of-thought prompting: An empirical study of what matters.arXiv preprint arXiv:2212.10001,

    arXiv

  14. [29]

    Towards better chain-of-thought prompting strategies: A survey.arXiv preprint arXiv:2310.04959,

    Zihan Yu, Liang He, Zhen Wu, Xinyu Dai, and Jiajun Chen. Towards better chain-of-thought prompting strategies: A survey.arXiv preprint arXiv:2310.04959,

    arXiv

  15. [30]

    Official implementation: https: //github.com/Tiiiger/bert_score

    URL https://arxiv.org/abs/1904.09675. Official implementation: https: //github.com/Tiiiger/bert_score. Yuxuan Zhou, Xien Liu, Chen Ning, Xiao Zhang, Chenwei Yan, Xiangling Fu, and Ji Wu. Revisiting the scaling effects of llms on medical reasoning capabilities. A DATASOURCE AND...

    1904 arXiv

  16. [31]

    Each CVE entry is enriched with exploitability status, malware connections, and actor attribution

    is a commercial threat intelligence sharing platform that provides in-depth reports on vulnerabilities, exploits, malware, and threat actors. Each CVE entry is enriched with exploitability status, malware connections, and actor attribution. We extract structured threat metadat...

    2024

  17. [32]

    Entries often include exploitability scores, attack vectors, exploitation status, and tags related to malware or campaigns

    is a commercial vulnerability intelligence service that provides insights into current exploits, threat actor behavior, and exploit trends. Entries often include exploitability scores, attack vectors, exploitation status, and tags related to malware or campaigns. We collect CV...

    2024

  18. [35]

    \n", "Q:

    D EXPERIMENTALSETTING This section details the experimental setup used to evaluate LLMs in the CyberTeam benchmark. Hyperparameters. Table 4 summarizes the key hyperparameters for querying LLMs during experi- ments. These settings were chosen to balance generation quality and ...

  19. [2013]

    Embodiedeval: Evaluate multimodal llms as embodied agents.arXiv preprint arXiv:2501.11858,

    Zhili Cheng, Yuge Tu, Ran Li, Shiqi Dai, Jinyi Hu, Shengding Hu, Jiahao Li, Yang Shi, Tianyu Yu, Weize Chen, et al. Embodiedeval: Evaluate multimodal llms as embodied agents.arXiv preprint arXiv:2501.11858,

    arXiv

  20. [2018]

    A survey on in-context learning.arXiv preprint arXiv:2301.00234,

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, et al. A survey on in-context learning.arXiv preprint arXiv:2301.00234,

    arXiv

  21. [2019]

    Improving discovery of known software vulnerability for enhanced cybersecurity.arXiv preprint arXiv:2412.16607,

    Devesh Sawant, Manjesh K Hanawal, and Atul Kabra. Improving discovery of known software vulnerability for enhanced cybersecurity.arXiv preprint arXiv:2412.16607,

    arXiv

  22. [2020]

    Turning the hunted into the hunter via threat hunting: Life cycle, ecosystem, challenges and the great promise of ai.arXiv preprint arXiv:2204.11076,

    Caroline Hillier and Talieh Karroubi. Turning the hunted into the hunter via threat hunting: Life cycle, ecosystem, challenges and the great promise of ai.arXiv preprint arXiv:2204.11076,

  23. [2021]

    Threatzoom: Cve2cwe using hierarchical neural network.arXiv preprint arXiv:2009.11501,

    Ehsan Aghaei, Waseem Shadid, and Ehab Al-Shaer. Threatzoom: Cve2cwe using hierarchical neural network.arXiv preprint arXiv:2009.11501,

    2009 arXiv

  24. [2022]

    Respact: Harmonizing reasoning, speaking, and acting towards building large language model-based conversational ai agents.arXiv preprint arXiv:2411.00927,

    Vardhan Dongre, Xiaocheng Yang, Emre Can Acikgoz, Suvodip Dey, Gokhan Tur, and Dilek Hakkani-Tür. Respact: Harmonizing reasoning, speaking, and acting towards building large language model-based conversational ai agents.arXiv preprint arXiv:2411.00927,

    arXiv

  25. [2023]

    Frameworks for querying databases using natural language: a literature review.arXiv preprint arXiv:1909.01822,

    Hafsa Shareef Dar, M Ikramullah Lali, Moin Ul Din, Khalid Mahmood Malik, and Syed Ahmad Chan Bukhari. Frameworks for querying databases using natural language: a literature review.arXiv preprint arXiv:1909.01822,

    1909 arXiv

  26. [2024]

    Ctibench: A benchmark for evaluating llms in cyber threat intelligence.arXiv preprint arXiv:2406.07599,

    Md Tanvirul Alam, Dipkamal Bhusal, Le Nguyen, and Nidhi Rastogi. Ctibench: A benchmark for evaluating llms in cyber threat intelligence.arXiv preprint arXiv:2406.07599,

    arXiv

  27. [2025]

    Leshem Choshen, Ariel Gera, Yotam Perlitz, Michal Shmueli-Scheuer, and Gabriel Stanovsky

    10 Preprint. Leshem Choshen, Ariel Gera, Yotam Perlitz, Michal Shmueli-Scheuer, and Gabriel Stanovsky. Navigating the modern evaluation landscape: Considerations in benchmarks and frameworks for large language models (llms). InProceedings of the 2024 Joint International Confer...

    2024

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.