Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Uncovering Vulnerabilities of LLM-Assisted Cyber Threat Intelligence

T0 review · 4 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLMs fail at cyber threat intelligence mainly because the threat landscape itself is messy—contradictory, volatile, and crowdsourced—not because of generic model flaws, and three specific failure modes capture most of the damage.

desk verdict A useful CTI failure taxonomy, but the abstract's causal-claims experiments are missing and the labeling reliability is unmeasured. read the letter →

arxiv 2509.23573 v5 pith:IB6QJ432 submitted 2025-09-28 cs.CR cs.AI

Yuqiao Meng , Luoxi Tang , Feiyang Yu , Jinyuan Jia , Guanhua Yan , Ping Yang , Zhaohan Xi This is my paper
classification cs.CRcs.AI
keywords largelanguagemodelscyberthreatintelligencespuriouscorrelationcontradictoryknowledgeconstrainedgeneralizationfailuretaxonomyhuman-in-the-looplabelinglandscape
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that when large language models fail at cyber threat intelligence (CTI), the dominant cause is not hallucination or weak model architecture but the nature of the threat evidence itself: CTI is heterogeneous, volatile, and fragmented, and its evidence is intertwined, crowdsourced, and temporally unstable. Drawing on a large evaluation spanning CTI benchmarks and real-world threat reports, the authors identify three fundamental failure modes—spurious correlations from superficial metadata, contradictory knowledge from conflicting sources, and constrained generalization to emerging threats—and break them into 15 subtypes across the CTI lifecycle of contextualization, attribution, prediction, and mitigation. The taxonomy is built with a human-in-the-loop framework that avoids relying on 'LLM-as-judge' labels. The authors' conclusion, if right, redirects the field: reliability cannot be engineered by prompts or scale alone; it requires data curation, adversarially aware fine-tuning, and inference-time safeguards aimed at the evidence conditions.

What carries the argument

The load-bearing object is a human-in-the-loop failure-categorization framework with three stages. First, stratification: because most CTI tasks lack hard correct/incorrect labels, outputs are scored against authoritative references and partitioned into quantile strata; human inspection of anchors (no more than 3% of instances) defines failed vs. correct score ranges, so the remaining instances can be labeled without per-item annotation. Second, autoregressive taxonomy refinement: humans seed an initial failure-mode taxonomy, an LLM classifies the rest, humans inspect every 'other' case, and new modes are added until the taxonomy converges. Third, multi-agent human-in-the-loop labeling: four

What would settle it

Two checks would settle the claim. First: a clean-evidence control—build a synthetic CTI corpus with no source conflicts, no temporal drift, no reused IOCs, and single-vendor provenance; if LLM failure rates remain comparable, the threat-landscape bottleneck claim is wrong. Second: an independent re-labeling audit—have fresh annotators, without access to the anchor taxonomy, classify a random sample of the machine-labeled instances; if a fourth mode (e.g., pure format or instruction-following errors) dominates or the three-mode distribution shifts materially, the converged taxonomy and Figure

Watch

Extended reading notes

Core claim

The paper's central claim is that three domain-specific cognitive failures—spurious correlations, contradictory knowledge, and constrained generalization—explain a large share of LLM failures across the CTI pipeline, and that these failures are rooted in the threat landscape rather than in generic model weaknesses. Spurious correlation means the model treats co-mention or shared metadata as causal evidence, e.g., linking a benign CVE to an active campaign because both appear in one bulletin. Contradictory knowledge means inconsistent or outdated sources destabilize reasoning, e.g., one advisory declaring a patch sufficient while another documents post-patch persistence. Constrained generaliz

Load-bearing premise

The framework assumes that manually inspecting at most 3% of instances as anchors, plus resolving fewer than 1.8% of uncertain cases, produces a complete and unbiased taxonomy that correctly classifies the remaining roughly 95%—if that anchor set is unrepresentative, the dominance of the three failure modes is not established.

Editorial extensions

If this is right

  • If the three-mode taxonomy is correct, CTI failure analysis can be decomposed by dominant vulnerability type, letting practitioners target the specific mechanism (e.g., filtering co-mentioned entities vs. reconciling source conflicts) rather than applying generic model fixes.
  • The abstract's claim that targeted defenses reduce failure rates significantly implies these vulnerabilities are addressable in practice, not immovable limits of LLMs.
  • Because the failures persist across both general-purpose and cybersecurity-specialized models, domain fine-tuning alone cannot close the gap; data curation and evidence conditioning become central to LLM-CTI systems.
  • The categorization framework itself is a reusable methodology for studying LLM failures in domains where hard labels are scarce, as an alternative to LLM-as-judge pipelines.
  • The propagation of upstream errors through the CTI pipeline (contextualization to attribution to prediction to mitigation) means defenses at early stages may pay off downstream.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An extension the paper leaves implicit: corpus-level diagnostics—such as contradiction density across sources, IOC reuse rate, or co-mention frequency—could predict an agent's expected failure mix before deployment, turning the taxonomy into a dataset assessment tool.
  • The same three-mode structure likely appears in other reasoning domains built on crowdsourced, volatile evidence (e.g., open-source medical or financial intelligence), though the paper only claims CTI scope; a transfer test would be a natural follow-up.
  • The paper's own observation that format errors are a universal gap suggests a possible fourth mode that the convergence criterion either absorbed into the three categories or omitted; checking where such errors were labeled would test the taxonomy's completeness.
  • If the threat-landscape framing is right, benchmark construction for LLM security should deliberately inject evidence noise (contradictions, temporal drift, skewed sources) rather than presenting clean ground truth, so that evaluation scores reflect operational readiness.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper evaluates 15 general-purpose and cyber-specialized LLMs on cyber threat intelligence (CTI) tasks spanning contextualization, attribution, prediction, and mitigation. It proposes a human-in-the-loop categorization pipeline (§3) that combines stratification, an autoregressive taxonomy-refinement loop, and multi-agent LLM labeling with human arbitration. Using this pipeline, the authors report three failure-mode categories—spurious correlations, contradictory knowledge, and constrained generalization—with subtype-level percentages in Figure 2 and qualitative root-cause case studies in §4 and Appendix D. The abstract further claims that these mechanisms are validated by causal interventions and that targeted defenses significantly reduce failure rates. The central thesis is that the threat landscape itself, rather than model architecture, is the dominant bottleneck for LLM-assisted CTI.

Significance. If the taxonomy and the claimed dominance ratios are reliable, the paper offers a valuable reframing of LLM failure in CTI: the problem is not only model reasoning but also the heterogeneous, volatile, and fragmented nature of threat evidence. The study is broad in scope, drawing on multiple benchmarks and real-world CTI databases, and the qualitative case studies in Appendix D are often concrete and informative. The authors also state that they release code, which aids reproducibility. However, the significance is conditional on two load-bearing points that are currently not established: the reliability and completeness of the failure taxonomy (Section 3 and Figure 2), and the causal-intervention/defense experiments promised in the abstract but absent from the manuscript.

major comments (4)
  1. [Abstract / §4] The abstract states: 'We validate these mechanisms via causal interventions and show that targeted defenses reduce failure rates significantly.' No such experiments appear anywhere in the manuscript. Sections 2–6 and Appendices A–E contain no intervention design, no control condition, no defense evaluation, and no effect-size measurement; §4's 'Design Insight' only recommends future directions. Because this validation claim is part of the paper's stated contribution, the central claim is currently unsupported. The authors must either add the missing experiments or revise the abstract to describe what was actually performed.
  2. [§3.2–3.3 / Figure 2] The taxonomy underpinning the dominance claims is labeled by four-LLM majority vote for approximately 95% of instances. §3.3 reports that fewer than 1.8% of cases are flagged as uncertain and human-verified, but the manuscript provides no inter-annotator agreement among the four models, no LLM-human agreement on a held-out sample, no confidence intervals for the proportions in Figure 2, and no sensitivity analysis for the initial taxonomy D0 or the coverage threshold ρ in Algorithm 2. If the voting models share a label bias or systematically under-report 'other', the loop can terminate with an incomplete taxonomy, and the claimed dominance of spurious correlations, contradictory knowledge, and constrained generalization is not established. Quantitative reliability evidence and a robustness check are needed.
  3. [§3.1 / Algorithm 1] The stratification step classifies instances as failed or correct by comparing similarity scores to anchor-derived ranges, but the manuscript does not report the score distributions for 'correct' versus 'failed' anchors, the size of the overlapping region, or the accuracy of the derived thresholds. The statement that 'practically, this process is efficient as it requires manually inspecting no more than 3% of instances' is an assertion rather than a measured outcome. Without stratification diagnostics, the failure set D itself may be contaminated, and this contamination propagates to every downstream ratio and percentage. Please report these diagnostics and the actual manual-inspection counts.
  4. [§3.3, Step 2] In the second round of multi-agent labeling, each LLM observes the first-round predictions of the other models. This can induce conformity and artificially reduce the size of the uncertainty set U, so the reported '<1.8%' human-verification rate may reflect deliberation dynamics rather than genuine labeling stability. The manuscript should report first-round agreement separately and validate the final labels on a random sample of agreed cases, not only on cases that the models themselves flagged as uncertain.
minor comments (5)
  1. [Figure 2] The percentages are given without denominators or confidence intervals, and because vulnerabilities may overlap, the legend is hard to parse. Please clarify how the 'General' and 'Agent' ratios are computed and what the base population is.
  2. [Table 2] The column headers run together (e.g., 'GRKFSC', 'LLY CBS SPT DHT'). Adding separators would greatly improve readability and avoid misreading model abbreviations.
  3. [Appendix E] The disclosure says LLMs were 'not employed in experimental execution,' but §3.3 uses four LLMs to label failure instances. Please clarify whether the disclosure refers only to manuscript preparation or also covers the LLM-assisted categorization pipeline.
  4. [Appendix A.3 / Table 3] There are scattered typos and spacing artifacts, e.g., 'Temporal forcasting' and 'Y ARA'. A copyedit pass would help.
  5. [§3.1 / Algorithm 1] The termination condition 'the distribution of failure modes ... converges to a stable ratio' is not formally defined. Please provide an operational criterion, e.g., a bound on the change in proportions between iterations.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the failure-mode taxonomy is inductively derived from data, and the only author-overlapping citation is a benchmark used as evaluation input, not as a load-bearing premise.

full rationale

The paper's central claim—that the threat landscape itself drives LLM failures in CTI through spurious correlation, contradictory knowledge, and constrained generalization—is not forced by construction. Failure instances are first identified through BLEU-based stratification with human anchors (§3.1), then the taxonomy is produced by an autoregressive human-in-the-loop process (§3.2–3.3) in which humans seed and refine categories and arbitrate only disagreement/fluctuation cases. The final three vulnerability families are an inductive summary of those labeled failures, not parameters fitted to the conclusion. The only notable author-overlapping citation is the CyberTeam benchmark (Liu et al., 2025), used as one of several evaluation datasets; this is an input to the study, not a result that presupposes the taxonomy or the three-vulnerability conclusion. The LLM majority-vote labeling of the majority of instances and the small human-inspected fraction raise reliability questions, but those are validity concerns, not circularity under this scorecard.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

No fitted parameters enter the central claim; the listed free parameters are procedure thresholds. The analysis depends on the validity of reference-similarity as a failure signal and on human annotation reliability. No new entities such as forces, particles, or dimensions are introduced. The abstract's implied causal-intervention experiments are missing from the text, which is a correctness gap, not an axiom.

free parameters (2)
  • Stratum quantile bin size delta = 0.05
    Chosen by hand in Algorithm 1 to partition instances by failure-depth similarity scores; affects which samples are picked as anchors and how failure boundaries are drawn.
  • Coverage threshold rho = 0.6 (example default)
    Hand-chosen in Algorithm 2 as the stopping criterion for taxonomy coverage in autoregressive failure-mode determination.
assumptions (3)
  • domain assumption Reference-based similarity metrics (e.g., BLEU) are a valid proxy for failure severity in open-ended CTI tasks.
    Section 3.1 uses BLEU-style scores to stratify instances into failed and correct groups, assuming that low similarity to authoritative references indicates failure.
  • domain assumption Human annotators provide trustworthy ground truth for failure-mode labels.
    Section 3.2 and 3.3 rely on human inspection for seeding the taxonomy and resolving uncertain labels; no inter-annotator agreement is reported.
  • domain assumption The four CTI stages and the benchmark plus real-world database mix represent the operational CTI lifecycle.
    Section 2.1 and Table 1 assume that contextualization, attribution, prediction, and mitigation, and the chosen datasets, cover the space of CTI tasks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Uncovering Vulnerabilities of LLM-Assisted Cyber Threat Intelligence." pith.science (2026). https://pith.science/paper/IB6QJ432

@misc{pith2026250923573,
  author       = {Pith},
  title        = {Pith review of: Uncovering Vulnerabilities of LLM-Assisted Cyber Threat Intelligence},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IB6QJ432}},
  note         = {Machine review of arXiv:2509.23573}
}
read the original abstract

Large language models (LLMs) are increasingly used to help security analysts manage the surge of cyber threats, automating tasks from vulnerability assessment to incident response. Yet in operational CTI workflows, reliability gaps remain substantial. Existing explanations often point to generic model issues (e.g., hallucination), but we argue the dominant bottleneck is the threat landscape itself: CTI is heterogeneous, volatile, and fragmented. Under these conditions, evidence is intertwined, crowdsourced, and temporally unstable, which are properties that standard LLM-based studies rarely capture. In this paper, we present a comprehensive empirical study of LLM vulnerabilities in CTI reasoning. We introduce a human-in-the-loop categorization framework that robustly labels failure modes across the CTI lifecycle, avoiding the brittleness of automated "LLM-as-a-judge" pipelines. We identify three domain-specific cognitive failures: spurious correlations from superficial metadata, contradictory knowledge from conflicting sources, and constrained generalization to emerging threats. We validate these mechanisms via causal interventions and show that targeted defenses reduce failure rates significantly. Together, these results offer a concrete roadmap for building resilient, domain-aware CTI agents.

Figures

Figures reproduced from arXiv: 2509.23573 by the authors.

Figure 1
Figure 1. (Left) Failure ratios of cybersecurity agents. (Right) Examples of vulnerabilities. Large Language Models (LLMs) have recently demonstrated strong performance in a broad range of cyber threat intelligence (CTI) tasks. By adapting models through instruction fine-tuning or prompt￾1 arXiv:2509.23573v2 [cs.CR] 1 Oct 2025 [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Summarization of the vulnerability types of LLMs in various CTI stages. Ratios are [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Overview of method to categorize failure instances (addressing RQ [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Varying proportions of vulnerabilities (types listed in Figure 2). Note that different [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ARENA: An Architecture for Measuring the Transferability of Autonomous Cyber Defense

    cs.CR 2026-06 unverdicted novelty 5.0 of 10

    ARENA creates anonymized SOC telemetry artifacts that reveal a measurable privacy-utility boundary when used both as training material for MITRE-mapped challenges and as a substrate to detect non-compliant LLM defende...

Reference graph

Works this paper leans on

33 extracted references · 16 linked inside Pith · cited by 1 Pith paper

  1. [1]

    phish→beacon→lateral→exfil

    and real-world ground-truth data. B.3 TASKDESCRIPTION ❶Contextualization. Affected Systems (F1).Binary decision per asset: is a listed host/application impacted by the described CVE/IOC set (yes/no).Example:decide whether Exchange 2019 CU12 is affected given a CVE vector and server build. Attack Infrastructure (F1).Binary decision per indicator: determine...

    2019

  2. [2]

    malware.exe

    4foreachx i ∈ Ddo 5Each modela∈Θobserves{ˆy (1,b) i }b∈Θ; 6Refine reasoning and outputˆy (2,a) i ; 7end // Step 3: Human verification of uncertain cases 8Define uncertain set: 9U={x i | ∃a:Var({ˆy(a) i }a∈Θ)>0orˆy (1,a) i ̸= ˆy(2,a) i }; 10Human annotatorsHinspect allx i ∈ Uand assign final labels; // Finalize results 11InstancesD \ Utake majority-agreed ...

    2021

  3. [5]

    The use of large language models (llm) for cyber threat intelligence (cti) in cybercrime forums.arXiv preprint arXiv:2408.03354,

    Vanessa Clairoux-Trepanier, Isa-May Beauchamp, Estelle Ruellan, Masarah Paquet-Clouston, Serge- Olivier Paquette, and Eric Clay. The use of large language models (llm) for cyber threat intelligence (cti) in cybercrime forums.arXiv preprint arXiv:2408.03354,

    arXiv

  4. [8]

    Enhancing cybersecurity in critical infrastructure with llm-assisted explainable iot systems.arXiv preprint arXiv:2503.03180,

    Ashutosh Ghimire, Ghazal Ghajari, Karma Gurung, Love K Sah, and Fathi Amsaad. Enhancing cybersecurity in critical infrastructure with llm-assisted explainable iot systems.arXiv preprint arXiv:2503.03180,

    arXiv

  5. [9]

    Validating llm-as-a-judge systems in the absence of gold labels.arXiv preprint arXiv:2503.05965,

    Luke Guerdan, Solon Barocas, Kenneth Holstein, Hanna Wallach, Zhiwei Steven Wu, and Alexandra Chouldechova. Validating llm-as-a-judge systems in the absence of gold labels.arXiv preprint arXiv:2503.05965,

    arXiv

  6. [10]

    Data interpreter: An llm agent for data science.arXiv preprint arXiv:2402.18679,

    Sirui Hong, Yizhang Lin, Bang Liu, Bangbang Liu, Binhao Wu, Ceyao Zhang, Chenxing Wei, Danyang Li, Jiaqi Chen, Jiayi Zhang, et al. Data interpreter: An llm agent for data science.arXiv preprint arXiv:2402.18679,

    arXiv

  7. [11]

    Jay Jacobs and Sasha Romanosky

    11 Preprint. Jay Jacobs and Sasha Romanosky. Predicting exploitation of disclosed software vulnerabilities using open-source data.arXiv preprint arXiv:1908.04832,

    arXiv 1908

  8. [12]

    Sevenllm: Benchmarking, eliciting, and enhancing abilities of large language models in cyber threat intelligence.arXiv preprint arXiv:2405.03446,

    Hangyuan Ji, Jian Yang, Linzheng Chai, Chaoren Wei, Liqun Yang, Yunlong Duan, Yunli Wang, Tianzhen Sun, Hongcheng Guo, Tongliang Li, et al. Sevenllm: Benchmarking, eliciting, and enhancing abilities of large language models in cyber threat intelligence.arXiv preprint arXiv:2405.03446,

    arXiv

Show all 33 references
  1. [13]

    Swe-bench: Can language models resolve real-world github issues?arXiv preprint arXiv:2310.06770,

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues?arXiv preprint arXiv:2310.06770,

    arXiv

  2. [14]

    Training language models to self-correct via reinforcement learning.arXiv preprint arXiv:2409.12917,

    Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, et al. Training language models to self-correct via reinforcement learning.arXiv preprint arXiv:2409.12917,

    arXiv

  3. [16]

    com/security/blog/2021/03/02/hafnium-targeting-exchange-servers/

    URL https://www.microsoft. com/security/blog/2021/03/02/hafnium-targeting-exchange-servers/ . Includes guidance, IOCs, and initial mitigation recommendation. MISP Project. Malware information sharing platform (misp). https://www.misp-project. org/,

    2021

  4. [17]

    Qualys reports 30% surge in cves for 2024 amid rising software complexity,

    Qualys. Qualys reports 30% surge in cves for 2024 amid rising software complexity,

    2024

  5. [18]

    Microsoft’s june 2025 patch tuesday addresses 65 cves (cve-2025-33053)

    Tenable Research. Microsoft’s june 2025 patch tuesday addresses 65 cves (cve-2025-33053). https://www.tenable.com/blog/ microsofts-june-2025-patch-tuesday-addresses-65-cves-cve-2025-33053 , June

    2025

  6. [19]

    OX Security

    URLhttps://arxiv.org/abs/2501.04227. OX Security. That was then, this is now: Modernizing appsec in fast-paced development environments,

    arXiv

  7. [22]

    A comprehensive survey in llm (-agent) full stack safety: Data, training and deployment.arXiv preprint arXiv:2504.15585,

    Kun Wang, Guibin Zhang, Zhenhong Zhou, Jiahao Wu, Miao Yu, Shiqian Zhao, Chenlong Yin, Jinhu Fu, Yibo Yan, Hanjun Luo, et al. A comprehensive survey in llm (-agent) full stack safety: Data, training and deployment.arXiv preprint arXiv:2504.15585,

    arXiv

  8. [25]

    Breaking agents: Compromising autonomous llm agents through malfunction amplification

    Boyang Zhang, Yicong Tan, Yun Shen, Ahmed Salem, Michael Backes, Savvas Zannettou, and Yang Zhang. Breaking agents: Compromising autonomous llm agents through malfunction amplification. arXiv preprint arXiv:2407.20859,

    arXiv

  9. [26]

    Cyber-zero: Training cybersecurity agents without runtime.arXiv preprint arXiv:2508.00910,

    Terry Yue Zhuo, Dingmin Wang, Hantian Ding, Varun Kumar, and Zijian Wang. Cyber-zero: Training cybersecurity agents without runtime.arXiv preprint arXiv:2508.00910,

    arXiv

  10. [27]

    14 Preprint

    A COMPLEMENTARYDETAILS OFCTI This appendix section provides additional details on how different CTI stages are conducted with the involvement of various techniques, complementing Section §2.1. 14 Preprint. A.1❶CONTEXTUALIZATION The contextualization stage transforms raw, fragm...

    2003

  11. [28]

    phishing campaigns leveraging Office macros

    are applied to large corpora of unstructured CTI text, including threat reports, incident tickets, and log annotations. These models group documents or paragraphs into coherent themes, enabling analysts to identify clusters of related activity such as “phishing campaigns lever...

    2018

  12. [29]

    Retrieval pipelines often combine lightweight keyword filters for precision with dense embedding search for semantic coverage

    and the CISA KEV catalog (Cybersecurity and Infrastructure Security Agency (CISA), 2024). Retrieval pipelines often combine lightweight keyword filters for precision with dense embedding search for semantic coverage. Analysts use these systems to verify whether a newly observe...

    2024

  13. [30]

    provides community-curated threat pulses (collections of IOCs associated with specific campaigns or malware). We use OTX primarily for ❷ attribution and ❸ prediction: pulses give us labeled groupings of IOCs tied to campaigns, which we then cross-check against LLM-predicted ca...

    2000

  14. [33]

    given X IOC or Y vulnerability, recommend Z patch or rule

    Co-mention bias (1.1) in ❷ attribution.In attribution, co-mention bias presents when reports describe overlapping infrastructure or techniques across multiple actors. A single campaign report may reference domains, malware families, or TTPs associated with different groups, no...

    2025

  15. [1999]

    Benchmarking llms in an embodied environment for blue team threat hunting.arXiv preprint arXiv:2505.11901,

    Xiaoqun Liu, Feiyang Yu, Xi Li, Guanhua Yan, Ping Yang, and Zhaohan Xi. Benchmarking llms in an embodied environment for blue team threat hunting.arXiv preprint arXiv:2505.11901,

    arXiv

  16. [2003]

    Llm agents for education: Advances and applications.arXiv preprint arXiv:2503.11733,

    Zhendong Chu, Shen Wang, Jian Xie, Tinghui Zhu, Yibo Yan, Jinheng Ye, Aoxiao Zhong, Xuming Hu, Jing Liang, Philip S Yu, et al. Llm agents for education: Advances and applications.arXiv preprint arXiv:2503.11733,

  17. [2012]

    An empirical study of llm-as-a-judge: How design choices impact evaluation reliability.arXiv preprint arXiv:2506.13639,

    Yusuke Yamauchi, Taro Yano, and Masafumi Oyamada. An empirical study of llm-as-a-judge: How design choices impact evaluation reliability.arXiv preprint arXiv:2506.13639,

    arXiv

  18. [2014]

    Targeted cyberattacks: a superset of advanced persistent threats

    Aditya K Sood and Richard J Enbody. Targeted cyberattacks: a superset of advanced persistent threats. InProceedings of the 2013 Conference on Security and Privacy in Communication Systems, pp. 427–439,

    2013

  19. [2019]

    Llama-3.1-foundationai- securityllm-8b-instruct technical report.arXiv preprint arXiv:2508.01059,

    Sajana Weerawardhena, Paul Kassianik, Blaine Nelson, Baturay Saglam, Anu Vellore, Aman Priyan- shu, Supriti Vijay, Massimo Aufiero, Arthur Goldblatt, Fraser Burch, et al. Llama-3.1-foundationai- securityllm-8b-instruct technical report.arXiv preprint arXiv:2508.01059,

    arXiv

  20. [2020]

    Llm security: Vulnerabilities, attacks, defenses, and countermeasures.arXiv preprint arXiv:2505.01177,

    Francisco Aguilera-Mart´ınez and Fernando Berzal. Llm security: Vulnerabilities, attacks, defenses, and countermeasures.arXiv preprint arXiv:2505.01177,

    arXiv

  21. [2021]

    Before we knew it: an empirical study of zero-day attacks in the real world

    Leyla Bilge and Tudor Dumitras. Before we knew it: an empirical study of zero-day attacks in the real world. InProceedings of the 2012 ACM Conference on Computer and Communications Security (CCS), pp. 833–844,

    2012

  22. [2022]

    Analysis of CVE-2020-1472 exploitation over time

    URLhttps://blog.eclecticiq.com/ long-term-analysis-illustrates-how-risk-posed-by-a-vulnerability-changes-as-exploits-develop-over-time . Analysis of CVE-2020-1472 exploitation over time. 13 Preprint. Rafael Uetz, Marco Herzog, Louis Hackl”ander, Simon Schwarz, and Martin Henze...

    2020

  23. [2023]

    ISBN 978-1-939133-37-3

    USENIX Association. ISBN 978-1-939133-37-3. URL https://www.usenix.org/conference/ usenixsecurity23/presentation/dong-feng. Xiaohu Du, Ming Wen, Jiahao Zhu, Zifan Xie, Bin Ji, Huijun Liu, Xuanhua Shi, and Hai Jin. Generalization-enhanced code vulnerability detection via multi-...

    arXiv

  24. [2024]

    Preemptive detection and correction of misaligned actions in llm agents.arXiv preprint arXiv:2407.11843, 2024a

    Haishuo Fang, Xiaodan Zhu, and Iryna Gurevych. Preemptive detection and correction of misaligned actions in llm agents.arXiv preprint arXiv:2407.11843, 2024a. Richard Fang, Rohan Bindu, Akul Gupta, and Daniel Kang. Llm agents can autonomously exploit one-day vulnerabilities.ar...

  25. [2025]

    Ctibench: A benchmark for evaluating llms in cyber threat intelligence.arXiv preprint arXiv:2406.07599,

    Md Tanvirul Alam, Dipkamal Bhusal, Le Nguyen, and Nidhi Rastogi. Ctibench: A benchmark for evaluating llms in cyber threat intelligence.arXiv preprint arXiv:2406.07599,

    arXiv

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.