Pith. sign in

REVIEW 4 major objections 5 minor 32 references

Rule-ATT&CK Mapper (RAM): Mapping SIEM Rules to TTPs Using LLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Pith's one-line read A multi-stage LLM pipeline maps SIEM rules to MITRE ATT&CK techniques at 0.75 recall without fine-tuning.

desk verdict Useful pipeline, but the headline precision is inflated by a test-set-tuned threshold; the paper deserves a serious referee if that gets fixed. read the letter →

arxiv 2502.02337 v1 pith:GSZSJMAH submitted 2025-02-04 cs.CR

classification cs.CR
keywords SIEMrulesMITREATT&CKLLMagentspromptchainingthreatdetectionTTPmappingGPT-4-TurboSplunkSecurityContent
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a multi-stage LLM pipeline, called Rule-ATT&CK Mapper (RAM), can automatically map structured SIEM rules to MITRE ATT&CK techniques without any fine-tuning or training data. On 360 recent Splunk endpoint rules, RAM with GPT-4-Turbo achieves an average recall of 0.75 and an average precision of 0.52, beating zero-shot GPT-4-Turbo, BERT/CodeBERT classifiers, and TTPxHunter. The central finding is that enriching a natural-language translation of each rule with web-retrieved context about its indicators of compromise is what lifts recall from 0.54 to 0.75. If correct, this makes LLM-based TTP mapping a practical alternative to manual annotation and supervised classifiers that require retraining.

What carries the argument

The central mechanism is a six-step prompt-chaining pipeline: zero-shot IoC extraction; a REACT web-search agent that pulls contextual information about each IoC; natural-language translation of the rule; agentic RAG to identify the relevant MITRE data source or mitigation; recommendation of probable techniques via another REACT agent; and final extraction of relevant techniques using chain-of-thought explanations and a confidence threshold. The pipeline's effectiveness rests on converting the structured rule into rich natural language before similarity comparison with technique descriptions.

What would settle it

Have several independent security analysts annotate the same 360 Splunk rules to a consensus ATT&CK gold standard, then recompute RAM's average recall and precision against that standard; if the scores move substantially from 0.75 and 0.52, the reported accuracy is an artifact of imperfect labels.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that the combination of prompt chaining and agent-based web retrieval lets an LLM map structured detection rules to the full MITRE ATT&CK technique taxonomy with usable accuracy. The reported numbers, AR 0.75 and AP 0.52 with GPT-4-Turbo, come from the complete RAM pipeline, while the same pipeline without contextual enrichment reaches only AR 0.54, and the raw rule as-is reaches 0.46. This shows the load-bearing role of external context over the LLM's implicit knowledge. The paper further claims that the pipeline is format-agnostic, works across hosted and local models of varying size, and produces chain-of-thought rationales that analysts can inspect.

Load-bearing premise

The evaluation assumes the Splunk Security Content dataset's ATT&CK labels are ground truth, but the paper itself shows a rule whose labels are incomplete; if such label noise is systematic, the reported recall and precision are not a true measure of mapping quality.

Editorial extensions

If this is right

  • Security teams could automate ATT&CK annotation of new SIEM rules without waiting for labeled training data.
  • The same pipeline should transfer to other rule definition languages such as KQL and Lucene, since the translation step normalizes them to text.
  • Because the method provides reasoning, analysts can audit and override mappings, unlike black-box classifiers.
  • New ATT&CK techniques can be accommodated without retraining, only by updating the technique descriptions in the RAG store and prompts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported gains may understate or overstate true accuracy because the Splunk labels are themselves incomplete; a multi-annotator gold standard would be needed to know the real headroom.
  • The web-search context stage likely depends on the freshness and quality of search results; a testable extension is to swap in a curated vulnerability and IoC knowledge base instead of live web search.
  • The same architecture could be applied to other structured detection artifacts such as Sigma rules or YARA, with a similar expected benefit from contextual enrichment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes RAM, a multi-stage LLM pipeline that maps structured Splunk SIEM rules to MITRE ATT&CK techniques and sub-techniques without fine-tuning. The pipeline extracts indicators of compromise, retrieves contextual web information through a ReAct agent, translates the rule into natural language, identifies data sources or mitigations via agentic RAG, recommends probable techniques, and filters those recommendations by a confidence threshold with chain-of-thought explanations. The evaluation uses 360 endpoint-domain Splunk Security Content rules with creation/modification dates after the models' knowledge cutoffs. With GPT-4-Turbo, RAM reportedly achieves AR 0.75 and AP 0.52, outperforming zero-shot GPT-4-Turbo, BERT/CodeBERT classifiers, and TTPxHunter. An ablation study attributes much of the gain to the web-enriched natural-language translation step.

Significance. If the reported results are valid, the paper would make a useful practical contribution: it demonstrates that an LLM-agent pipeline, without training data, can produce explainable SIEM-rule-to-ATT&CK mappings that are competitive with or better than supervised classifiers. The temporal-filtering design to avoid data leakage, the clear ablation isolating the effect of contextual enrichment, and the inclusion of chain-of-thought rationales are all strengths. The claims are empirical rather than formal, and the central quantitative claims currently rest on a hyperparameter-selection protocol that is not adequately separated from the test set, so the significance is conditional on the evaluation being repaired.

major comments (4)
  1. [§5.5, Table 5] The dynamic-k configuration with a confidence threshold of 0.8 appears to be selected using the same 360-rule test set on which the final metrics are reported. The text says the threshold 'effectively filtered low-confidence recommendations' after observing performance, but no validation split or independent threshold-selection procedure is described. This is load-bearing because Table 5 shows that without the dynamic-k filter the best AP is 0.39 (at k=11), which is below the CodeBERT baseline AP of 0.47 reported in Table 3. The claimed AP of 0.52 is therefore not established as a fair test-set measurement. The authors should choose the threshold on a held-out validation set (or with nested cross-validation), then report test-set AP/AR for that fixed configuration, together with repeated-run variability.
  2. [§5.3 and §5.1] The training and evaluation protocol for the BERT, CodeBERT, and TTPxHunter baselines is not specified. The paper states that BERT and CodeBERT classifiers were trained and that TTPxHunter's public code was implemented, but it does not state what data they were trained on, whether the 360 test rules were part of that training set, what label set the classifiers output, or what hyperparameters were used. Without this information, the comparison in Table 3 cannot be interpreted. The authors must provide the exact train/validation/test split and confirm that none of the 360 evaluation rules appeared in any baseline's training data.
  3. [§6, 'Dataset Mislabeling'] The paper itself provides a concrete example of incomplete ground truth: an account-lockout rule labeled only T1078 while T1110 (Brute Force) is also relevant. Since every AR and AP number is computed against these labels, the absolute metrics are uncalibrated and the relative ranking could shift if labels were corrected. The acknowledgment is honest, but the paper should quantify the impact, for example by manually auditing a random sample of the 360 rules, reporting corrected-label metrics for all methods, or performing a sensitivity analysis. This is especially important because missing labels penalize methods that predict additional relevant techniques, and RAM tends to output a larger candidate set.
  4. [§5.2 and §5.5] No error bars, confidence intervals, or significance tests are reported for any of the AR/AP/WAR/WAP numbers. LLM outputs are stochastic and the baselines involve trained models, so single-run point estimates may not be stable. The authors should run the pipeline multiple times (with the same fixed threshold chosen on validation data) and report means and standard deviations or confidence intervals, at least for the GPT-4-Turbo configuration and the leading baselines.
minor comments (5)
  1. [Throughout] The manuscript contains several typos and grammatical errors, including 'In the next sstep' (§4.2), 'Howerver' (Related Work), and 'None that none of the hosted or local models...' (§5.1). A careful proofreading pass is needed.
  2. [Table 5] The table has two rows labeled '7': one for k=13 and one for dynamic-k. Renumber the rows for clarity.
  3. [§4.5 and §5.5] Section 4.5 says a k-value of 11 was selected to ensure high recall, while Section 5.5 says the hard limit was replaced by a confidence-based dynamic filter. The relationship between the fixed-k results in Table 5 and the final dynamic-k configuration should be stated explicitly, including how k is involved in the dynamic-k setting.
  4. [Figure 4] The average precision vs. average recall curve in Figure 4 is not referenced anywhere in the text. The authors should either discuss how it was generated and what it shows, or remove it.
  5. [§5.1] The authors selected rules 'created or modified' after the knowledge cutoff. Since modification can touch only a small part of an older rule, the temporal-leakage protection is weaker than stated if the rule body predates the cutoff. This should be acknowledged and ideally the analysis repeated with only newly created rules.

Circularity Check

1 steps flagged · score 6.0 of 10

RAM's headline AP of 0.52 is a test-set-fitted optimum: the dynamic-k confidence threshold (0.8) and k selection are made on the same 360-rule evaluation set, so the reported precision is not an independent prediction.

  1. fitted input called prediction [Section 5.5 ('Effect of k on relevant recommendations') and Section 4.5 ('Probable Technique Recommendation')]
    "To address this, we replaced the hard limit on k with a filtering mechanism based on the confidence (relevance) score generated in the final stage of the pipeline. Recommendations with scores below a predefined threshold were excluded. We used a threshold of 0.8, which effectively filtered low-confidence recommendations while retaining the most relevant results. ... Based on these observations(please refer Table 5), we selected a k-value of 11 to ensure a high recall."

    The configuration that produces the headline numbers is chosen by inspecting AR/AP on the same 360-rule set on which the numbers are then reported. No validation split or independent threshold-selection procedure is described. Table 5 makes the selection visible: fixed-k rows have AP at most 0.45, while dynamic-k at the data-selected 0.8 threshold reports AP 0.52. The reported AP is therefore an optimized value on the evaluation set, not an unbiased estimate; the claimed precision advantage over CodeBERT's 0.47 AP is statistically forced by the data-dependent choice of the threshold rather than demonstrated as a prediction on unseen rules.

full rationale

This is an empirical system paper, not a derivation, and most of its pipeline is self-contained: the multi-stage LLM design, ablations, and baseline comparisons are evaluated with explicit metrics on the Splunk Security Content dataset. I found no load-bearing self-citation chain, no imported uniqueness theorem, and no renaming of a known result. The one circular element is the selection of the dynamic-k threshold (0.8) and the related k value on the same data used to compute the final AR/AP; the paper never reports a validation split or a threshold-selection procedure independent of the test rules. The acknowledged mislabeling in Section 6 is a ground-truth validity concern rather than circularity, and it applies to all compared methods equally. The web-search retrieval stage is a potential answer-leakage channel, but the paper does not supply enough detail to establish a specific reduction, so I do not count it as a circular step. Overall, the central performance claim is partially circular via test-set-fitted hyperparameters, while the framework itself retains independent empirical content.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities. The load-bearing assumptions are about ground-truth quality, data leakage, and the reliability of MITRE relationships. The free parameters (k and threshold) are chosen from test-set performance, which is the most significant circularity concern.

free parameters (3)
  • k (number of probable techniques) = 11
    Selected based on observed recall/precision trade-off in preliminary experiments (Section 4.5, Table 5); later replaced by a confidence threshold.
  • Confidence threshold for dynamic-k filtering = 0.8
    Used in the final 'dynamic-k' configuration to filter recommendations; appears to be chosen from test-set outcomes (Table 5), which is a post-hoc fit to the evaluation data.
  • RAG top-k retrieved documents = 5
    The agent retrieves the five most similar documents from the MITRE vector database (Section 4.4); this is a hand-chosen hyperparameter.
assumptions (4)
  • domain assumption The Splunk Security Content dataset labels are accurate and complete enough to serve as ground truth.
    The paper's metrics are all computed against these labels, yet Section 6 provides a concrete example of a missing label (T1110), undermining the assumption.
  • domain assumption Rules created or modified after the LLM knowledge cut-off dates are not present in the models' training data and web search does not leak ground-truth labels.
    The leakage-prevention strategy in Section 5.1 depends on this; no audit of web-search outputs is described.
  • standard math MITRE ATT&CK data source and mitigation relationships accurately connect to techniques and can serve as a filtering signal.
    The RAG step in Section 4.4 relies on these relationships being correct and sufficiently discriminative for the task.
  • domain assumption LLMs are effective at text similarity matching for filtering candidate techniques.
    Section 4.6 states this assumption explicitly as the basis for the confidence-scoring step.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rule-ATT&CK Mapper (RAM): Mapping SIEM Rules to TTPs Using LLMs." pith.science (2026). https://pith.science/paper/GSZSJMAH

@misc{pith2026250202337,
  author       = {Pith},
  title        = {Pith review of: Rule-ATT&CK Mapper (RAM): Mapping SIEM Rules to TTPs Using LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GSZSJMAH}},
  note         = {Machine review of arXiv:2502.02337}
}
read the original abstract

The growing frequency of cyberattacks has heightened the demand for accurate and efficient threat detection systems. SIEM platforms are important for analyzing log data and detecting adversarial activities through rule-based queries, also known as SIEM rules. The efficiency of the threat analysis process relies heavily on mapping these SIEM rules to the relevant attack techniques in the MITRE ATT&CK framework. Inaccurate annotation of SIEM rules can result in the misinterpretation of attacks, increasing the likelihood that threats will be overlooked. Existing solutions for annotating SIEM rules with MITRE ATT&CK technique labels have notable limitations: manual annotation of SIEM rules is both time-consuming and prone to errors, and ML-based approaches mainly focus on annotating unstructured free text sources rather than structured data like SIEM rules. Structured data often contains limited information, further complicating the annotation process and making it a challenging task. To address these challenges, we propose Rule-ATT&CK Mapper (RAM), a novel framework that leverages LLMs to automate the mapping of structured SIEM rules to MITRE ATT&CK techniques. RAM's multi-stage pipeline, which was inspired by the prompt chaining technique, enhances mapping accuracy without requiring LLM pre-training or fine-tuning. Using the Splunk Security Content dataset, we evaluate RAM's performance using several LLMs, including GPT-4-Turbo, Qwen, IBM Granite, and Mistral. Our evaluation highlights GPT-4-Turbo's superior performance, which derives from its enriched knowledge base, and an ablation study emphasizes the importance of external contextual knowledge in overcoming the limitations of LLMs' implicit knowledge for domain-specific tasks. These findings demonstrate RAM's potential in automating cybersecurity workflows and provide valuable insights for future advancements in this field.

Figures

Figures reproduced from arXiv: 2502.02337 by the authors.

Figure 1
Figure 1. Overview of our AI Agent-based RAM pipeline. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. An illustration of the different steps in RAM. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Overview of prompt structure used in all steps of [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Average precision vs average recall curve. [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Chain-of-thought reasoning provided by RAM. [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Distribution of length of labels in test samples. [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 17 canonical work pages

  1. [1]

    [n. d.]. ReAct Prompting. https://www.promptingguide.ai/techniques/react

  2. [2]

    Bader Al-Sada, Alireza Sadighian, and Gabriele Oligeri. 2024. Mitre att&ck: State of the art and way forward. Comput. Surveys 57, 1 (2024), 1–37

  3. [3]

    Md Tanvirul Alam, Dipkamal Bhusal, Youngja Park, and Nidhi Rastogi. 2023. Looking beyond IoCs: Automatically extracting attack patterns from external CTI. In Proceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses. 92–108

  4. [4]

    Paulo MMR Alves, PR Geraldo Filho, and Vinícius P Gonçalves. 2022. Leveraging BERT’s Power to Classify TTP from Unstructured Text. In 2022 Workshop on Communication Networks and Power Systems (WCNPS) . IEEE, 1–7

  5. [5]

    Checkpoint. [n. d.]. Check Point Research. https://blog.checkpoint.com/research/ check-point-research-reports-highest-increase-of-global-cyber-attacks-seen- in-last-two-years-a-30-increase-in-q2-2024-global-cyber-attacks/

  6. [6]

    Xiang Chen, Chaoyang Gao, Chunyang Chen, Guangbei Zhang, and Yong Liu

  7. [7]

    CISA. [n. d.]. Best Practices for Mapping to MITRE ATT&CK . https://www.cisa. gov/news-events/news/best-practices-mitre-attckr-mapping

  8. [8]

    Cybero. [n. d.]. SIEM Optimization Through MITRE ATT&CK . https: //www.cyrebro.io/blog/siem-optimization-through-mitre-attck-staying- ahead-of-threats-with-cyrebro/

Show all 32 references
  1. [9]

    Nir Daniel, Florian Klaus Kaiser, Anton Dzega, Aviad Elyashar, and Rami Puzis

  2. [10]

    Jacob Devlin. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)

  3. [11]

    Exabeam. [n. d.]. What is SIEM and How Does it Work? https://www.exabeam. com/explainers/siem-tools/siem-solutions/

  4. [12]

    Reza Fayyazi, Rozhina Taghdimi, and Shanchieh Jay Yang. 2023. Advancing TTP Analysis: Harnessing the Power of Encoder-Only and Decoder-Only Language Models with Retrieval Augmented Generation. arXiv preprint arXiv:2401.00280 (2023)

  5. [13]

    Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. 2020. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155 (2020)

  6. [14]

    Yu Fengrui and Yanhui Du. 2024. Few-Shot Learning of TTPs Classification Using Large Language Models. (2024)

  7. [15]

    MITRE ATT&CK framework. [n. d.]. Data Sources. https://attack.mitre.org/ datasources/

  8. [16]

    Scott Freitas, Jovan Kalajdjieski, Amir Gharib, and Robert McCann. 2024. AI- Driven Guided Response for Security Operation Centers with Microsoft Copilot for Security. arXiv:2407.09017 [cs.LG] https://arxiv.org/abs/2407.09017

  9. [17]

    Xu Huang, Weiwen Liu, Xiaolong Chen, Xingmei Wang, Hao Wang, Defu Lian, Yasheng Wang, Ruiming Tang, and Enhong Chen. 2024. Understanding the planning of LLM agents: A survey. arXiv:2402.02716 [cs.AI] https://arxiv.org/ abs/2402.02716

  10. [18]

    Roman Kryukov, Vladimir Zima, Elena Fedorchenko, Evgenia Novikova, and Igor Kotenko. 2022. Mapping the Security Events to the MITRE ATT &CK Attack Patterns to Forecast Attack Propagation. In International Workshop on Attacks and Defenses for Internet-of-Things. Springer, 165–176

  11. [19]

    Chenjing Liu, Junfeng Wang, and Xiangru Chen. 2022. Threat intelligence ATT&CK extraction based on the attention transformer hierarchical recurrent neural network. Applied Soft Computing 122 (2022), 108826

  12. [20]

    Marius Mărmureanu and Ciprian Oprişa. 2023. MITRE Tactics Inference from Splunk Queries. In2023 IEEE 19th International Conference on Intelligent Computer Communication and Processing (ICCP) . 277–283. doi:10.1109/ICCP60212.2023. 10398612

  13. [21]

    Nanda Rani, Bikash Saha, Vikas Maurya, and Sandeep Kumar Shukla. 2023. TTPHunter: Automated Extraction of Actionable Intelligence as TTPs from Narrative Threat Reports. In Proceedings of the 2023 Australasian Computer Sci- ence Week (Melbourne, VIC, Australia) (ACSW ’23). Asso...

  14. [22]

    Nanda Rani, Bikash Saha, Vikas Maurya, and Sandeep Kumar Shukla. 2024. TTPXHunter: Actionable Threat Intelligence Extraction as TTPs form Finished Cyber Threat Reports. arXiv preprint arXiv:2403.03267 (2024)

  15. [23]

    Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Mondal, and Aman Chadha. 2024. A systematic survey of prompt engineering in large language models: Techniques and applications. arXiv preprint arXiv:2402.07927 (2024)

  16. [24]

    Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2019. How to fine-tune bert for text classification?. In Chinese computational linguistics: 18th China national conference, CCL 2019, Kunming, China, October 18–20, 2019, proceedings

  17. [25]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reason- ing in large language models. Advances in neural information processing systems 35 (2022), 24824–24837

  18. [26]

    Thomas Wolf. 2020. Transformers: State-of-the-Art Natural Language Processing. arXiv preprint arXiv:1910.03771 (2020)

  19. [27]

    Tongshuang Wu, Ellen Jiang, Aaron Donsbach, Jeff Gray, Alejandra Molina, Michael Terry, and Carrie J Cai. 2022. Promptchainer: Chaining large language model prompts through visual programming. InCHI Conference on Human Factors in Computing Systems Extended Abstracts . 1–10

  20. [28]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. arXiv:2210.03629 [cs.CL] https://arxiv.org/abs/2210.03629

  21. [29]

    Yizhe You, Jun Jiang, Zhengwei Jiang, Peian Yang, Baoxu Liu, Huamin Feng, Xuren Wang, and Ning Li. 2022. TIM: threat context-enhanced TTP intelligence mining on unstructured threat data. Cybersecurity 5, 1 (2022), 3

  22. [30]

    Yongheng Zhang, Tingwen Du, Yunshan Ma, Xiang Wang, Yi Xie, Guozheng Yang, Yuliang Lu, and Ee-Chien Chang. 2024. AttacKG+: Boosting Attack Knowledge Graph Construction with Large Language Models.arXiv preprint arXiv:2405.04753 (2024). A Labels Distribution Figure 6: Distributi...

  23. [2023]

    In European Symposium on Research in Computer Security

    Labeling NIDS Rules with MITRE ATT &CK Techniques Using ChatGPT. In European Symposium on Research in Computer Security . Springer, 76–91

  24. [2024]

    arXiv:2408.05002 [cs.SE] https://arxiv.org/abs/2408.05002

    An Empirical Study on Challenges for LLM Application Developers. arXiv:2408.05002 [cs.SE] https://arxiv.org/abs/2408.05002

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.