REVIEW 4 major objections 5 minor 32 references
Rule-ATT&CK Mapper (RAM): Mapping SIEM Rules to TTPs Using LLMs
T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read A multi-stage LLM pipeline maps SIEM rules to MITRE ATT&CK techniques at 0.75 recall without fine-tuning.
desk verdict Useful pipeline, but the headline precision is inflated by a test-set-tuned threshold; the paper deserves a serious referee if that gets fixed. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is a six-step prompt-chaining pipeline: zero-shot IoC extraction; a REACT web-search agent that pulls contextual information about each IoC; natural-language translation of the rule; agentic RAG to identify the relevant MITRE data source or mitigation; recommendation of probable techniques via another REACT agent; and final extraction of relevant techniques using chain-of-thought explanations and a confidence threshold. The pipeline's effectiveness rests on converting the structured rule into rich natural language before similarity comparison with technique descriptions.
What would settle it
Have several independent security analysts annotate the same 360 Splunk rules to a consensus ATT&CK gold standard, then recompute RAM's average recall and precision against that standard; if the scores move substantially from 0.75 and 0.52, the reported accuracy is an artifact of imperfect labels.
Extended reading notes
Core claim
On its own terms, the paper's discovery is that the combination of prompt chaining and agent-based web retrieval lets an LLM map structured detection rules to the full MITRE ATT&CK technique taxonomy with usable accuracy. The reported numbers, AR 0.75 and AP 0.52 with GPT-4-Turbo, come from the complete RAM pipeline, while the same pipeline without contextual enrichment reaches only AR 0.54, and the raw rule as-is reaches 0.46. This shows the load-bearing role of external context over the LLM's implicit knowledge. The paper further claims that the pipeline is format-agnostic, works across hosted and local models of varying size, and produces chain-of-thought rationales that analysts can inspect.
Load-bearing premise
The evaluation assumes the Splunk Security Content dataset's ATT&CK labels are ground truth, but the paper itself shows a rule whose labels are incomplete; if such label noise is systematic, the reported recall and precision are not a true measure of mapping quality.
Editorial extensions
If this is right
- Security teams could automate ATT&CK annotation of new SIEM rules without waiting for labeled training data.
- The same pipeline should transfer to other rule definition languages such as KQL and Lucene, since the translation step normalizes them to text.
- Because the method provides reasoning, analysts can audit and override mappings, unlike black-box classifiers.
- New ATT&CK techniques can be accommodated without retraining, only by updating the technique descriptions in the RAG store and prompts.
Reading between the lines
- The reported gains may understate or overstate true accuracy because the Splunk labels are themselves incomplete; a multi-annotator gold standard would be needed to know the real headroom.
- The web-search context stage likely depends on the freshness and quality of search results; a testable extension is to swap in a curated vulnerability and IoC knowledge base instead of live web search.
- The same architecture could be applied to other structured detection artifacts such as Sigma rules or YARA, with a similar expected benefit from contextual enrichment.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes RAM, a multi-stage LLM pipeline that maps structured Splunk SIEM rules to MITRE ATT&CK techniques and sub-techniques without fine-tuning. The pipeline extracts indicators of compromise, retrieves contextual web information through a ReAct agent, translates the rule into natural language, identifies data sources or mitigations via agentic RAG, recommends probable techniques, and filters those recommendations by a confidence threshold with chain-of-thought explanations. The evaluation uses 360 endpoint-domain Splunk Security Content rules with creation/modification dates after the models' knowledge cutoffs. With GPT-4-Turbo, RAM reportedly achieves AR 0.75 and AP 0.52, outperforming zero-shot GPT-4-Turbo, BERT/CodeBERT classifiers, and TTPxHunter. An ablation study attributes much of the gain to the web-enriched natural-language translation step.
Significance. If the reported results are valid, the paper would make a useful practical contribution: it demonstrates that an LLM-agent pipeline, without training data, can produce explainable SIEM-rule-to-ATT&CK mappings that are competitive with or better than supervised classifiers. The temporal-filtering design to avoid data leakage, the clear ablation isolating the effect of contextual enrichment, and the inclusion of chain-of-thought rationales are all strengths. The claims are empirical rather than formal, and the central quantitative claims currently rest on a hyperparameter-selection protocol that is not adequately separated from the test set, so the significance is conditional on the evaluation being repaired.
major comments (4)
- [§5.5, Table 5] The dynamic-k configuration with a confidence threshold of 0.8 appears to be selected using the same 360-rule test set on which the final metrics are reported. The text says the threshold 'effectively filtered low-confidence recommendations' after observing performance, but no validation split or independent threshold-selection procedure is described. This is load-bearing because Table 5 shows that without the dynamic-k filter the best AP is 0.39 (at k=11), which is below the CodeBERT baseline AP of 0.47 reported in Table 3. The claimed AP of 0.52 is therefore not established as a fair test-set measurement. The authors should choose the threshold on a held-out validation set (or with nested cross-validation), then report test-set AP/AR for that fixed configuration, together with repeated-run variability.
- [§5.3 and §5.1] The training and evaluation protocol for the BERT, CodeBERT, and TTPxHunter baselines is not specified. The paper states that BERT and CodeBERT classifiers were trained and that TTPxHunter's public code was implemented, but it does not state what data they were trained on, whether the 360 test rules were part of that training set, what label set the classifiers output, or what hyperparameters were used. Without this information, the comparison in Table 3 cannot be interpreted. The authors must provide the exact train/validation/test split and confirm that none of the 360 evaluation rules appeared in any baseline's training data.
- [§6, 'Dataset Mislabeling'] The paper itself provides a concrete example of incomplete ground truth: an account-lockout rule labeled only T1078 while T1110 (Brute Force) is also relevant. Since every AR and AP number is computed against these labels, the absolute metrics are uncalibrated and the relative ranking could shift if labels were corrected. The acknowledgment is honest, but the paper should quantify the impact, for example by manually auditing a random sample of the 360 rules, reporting corrected-label metrics for all methods, or performing a sensitivity analysis. This is especially important because missing labels penalize methods that predict additional relevant techniques, and RAM tends to output a larger candidate set.
- [§5.2 and §5.5] No error bars, confidence intervals, or significance tests are reported for any of the AR/AP/WAR/WAP numbers. LLM outputs are stochastic and the baselines involve trained models, so single-run point estimates may not be stable. The authors should run the pipeline multiple times (with the same fixed threshold chosen on validation data) and report means and standard deviations or confidence intervals, at least for the GPT-4-Turbo configuration and the leading baselines.
minor comments (5)
- [Throughout] The manuscript contains several typos and grammatical errors, including 'In the next sstep' (§4.2), 'Howerver' (Related Work), and 'None that none of the hosted or local models...' (§5.1). A careful proofreading pass is needed.
- [Table 5] The table has two rows labeled '7': one for k=13 and one for dynamic-k. Renumber the rows for clarity.
- [§4.5 and §5.5] Section 4.5 says a k-value of 11 was selected to ensure high recall, while Section 5.5 says the hard limit was replaced by a confidence-based dynamic filter. The relationship between the fixed-k results in Table 5 and the final dynamic-k configuration should be stated explicitly, including how k is involved in the dynamic-k setting.
- [Figure 4] The average precision vs. average recall curve in Figure 4 is not referenced anywhere in the text. The authors should either discuss how it was generated and what it shows, or remove it.
- [§5.1] The authors selected rules 'created or modified' after the knowledge cutoff. Since modification can touch only a small part of an older rule, the temporal-leakage protection is weaker than stated if the rule body predates the cutoff. This should be acknowledged and ideally the analysis repeated with only newly created rules.
Circularity Check
RAM's headline AP of 0.52 is a test-set-fitted optimum: the dynamic-k confidence threshold (0.8) and k selection are made on the same 360-rule evaluation set, so the reported precision is not an independent prediction.
-
fitted input called prediction
[Section 5.5 ('Effect of k on relevant recommendations') and Section 4.5 ('Probable Technique Recommendation')]
"To address this, we replaced the hard limit on k with a filtering mechanism based on the confidence (relevance) score generated in the final stage of the pipeline. Recommendations with scores below a predefined threshold were excluded. We used a threshold of 0.8, which effectively filtered low-confidence recommendations while retaining the most relevant results. ... Based on these observations(please refer Table 5), we selected a k-value of 11 to ensure a high recall."
The configuration that produces the headline numbers is chosen by inspecting AR/AP on the same 360-rule set on which the numbers are then reported. No validation split or independent threshold-selection procedure is described. Table 5 makes the selection visible: fixed-k rows have AP at most 0.45, while dynamic-k at the data-selected 0.8 threshold reports AP 0.52. The reported AP is therefore an optimized value on the evaluation set, not an unbiased estimate; the claimed precision advantage over CodeBERT's 0.47 AP is statistically forced by the data-dependent choice of the threshold rather than demonstrated as a prediction on unseen rules.
full rationale
This is an empirical system paper, not a derivation, and most of its pipeline is self-contained: the multi-stage LLM design, ablations, and baseline comparisons are evaluated with explicit metrics on the Splunk Security Content dataset. I found no load-bearing self-citation chain, no imported uniqueness theorem, and no renaming of a known result. The one circular element is the selection of the dynamic-k threshold (0.8) and the related k value on the same data used to compute the final AR/AP; the paper never reports a validation split or a threshold-selection procedure independent of the test rules. The acknowledged mislabeling in Section 6 is a ground-truth validity concern rather than circularity, and it applies to all compared methods equally. The web-search retrieval stage is a potential answer-leakage channel, but the paper does not supply enough detail to establish a specific reduction, so I do not count it as a circular step. Overall, the central performance claim is partially circular via test-set-fitted hyperparameters, while the framework itself retains independent empirical content.
Assumptions & free parameters
free parameters (3)
- k (number of probable techniques) =
11
- Confidence threshold for dynamic-k filtering =
0.8
- RAG top-k retrieved documents =
5
assumptions (4)
- domain assumption The Splunk Security Content dataset labels are accurate and complete enough to serve as ground truth.
- domain assumption Rules created or modified after the LLM knowledge cut-off dates are not present in the models' training data and web search does not leak ground-truth labels.
- standard math MITRE ATT&CK data source and mitigation relationships accurately connect to techniques and can serve as a filtering signal.
- domain assumption LLMs are effective at text similarity matching for filtering candidate techniques.
Cite this review
Pith. "Pith review of Rule-ATT&CK Mapper (RAM): Mapping SIEM Rules to TTPs Using LLMs." pith.science (2026). https://pith.science/paper/GSZSJMAH
@misc{pith2026250202337,
author = {Pith},
title = {Pith review of: Rule-ATT&CK Mapper (RAM): Mapping SIEM Rules to TTPs Using LLMs},
year = {2026},
howpublished = {\url{https://pith.science/paper/GSZSJMAH}},
note = {Machine review of arXiv:2502.02337}
}
read the original abstract
The growing frequency of cyberattacks has heightened the demand for accurate and efficient threat detection systems. SIEM platforms are important for analyzing log data and detecting adversarial activities through rule-based queries, also known as SIEM rules. The efficiency of the threat analysis process relies heavily on mapping these SIEM rules to the relevant attack techniques in the MITRE ATT&CK framework. Inaccurate annotation of SIEM rules can result in the misinterpretation of attacks, increasing the likelihood that threats will be overlooked. Existing solutions for annotating SIEM rules with MITRE ATT&CK technique labels have notable limitations: manual annotation of SIEM rules is both time-consuming and prone to errors, and ML-based approaches mainly focus on annotating unstructured free text sources rather than structured data like SIEM rules. Structured data often contains limited information, further complicating the annotation process and making it a challenging task. To address these challenges, we propose Rule-ATT&CK Mapper (RAM), a novel framework that leverages LLMs to automate the mapping of structured SIEM rules to MITRE ATT&CK techniques. RAM's multi-stage pipeline, which was inspired by the prompt chaining technique, enhances mapping accuracy without requiring LLM pre-training or fine-tuning. Using the Splunk Security Content dataset, we evaluate RAM's performance using several LLMs, including GPT-4-Turbo, Qwen, IBM Granite, and Mistral. Our evaluation highlights GPT-4-Turbo's superior performance, which derives from its enriched knowledge base, and an ablation study emphasizes the importance of external contextual knowledge in overcoming the limitations of LLMs' implicit knowledge for domain-specific tasks. These findings demonstrate RAM's potential in automating cybersecurity workflows and provide valuable insights for future advancements in this field.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
[n. d.]. ReAct Prompting. https://www.promptingguide.ai/techniques/react
-
[2]
Bader Al-Sada, Alireza Sadighian, and Gabriele Oligeri. 2024. Mitre att&ck: State of the art and way forward. Comput. Surveys 57, 1 (2024), 1–37
work page 2024
-
[3]
Md Tanvirul Alam, Dipkamal Bhusal, Youngja Park, and Nidhi Rastogi. 2023. Looking beyond IoCs: Automatically extracting attack patterns from external CTI. In Proceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses. 92–108
work page 2023
-
[4]
Paulo MMR Alves, PR Geraldo Filho, and Vinícius P Gonçalves. 2022. Leveraging BERT’s Power to Classify TTP from Unstructured Text. In 2022 Workshop on Communication Networks and Power Systems (WCNPS) . IEEE, 1–7
work page 2022
-
[5]
Checkpoint. [n. d.]. Check Point Research. https://blog.checkpoint.com/research/ check-point-research-reports-highest-increase-of-global-cyber-attacks-seen- in-last-two-years-a-30-increase-in-q2-2024-global-cyber-attacks/
work page 2024
-
[6]
Xiang Chen, Chaoyang Gao, Chunyang Chen, Guangbei Zhang, and Yong Liu
-
[7]
CISA. [n. d.]. Best Practices for Mapping to MITRE ATT&CK . https://www.cisa. gov/news-events/news/best-practices-mitre-attckr-mapping
-
[8]
Cybero. [n. d.]. SIEM Optimization Through MITRE ATT&CK . https: //www.cyrebro.io/blog/siem-optimization-through-mitre-attck-staying- ahead-of-threats-with-cyrebro/
Show all 32 references
-
[9]
Nir Daniel, Florian Klaus Kaiser, Anton Dzega, Aviad Elyashar, and Rami Puzis
-
[10]
Jacob Devlin. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
2018 arXiv
-
[11]
Exabeam. [n. d.]. What is SIEM and How Does it Work? https://www.exabeam. com/explainers/siem-tools/siem-solutions/
-
[12]
Reza Fayyazi, Rozhina Taghdimi, and Shanchieh Jay Yang. 2023. Advancing TTP Analysis: Harnessing the Power of Encoder-Only and Decoder-Only Language Models with Retrieval Augmented Generation. arXiv preprint arXiv:2401.00280 (2023)
2023
-
[13]
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. 2020. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155 (2020)
2020 arXiv
-
[14]
Yu Fengrui and Yanhui Du. 2024. Few-Shot Learning of TTPs Classification Using Large Language Models. (2024)
2024
-
[15]
MITRE ATT&CK framework. [n. d.]. Data Sources. https://attack.mitre.org/ datasources/
-
[16]
Scott Freitas, Jovan Kalajdjieski, Amir Gharib, and Robert McCann. 2024. AI- Driven Guided Response for Security Operation Centers with Microsoft Copilot for Security. arXiv:2407.09017 [cs.LG] https://arxiv.org/abs/2407.09017
2024 arXiv
-
[17]
Xu Huang, Weiwen Liu, Xiaolong Chen, Xingmei Wang, Hao Wang, Defu Lian, Yasheng Wang, Ruiming Tang, and Enhong Chen. 2024. Understanding the planning of LLM agents: A survey. arXiv:2402.02716 [cs.AI] https://arxiv.org/ abs/2402.02716
2024 arXiv
-
[18]
Roman Kryukov, Vladimir Zima, Elena Fedorchenko, Evgenia Novikova, and Igor Kotenko. 2022. Mapping the Security Events to the MITRE ATT &CK Attack Patterns to Forecast Attack Propagation. In International Workshop on Attacks and Defenses for Internet-of-Things. Springer, 165–176
2022
-
[19]
Chenjing Liu, Junfeng Wang, and Xiangru Chen. 2022. Threat intelligence ATT&CK extraction based on the attention transformer hierarchical recurrent neural network. Applied Soft Computing 122 (2022), 108826
2022
-
[20]
Marius Mărmureanu and Ciprian Oprişa. 2023. MITRE Tactics Inference from Splunk Queries. In2023 IEEE 19th International Conference on Intelligent Computer Communication and Processing (ICCP) . 277–283. doi:10.1109/ICCP60212.2023. 10398612
2023
-
[21]
Nanda Rani, Bikash Saha, Vikas Maurya, and Sandeep Kumar Shukla. 2023. TTPHunter: Automated Extraction of Actionable Intelligence as TTPs from Narrative Threat Reports. In Proceedings of the 2023 Australasian Computer Sci- ence Week (Melbourne, VIC, Australia) (ACSW ’23). Asso...
2023
-
[22]
Nanda Rani, Bikash Saha, Vikas Maurya, and Sandeep Kumar Shukla. 2024. TTPXHunter: Actionable Threat Intelligence Extraction as TTPs form Finished Cyber Threat Reports. arXiv preprint arXiv:2403.03267 (2024)
2024 arXiv
-
[23]
Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Mondal, and Aman Chadha. 2024. A systematic survey of prompt engineering in large language models: Techniques and applications. arXiv preprint arXiv:2402.07927 (2024)
2024 arXiv
-
[24]
Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2019. How to fine-tune bert for text classification?. In Chinese computational linguistics: 18th China national conference, CCL 2019, Kunming, China, October 18–20, 2019, proceedings
2019
-
[25]
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reason- ing in large language models. Advances in neural information processing systems 35 (2022), 24824–24837
2022
-
[26]
Thomas Wolf. 2020. Transformers: State-of-the-Art Natural Language Processing. arXiv preprint arXiv:1910.03771 (2020)
2020 arXiv
-
[27]
Tongshuang Wu, Ellen Jiang, Aaron Donsbach, Jeff Gray, Alejandra Molina, Michael Terry, and Carrie J Cai. 2022. Promptchainer: Chaining large language model prompts through visual programming. InCHI Conference on Human Factors in Computing Systems Extended Abstracts . 1–10
2022
-
[28]
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. arXiv:2210.03629 [cs.CL] https://arxiv.org/abs/2210.03629
2023 arXiv
-
[29]
Yizhe You, Jun Jiang, Zhengwei Jiang, Peian Yang, Baoxu Liu, Huamin Feng, Xuren Wang, and Ning Li. 2022. TIM: threat context-enhanced TTP intelligence mining on unstructured threat data. Cybersecurity 5, 1 (2022), 3
2022
-
[30]
Yongheng Zhang, Tingwen Du, Yunshan Ma, Xiang Wang, Yi Xie, Guozheng Yang, Yuliang Lu, and Ee-Chien Chang. 2024. AttacKG+: Boosting Attack Knowledge Graph Construction with Large Language Models.arXiv preprint arXiv:2405.04753 (2024). A Labels Distribution Figure 6: Distributi...
2024 arXiv
-
[2023]
In European Symposium on Research in Computer Security
Labeling NIDS Rules with MITRE ATT &CK Techniques Using ChatGPT. In European Symposium on Research in Computer Security . Springer, 76–91
-
[2024]
arXiv:2408.05002 [cs.SE] https://arxiv.org/abs/2408.05002
An Empirical Study on Challenges for LLM Application Developers. arXiv:2408.05002 [cs.SE] https://arxiv.org/abs/2408.05002
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.