REVIEW 3 major objections 5 minor 26 references
Multi-LLM Consensus Framework for Evaluating Banking-Sector NIDS Dataset Coverage of MITRE ATT&CK Techniques
T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read The paper establishes a priority-weighted, NIST-constrained scoring method for NIDS datasets against banking-relevant MITRE ATT&CK techniques, and finds UNSW-NB15 highest at 82.2% weighted coverage with CIC-DDoS2019 at 10.1%.
desk verdict Useful framework (NIST-constrained ATT&CK filtering plus a banking baseline) whose headline coverage numbers are unvalidated LLM consensus over dataset text profiles — worth refereeing, but the percentages should be read as estimates. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a three-stage evaluation pipeline. Stage one extracts banking-relevant ATT&CK techniques from the knowledge base. Stage two applies NIST SP 800-94 passive-sensor constraints, labeling each technique YES, PARTIAL, or NO and retaining the 68 that are network-observable. Stage three maps those 68 techniques to each dataset as Direct, Indirect, or None, using four independently prompted large language models with a 75% agreement threshold and conservative tie-break rules, then scores datasets with the weighted coverage formula $\frac{\sum_i w_i c_i}{\sum_i w_i} \times 100$ where $w_i$ comes from the hand-set Priority Score $PS = U_e + I_t + D_g$ (weights 40/40/20). The 75% consensus threshold and the Indirect-versus-None exclusion rules are what convert noisy model judgments into a stable coverage number.
What would settle it
Recompute all weighted coverage scores under alternative Priority Score weightings, such as equal weights on entity usage, tactic importance, and group diversity, or weights fitted to real banking intrusion reports, and test whether UNSW-NB15 still ranks first and CIC-DDoS2019 still ranks last; the headline result stands only if the ordering survives plausible reweighting.
Extended reading notes
Core claim
The central claim is that existing public NIDS benchmarks give uneven, mostly indirect coverage of the adversary behaviors that matter in core banking, and that this can be measured in a reproducible way. Under the paper's NIST-constrained definition of passive NIDS visibility, only 68 of 210 banking-relevant ATT&CK techniques are observable in network traffic at all. Mapping those 68 techniques to five benchmarks, UNSW-NB15 is the most useful at 82.2% weighted coverage (18.4% direct, 63.8% indirect), CTU-13 is dominated by indirect coverage (40.4% indirect versus 9.0% direct), and CIC-DDoS2019 covers only 10.1%, an 89.9% blind spot for core banking behaviors. At tactic level, Discovery and Exfiltration are best covered while Initial Access and Impact are systematically underrepresented, and no dataset provides direct coverage of T1071 (Application Layer Protocol), a key command-and-control technique. The paper concludes that high benchmark accuracy can create false assurance and that banking-native datasets with explicit ATT&CK labels are needed.
Load-bearing premise
The ranking rests on the hand-selected 40/40/20 weights in the Priority Score formula; if those weights were changed, the claim that UNSW-NB15 is the best banking-sector benchmark and the size of every coverage gap could change.
Editorial extensions
If this is right
- UNSW-NB15 should be read as an anomaly-detection benchmark for banking, not a technique-classification benchmark, because 63.8% of its 82.2% score comes from indirect artifacts that do not identify specific tradecraft such as SWIFT message manipulation or ATM jackpotting.
- CIC-DDoS2019 alone cannot validate a banking NIDS: its 10.1% weighted coverage means models trained on it have not seen 89.9% of the network-observable banking attack baseline.
- Passive NIDS will systematically under-cover Initial Access and Impact tactics, so closing the validation gap requires host-based telemetry and TLS inspection in addition to better datasets.
- The published 68-technique baseline gives dataset builders a concrete checklist: a banking-native benchmark should add explicit SWIFT and ATM traffic with ATT&CK labels for currently missing techniques.
- The same NIST-constrained consensus procedure can be rerun as ATT&CK evolves or as new datasets appear, making the coverage audit repeatable rather than a one-time result.
Reading between the lines
- If the Priority Score weights were recalibrated (for example, made equal across entity usage, tactic importance, and group diversity, or fitted to real banking intrusion reports), the dataset ordering might change; the 40/40/20 choice is a modelling decision, not a measured quantity.
- Because the underlying LLMs are accessed through non-deterministic consumer interfaces, a second research group rerunning the same prompts could obtain slightly different YES/PARTIAL/NO boundaries or mapping labels; the published dispute rules make the procedure reproducible in structure but not guaranteed in exact output.
- The same pipeline could be applied to host-based telemetry (process execution, file and registry changes) to build a complementary endpoint-observable baseline, which would turn the paper's NIDS-only blind spots into quantifiable comparisons across sensor types.
- The universal absence of direct T1071 coverage suggests a concrete acceptance test for any future banking-native dataset: include labelled application-layer command-and-control traffic and check whether the coverage score moves from Indirect to Direct.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a three-stage framework for evaluating how well network intrusion detection benchmark datasets cover MITRE ATT&CK techniques relevant to banking. The authors filter 210 banking-related techniques through NIST SP 800-94 passive sensor constraints using a four-LLM consensus engine (75% agreement), obtaining 68 network-observable techniques; then map those techniques to five datasets using the same LLM consensus procedure over text dataset profiles. A hand-weighted Priority Score (Eq. 1, weights 40/40/20) and a weighted coverage metric (Eq. 2, counting Direct and Indirect equally) yield the headline results: UNSW-NB15 achieves 82.2% weighted coverage (18.4% direct) and CIC-DDoS2019 achieves only 10.1%.
Significance. If the coverage estimates were validated against actual packet contents, this would be a valuable contribution to sector-aware NIDS evaluation: the NIST-constrained baseline is a sensible way to avoid overclaiming detectability, the consensus prompt design is clearly documented, and the artifacts (GitHub repository, prompt templates) support reproducibility. The paper also makes falsifiable predictions (e.g., no dataset directly covers T1071; Initial Access and Impact are underrepresented) that could be tested by future work. However, the current quantitative conclusions rest on two unvalidated choices: LLM labels without ground-truth checking, and hand-picked weights without sensitivity analysis.
major comments (3)
- Section 3.3, Eq. (2): The Direct/Indirect/None mapping labels are produced entirely by four LLMs reading text dataset profiles; no label is verified against the actual packets, flows, or labels of any dataset. Because Eq. (2) sums these labels with equal credit for Direct and Indirect (c_i = 1 for both), a handful of mislabeled high-priority techniques can materially change the reported coverage. For instance, UNSW-NB15's 82.2% coverage contains 63.8% indirect evidence; if half of those indirect labels are wrong, its score falls toward the 18.4% direct-only figure. The authors should either (a) validate a sample of labels by inspecting actual dataset artifacts, or (b) reframe the percentages as 'LLM-profile-based coverage estimates' and temper the dataset-ranking claims accordingly.
- Section 3.3, Eq. (1): The Priority Score weights (40/40/20 for Entity Usage, Tactic Importance, Group Diversity) are asserted without derivation, calibration, or sensitivity analysis. These weights directly determine the weighted coverage in Eq. (2), so the headline conclusion that UNSW-NB15 is the best dataset is conditional on this hand-picked choice. The paper should provide a sensitivity analysis (e.g., vary the weights over a plausible range and report whether the ranking persists) or derive the weights from an explicit, reproducible model. Without this, the main quantitative claim is not robust.
- Sections 3.2 and 4.1: The 68-technique baseline is itself an LLM consensus output, with 21 of 210 techniques requiring manual adjudication after 2-2 or 2-1-1 splits. No ground truth (e.g., expert labeling or actual attack traces) checks whether the Stage-1 YES/PARTIAL/NO labels are correct. Since the coverage percentages in Section 4.3 are ratios over this baseline, any error in the baseline changes both numerator and denominator. Please report the per-model Stage-1 classifications for all 210 techniques, describe the manual adjudication protocol, and quantify how the 68-technique count and the Section 4.3 coverage scores change when the 75% consensus threshold is varied.
minor comments (5)
- References: Reference [10] (Hasan et al., 'Retrieval of surface reflectance from NOAA-AVHRR satellite data') appears unrelated to the ATT&CK/NIDS context in which it is cited (Section 1, paragraph 4); please correct or remove this citation.
- Section 3.3: The priority tier definitions (High/Common/Medium/Rare) are introduced but never used in the coverage calculation or results; clarify their role or remove them.
- Section 3.2: The manual adjudication of the 21 Stage-1 disputes is mentioned but not described (who performed it, what criteria were used, what the outcomes were for each disputed technique). Provide this detail for reproducibility.
- Section 4.3: The text reports selected weighted coverage numbers (UNSW-NB15 82.2%, CTU-13, CIC-DDoS2019 10.1%) but does not provide a table with the full breakdown (total, direct, indirect, excluded count) for all five datasets; adding such a table would let readers verify the ranking.
- Section 5: The claim that a 50% consensus threshold 'would expand the detected techniques by 12%' and a unanimous threshold 'would have excluded 35% of the baseline' is not supported by any table or calculation in the paper; either present the underlying data or soften the quantitative claim.
Circularity Check
No significant circularity: the coverage ranking is a transparently defined metric computed from announced inputs, not a prediction that reduces to its own inputs.
full rationale
The paper's central quantity, weighted coverage, is explicitly introduced as a scoring rule in Eq. (2): Coverage(Weighted) = sum(w_i*c_i)/sum(w_i)*100. The priority weights are defined in Eq. (1) as PS = Ue + It + Dg, and the coverage tags c_i come from a stated multi-LLM consensus procedure operating on dataset profiles. There is no step in which an output of the evaluation is used to define its own inputs. The 210-technique scope is anchored to MITRE ATT&CK v18 and a banking-sector definition, and the 68-technique baseline is filtered through NIST SP 800-94 constraints, which are external standards. Dataset labels are generated from dataset profiles, but those profiles are inputs to the coverage calculation, not consequences of it. The 40/40/20 weights and the 75% consensus threshold are hand-selected modeling choices; they affect the numeric results and could change the ranking, but that is a robustness and validity limitation, not circularity. The self-citations to [19] and [24] describe prior methodology and related work; they are not invoked as an authority that forces the conclusion, and the paper does not claim a uniqueness theorem from its own prior work. The paper also candidly acknowledges 'dependence on model consensus' as a limitation. In short, the reported 82.2% figure is a transparent calculation from declared assumptions and labeled LLM judgments, not a fitted parameter renamed as a prediction.
Assumptions & free parameters
free parameters (3)
- Priority Score weights (Ue, It, Dg) =
40, 40, 20
- LLM consensus threshold =
75%
- Priority tier thresholds =
High (>=6 entities), Common (4-5), Medium (2-3), Rare (1)
assumptions (4)
- domain assumption MITRE ATT&CK v18 accurately enumerates banking-relevant adversary techniques
- domain assumption NIST SP 800-94 passive NIDS constraints are correctly operationalized (no TLS decryption, passive-only, no endpoint visibility)
- ad hoc to paper 75% agreement among four LLMs is a valid substitute for ground-truth classification
- domain assumption The 24 selected threat actors are representative of banking-sector adversaries
Cite this review
Pith. "Pith review of Multi-LLM Consensus Framework for Evaluating Banking-Sector NIDS Dataset Coverage of MITRE ATT&CK Techniques." pith.science (2026). https://pith.science/paper/7KIIGNUF
@misc{pith2026260800895,
author = {Pith},
title = {Pith review of: Multi-LLM Consensus Framework for Evaluating Banking-Sector NIDS Dataset Coverage of MITRE ATT&CK Techniques},
year = {2026},
howpublished = {\url{https://pith.science/paper/7KIIGNUF}},
note = {Machine review of arXiv:2608.00895}
}
read the original abstract
The systemic criticality of global banking networks has ren-dered them high-priority targets for advanced persistent threats, neces-sitating Network Intrusion Detection Systems (NIDS) whose operational effectiveness must extend beyond statistical accuracy. However, a signif-icant validation gap persists between experimental NIDS performance and real-world effectiveness: NIDS models that achieve high accuracy on standard benchmarks often fail in operational banking environments because generic datasets lack sector-specific patterns, such as SWIFT and ATM-related intrusions, that characterize real financial threats. To address this, the paper investigates a sector-aware evaluation method-ology that systematically assesses how well existing NIDS benchmark datasets cover the attack behaviors most relevant to banking infrastruc-ture. The methodology maps documented adversary behaviors from the MITRE ATT&CK knowledge base to NIDS benchmarks while enforcing the realistic sensor limitations defined by NIST SP 800-94. Leveraging a multi-LLM consensus engine with four state-of-the-art models, we evalu-ated 210 banking-specific adversary techniques to derive a baseline of 68 network-observable behaviors for systematic coverage analysis. Results across five benchmark datasets demonstrate that UNSW-NB15 achieves the highest utility with an 82.2% weighted coverage score (though only 18.4% reflects direct, technique-level evidence), while CIC-DDoS2019 re-veals an 89.9% blind spot for core banking behaviors. These findings es-tablish a reproducible foundation for sector-aware NIDS evaluation and highlight the urgent need for banking-native datasets.
Reference graph
Works this paper leans on
-
[1]
Alsaedi, A., Gupta, V., Gupta, B.B.: An overview of ransomware in the financial sector. Journal of Computer Virology and Hacking Techniques 20, 421– 432 (2024), https://doi.org/10.1007/s11416-024-00522-8
-
[2]
claude documentation (2024), https://docs.anthropic.com/en/docs/use -xml-tags
Anthropic: Use xml tags to structure your prompts. claude documentation (2024), https://docs.anthropic.com/en/docs/use -xml-tags
work page 2024
-
[3]
Data 8 (1), 18 (2023), https://www.mdpi.com/ 2306-5729/8/1/18
Bagui, S.S., Mink, D., Bagui, S.C., Ghosh, T., Plenkers, R., McElroy, T., Shabanali, S.: Introducing uwf -zeekdata22: A comprehensive network traffic dataset based on the mitre att&ck framework. Data 8 (1), 18 (2023), https://www.mdpi.com/ 2306-5729/8/1/18
work page 2023
-
[4]
Bank for International Settlements: Principles for financial market infrastructures: Executive summary (2023), https://www.bis.org/fsi/fsisumm aries/pfmi.htm
work page 2023
-
[5]
Bsharat, S., Myrzakhan, A., Shen, Z.: Principled instructions are all you need for questioning llama-1/2, gpt-3.5/4. arXiv preprint arXiv:2312.16171 (2023) 16 Accepted in the 8th International Conference on Machine Learning for Cyber Security 2026
arXiv 2023
-
[6]
Canadian Institute for Cybersecurity, University of New Brunswick: Cic-ids2017 in- trusion detection evaluation dataset (2017), https://www.unb.ca/cic/datasets/ ids-2017.html
work page 2017
-
[7]
Committee on Payments and Market Infrastructures & International Organization of Securities Commissions: Principles for financial market infrastructures (2012), https://www.bis.org/cpmi/publ/d101a.pdf
work page 2012
-
[8]
Daniel, N., Kaiser, F.K., Giladi, S., Sharabi, S., Moyal, R., Shpolyansky, S., Puzis, R.: Labeling network intrusion detection system (nids) rules with mitre att&ck techniques: Machine learning vs. large language models. Big Data and Cognitive Computing 9(2), 23 (2025), https://www.mdpi.com /2504 -2289/9/2/23
work page 2025
Show all 26 references
-
[9]
Computers & Security 45, 100–123 (2014)
García, S., Grill, M., Stiborek, J., Zunino, A.: An empirical comparison of botnet detection methods. Computers & Security 45, 100–123 (2014)
2014
-
[10]
Dhaka University Journal of Engineering and Technology 1(2), 121– 124 (2011)
Hasan, K., Ali, M., Nessa, M., Aditya, S., Mazumder, R.: Retrieval of surface reflectance from noaa-avhrr satellite data. Dhaka University Journal of Engineering and Technology 1(2), 121– 124 (2011)
2011
-
[11]
In: Pro- ceedings of the 49th Annual Precise Time and Time Interval Systems and Appli- cations Meeting
Hasan, K.F., Feng, Y., Tian, Y.C.: Exploring the potential and feasibility of time synchronization using gnss receivers in vehicleto-vehicle communications. In: Pro- ceedings of the 49th Annual Precise Time and Time Interval Systems and Appli- cations Meeting. pp. 80–90 (2018)
2018
-
[12]
IEEE Access 13, 217316– 217348 (2025)
Hasan, K.F., Shajeeb, H.H., Abeydeera, C., Turnbull, B., Warren, M.: Isadm: An integrated stride, att&ck, and d3fend model for threat modeling against real-world adversaries. IEEE Access 13, 217316– 217348 (2025)
2025
-
[13]
Cybersecurity 2(1), 1–22 (2019), https://link.springer.com/article/10.1186/s42400- 019- 0038- 7
Khraisat, A., Gondal, I., Vamplew, P., Kamruzzaman, J.: Survey of intrusion detec- tion systems: techniques, datasets and challenges. Cybersecurity 2(1), 1–22 (2019), https://link.springer.com/article/10.1186/s42400- 019- 0038- 7
2019 doi
-
[14]
Kinnunen, J.: Threat detection gap analysis using mitre att&ck framework (2022), https://www.theseus.fi/handle/10024/745250
2022
-
[15]
MITRE: Mitre att&ck v18 (2025), https://attack.mitre.org/
2025
-
[16]
South African Journal of Information Management 27(1), 2019 (2025), https: //journals.co.za/doi/full/10.4102/sajim.v27i1.2019
Miya, N.F., Joseph, N.: Banking on resilience: 20 years of cybersecurity evolution. South African Journal of Information Management 27(1), 2019 (2025), https: //journals.co.za/doi/full/10.4102/sajim.v27i1.2019
2025 doi
-
[17]
In: 2015 military communications and information systems conference (MilCIS)
Moustafa, N., Slay, J.: Unsw -nb15: a comprehensive data set for network intrusion detection systems (unsw-nb15 network data set). In: 2015 military communications and information systems conference (MilCIS). pp. 1 –6. IEEE (2015), https:// ieeexplore.ieee.org/abstract/doc ume...
2015
-
[18]
arXiv preprint arXiv:2411.06535 (2024)
Naik, N.: Probabilistic consensus through ensemble validation: A framework for llm reliability. arXiv preprint arXiv:2411.06535 (2024)
2024 arXiv
-
[19]
Computers & Security 153, 104777 (2025), https://doi.org/10.1016/j.cose.2025.104777
Rahman Tory, A., Hasan, K.F.: An evaluation framework for network ids/ips datasets: Leveraging mitre att&ck and industry relevance metrics. Computers & Security 153, 104777 (2025), https://doi.org/10.1016/j.cose.2025.104777
2025
-
[20]
Jour- nal of Big Data 12(1), 264 (2025), https://link.springer.com/article/10
Rehman, H.M.R.U., Liaquat, S., Gul, M.J., Jhandir, M.Z., Gavilanes, D., Vergara, M.M., Ashraf, I.: A systematic literature study of machine learning techniques based intrusion detection: datasets, models, challenges, and future directions. Jour- nal of Big Data 12(1), 264 (202...
2025
-
[21]
Scarfone, K., Mell, P.: Guide to intrusion detection and prevention systems (idps). Tech. Rep. NIST SP 800 -94, National Institute of Standards and Technology (2007), https://doi.org/10.6028/NIST.SP.800- 94
2007 doi
-
[22]
In: 2019 Inter - national Carnahan Conference on Security Technology (ICCST)
Sharafaldin, I., Lashkari, A.H., Hakak, S., Ghorbani, A.A.: Developing realistic distributed denial of service (ddos) attack dataset and taxonomy. In: 2019 Inter - national Carnahan Conference on Security Technology (ICCST). IEEE (2019) 17 Accepted in the 8th International Con...
2019
-
[23]
In: 2009 IEEE Symposium on Computational Intelligence for Security and Defense Applications
Tavallaee, M., Bagheri, E., Lu, W., Ghorbani, A.A.: A detailed analysis of the kdd cup 99 data set. In: 2009 IEEE Symposium on Computational Intelligence for Security and Defense Applications. IEEE (2009), https://ieeexplore.ieee.org/ abstract/document/5356528
2009
-
[24]
arXiv preprint arXiv:2511.00360 (2025), https://doi.org/10.48550/arXiv.2511.00360
Tory, A.R., Hasan, K.F., Rahman, M.S., Koroniotis, N., Moni, M.A.: Mind the gap: Missing cyber threat coverage in nids datasets for the energy sector. arXiv preprint arXiv:2511.00360 (2025), https://doi.org/10.48550/arXiv.2511.00360
2025 doi
-
[25]
In: Proceedings of the Fifteenth ACM Conference on Data and Application Secu - rity and Privacy
Wagner, E., Bader, L., Wolsing, K., Serror, M.: Sherlock: A dataset for process - aware intrusion detection research on power grid networks: Dataset paper. In: Proceedings of the Fifteenth ACM Conference on Data and Application Secu - rity and Privacy. pp. 419– 424 (2024), htt...
2024
-
[26]
arXiv preprint arXiv:2411.17058 (2024), https://arxiv.org/abs/2411.17058
Wu, T., Yang, S., Liu, S., Nguyen, D., Jang, S., Abuadbba, A.: Threatmodeling - llm: Automating threat modeling using large language models for banking system. arXiv preprint arXiv:2411.17058 (2024), https://arxiv.org/abs/2411.17058
2024 arXiv
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.