Recognition: unknown
SoK: Reshaping Research on Network Intrusion Detection Systems
Pith reviewed 2026-05-10 06:07 UTC · model grok-4.3
The pith
Misunderstandings of NIDS core properties create a wide gap between academic research and operational security practice.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The disconnection between NIDS research and practice arises from a fundamental misunderstanding of intrinsic NIDS characteristics, which the paper captures in three Assertions: a compromised NIDS cannot be expected to work well, evaluations must involve experiments in real or synthetic networks, and operators triage high-level reports rather than individual flagged samples. Recommendations follow to realign future work.
What carries the argument
Three Assertions that state quintessential properties of NIDS without criticizing specific prior works, serving as the foundation for recommendations and the case study.
Load-bearing premise
The three Assertions capture the primary causes of the research-practice gap and that following the recommendations will meaningfully reshape NIDS research toward operational relevance.
What would settle it
A large-scale survey of NIDS operators and researchers that measures the extent to which current papers address the three Assertions, or a controlled experiment showing that papers following the recommendations see higher adoption rates in practice.
Figures
read the original abstract
Network Intrusion Detection Systems (NIDS) have been studied for decades. Hundreds of papers have, e.g., proposed ways to enhance, harden or bypass NIDS. However, the findings of prior literature are hardly reflected in real-world operational contexts. Such a disconnection is problematic for research itself: it is unclear what scenario envisioned by prior work can be used as a baseline for future advancements. We argue that a key reason for this disconnection is a fundamental misunderstanding of intrinsic characteristics of NIDS. For instance, the fact that a compromised NIDS cannot be expected to work well; the fact that some evaluations are done without carrying out any experiment in a (even synthetic) "real" network; the fact that security operators triage high-level reports -- and not individual samples flagged by some classifier. In this SoK, which is primarily a reflective piece, we first constructively highlight such quintessential properties (without criticizing _any_ work by different authors) by stating three Assertions. Then, we provide recommendations -- further emphasized through an original and reproducible case study that challenges some established practices. Ultimately, we seek to lay a foundation to reshape research on NIDS.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript is a reflective Systematization of Knowledge (SoK) on Network Intrusion Detection Systems (NIDS). It argues that the long-standing disconnection between academic NIDS research and real-world operational practice stems from a fundamental misunderstanding of intrinsic NIDS characteristics. The authors constructively articulate three Assertions—(1) a compromised NIDS cannot be expected to work well, (2) many evaluations proceed without experiments in even synthetic real networks, and (3) operators triage high-level reports rather than individual classifier outputs—then derive recommendations for more relevant research, illustrated by an original reproducible case study that challenges established practices. The goal is to lay a foundation for reshaping future NIDS work toward operational utility.
Significance. If the three Assertions hold as representative observations of operational realities, the paper offers meaningful significance by shifting NIDS research away from incremental classifier tweaks toward scenarios that respect deployment constraints. The reproducible case study provides concrete, falsifiable grounding that strengthens the recommendations and could serve as a template for future work. This constructive, non-critical framing and emphasis on reproducibility are clear strengths that may help close the research-practice gap without requiring new empirical data.
minor comments (2)
- [Recommendations section] The transition from the three Assertions to the specific recommendations would benefit from an explicit mapping (e.g., which recommendation addresses which Assertion) to make the logical flow more transparent for readers.
- [Case study] In the case-study section, additional detail on the synthetic network topology, traffic generation parameters, and exact metrics used to demonstrate the challenge to established practices would improve replicability, even though the study is stated to be reproducible.
Simulated Author's Rebuttal
We thank the referee for the positive and constructive review. We are pleased that the significance of the three Assertions, the emphasis on operational realities, and the reproducible case study have been recognized as a foundation for reshaping NIDS research. The recommendation for minor revision is noted.
Circularity Check
No significant circularity
full rationale
The paper is a reflective SoK that states three Assertions as observations drawn from operational NIDS realities (compromised NIDS unreliability, lack of real-network experiments, operator triage of high-level reports) and derives recommendations from them. No mathematical derivations, equations, fitted parameters, or predictions appear; the Assertions are not defined in terms of the paper's outputs, nor are they justified via self-citation chains that reduce to unverified inputs. The case study is presented as original and reproducible, providing independent illustrative content. The derivation chain is therefore self-contained against external benchmarks of real-world security operations.
Axiom & Free-Parameter Ledger
axioms (2)
- domain assumption NIDS in real networks are subject to compromise and therefore cannot be assumed to function correctly when attacked.
- domain assumption Security operators triage high-level reports rather than individual classifier outputs.
Forward citations
Cited by 2 Pith papers
-
MCPShield: Content-Aware Attack Detection for LLM Agent Tool-Call Traffic
MCPShield models MCP tool-call sessions as graphs with SBERT embeddings and shows that content features raise AUROC above 0.89 while tree ensembles on pooled embeddings reach 0.975, outperforming GNNs and exposing inf...
-
MCPShield: Content-Aware Attack Detection for LLM Agent Tool-Call Traffic
MCPShield detects attacks on LLM agent tool-call traffic by encoding sessions as graphs enriched with SBERT content embeddings, achieving AUROC above 0.89 with content features versus 0.64 for metadata alone.
Reference graph
Works this paper leans on
-
[1]
Internet Assigned Numbers Authority
2011. Internet Assigned Numbers Authority. https://www.iana.org/assignmen ts/service-names-port-numbers/service-names-port-numbers.txt
2011
-
[2]
2012. Hulk. https://github.com/grafov/hulk
2012
-
[3]
GoldenEye
2014. GoldenEye. https://github.com/jseidl/GoldenEye
2014
-
[4]
slowhttptest
2016. slowhttptest. https://github.com/shekyan/slowhttptest
2016
-
[5]
2017. Ares. https://github.com/sweetsoftware/Ares
2017
-
[6]
Intrusion detection evaluation dataset (CIC-IDS2017)
2017. Intrusion detection evaluation dataset (CIC-IDS2017). https://www.unb. ca//cic/datasets/ids-2017.html
2017
-
[7]
2017. Patator. https://github.com/lanjelot/patator
2017
-
[8]
CSE-CIC-IDS2018 on AWS
2018. CSE-CIC-IDS2018 on AWS. https://www.unb.ca//cic/datasets/ids- 2018.html
2018
-
[9]
2018. Loic. https://www.cloudflare.com/learning/ddos/ddos-attack-tools/low- orbit-ion-cannon-loic/
2018
-
[10]
HeartBleed
2020. HeartBleed. https://cheese-hub.github.io/secure-coding/03-heartbleed/i ndex.html
2020
-
[11]
2024. Vulnhub. https://github.com/vulhub/vulhub
2024
-
[12]
Repository of this paper
2026. Repository of this paper. https://github.com/hihey54/asiaccs26_sok
2026
-
[13]
Cristina Abad, Jed Taylor, Cigdem Sengul, William Yurcik, Yuanyuan Zhou, and Ken Rowe. 2003. Log correlation for intrusion detection: A proof of concept. In ACSAC
2003
-
[14]
Bushra A Alahmadi, Louise Axon, and Ivan Martinovic. 2022. 99% False Positives: A Qualitative Study of SOC Analysts’ Perspectives on Security Alarms. In USENIX SEC
2022
-
[15]
Hisham Alasmary, Aminollah Khormali, Afsah Anwar, Jeman Park, Jinchun Choi, Ahmed Abusnaina, Amro Awad, Daehun Nyang, and Aziz Mohaisen. 2019. Analyzing and detecting emerging Internet of Things malware: A graph-based approach.IEEE Internet of Things Journal6, 5 (2019), 8977–8988
2019
-
[16]
Abdulellah Alsaheel, Yuhong Nan, Shiqing Ma, Le Yu, Gregory Walkup, Z Berkay Celik, Xiangyu Zhang, and Dongyan Xu. 2021. ATLAS : A sequence-based learning approach for attack investigation. In30th USENIX security symposium (USENIX security 21). 3005–3022
2021
-
[17]
2001.Security engineering: a guide to building dependable distributed systems
Ross J Anderson. 2001.Security engineering: a guide to building dependable distributed systems
2001
-
[18]
Giuseppina Andresini, Feargus Pendlebury, Fabio Pierazzi, Corrado Loglisci, Annalisa Appice, and Lorenzo Cavallaro. 2021. Insomnia: Towards concept- drift robustness in network intrusion detection. InACM workshop on Artificial Intelligence and Security. 111–122
2021
-
[19]
Real Attackers Don’t Compute Gradients
Giovanni Apruzzese, Hyrum S Anderson, Savino Dambra, David Freeman, Fabio Pierazzi, and Kevin Roundy. 2023. “Real Attackers Don’t Compute Gradients”: Bridging the Gap Between Adversarial ML Research and Practice. InSaTML
2023
-
[20]
Giovanni Apruzzese, Mauro Andreolini, Luca Ferretti, Mirco Marchetti, and Michele Colajanni. 2021. Modeling realistic adversarial attacks against network intrusion detection systems.ACM Digital Threats: Research and Practice(2021)
2021
-
[21]
Giovanni Apruzzese, Mauro Andreolini, Mirco Marchetti, Andrea Venturi, and Michele Colajanni. 2020. Deep reinforcement adversarial learning against botnet evasion attacks.IEEE Transactions on Network and Service Management(2020)
2020
-
[22]
Giovanni Apruzzese, Aurore Fass, and Fabio Pierazzi. 2024. When adversarial perturbations meet concept drift: an exploratory analysis on ml-nids. InACM AISec
2024
-
[23]
Giovanni Apruzzese, Pavel Laskov, and Johannes Schneider. 2023. Sok: Prag- matic assessment of machine learning for network intrusion detection. InIEEE EuroS&P
2023
-
[24]
Giovanni Apruzzese, Luca Pajola, and Mauro Conti. 2022. The cross-evaluation of machine learning-based network intrusion detection systems.IEEE TNSM (2022)
2022
-
[25]
Giovanni Apruzzese et al. 2022. The Role of Machine Learning in Cybersecurity. ACM DTRAP(2022)
2022
-
[26]
Systems for Machine Learning
Ignacio Arnaldo and Kalyan Veeramachaneni. 2019. The Holy Grail of "Systems for Machine Learning" Teaming humans and machine learning for detecting cyber threats.ACM SIGKDD Explorations Newsletter(2019)
2019
-
[27]
Daniel Arp, Erwin Quiring, Feargus Pendlebury, Alexander Warnecke, Fabio Pierazzi, Christian Wressnegger, Lorenzo Cavallaro, and Konrad Rieck. 2022. Dos and don’ts of machine learning in computer security. InUSENIX Security
2022
-
[28]
2018.Operating sys- tems: Three easy pieces
Remzi H Arpaci-Dusseau and Andrea C Arpaci-Dusseau. 2018.Operating sys- tems: Three easy pieces
2018
-
[29]
Stefan Axelsson. 2000. The base-rate fallacy and the difficulty of intrusion detection.ACM Transactions on Information and System Security (TISSEC)(2000)
2000
-
[30]
Md Ahsan Ayub, William A Johnson, Douglas A Talbert, and Ambareen Siraj
-
[31]
In2020 54th annual conference on information sciences and systems (CISS)
Model evasion attack on intrusion detection systems using adversarial machine learning. In2020 54th annual conference on information sciences and systems (CISS). 1–6
-
[32]
Rebecca Bace and Peter Mell. 2001. Intrusion Detection Systems.NIST Special Publication on Intrusion Detection Systems(2001)
2001
-
[33]
Tao Ban, Takeshi Takahashi, Samuel Ndichu, and Daisuke Inoue. 2023. Break- ing alert fatigue: AI-assisted SIEM framework for effective incident response. Applied Sciences13, 11 (2023), 6610
2023
-
[34]
Diogo Barradas, Nuno Santos, Luís Rodrigues, Salvatore Signorello, Fer- nando MV Ramos, and André Madeira. 2021. FlowLens: Enabling Efficient Flow Classification for ML-based Network Security Applications.. InNDSS
2021
-
[35]
Mohan Baruwal Chhetri, Shahroz Tariq, Ronal Singh, Fatemeh Jalalvand, Cecile Paris, and Surya Nepal. 2024. Towards human-AI teaming to mitigate alert fatigue in security operations centres.ACM Transactions on Internet Technology 24, 3 (2024), 1–22
2024
-
[36]
Elmarie Biermann, Elsabe Cloete, and Lucas M Venter. 2001. A comparison of intrusion detection systems.Computers & Security(2001)
2001
-
[37]
Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šrndić, Pavel Laskov, Giorgio Giacinto, and Fabio Roli. 2013. Evasion attacks against machine learning at test time. InECMLKDD
2013
-
[38]
Battista Biggio and Fabio Roli. 2018. Wild patterns: Ten years after the rise of adversarial machine learning.Pattern Recognition(2018)
2018
-
[39]
Abdullah Bin Jasni, Akiko Manada, and Kohei Watabe. 2024. DiffuPac: Contex- tual Mimicry in Adversarial Packets Generation via Diffusion Model.NeurIPS (2024)
2024
-
[40]
Philipp Bönninghausen, Rafael Uetz, and Martin Henze. 2024. Introducing a Comprehensive, Continuous, and Collaborative Survey of Intrusion Detection Datasets. InCyber Secur. Exp. and Test Workshop
2024
-
[41]
Robert A Bridges, Tarrah R Glass-Vanderlan, Michael D Iannacone, Maria S Vin- cent, and Qian Chen. 2019. A survey of intrusion detection systems leveraging host data.ACM CSUR(2019)
2019
-
[42]
Blake D Bryant and Hossein Saiedian. 2020. Improving SIEM alert metadata aggregation with a novel kill-chain based classification model.Computers & Security94 (2020), 101817
2020
-
[43]
Marta Catillo, Antonio Pecchia, Antonio Repola, and Umberto Villano. 2024. Towards realistic problem-space adversarial attacks against machine learning in network intrusion detection. InARES
2024
-
[44]
Marta Catillo, Antonio Pecchia, and Umberto Villano. 2023. Machine learning on public intrusion datasets: Academic hype or concrete advances in NIDS?. In 12 SoK: Reshaping Research on Network Intrusion Detection Systems ASIA CCS ’26, June 1–5, 2026, Bangalore, India DSN-S
2023
-
[45]
Paolo Cerracchio, Stefano Longari, Michele Carminati, Stefano Zanero, et al
-
[46]
InSymposium on Vehicles Security and Privacy (VehicleSec)
Investigating the impact of evasion attacks against automotive intrusion detection systems. InSymposium on Vehicles Security and Privacy (VehicleSec)
-
[47]
Fabrício Ceschin, Marcus Botacin, Albert Bifet, Bernhard Pfahringer, Luiz S Oliveira, Heitor Murilo Gomes, and André Grégio. 2024. Machine learning (in) security: A stream of problems.DTRAP(2024)
2024
-
[48]
Fabrício Ceschin, Marcus Botacin, Heitor Murilo Gomes, Luiz S Oliveira, and André Grégio. 2019. Shallow security: On the creation of adversarial variants to evade machine learning-based malware detectors. InROOTS
2019
-
[49]
Pin-Yu Chen, Shin-Ming Cheng, and Kwang-Cheng Chen. 2012. Smart attacks in smart grid communication networks.IEEE Communications Magazine50, 8 (2012), 24–29
2012
-
[50]
Zijun Cheng, Qiujian Lv, Jinyuan Liang, Yan Wang, Degang Sun, Thomas Pasquier, and Xueyuan Han. 2024. Kairos: Practical intrusion detection and investigation using whole-system provenance. InIEEE Symposium on Security and Privacy (SP)
2024
-
[51]
Henry Clausen, Robert Flood, and David Aspinall. 2019. Traffic generation using containerization for machine learning. InWorkshop on DYnamic and Novel Advances in Machine Learning and Intelligent Cyber Security
2019
-
[52]
Carlos Garcia Cordero, Sascha Hauke, Max Mühlhäuser, and Mathias Fischer
-
[53]
InIEEE PST
Analyzing flow-based anomaly intrusion detection using replicator neural networks. InIEEE PST
-
[54]
Carlos Garcia Cordero, Emmanouil Vasilomanolakis, Aidmar Wainakh, Max Mühlhäuser, and Simin Nadjm-Tehrani. 2021. On generating network traffic datasets with synthetic attacks for intrusion detection.ACM Transactions on Privacy and Security(2021)
2021
-
[55]
Jordan Cropper, Johanna Ullrich, Peter Frühwirt, and Edgar Weippl. 2015. The role and security of firewalls in iaas cloud computing. InARES
2015
-
[56]
Levente Csikor, Himanshu Singh, Min Suk Kang, and Dinil Mon Divakaran. 2021. Privacy of DNS-over-HTTPS: Requiem for a Dream?. In2021 IEEE European Symposium on Security and Privacy (EuroS&P). IEEE Computer Society, 252–271
2021
-
[57]
Savino Dambra, Yufei Han, Simone Aonzo, Platon Kotzias, Antonino Vitale, Juan Caballero, Davide Balzarotti, and Leyla Bilge. 2023. Decoding the secrets of machine learning in malware classification: A deep dive into datasets, feature extraction, and model performance. InCCS
2023
-
[58]
Hervé Debar, Marc Dacier, and Andreas Wespi. 1999. Towards a taxonomy of intrusion-detection systems.Computer networks(1999)
1999
-
[59]
Hervé Debar and Andreas Wespi. 2001. Aggregation and correlation of intrusion- detection alerts. InRAID
2001
-
[60]
Dorothy E Denning. 1987. An intrusion-detection model.IEEE TSE(1987)
1987
-
[61]
Alec F Diallo and Paul Patras. 2024. Sabre: Cutting through Adversarial Noise with Adaptive Spectral Filtering and Input Reconstruction. InIEEE S&P
2024
-
[62]
Christian Dietz, Raphael Labaca Castro, Jessica Steinberger, Cezary Wilczak, Marcel Antzek, Anna Sperotto, and Aiko Pras. 2018. IoT-botnet detection and isolation by access routers. In2018 9th International Conference on the Network of the Future (NOF). IEEE, 88–95
2018
-
[63]
Manuel Egele, Martin Szydlowski, Engin Kirda, and Christopher Kruegel. 2006. Using static program analysis to aid intrusion detection. InDIMV A
2006
- [64]
-
[65]
Gints Engelen, Vera Rimmer, and Wouter Joosen. 2021. Troubleshooting an intrusion detection dataset: the CICIDS2017 case study. InIEEE S&PW
2021
-
[66]
Alessandro Erba, Andres F Murillo, Riccardo Taormina, Stefano Galelli, and Nils Ole Tippenhauer. 2024. On Practical Realization of Evasion Attacks for Industrial Control Systems. InProceedings of the 2024 Workshop on Re-design Industrial Control Systems with Security
2024
-
[67]
Alessandro Erba, Riccardo Taormina, Stefano Galelli, Marcello Pogliani, Michele Carminati, Stefano Zanero, and Nils Ole Tippenhauer. 2020. Constrained con- cealment attacks against reconstruction-based anomaly detectors in industrial control systems. InACSAC
2020
-
[68]
Robert Flood, Gints Engelen, David Aspinall, and Lieven Desmet. 2024. Bad design smells in benchmark nids datasets. InEuroS&P
2024
-
[69]
Anderson Frasão, Tiago Heinrich, Vinicius Fulber-Garcia, Newton C Will, Rafael R Obelheiro, and Carlos A Maziero. 2024. I See Syscalls by the Seashore: An Anomaly-based IDS for Containers Leveraging Sysdig Data. InISCC
2024
-
[70]
Clement Fung, Eric Zeng, and Lujo Bauer. 2024. Attributions for ML-based ICS anomaly detection: From theory to practice. InProc. 31st Netw. Distrib. Syst. Secur. Symp
2024
-
[71]
Gustavo González-Granadillo, Susana González-Zarzosa, and Rodrigo Diaz
-
[72]
Security information and event management (SIEM): Analysis, trends, and usage in critical infrastructures.Sensors(2021)
2021
-
[73]
David Grochocki, Jun Ho Huh, Robin Berthier, Rakesh Bobba, William H Sanders, Alvaro A Cárdenas, and Jorjeta G Jetcheva. 2012. AMI threats, intrusion de- tection requirements and deployment recommendations. In2012 IEEE Third International Conference on Smart Grid Communications (SmartGridComm). IEEE, 395–400
2012
-
[74]
Eric Gyamfi and Anca Delia Jurcut. 2022. Novel online network intrusion detection system for industrial IoT based on OI-SVDD and AS-ELM.IEEE Internet of Things Journal(2022)
2022
-
[75]
Dongqi Han, Zhiliang Wang, Ying Zhong, Wenqi Chen, Jiahai Yang, Shuqiang Lu, Xingang Shi, and Xia Yin. 2021. Evaluating and improving adversarial robustness of machine learning-based network intrusion detectors.IEEE Journal on Selected Areas in Communications(2021)
2021
-
[76]
Mark Handley, Vern Paxson, and Christian Kreibich. 2001. Network Intru- sion Detection: Evasion, Traffic Normalization, and End-to-End Protocol Semantics. InUSENIX SEC
2001
-
[77]
Ahmad Hariri, Murat Yuksel, and David Mohaisen. 2024. RL-Based Speculative Installation of Unseen Flows in SDNs for Low-Latency Applications. In2024 IEEE International Conference on Machine Learning for Communication and Networking (ICMLCN). IEEE, 250–256
2024
-
[78]
Yiling He, Jian Lou, Zhan Qin, and Kui Ren. 2023. Finer: Enhancing state- of-the-art classifiers with feature attribution to facilitate security analysis. In CCS
2023
-
[79]
Hwanjo Heo and Seungwon Shin. 2018. Who is knocking on the telnet port: A large-scale empirical study of network scanning. InProceedings of the 2018 on Asia Conference on Computer and Communications Security. 625–636
2018
-
[80]
Grant Ho, Mayank Dhiman, Devdatta Akhawe, Vern Paxson, Stefan Savage, Geoffrey M Voelker, and David Wagner. 2021. Hopper: Modeling and detecting lateral movement. InUSENIX Security Symposium
2021
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.