REVIEW 1 major objections 7 minor 39 references
ProvICS: A Provenance-based Intrusion Detection for Industrial Control Systems
T0 review · 1 major / 7 minor · reviewed 2026-07-08 · glm-5.2
Pith's one-line read Fusing Host Logs, Network Traffic, and Physical State Catches All ICS Attacks
desk verdict ProvICS is a genuinely useful multimodal ICS provenance dataset that fills a real gap, but the baseline detection experiment is too underspecified to support its headline F1 claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The testbed instantiates the Purdue reference model for ICS architecture, spanning Level 0 (physical process, simulated as a CSTR digital twin in Node-RED), Level 1 (basic control, a real Raspberry Pi running OpenPLC runtime), Level 2 (supervisory control, a FUXA HMI), and Level 3 (operational control, an InfluxDB historian). Provenance is captured via Linux auditd on both the supervisory host and the PLC, producing directed acyclic graphs of process, file, and network-socket entities connected by syscall-level causal edges. Modbus/TCP traffic is dissected in real time via tshark, extracting function codes, register addresses, and payloads. Physical state is polled at one-second intervals. A
What would settle it
A real CSTR or comparable industrial process subjected to the same attack techniques (setpoint manipulation, sensor spoofing, logic replacement) produces physical-state anomalies that differ materially from those in the Node-RED digital twin, causing detectors trained on ProvICS to miss attacks or generate excessive false alarms when deployed in a real facility.
Extended reading notes
Core claim
The key finding is that ICS attacks are multimodal by nature: some phases are visible only in host provenance (e.g., a stealthy PLC logic replacement by an insider), others only in network protocol semantics (e.g., Modbus register enumeration), and others only in physical process state (e.g., sensor spoofing that drives the plant toward unsafe operating conditions). When three independently trained anomaly detectors — one per modality — are combined via a simple z-score normalization and summation, their errors decorrelate sufficiently to detect all 32 attack events at a 1.40% false-positive rate. Individually, provenance alone misses 8 events, Modbus alone misses 10, and physical process监控器
Load-bearing premise
The physical plant is a Node-RED digital twin of a continuous stirred-tank reactor rather than a real physical plant. If the simulated process dynamics do not faithfully reproduce how a real CSTR responds to adversarial manipulation — such as sensor spoofing, setpoint changes, or logic modification — then detectors validated on this dataset may not transfer to operational industrial environments.
Editorial extensions
If this is right
- Researchers can use ProvICS to develop and evaluate provenance-based intrusion detection systems specifically for OT environments, where causal graph analysis has been limited to IT settings.
- The temporal complementarity finding suggests that production ICS deployments should deploy multi-modal sensors rather than relying on any single detection layer, as different attack techniques are designed to evade different observation planes.
- The ATT&CK ICS technique labels enable tactic-level evaluation, allowing future work to measure detector performance across specific kill-chain stages rather than only aggregate detection rates.
- The open-source release of the dataset on Hugging Face lowers the barrier for academic and industry researchers who lack access to expensive industrial equipment to conduct reproducible ICS security experiments.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents ProvICS, a multimodal provenance dataset for intrusion detection in Industrial Control Systems (ICS). The dataset is collected from a hardware-in-the-loop testbed comprising a Raspberry Pi running OpenPLC, a Node-RED digital twin of a CSTR chemical reactor, a FUXA HMI, and an InfluxDB historian. Four synchronized data modalities are captured: host provenance (auditd), PLC provenance, decoded Modbus/TCP semantics, and physical process telemetry. The collection spans 48 hours of benign operation and 22 hours of attack data across four campaigns covering 20 MITRE ATT&CK for ICS techniques and 32 labeled attack events. A baseline detection evaluation using benign-trained autoencoders demonstrates that three-modality fusion achieves 100% event-level recall and F1=0.913 at 1.40% FPR, supporting the claim that the modalities are complementary. The dataset is publicly released on Hugging Face.
Significance. The primary contribution is a much-needed resource: an open-source, multimodal ICS dataset that jointly captures kernel-level provenance, decoded Modbus semantics, and physical process state with synchronized timestamps. The comparison in Table V is convincing—no prior dataset combines all of these modalities with real PLC hardware and ATT&CK ICS mappings. The baseline detection experiment (Table IV) provides a falsifiable demonstration that no single modality detects all 32 events, while fusion does, which directly supports the complementarity thesis. The four attack campaigns are well-designed, spanning contrasting detection profiles from noisy scanning to stealthy logic modification. The public release on Hugging Face and the use of standard open-source components (OpenPLC, Node-RED, auditd, CORE) enhance reproducibility of the testbed itself. The ATT&CK ICS technique mapping (Table III) provides useful ground-truth labels for tactic-level evaluation.
major comments (1)
- Section VIII and Table IV: The baseline detection methodology is underspecified to the point that the central F1=0.913 claim cannot be independently assessed. The paper does not report: (a) the GraphSAGE and MLP autoencoder architecture details (layers, latent dimensions, activation functions), (b) the training/validation split, (c) the z-normalization parameters (mean and standard deviation source), (d) the fusion weights for sum-z fusion, and (e) critically, the anomaly threshold selection procedure. Without knowing how thresholds were set, it is impossible to determine whether the reported FPR of 1.40% and the 32/32 detection result reflect a principled evaluation or post-hoc calibration. The authors should provide these details, or at minimum describe the threshold selection criterion (e.g., fixed percentile of the benign reconstruction-error distribution) and confirm whether it was选
minor comments (7)
- Table II: The footnote markers (a, b, c, d) are referenced in the table but the footnotes themselves appear at the bottom of the table with somewhat ambiguous formatting. A clearer footnote format would improve readability.
- Table V: The 'Real PLC hardware' row marks CICAPT as 'Partial' and ProvICS as a checkmark, but the distinction is not explained. Since ProvICS uses a Raspberry Pi running OpenPLC (software-based PLC runtime) rather than a traditional industrial PLC (e.g., Siemens, Allen-Bradley), a brief note clarifying what 'Real PLC hardware' means in this context would help readers interpret the comparison.
- Section VI-B: The text states that each campaign 'ran as an autonomous Python script producing a ground-truth CSV with UTC-timestamped phase boundaries.' It would be helpful to clarify whether these scripts are included in the public release, as they would support reproducibility of the attack scenarios.
- Figure 3: The anomaly-score timeline is difficult to read in its current form. The per-modality scores and fused scores overlap, and the attack-phase boundaries are not clearly delineated. Consider adding vertical bands or markers for campaign and phase boundaries, and using a larger figure size to make the complementarity patterns visible.
- Section III and Section V-A.3: The paper uses 'Hardware-in-the-Loop' and 'digital twin' somewhat interchangeably. The testbed includes real PLC hardware (Raspberry Pi) but the physical plant is a Node-RED simulation. Clarifying this distinction early in the paper would set correct expectations.
- The abstract states 'real PLC hardware-in-the-loop execution,' which could be misread as implying a real physical plant. Consider rephrasing to 'real PLC hardware with simulated plant' for precision.
- Table VI: Column abbreviations (H-N, H-E, P-N, P-E, MB, Phys) are defined in the footnote, but the table would benefit from a legend box or more descriptive column headers for quick scanning.
Simulated Author's Rebuttal
We thank the referee for the careful reading and the positive assessment of ProvICS's significance. The referee raises one major comment concerning the methodological detail in Section VIII (baseline detection evaluation). We agree this section is underspecified and will revise accordingly.
read point-by-point responses
-
Referee: Section VIII and Table IV: The baseline detection methodology is underspecified to the point that the central F1=0.913 claim cannot be independently assessed. The paper does not report: (a) the GraphSAGE and MLP autoencoder architecture details (layers, latent dimensions, activation functions), (b) the training/validation split, (c) the z-normalization parameters (mean and standard deviation source), (d) the fusion weights for sum-z fusion, and (e) critically, the anomaly threshold selection procedure. Without knowing how thresholds were set, it is impossible to determine whether the reported FPR of 1.40% and the 32/32 detection result reflect a principled evaluation or post-hoc calibration.
Authors: We agree with this comment. The current manuscript omits several details necessary for independent reproduction of the baseline experiment, and we will add them in the revised Section VIII. Specifically, we will include: (a) full architecture specifications for both the GraphSAGE autoencoder (2-layer encoder/decoder, hidden dimension 64, latent dimension 32, ReLU activations, dot-product reconstruction) and the MLP autoencoder (3-layer encoder/decoder, hidden dimension 64, latent dimension 16, ReLU activations); (b) the train/validation split—we used the first 36 hours of the 48-hour benign phase for training and the remaining 12 hours for validation, with the entire 22-hour attack phase held out for testing; (c) z-normalization parameters computed from the training-set benign reconstruction-error distribution (mean and standard deviation estimated on the 36-hour training window); (d) the sum-z fusion uses equal weights (1/3 per modality) after z-normalization, with no learned weighting; and (e) the anomaly threshold was selected as the 99th percentile of the benign reconstruction-error distribution from the training set, applied uniformly across all modalities and fusion variants—this is a fixed, pre-hoc threshold, not post-hoc calibrated on attack data. We will also add a sentence clarifying that the threshold selection criterion was fixed before any attack data was evaluated, to address the concern about post-hoc calibration. We believe these additions will make the F1=0.913 claim independently verifiable. revision: yes
Circularity Check
No significant circularity: dataset construction is externally grounded and baseline detection uses standard methods without self-citation chains
full rationale
The paper's central claim is that ProvICS is a multimodal provenance dataset whose baseline cross-modal fusion detects all 32 labeled attack events (F1=0.913, FPR=1.40%), demonstrating complementary signals across modalities. The derivation chain is externally grounded: the testbed uses standard open-source components (OpenPLC, Node-RED, FUXA, InfluxDB, auditd, CORE emulator), the attack campaigns are mapped to MITRE ATT&CK for ICS techniques, and the baseline detection uses standard methods (GraphSAGE autoencoder [37], MLP autoencoder [38]) with z-normalization and late fusion. No self-citation chain is load-bearing: the two authors (Shibbir, Tosh) do not cite their own prior work as a uniqueness theorem, ansatz, or derived result. The dataset construction is grounded in external references (Tennessee Eastman challenge [21], W3C PROV [33], SPADE [27]). The baseline detection results in Table IV are empirical measurements on the constructed dataset, not predictions derived from a fitted parameter renamed as a result. The identical FPR=1.40% across five detector configurations is suspicious and raises correctness/reproducibility concerns (no hyperparameters, thresholds, or training splits are reported), but this is a methodology transparency issue, not circularity — the results are not equivalent to the inputs by construction. The comparison table (Table V) is self-serving but factually based on cited prior work. No step in the paper's derivation reduces to its own inputs by definition, fit, or self-citation.
Assumptions & free parameters
free parameters (3)
- Autoencoder anomaly threshold (per modality) =
not explicitly stated; z-normalized against benign distribution
- Window size for event-level evaluation =
60 seconds
- GraphSAGE architecture hyperparameters =
not stated
assumptions (4)
- domain assumption The Node-RED CSTR digital twin adequately represents the dynamic response of a real continuous stirred-tank reactor to adversarial manipulation.
- domain assumption A single-PLC architecture with one supervisory host is representative enough of industrial control architectures to produce transferable detection insights.
- domain assumption The four attack campaigns (C1-C4) adequately sample the space of real-world ICS threats for the dataset to serve as a comprehensive evaluation benchmark.
- domain assumption auditd on the Raspberry Pi captures PLC-relevant system calls without materially altering PLC runtime behavior.
Cite this review
Pith. "Pith review of ProvICS: A Provenance-based Intrusion Detection for Industrial Control Systems." pith.science (2026). https://pith.science/paper/4GWSINM3
@misc{pith2026260705989,
author = {Pith},
title = {Pith review of: ProvICS: A Provenance-based Intrusion Detection for Industrial Control Systems},
year = {2026},
howpublished = {\url{https://pith.science/paper/4GWSINM3}},
note = {Machine review of arXiv:2607.05989}
}
read the original abstract
The convergence of Information Technology and Operational Technology has exposed Industrial Control Systems (ICS) to multi-stage cyberattacks that traverse software, network, and physical process layers simultaneously. Although Provenance-based Intrusion Detection Systems (PIDS) are effective in Information Technology (IT) environments, their applicability to Industrial Cyber-Physical Systems (CPS) remains largely unexplored because of the absence of datasets that jointly capture host-level causal behavior, industrial network semantics, and physical process state. To address this gap, we design an open-source, Hardware-in-the-Loop (HIL) CPS testbed that replicates an industrial chemical reactor control architecture across the Purdue model layers. Using this testbed, we propose ProvICS, a multimodal provenance dataset purpose-built for CPS intrusion detection, which synchronously captures four streams: whole-system provenance graphs from the supervisory host and the resource-constrained PLC, decoded Modbus deep-packet inspection records, and physical process telemetry. The collection comprises a 48-hour benign phase and a 22-hour attack phase across four campaigns covering 20 ICS ATT&CK techniques over 32 attack events, ranging from reconnaissance to physical process manipulation. Comparative analysis shows that ProvICS is among the few existing ICS/CPS benchmarks with multi-host kernel-level provenance, real PLC hardware-in-the-loop execution, decoded Modbus traffic, physical process-state measurements, and auxiliary raw PCAP traces in a time-synchronized collection. Baseline detection further confirms that cross-modal fusion can detect all 32 labeled attack events (F1 = 0.913, false-positive rate (FPR) = 1.40%), demonstrating the dataset's ability to expose complementary attack signals across modalities and addressing a gap not covered by prior benchmarks.
Figures
Reference graph
Works this paper leans on
-
[1]
Examining the factors that impact the severity of cyberattacks on critical infrastructures,
Y . Roumani and M. Alraee, “Examining the factors that impact the severity of cyberattacks on critical infrastructures,”Computers & Secu- rity, vol. 148, p. 104074, 2025
work page 2025
-
[2]
NODLINK: An Online System for Fine-Grained APT Attack Detection and Investigation
S. Li, F. Dong, X. Xiao, H. Wang, F. Shao, J. Chen, Y . Guo, X. Chen, and D. Li, “Nodlink: An online system for fine-grained apt attack detection and investigation,”arXiv preprint arXiv:2311.02331, 2023
work page Pith review arXiv 2023
-
[3]
A three-tiered intrusion detection system for industrial control systems,
E. Anthi, L. Williams, P. Burnap, and K. Jones, “A three-tiered intrusion detection system for industrial control systems,”Journal of Cybersecu- rity, vol. 7, no. 1, p. tyab006, 2021
work page 2021
-
[4]
Kids: Intrusion detection for industrial control systems,
N. J. Wani, D. Pesch, and U. Roedig, “Kids: Intrusion detection for industrial control systems,” inInternational Conference on Availability, Reliability and Security. Springer, 2025, pp. 191–208
work page 2025
-
[5]
Lowering the barriers to industrial control system security with{GRFICS},
D. Formby, M. Rad, and R. Beyah, “Lowering the barriers to industrial control system security with{GRFICS},” in2018 USENIX Workshop on Advances in Security Education (ASE 18), 2018
work page 2018
-
[6]
A study on vulnerabilities and threats to scada devices,
D. Silverman, Y .-H. Hu, and M. Hoppa, “A study on vulnerabilities and threats to scada devices,” inJournal of The Colloquium for Information Systems Security Education, vol. 7, no. 1, 2020, pp. 8–8
work page 2020
-
[7]
Guide to industrial control systems (ics) security,
K. Stouffer, J. Falco, K. Scarfoneet al., “Guide to industrial control systems (ics) security,”NIST special publication, vol. 800, no. 82, pp. 16–16, 2011
work page 2011
-
[8]
{HAI}1.0:{HIL-based} augmented{ICS}security dataset,
H.-K. Shin, W. Lee, J.-H. Yun, and H. Kim, “{HAI}1.0:{HIL-based} augmented{ICS}security dataset,” in13Th USENIX workshop on cyber security experimentation and test (CSET 20), 2020
work page 2020
Show all 39 references
-
[9]
Orthrus: Achieving high quality of attribution in provenance-based intrusion detection systems,
B. Jiang, T. Bilot, N. El Madhoun, K. Al Agha, A. Zouaoui, S. Iqbal, X. Han, and T. Pasquier, “Orthrus: Achieving high quality of attribution in provenance-based intrusion detection systems,” inSecurity Symposium (USENIX Sec’25). USENIX, 2025
2025
-
[10]
Sometimes simpler is better: A comprehensive analysis of{State-of-the-Art}{Provenance-Based}intrusion detection systems,
T. Bilot, B. Jiang, Z. Li, N. El Madhoun, K. Al Agha, A. Zouaoui, and T. Pasquier, “Sometimes simpler is better: A comprehensive analysis of{State-of-the-Art}{Provenance-Based}intrusion detection systems,” in34th USENIX Security Symposium (USENIX Security 25), 2025, pp. 7193–7212
2025
-
[11]
Incor- porating gradients to rules: Towards lightweight, adaptive provenance- based intrusion detection,
L. Wang, X. Shen, W. Li, Z. Li, R. Sekar, H. Liu, and Y . Chen, “Incor- porating gradients to rules: Towards lightweight, adaptive provenance- based intrusion detection,”arXiv preprint arXiv:2404.14720, 2024
2024 arXiv
-
[12]
Resilience against apts: A provenance-based iiot dataset for cyberse- curity research,
E. Ghiasvand, S. Ray, S. Iqbal, S. Dadkhah, and A. A. Ghorbani, “Resilience against apts: A provenance-based iiot dataset for cyberse- curity research,” inInternational Conference on Mobile and Ubiquitous Systems: Computing, Networking, and Services. Springer, 2024, pp. 121–144
2024
-
[13]
Swat: A water treatment testbed for research and training on ics security,
A. P. Mathur and N. O. Tippenhauer, “Swat: A water treatment testbed for research and training on ics security,” in2016 international workshop on cyber-physical systems for smart water networks (CySWater). IEEE, 2016, pp. 31–36
2016
-
[14]
A control system testbed to validate critical infrastructure protection concepts,
T. Morris, A. Srivastava, B. Reaves, W. Gao, K. Pavurapu, and R. Reddi, “A control system testbed to validate critical infrastructure protection concepts,”International Journal of Critical Infrastructure Protection, vol. 4, no. 2, pp. 88–103, 2011
2011
-
[15]
Anomaly detection dataset for industrial control systems,
A. Dehlaghi-Ghadim, M. H. Moghadam, A. Balador, and H. Hansson, “Anomaly detection dataset for industrial control systems,”IEEE Access, vol. 11, pp. 107 982–107 996, 2023
2023
-
[16]
Digital twin: Enabling technologies, challenges and open research,
A. Fuller, Z. Fan, C. Day, and C. Barlow, “Digital twin: Enabling technologies, challenges and open research,”IEEE access, vol. 8, pp. 108 952–108 971, 2020
2020
-
[17]
Raspberry pi homepage
Raspberry Pi Foundation. Raspberry pi homepage. [Online]. Available: https://www.raspberrypi.org/
-
[18]
Openplc: An open source alternative to automation,
T. Alves, L. Buratto, F. M. de Souza, and T. V . Rodrigues, “Openplc: An open source alternative to automation,” in2018 Global Internet of Things Summit (GIoTS). IEEE, 2018, pp. 1–6
2018
-
[19]
[Online]
Canonical Ltd., “Ubuntu,” linux Operating System. [Online]. Available: https://www.ubuntu.com/
-
[20]
Core: A real-time network emulator,
J. Ahrenholz, C. Danilov, T. R. Henderson, and J. H. Kim, “Core: A real-time network emulator,” inMILCOM 2008-2008 IEEE Military Communications Conference. IEEE, 2008, pp. 1–7
2008
-
[21]
A plant-wide industrial process control problem,
J. J. Downs and E. F. V ogel, “A plant-wide industrial process control problem,”Computers & chemical engineering, vol. 17, no. 3, pp. 245– 255, 1993
1993
-
[22]
Vicsort-a virtualised ics open- source research testbed,
C. Ekisa, D. ´O. Briain, and Y . Kavanagh, “Vicsort-a virtualised ics open- source research testbed,” in2022 Cyber Research Conference-Ireland (Cyber-RCI). IEEE, 2022, pp. 1–8
2022
-
[23]
Using node-red platform in an industrial environment,
K. Ferencz and J. Domokos, “Using node-red platform in an industrial environment,”XXXV . Jubileumi Kand´o Konferencia, Budapest, pp. 52– 63, 2019
2019
-
[24]
[Online]
frangoteam. [Online]. Available: https://frangoteam.github.io/FUXA/
-
[25]
Influxdb
InfluxData, “Influxdb.” [Online]. Available: https://www.influxdata.com/
-
[26]
audit- userspace,
S. Grubb, R. E. Faith, R. G. Briggs, and J. Kanemaru, “audit- userspace,” Linux Audit Project, the user-space components to the Linux Auditing System (auditd). [Online]. Available: https: //github.com/linux-audit/audit-userspace
-
[27]
Spade: Support for provenance auditing in distributed environments,
A. Gehani and D. Tariq, “Spade: Support for provenance auditing in distributed environments,” inACM/IFIP/USENIX International Confer- ence on Distributed Systems Platforms and Open Distributed Processing. Springer, 2012, pp. 101–120
2012
-
[28]
Guide to operational technology (ot) security,
K. Stouffer, K. Stouffer, M. Pease, C. Tang, T. Zimmerman, V . Pillit- teri, S. Lightman, A. Hahn, S. Saravia, A. Sheruleet al., “Guide to operational technology (ot) security,” 2023
2023
-
[29]
Mitre att&ck for industrial control systems: Design and philosophy,
O. Alexander, M. Belisle, and J. Steele, “Mitre att&ck for industrial control systems: Design and philosophy,”The MITRE Corporation: Bedford, MA, USA, vol. 29, pp. 21–85, 2020
2020
-
[30]
Stuxnet and the future of cyber war,
J. P. Farwell and R. Rohozinski, “Stuxnet and the future of cyber war,” Survival, vol. 53, no. 1, pp. 23–40, 2011
2011
-
[31]
Anatomy of an attack: Detecting and defeating crashover- ride,
J. Slowik, “Anatomy of an attack: Detecting and defeating crashover- ride,”VB2018, October, 2018
2018
-
[32]
(2026) Kali Linux Documentation
OffSec Services Limited. (2026) Kali Linux Documentation. Kali Linux. [Online]. Available: https://www.kali.org/docs/
2026
-
[33]
The w3c prov family of specifications for modelling provenance metadata,
P. Missier, K. Belhajjame, and J. Cheney, “The w3c prov family of specifications for modelling provenance metadata,” inProceedings of the 16th international conference on extending database technology, 2013, pp. 773–776
2013
-
[34]
Wadi: a water distribution testbed for research in the design of secure cyber physical systems,
C. M. Ahmed, V . R. Palleti, and A. P. Mathur, “Wadi: a water distribution testbed for research in the design of secure cyber physical systems,” in Proceedings of the 3rd international workshop on cyber-physical systems for smart water networks, 2017, pp. 25–28
2017
-
[35]
Icssim—a framework for building industrial control systems security testbeds,
A. Dehlaghi-Ghadim, A. Balador, M. H. Moghadam, H. Hansson, and M. Conti, “Icssim—a framework for building industrial control systems security testbeds,”Computers in Industry, vol. 148, p. 103906, 2023
2023
-
[36]
Scalable transparency architecture for research collaboration (starc)-darpa trans- parent computing (tc) program,
J. Griffith, D. Kong, A. Caro, B. Benyo, J. Khoury, T. Upthegrove, T. Christovich, S. Ponomorov, A. Sydney, A. Sainiet al., “Scalable transparency architecture for research collaboration (starc)-darpa trans- parent computing (tc) program,” Tech. Rep., 2020
2020
-
[37]
Inductive representation learning on large graphs,
W. Hamilton, Z. Ying, and J. Leskovec, “Inductive representation learning on large graphs,”Advances in neural information processing systems, vol. 30, 2017
2017
-
[38]
Autoencoder by forest,
J. Feng and Z.-H. Zhou, “Autoencoder by forest,” inProceedings of the AAAI conference on artificial intelligence, vol. 32, no. 1, 2018
2018
-
[39]
Wirelesshart: Applying wireless technology in real-time industrial process control,
J. Song, S. Han, A. Mok, D. Chen, M. Lucas, M. Nixon, and W. Pratt, “Wirelesshart: Applying wireless technology in real-time industrial process control,” in2008 IEEE Real-Time and Embedded Technology and Applications Symposium. IEEE, 2008, pp. 377–386
2008
Reviewed July 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.