REVIEW 4 major objections 6 minor 14 references
Poster: Enhancing GNN Robustness for Network Intrusion Detection via Agent-based Analysis
T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A large language model acting as a graph pre-filter can identify nearly all adversarially injected nodes in a netflow graph and restore the downstream GNN intrusion detector's accuracy to clean levels.
desk verdict A plausible poster-scale idea with an under-controlled single experiment; the LLM pre-filtering result needs baselines and prompt/attack disclosure before the robustness claim is credible. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the LLM as a textual pre-filter. Network flows are converted into an IP-centric graph, the graph's elements are rendered as text describing node interactions and flow characteristics, and the LLM is prompted to assess each element's relevance or maliciousness and return a flag or score. Flagged elements are removed or down-weighted before the GNN runs, so the attack is neutralized in the input space rather than by retraining the detector. The node-injection attack itself, where an attacker adds extra nodes to the graph, is the test case that makes the mechanism observable, and the unified dataset built from three netflow corpora is what makes the drift comparison possible.
What would settle it
Run the Table II pipeline with injected nodes whose degree, port, and byte statistics are matched to the benign distribution and with the LLM prompt blinded to any attack metadata; if correct-flagging falls to chance while the GNN accuracy stays degraded, the central claim fails.
Extended reading notes
Core claim
The core experimental result is that putting an LLM in front of a GNN restores the GNN's accuracy after a node-injection attack. On a 1000-node base graph with 200 injected nodes, GPT-4o and LLaMA 4 correctly flagged a high percentage of the injected nodes (197 and 200 in the reported 'correctly flagged' counts, with 58 and 69 false positives), and the CAGN-GAT Fusion GNN's accuracy and F1 recovered from the attacked state to 0.857/0.838 for GPT-4o and 0.859/0.834 for LLaMA 4, compared with a clean baseline of 0.842/0.821. The authors interpret this as evidence that individual LLMs can serve as complementary expert-analysis layers for GNN-based intrusion detection, and they generalize the idea to a larger agentic pipeline in which a strong model generates context for cheaper models.
Load-bearing premise
The approach assumes that an LLM given a textual description of a network graph can identify injected malicious nodes by genuine reasoning about flow structure rather than through an accidental cue such as prompt leakage or a statistically obvious outlier pattern.
Editorial extensions
If this is right
- Because the LLM filter operates on the input graph before classification, it can be attached to any GNN-based intrusion detector without retraining the underlying model.
- The unified-dataset results support the paper's warning that single-dataset benchmarks overstate GNN robustness under distribution drift.
- The method's task definition covers edges and flows as well as nodes, so the node-injection experiment is presented as the first test of a filter mechanism designed to generalize to other structural attacks.
- The authors argue that because the LLM reasons about flow interactions rather than matching fixed signatures, the same filter could catch zero-day attacks or drift-induced anomalies that no training set contains.
- The agentic pipeline outlined as future work pairs a powerful model for analysis with cheaper models for bulk filtering, which the authors propose as the route to practical cost and latency.
Reading between the lines
- A direct test the paper does not include is comparing the LLM filter against a simple statistical outlier detector; if a cheap baseline recovers the same accuracy, the mechanism is generic anomaly rejection rather than expert analysis.
- The reported recovery that slightly exceeds the clean baseline suggests the filter may also be pruning noisy benign nodes, so measuring the pipeline on un-attacked traffic would separate attack repair from general graph cleaning.
- The paper's stated generalization to zero-day and drift anomalies is plausible but untested; a stronger experiment would introduce a novel attack family or an unseen data source and ask the LLM to flag it without any examples.
- The 'LLM Recall' column and the correctly-flagged counts in Table II are not internally consistent enough to reconstruct the true number of injected nodes in the test graph, so a re-run that reports the attack size alongside per-model recall would make the result easier to verify.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This poster paper proposes to insert an LLM-based agentic filter between a network-flow graph and a GNN classifier. The authors construct a unified dataset from three NetFlow datasets to test distribution drift, evaluate three GNN models, and report that accuracy/F1 degrade on the unified data. They then simulate a 20% node-injection attack and use GPT-4o, LLaMA 4, Claude 3.5 Haiku, and Claude 3.7 Sonnet to score or flag graph nodes before classification. Table II reports that GPT-4o and LLaMA 4 filtering raises accuracy/F1 back to about the clean level, and the paper concludes that LLM agents can enhance GNN robustness.
Significance. The idea of using LLMs as simulated security experts to sanitize graph inputs is timely and potentially useful, and the observation that two strong LLMs can flag most injected nodes is an interesting empirical data point. The downstream GNN accuracy after filtering is an independent measurement, which is a genuine strength. However, the paper currently provides no non-LLM baseline, no attack-generation details, no prompt disclosure, and no statistical error bars; as a result the central claim is not yet supported beyond a single synthetic attack scenario. The broad claims in the abstract about 'a variety of adversarial attacks' and physical testbed experiments are not supported by the body.
major comments (4)
- [Section IV, Table II] The central mitigation claim is not supported without a non-LLM baseline or an attacked-without-mitigation row. The table reports only Clean accuracy/F1 and the accuracy/F1 after LLM filtering; the reader cannot tell how much degradation the 20% node injection caused or whether random removal of the same number of nodes (or removal of nodes with largest feature deviation) would produce the same or better recovery. Please add at least a 'No mitigation (attacked)' row and a random-removal or heuristic-filter baseline, with the number of removed nodes matched.
- [Section III-B and III-C] The node-injection attack is not specified. The text states 200 injected nodes into a base graph of 1000 but does not describe how these nodes are generated, what features they receive, or whether the injection respects network-flow constraints. If the injected nodes are statistically distinct from benign nodes, the LLM may simply be reporting an obvious outlier pattern. Please describe the attack construction and add a simple feature-distance/degree/kNN outlier detector as a baseline to control for injected-node distinguishability.
- [Section III-B and Table II] The LLM prompt and the textual graph representation are not disclosed, and the table's metric definitions are inconsistent. With 200 injected nodes, LLaMA 4's CF=200 implies a recall of 1.0, yet the table reports LLM Recall 0.758; Claude 3.5 Haiku reports CF=0 but LLM Recall 0.698. The column meanings and the recall computation need to be clarified, and the exact prompt must be included in the appendix or supplement, otherwise prompt leakage cannot be ruled out and the numbers are not reproducible.
- [Abstract and Section V] The abstract claims 'a variety of adversarial attacks including a dataset collected from physical testbed experiments,' but Section V defers testbed validation to future work, and Section IV/Table II report only a single node-injection scenario. The PGD feature attacks and Edge Removal listed in Section III-C are not reported. Please limit the claims to what is demonstrated or add the missing results.
minor comments (6)
- [Table I] The comparison mixes results reported in the original papers with the authors' unified-dataset runs; the preprocessing and evaluation protocol are not matched. Please re-run the same models on the individual datasets under the same protocol or state this as a rough comparison.
- [Table II] Table formatting issues should be corrected: entries such as '0 .821' and '17B0.859' appear with stray spaces, and the caption should define Nodes, CF, IF, and LLM Recall explicitly.
- [Section IV] Figure 2 is referenced as the design diagram but the figure is not present in the submitted text; please ensure all figures are included and described.
- [Section V] The phrase 'integrating individual LLM as expert analyzers' should read 'integrating individual LLMs as expert analyzers'; minor grammar issue.
- [Section II] Reference [14] appears closely related to this work (LLMs improving adversarial robustness of GNNs); the related-work discussion should state explicitly how the proposed agentic NIDS pipeline differs.
- [General] No code, data, or artifact link is provided. For a results-driven poster, releasing the unified dataset preprocessing and evaluation scripts would greatly improve reproducibility.
Circularity Check
No significant circularity: the LLM mitigation result is an empirical comparison with independent downstream measurements.
full rationale
The paper's derivation chain is: standard GNNs degrade under distribution drift and a synthetic node-injection attack; an LLM is asked to flag suspicious graph elements; flagged nodes are filtered; downstream GNN accuracy is remeasured. Table II reports both LLM recall against the known injected-node ground truth and the resulting GNN accuracy/F1. Neither quantity is an input to the other by construction: the LLM weights are not fitted to the GNN objective or to the attack labels, and the post-filter GNN accuracy is an independent model output on the remaining graph. The only overlapping-author citation is [13], in the related-work sentence 'Ideas leveraging LLMs for robust evaluation or enhancing GNNs are emerging [13], [14]'; it is contextual and not load-bearing for the claimed mitigation gain. The abstract's assertion that testbed data was used is contradicted by Section V, which says 'we plan to validate these findings against diverse, realistic attack scenarios within a physical IoT testbed' — but that is an over-claim/correctness issue, not circularity. Absent the undisclosed prompt or attack-generation code, one cannot exhibit a specific reduction of Table II to its inputs; therefore no circular step is established under the required evidence standard.
Assumptions & free parameters
assumptions (3)
- domain assumption NetFlow datasets (NF-BoT-IoT, NF-CSE-CIC-IDS2018, NF-UNSW-NB15) can be merged and standardized to simulate distribution drift.
- domain assumption IP-centric graphs with flow features preserve the information needed to judge node maliciousness.
- ad hoc to paper LLM relevance scores can be thresholded to filter nodes without calibration.
Cite this review
Pith. "Pith review of Poster: Enhancing GNN Robustness for Network Intrusion Detection via Agent-based Analysis." pith.science (2026). https://pith.science/paper/YSADCDQ2
@misc{pith2026250620806,
author = {Pith},
title = {Pith review of: Poster: Enhancing GNN Robustness for Network Intrusion Detection via Agent-based Analysis},
year = {2026},
howpublished = {\url{https://pith.science/paper/YSADCDQ2}},
note = {Machine review of arXiv:2506.20806}
}
read the original abstract
Graph Neural Networks (GNNs) show great promise for Network Intrusion Detection Systems (NIDS), particularly in IoT environments, but suffer performance degradation due to distribution drift and lack robustness against realistic adversarial attacks. Current robustness evaluations often rely on unrealistic synthetic perturbations and lack demonstrations on systematic analysis of different kinds of adversarial attack, which encompass both black-box and white-box scenarios. This work proposes a novel approach to enhance GNN robustness and generalization by employing Large Language Models (LLMs) in an agentic pipeline as simulated cybersecurity expert agents. These agents scrutinize graph structures derived from network flow data, identifying and potentially mitigating suspicious or adversarially perturbed elements before GNN processing. Our experiments, using a framework designed for realistic evaluation and testing with a variety of adversarial attacks including a dataset collected from physical testbed experiments, demonstrate that integrating LLM analysis can significantly improve the resilience of GNN-based NIDS against challenges, showcasing the potential of LLM agent as a complementary layer in intrusion detection architectures.
Figures
Reference graph
Works this paper leans on
-
[1]
Insomnia: Towards concept-drift robustness in network intrusion detection,
G. Andresini, F. Pendlebury, F. Pierazzi, C. Loglisci, A. Appice, and L. Cavallaro, “Insomnia: Towards concept-drift robustness in network intrusion detection,” in Proceedings of the 14th ACM workshop on artificial intelligence and security , 2021, pp. 111–122
2021
-
[2]
Unsw-nb15: a comprehensive data set for network intrusion detection systems (unsw-nb15 network data set),
N. Moustafa and J. Slay, “Unsw-nb15: a comprehensive data set for network intrusion detection systems (unsw-nb15 network data set),” in 2015 military communications and information systems conference (MilCIS). IEEE, 2015, pp. 1–6
2015
-
[3]
Toward generating a new intrusion detection dataset and intrusion traffic characterization
I. Sharafaldin, A. H. Lashkari, A. A. Ghorbani et al., “Toward generating a new intrusion detection dataset and intrusion traffic characterization.” ICISSp, vol. 1, no. 2018, pp. 108–116, 2018
work page 2018
-
[4]
Towards the development of realistic botnet dataset in the internet of things for network forensic analytics: Bot-iot dataset,
N. Koroniotis, N. Moustafa, E. Sitnikova, and B. Turnbull, “Towards the development of realistic botnet dataset in the internet of things for network forensic analytics: Bot-iot dataset,” Future Generation Computer Systems, vol. 100, pp. 779–796, 2019
2019
-
[5]
E- graphsage: A graph neural network based intrusion detection system for iot,
W. W. Lo, S. Layeghy, M. Sarhan, M. Gallagher, and M. Portmann, “E- graphsage: A graph neural network based intrusion detection system for iot,” in NOMS 2022-2022 IEEE/IFIP Network Operations and Management Symposium. IEEE, 2022, pp. 1–9
work page 2022
-
[6]
Anomal-e: A self- supervised network intrusion detection system based on graph neural networks,
E. Caville, W. W. Lo, S. Layeghy, and M. Portmann, “Anomal-e: A self- supervised network intrusion detection system based on graph neural networks,” Knowledge-based systems, vol. 258, p. 110030, 2022
work page 2022
-
[7]
M. A. Jahin, S. Soudeep, M. Mridha, R. Kabir, M. R. Islam, and Y . Watanobe, “Cagn-gat fusion: A hybrid contrastive attentive graph neural network for network intrusion detection,” arXiv preprint arXiv:2503.00961, 2025
arXiv 2025
-
[8]
Towards more practical adversarial attacks on graph neural networks,
J. Ma, S. Ding, and Q. Mei, “Towards more practical adversarial attacks on graph neural networks,” Advances in neural information processing systems, vol. 33, pp. 4756–4766, 2020
work page 2020
Show all 14 references
-
[9]
Iotsafe: Enforcing safety and security policy withreal iot physical interaction discovery,
W. Ding, H. Hu, and L. Cheng, “Iotsafe: Enforcing safety and security policy withreal iot physical interaction discovery,” in Network and Distributed System Security Symposium , 2021
2021
-
[10]
Security testbed for the internet of things,
S. Siboni, V . Sachidananda, A. Shabtai, and Y . Elovici, “Security testbed for the internet of things,” arXiv preprint arXiv:1610.05971 , 2016
2016 arXiv
-
[11]
Are we there yet? unraveling the state-of-the-art graph network intrusion detection systems,
C. Wang, P. Zheng, J. Gui, C. Hua, and W. U. Hassan, “Are we there yet? unraveling the state-of-the-art graph network intrusion detection systems,” arXiv preprint arXiv:2503.20281 , 2025
2025 arXiv
-
[12]
Graph robustness benchmark: Benchmarking the adversarial robustness of graph machine learning,
Q. Zheng, X. Zou, Y . Dong, Y . Cen, D. Yin, J. Xu, Y . Yang, and J. Tang, “Graph robustness benchmark: Benchmarking the adversarial robustness of graph machine learning,” arXiv preprint arXiv:2111.04314 , 2021
2021 arXiv
-
[13]
Trustrag: Enhancing robustness and trustworthiness in rag,
H. Zhou, K.-H. Lee, Z. Zhan, Y . Chen, Z. Li, Z. Wang, H. Haddadi, and E. Yilmaz, “Trustrag: Enhancing robustness and trustworthiness in rag,” arXiv preprint arXiv:2501.00879 , 2025
2025 arXiv
-
[14]
Can large language models improve the adversarial robustness of graph neural networks?
Z. Zhang, X. Wang, H. Zhou, Y . Yu, M. Zhang, C. Yang, and C. Shi, “Can large language models improve the adversarial robustness of graph neural networks?” arXiv preprint arXiv:2408.08685 , 2024
2024 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.