Analysis of Zero Day Attack Detection Using MLP and XAI

Ashim Dahal; Nick Rahimi; Prabin Bajgai

REVIEW 4 major objections 4 minor 14 references

Analysis of Zero Day Attack Detection Using MLP and XAI

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Small multilayer perceptrons trained on four grouped KDD99 classes achieve 99.62% accuracy, and adding class weights trades slight accuracy for much better recall on rare attack classes while SHAP exposes the feature shift.

desk verdict A small KDD99 MLP experiment with a plausible but unremarkable trade-off result, wrapped in an unsupported zero-day claim and marred by a real label-swap error. read the letter →

arxiv 2501.16638 v1 pith:YGMOJ2WU submitted 2025-01-28 cs.LG cs.CR

Ashim Dahal , Prabin Bajgai , Nick Rahimi This is my paper · ORCID

classification cs.LGcs.CR

keywords zero-dayattacksKDD99multilayerperceptronintrusiondetectionsystemclassimbalanceSHAPexplainableAInetworksecurity

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that an intrusion detection system for zero-day attacks can be built from a small multilayer perceptron trained on KDD99, provided the 23 fine-grained attack labels are merged into four parent categories and the loss is class-weighted. The truncated model reaches 99.62% accuracy, precision, and recall, while the weighted truncated model reaches 97.26% accuracy but lifts recall on the rare Unauthorized Access class from zero to about 0.34. The paper reads this trade-off as evidence that class weights reduce bias, and it uses SHAP to show that the weighted model attends to different features than the unweighted one. A sympathetic reader would care because the paper offers a concrete recipe—semantic label grouping plus loss weighting plus a compact architecture—for making ML-based intrusion detection both accurate and less biased on imbalanced attack data.

What carries the argument

The central machinery is the pairing of label truncation with class weighting in the loss function. The 23 original KDD99 attack labels are merged into four parent categories—Normal, DoS, Probe, and Unauthorized Access—which reduces the heavy imbalance (Smurf alone is about 60% of the data) and lets a much smaller MLP (13,892 parameters versus 62,871) learn effectively. Class weights then make the loss treat the tiny Unauthorized Access and DoS classes as more important, and a SHAP kernel explainer trained on 50 random test samples attributes each prediction to input features, exposing how weighting changes the model's decision logic.

What would settle it

Hold out an entire family of KDD99 attack classes (for example, all user-to-root and remote-to-local attacks) during training, then test the truncated MLP on those held-out classes; if recall on them falls to near zero, the model is recognizing known classes, not detecting zero-day attacks. Repeating the same test on a newer intrusion dataset with attack types not in KDD99 would settle the zero-day claim directly.

Watch

Extended reading notes

Core claim

The central claim is that reorganizing the 23 KDD99 classes into four coarse categories (Normal, DoS, Probe, and Unauthorized Access) and training a compact multilayer perceptron yields a model that is more accurate, less biased, and smaller than models trained on the original labels. On a stratified 67/33 split, the truncated MLP achieves 99.62% overall accuracy, precision, and recall, while the weighted truncated MLP achieves 97.26% accuracy but improves recall on the rare Unauthorized Access class from 0 to roughly 0.34. SHAP feature attribution shows the unweighted model focuses on features tied to the majority Probe class, whereas the weighted model redistributes attention toward the underrepresented class. The paper presents these results as establishing robustness, interpretability, and scalability for zero-day attack detection.

Load-bearing premise

The entire zero-day conclusion rests on the premise that performance on a random split of KDD99—where every attack type is known and present in training—predicts performance on attacks the model has never seen; if novel exploits follow patterns absent from the training set, the reported accuracies would not transfer.

Editorial extensions

If this is right

A combined or ensemble model using both the truncated and weighted truncated MLPs could preserve the 99.62% overall accuracy while retaining the weighted model's better rare-class recall.
The truncated architecture's 13,892 parameters make it fast to train and cheap to deploy, supporting the paper's scalability argument for real-world intrusion detection systems.
The SHAP results imply that class weights change which features the network relies on, so XAI summaries can serve as a check that a model is actually learning minority classes.
Grouping fine-grained attack labels into semantic parent categories is a preprocessing step that other imbalanced intrusion datasets could adopt.
The trade-off between weighted and unweighted models shows accuracy alone is insufficient as an intrusion detection evaluation metric.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The leap from KDD99 splits to 'zero-day detection' is an extrapolation the experiments do not test; a fair test would train on known classes and evaluate on attack families not present in training, such as newer datasets with novel exploit types.
The same recipe—semantic label grouping plus loss weighting—could generalize to other imbalanced security domains, where rare-event recall matters more than aggregate accuracy.
The SHAP feature shift between weighted and unweighted models suggests a diagnostic: if weighting does not change the top features, the weights may be ineffective regardless of loss values.
Tuning the class weights could trace a frontier between overall accuracy and rare-class recall, letting an operator choose an operating point for a given threat profile; this is a testable extension of the reported two-point comparison.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a modest, workmanlike KDD99 experiment whose title claim to zero-day detection is not backed by the evaluation. If you treat it as a confirmatory data point on imbalanced classification, the numbers are probably fine. If you treat it as a zero-day paper, it fails.

What's actually new: not much. The authors themselves cite 99.3% and 99.5% accuracies on KDD99, so the headline accuracy is known territory. Merging the 23 classes into four groups and adding class weights are standard tricks. The one thing of interest is the comparison between a small 13,892-parameter truncated MLP and a larger base model: the truncated model keeps accuracy while the weighted truncated model trades some accuracy (97.26% vs 99.62%) for better recall on the rare class. That observation is credible and consistent with the imbalanced-learning literature. The SHAP analysis on 50 test points is illustrative and nothing more.

Where it falls down. First, the zero-day claim. The evaluation uses a stratified 67/33 split, so every one of the 23 attack classes appears in both training and test. No attack family is withheld, no temporally later data is used, and the 'zero-day' framing is pure extrapolation. That alone sinks the title claim. Second, the class definitions in Section 3.2 are swapped: Probe is listed as containing Back, Neptune, Smurf, etc., and DoS as containing Satan, Ipsweep, etc., which is backwards. The counts in Table 1 match the standard grouping (3.88M DoS, 41K Probe), so the model likely trained on the correct labels, but Table 3's per-class results are therefore mislabeled. That is a real, visible error, not a nitpick. Third, the models are under-specified: class weights are not disclosed, and there is a single stratified split with no repeats or error bars, so the 99.62% and 97.26% figures are point estimates without variance. No code is provided.

Soft spots in proportion: the accuracy-vs-recall trade-off is a minor, likely reproducible finding; the zero-day conclusion is load-bearing and empty; the label swap is embarrassing but correctable; under-specification is fixable in principle.

Who gets value: someone teaching a class on how to critically read an IDS paper, or a researcher wanting a quick sanity check that class weighting improves rare-class recall on KDD99. Not a researcher looking for new zero-day methodology.

Recommendation: I would not send this to peer review as-is. The title claim is unsupported and the labeling error needs correcting before the results can be interpreted. If the authors drop the zero-day language, fix the swap, disclose weights, and add repeated runs, it could be a legitimate workshop-level contribution. Until then, it's a desk reject.

Referee Report

4 major / 4 minor

Summary. The manuscript reports a multi-class intrusion-detection study on the KDD99 dataset. The authors group the 23 KDD99 classes into four categories (Normal, DoS, Probe, Unauthorized Access), train four MLPs (base unweighted, base weighted, truncated unweighted, and truncated weighted) for 20 epochs, and evaluate on a stratified 67/33 split. They report 99.62% accuracy for the truncated model and 97.26% for the weighted truncated model, with improved recall on rare classes after weighting, and use SHAP on 50 test samples to inspect feature importance. The paper's central claim is that the truncated models provide 'robust and dependable results in zero-day threat detection.'

Significance. If the central claim were supported, the paper would offer a useful recipe for designing compact MLP-based IDSs that preserve overall accuracy while improving rare-class recall, with interpretability via SHAP. The parameter-count comparison (13,892 vs 62,871 parameters) and the explicit attention to class imbalance are strengths. However, the significance is severely limited by the absence of any unseen-attack evaluation, by the swapped Probe/DoS labels, and by missing experimental details that prevent independent reproduction. As it stands, the claims are not supported at the level claimed.

major comments (4)

[Section 3.3, Section 5] Section 3.3 creates a stratified 67/33 split, so every one of the 23 original KDD99 classes is present in both the training and test sets. The 99.62% and 97.26% accuracies in Section 4 therefore measure classification of previously seen attack families, not detection of zero-day exploits. The conclusion's statement that the models yield 'robust and dependable results in zero-day threat detection' is an unsupported extrapolation. To support the title-level claim, the evaluation would need to withhold an entire attack family during training and test on it, or use a temporally later dataset with novel attacks.
[Section 3.2, Table 3] The category definitions in Section 3.2 are swapped relative to the KDD99 attack taxonomy: the classes listed under Probe (Back, Land, Neptune, Pod, Smurf, Teardrop, Apache2, Udp Storm, Processtable, Worm) are DoS attacks, while the classes listed under DoS (Satan, Ipsweep, Nmap, Portsweep, Mscan, Saint) are Probe attacks. This error propagates to Table 3, where the row labeled "Probe" has support 1,281,513 and the row labeled "DOS" has support 13,563, exactly matching the swapped test-set counts. All per-category precision, recall, and F1 values in Table 3 are therefore mislabeled, and the statements about which classes benefit from weighting must be re-examined after correcting the labels.
[Section 3.4, Tables 2-3] The paper does not disclose the MLP architecture, activation functions, learning rate, batch size, regularization, the actual class-weight values ("newly curated weights"), or the number of random seeds. All results come from a single stratified split with no confidence intervals or repeated runs. Consequently, the reported 99.62% and 97.26% figures cannot be reproduced or assessed for statistical significance, and the claim of "low variance" in the introduction is unsupported.
[Section 4, Table 3] The claim that the weighted truncated model offers "better class representation (less bias)" rests almost entirely on the Unauthorized Access row, whose recall improves from 0 to 0.3368 but whose F1 score is only 0.0149 with precision 0.0076. This level of performance is not convincingly "less biased"; the paper should either report a meaningful utility threshold or temper the claim.

minor comments (4)

[Abstract] "Shapely" is a misspelling; SHAP stands for "SHapley Additive exPlanations."
[Section 4] The sentence "shown in Fig 2 and Fig 5" appears to reference the wrong figures; the loss curves are in Fig. 2, while the SHAP plots are Figs. 4 and 5.
[Section 4] The statement about the maximum change in F1-score of "0.01%" is unclear; Table 2 shows a change from 0 to 0.0105 for Warezclient, which is a fraction, not 0.01%.
[Section 3.6] The SHAP explainer is trained on only 50 random samples; the choice of sample size should be justified, since small samples can make the explanations unstable.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the reported accuracies are empirical held-out measurements, and the unsupported zero-day generalization is an external-validity issue, not a circular derivation.

full rationale

The paper's load-bearing numbers (99.62% and 97.26% accuracy) are empirical measurements on a held-out stratified 67/33 split of KDD99, not quantities that reduce by construction to the training labels, loss weights, or the authors' own citations. The four MLPs are trained on the training portion and evaluated on the test portion, so the classification report is a genuine measurement rather than a renamed training objective. The class-weighting scheme is a design choice that predictably emphasizes minority classes, but the paper reports the resulting test-set recall and accuracy as observations, not as a prediction derived from the weighting formula; no equation in the paper makes the weighted model's better minority recall equivalent to its loss weights by definition. The SHAP explainer is trained on 50 random test samples to describe model behavior, but it does not define or force any reported performance metric, so it introduces no circularity. There is no load-bearing self-citation chain: the paper cites standard external works for SHAP, KDD99, and prior IDS results, and none of those citations is used to forbid alternatives or to justify the paper's own conclusions. The paper's main weakness is that the title-level 'zero-day threat detection' claim is unsupported because KDD99 contains only known attack classes and all classes appear in both training and test splits; however, that is a threat to external validity or correctness, not circularity, because the measured accuracies are not defined in terms of the zero-day claim. Overall, the derivation chain is self-contained and no specific step reduces to its own inputs.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The load-bearing axioms are that KDD99 labels are a valid attack taxonomy, that same-distribution accuracy says something about zero-day attacks, and that 50-sample SHAP values explain model behavior. The zero-day assumption is the most fragile: it is asserted by the title and conclusion but never tested. The class weights, grouping, and architecture are hand-chosen and under-specified, and the grouping is implemented with swapped labels.

free parameters (4)

Class grouping scheme = 23 classes mapped to 4 categories, with Probe and DoS names swapped
Section 3.2 defines the four groups by hand; the mapping determines the classification task, and the printed assignment is internally inconsistent with the support counts in Table 3.
Class weights = not reported
Section 3.4 mentions class weights for the weighted models but gives no values, so the weighted model results cannot be recomputed or audited.
MLP architecture = 13892 and 62871 parameters; exact layer sizes not specified
Only parameter counts and 20 training epochs are given, with no layer widths, activation, optimizer, learning rate, batch size, or seed.
SHAP explainer sample = 50 random test samples
Section 3.6 fixes the explainer training set at 50 samples; this is a hand-chosen size and makes the SHAP summaries high-variance.

assumptions (3)

domain assumption KDD99 labels and their four-group merging are valid representations of network attack categories.
Section 3.1 treats KDD99 labels as ground truth attack types and Section 3.2 merges them into four parent categories; no external validation of this taxonomy is provided, and the merge is implemented with Probe and DoS names swapped.
ad hoc to paper High accuracy on a stratified same-distribution split of KDD99 indicates zero-day attack detection capability.
Sections 3.3 and 4 evaluate only on a random split of the same known attacks, while the abstract and conclusion claim zero-day detection; this untested premise is required for the title claim to hold.
domain assumption SHAP feature importance computed on 50 test samples reliably explains model decisions.
Section 3.6 uses a SHAP kernel explainer on 50 randomly chosen samples and Section 4 interprets the resulting top features as the model's reasoning; the paper provides no variance or convergence check for these estimates.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Analysis of Zero Day Attack Detection Using MLP and XAI." pith.science (2026). https://pith.science/paper/YGMOJ2WU

@misc{pith2026250116638,
  author       = {Pith},
  title        = {Pith review of: Analysis of Zero Day Attack Detection Using MLP and XAI},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YGMOJ2WU}},
  note         = {Machine review of arXiv:2501.16638}
}

read the original abstract

Any exploit taking advantage of zero-day is called a zero-day attack. Previous research and social media trends show a massive demand for research in zero-day attack detection. This paper analyzes Machine Learning (ML) and Deep Learning (DL) based approaches to create Intrusion Detection Systems (IDS) and scrutinizing them using Explainable AI (XAI) by training an explainer based on randomly sampled data from the testing set. The focus is on using the KDD99 dataset, which has the most research done among all the datasets for detecting zero-day attacks. The paper aims to synthesize the dataset to have fewer classes for multi-class classification, test ML and DL approaches on pattern recognition, establish the robustness and dependability of the model, and establish the interpretability and scalability of the model. We evaluated the performance of four multilayer perceptron (MLP) trained on the KDD99 dataset, including baseline ML models, weighted ML models, truncated ML models, and weighted truncated ML models. Our results demonstrate that the truncated ML model achieves the highest accuracy (99.62%), precision, and recall, while weighted truncated ML model shows lower accuracy (97.26%) but better class representation (less bias) among all the classes with improved unweighted recall score. We also used Shapely Additive exPlanations (SHAP) to train explainer for our truncated models to check for feature importance among the two weighted and unweighted models.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 12 canonical work pages

[1]

This organization helps reduce the huge discrepancy between the classes. Probe: This category encompasses network probing and reconnaissance activities involving scanning, testing, and probing for vulnerabilities to gather information about target systems. The following classes from the dataset were included in this category: Back, Land, Neptune, Pod, Smu...

work page 1965
[2]

Goodfellow, I., Bengio, Y., Courville, A. (2016). Deep Learning. MIT Press

work page 2016
[3]

Holm, H. (2014). Signature Based Intrusion Detection for Zero -Day Attacks: (Not) A Closed Chapter?. In: 2014 47th Hawaii International Conference on System Sciences. Waikoloa, HI, USA, 2014, pp. 4895 -4904. doi: 10.1109/HICSS.2014.600

work page doi:10.1109/hicss.2014.600 2014
[4]

Guo, Y., Gao, J., Li, H. (2022). A Review of Machine Learning-based Zero-day Attack Detection: Challenges and Future Directions. Journal of Computer Security, 30(2), pp. 1-25

work page 2022
[5]

Popescu, M., Balas, V., Perescu-Popescu, L., Mastorakis, N. (2009). Multilayer perceptron and neural networks. Semantic Scholar, 8(7). doi: https://doi.org/10.5555/1639537.1639542

work page arXiv 2009
[7]

K., Elsayed, M

Hairab, I., Aslan, H. K., Elsayed, M. S., Jurcut, A. D., Azer, M. A. (2023). Anomaly Detection of Zero-Day Attacks Based on CNN and Regularization Techniques. Electronics, 12(3), p. 573. doi: 10.3390/electronics12030573

work page doi:10.3390/electronics12030573 2023
[8]

Gaber, M., Zakaria, N., Kumar, S. (2019). The Problem with Accuracy: Lessons from the KDD99 Dataset. IEEE Trans- actions on Knowledge and Data Engineering, 31(12), pp. 2305-2319. doi: 10.1109/TKDE.2018.2873826

work page arXiv 2019
[9]

T., Islam, M

Rahman, M. T., Islam, M. T., Nahar, N., Hossain, M. J. (2020). Beyond Accuracy: A Review of Evaluation Metrics for Network Intrusion Detection. IEEE Access, 8, pp. 148609-148629. doi: 10.1109/ACCESS.2020.3017641

work page arXiv 2020

Show all 14 references

[10]

Mukunth, S., Srinivasa Murthy, H. N. (2014). Anomaly Detection of Network Intrusions in KDD99 Using Support Vector Machines. International Journal of Computer Applications, 91(1), pp. 13-18

2014
[11]

However, this paper also does not address the class imbalance in the KDD99 dataset

achieved an accuracy of 99.5% on the KDD99 dataset using a genetic algorithm to optimize the features and parameters of an SVM classifier. However, this paper also does not address the class imbalance in the KDD99 dataset. In a dataset like the KDD99 dataset with a heavy class...
[12]

Stolfo, S., Fan, W., Lee, W., Prodromidis, A., Chan, P. (1999). KDD Cup 1999 Data. UCI Machine Learning Repository. doi: 10.24432/C51C7N

1999 doi
[13]

Kumar, S., Minz, S. K. (2014). Application of Genetic Algorithm for Feature Selection and Parameter Optimization of Support Vector Machine for Network Intrusion Detection in KDD99 Dataset. International Journal of Computer Science and Information Technologies, 5(1), pp. 168-173

2014
[14]

M., Lee, S

Lundberg, S. M., Lee, S. (2017). A Unified Approach to Interpreting Model Predictions. In: Advances in Neural Infor- mation Processing Systems, vol. 30, Guyon, I., et al., Eds. Curran Associates, Inc., pp. 30-38. Available: https://proceed- ings.neurips.cc/paper_files/paper/20...

2017
[15]

H., Bhattacharya, S., Pandya, S., Maddikunta, P

Srivastava, G., Jhaveri, R. H., Bhattacharya, S., Pandya, S., Maddikunta, P. K. R., Yenduri, G., Hall, J. G., Alazab, M., Gadekallu, T. R., et al. (2022). XAI for cybersecurity: state of the art, challenges, open issues and future directions. arXiv preprint arXiv:2206.03585

2022 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

This organization helps reduce the huge discrepancy between the classes. Probe: This category encompasses network probing and reconnaissance activities involving scanning, testing, and probing for vulnerabilities to gather information about target systems. The following classes from the dataset were included in this category: Back, Land, Neptune, Pod, Smu...

work page 1965

[2] [2]

Goodfellow, I., Bengio, Y., Courville, A. (2016). Deep Learning. MIT Press

work page 2016

[3] [3]

Holm, H. (2014). Signature Based Intrusion Detection for Zero -Day Attacks: (Not) A Closed Chapter?. In: 2014 47th Hawaii International Conference on System Sciences. Waikoloa, HI, USA, 2014, pp. 4895 -4904. doi: 10.1109/HICSS.2014.600

work page doi:10.1109/hicss.2014.600 2014

[4] [4]

Guo, Y., Gao, J., Li, H. (2022). A Review of Machine Learning-based Zero-day Attack Detection: Challenges and Future Directions. Journal of Computer Security, 30(2), pp. 1-25

work page 2022

[5] [5]

Popescu, M., Balas, V., Perescu-Popescu, L., Mastorakis, N. (2009). Multilayer perceptron and neural networks. Semantic Scholar, 8(7). doi: https://doi.org/10.5555/1639537.1639542

work page arXiv 2009

[6] [7]

K., Elsayed, M

Hairab, I., Aslan, H. K., Elsayed, M. S., Jurcut, A. D., Azer, M. A. (2023). Anomaly Detection of Zero-Day Attacks Based on CNN and Regularization Techniques. Electronics, 12(3), p. 573. doi: 10.3390/electronics12030573

work page doi:10.3390/electronics12030573 2023

[7] [8]

Gaber, M., Zakaria, N., Kumar, S. (2019). The Problem with Accuracy: Lessons from the KDD99 Dataset. IEEE Trans- actions on Knowledge and Data Engineering, 31(12), pp. 2305-2319. doi: 10.1109/TKDE.2018.2873826

work page arXiv 2019

[8] [9]

T., Islam, M

Rahman, M. T., Islam, M. T., Nahar, N., Hossain, M. J. (2020). Beyond Accuracy: A Review of Evaluation Metrics for Network Intrusion Detection. IEEE Access, 8, pp. 148609-148629. doi: 10.1109/ACCESS.2020.3017641

work page arXiv 2020