{"id":"322b8345-e023-40b3-b71d-86d65caccf62","arxiv_id":"2602.13271","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A CNN/LSTM intrusion detection framework with SHAP explanations and a 15-expert trust survey reports 99% accuracy on NSL-KDD, but its claimed superiority is unmeasured.","lead":"The paper pairs CNN and LSTM intrusion-detection models with SHAP explanations and tests them with a small expert survey, reporting 99% accuracy on NSL-KDD. The claimed superiority over other systems is not backed by any baseline comparison, and the dataset handling contains internal errors.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Dataset handling errors (swapped R2L/Probe counts and non-standard split) invalidate the 99% accuracy claim.","rationale":"The reader's weakest assumption identified the dataset handling as the load-bearing premise. I agree. The swapped R2L/Probe counts in Section III-A are a concrete error that, if replicated in the label mapping, would invert two attack classes, making the 99% accuracy and the SHAP feature importance results for DoS detection (Section IV-B) meaningless. The random 80/20 split instead of the standard KDDTrain+/KDDTest+ partitions further invalidates any comparison with published NSL-KDD results, because the test set is not the out-of-distribution benchmark that the dataset was designed to provide. These issues are more fundamental than the absence of explicit baselines; even if baselines were added, the reported accuracy itself would be unreliable. The paper also contains internal inconsistencies (e.g., text reporting CNN macro recall 0.99 vs Table I 0.84) that reinforce the impression of unchecked errors. Given the central claim of superior performance on a benchmark dataset rests on flawed data handling, the REJECT verdict is appropriate. The paper could be revised, but as submitted the evidence does not support the claim.","tokens_in":11150,"tokens_out":7973,"duration_ms":78257,"concrete_test":"Obtain the official NSL-KDD KDDTrain+ and KDDTest+ files. Verify the class counts: KDDTrain+ has 67,343 Normal, 45,927 DoS, 11,656 Probe, 995 R2L, 52 U2R. Re-run the CNN and LSTM preprocessing with the correct mapping and train on KDDTrain+, test on KDDTest+. If the reported confusion matrices cannot be reproduced with the correct labels (e.g., the per-class counts change or accuracy drops significantly), the original results are invalid. Also verify whether the original project's label encoding swapped the R2L and Probe classes by inspecting the code if available.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section III-A, the paper states the NSL-KDD class distribution as \"67343, 45927, 11647, 995, 52 for Normal, Dos, R2L, Probe and U2R\". The canonical NSL-KDD KDDTrain+ distribution is Normal 67,343, DoS 45,927, Probe 11,656, R2L 995, U2R 52. Thus the counts for R2L and Probe are swapped. If this swap reflects the label mapping used in preprocessing (Section III-D), then the models were trained with mislabeled classes. The reported 99% accuracy (Table I) would then be on incorrect targets, and the SHAP explanations in Section IV-B would be explaining decisions for wrong categories. Additionally, Section III-C states an 80/20 random split of the full 125,973 rows, whereas the standard benchmark protocol uses KDDTrain+ and KDDTest+ with different distributions (including novel attack types). Randomly splitting the training set inflates accuracy because test samples are drawn from the same distribution. Both errors undermine the central claim of superiority on a benchmark dataset. If the labels are truly swapped, the model's performance and interpretability results are meaningless.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes an explainable intrusion detection framework combining CNN and LSTM models on the NSL-KDD dataset, with SHAP for post-hoc explanations and a 15-participant user study measuring trust, reliability, and usability. The authors report 99% accuracy for both models, claim superior performance over traditional IDS and black-box deep learning models, and use SHAP to identify influential features for DoS detection. A human-centered evaluation via an interactive UI is also presented.","tokens_in":11322,"tokens_out":6263,"duration_ms":61979,"significance":"If the results were valid, the paper would contribute a useful comparison of CNN and LSTM architectures for intrusion detection and a rare attempt to evaluate SHAP-based explanations with human experts. The inclusion of confusion matrices, ROC curves, and a user study based on validated psychometric instruments is commendable. However, the central quantitative claims are currently not supported by the experimental design: the non-standard data split, absence of baseline comparisons, and internal contradictions in the dataset description and reported experimental setup prevent any reliable assessment of the claimed superiority.","major_comments":[{"comment":"The evaluation uses an 80/20 random split of the 125,973-row NSL-KDD training set instead of the standard KDDTrain+/KDDTest+ protocol. KDDTest+ contains attack types not present in KDDTrain+, so a random split from the same distribution inflates accuracy and makes the reported 99% figure non-comparable with published NSL-KDD results. The abstract's central claim of 'superior performance compared to traditional IDS and black-box deep learning models' is therefore not supported by the chosen experimental design.","section":"Section III-C / IV-A"},{"comment":"No baseline models are compared. The paper only compares the proposed CNN and LSTM models to each other; the claim of superiority over traditional IDS and black-box deep learning models is asserted without any empirical baseline, confidence interval, or statistical significance test. This is a load-bearing omission for the paper's main claim.","section":"Abstract / Section IV-A"},{"comment":"The class distribution statement lists counts as '67343, 45927, 11647, 995, 52 for Normal, Dos, R2L, Probe and U2R', which swaps the canonical Probe and R2L counts (canonical KDDTrain+: Probe=11,656, R2L=995). The confusion matrices' row totals (Probe≈2355, R2L≈202 on the 20% test split) correspond to the canonical counts, suggesting this is a typo rather than an actual label swap. Nevertheless, because Section III-D defines the target encoding, the paper must explicitly state the mapping used and resolve the ambiguity; as written, the data pipeline is unclear.","section":"Section III-A / III-D"},{"comment":"The conclusion states that 'due to hardware limitations, our experiments were constrained to simplified models and downsampled datasets,' but the methodology describes training and testing on the full 125,973-row dataset. This is a direct contradiction. If the data were downsampled, the confusion matrices and 99% accuracy do not correspond to the stated dataset and the reported results are not reproducible. This must be clarified and the experiments must be described consistently.","section":"Section V vs. Section III-A"},{"comment":"The SHAP analysis is reported only for class 0 (DoS), and for the LSTM the three-dimensional sequential inputs are flattened to two dimensions before applying KernelExplainer. Flattening destroys the temporal structure that the LSTM is claimed to exploit, so the resulting feature attributions may not reflect the model's actual decision process. Additionally, using 100 samples for a 'global' interpretation is insufficient; the paper should justify the procedure and limit the claims made from the summary plots.","section":"Section IV-B"}],"minor_comments":[{"comment":"Typos and grammatical issues: 'decisoin' in the Introduction, 'In section' with a missing number, and inconsistent capitalization of 'SHapley Additive exPlanations'.","section":"General"},{"comment":"The prose states the CNN achieved a macro recall of 0.99, but Table I reports 0.84; the confusion-matrix-based macro recall is 0.84, so the text should be corrected.","section":"Section IV-A / Table I"},{"comment":"For the CNN, the U2R confusion-matrix row totals give a recall of 0.30 (3/10), not 0.33 as in Table II; the corresponding F1-score would be 0.375, not 0.40. The source of these discrepancies should be checked.","section":"Section IV-A / Table II / Fig. 2"},{"comment":"The title 'Accuracy Per Attack Type' is misleading since the table reports precision, recall, and F1-score, not accuracy. A more descriptive title is needed.","section":"Table II"},{"comment":"Figure 1 is low-resolution and the text is difficult to read; a vector version or higher-resolution figure would improve clarity.","section":"Figure 1"},{"comment":"The dataset description says NSL-KDD has '43 columns along with 125973 rows'; the standard NSL-KDD KDDTrain+ has 42 columns (41 features + label). Please clarify the column count and exactly which file(s) were used.","section":"Section III-C"},{"comment":"The user study reports Cronbach's alpha of 0.60 for usability, which is below the commonly accepted 0.70 threshold; this limitation should be discussed in the text, not merely mentioned as 'moderate consistency.'","section":"Section IV-C"}],"recommendation":"reject","confidential_remarks":"The paper addresses a relevant topic and includes some useful components, but the experimental evaluation is not reliable: the non-standard data split, missing baselines, internal contradictions about the dataset, and inconsistent results undermine the central claims. A full re-evaluation with the standard NSL-KDD protocol and proper baselines would be needed before the paper could be considered for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nThe headline is this: the paper's actual novelty is small — the 15-person trust survey on a SHAP-augmented NIDS interface — and the rest is standard CNN/LSTM on NSL-KDD with SHAP. The 99% accuracy is believable for a random split, but it does not support the abstract's 'superior performance' claim because no baselines are run and the split is non-standard. The stress-test's swapped-label worry doesn't hold up: the encoding in Section III-D and the confusion matrices show Probe and R2L are in their correct places; the sentence in Section III-A has a typo in the counts, but it doesn't corrupt the experiments.\n\nWhat the paper does well: it's honest about its limitations (SHAP compatibility issues, hardware constraints, small sample), the preprocessing is described clearly enough to reproduce, and the survey is a genuine attempt to address human trust in XAI, which is rare in this subfield. The SHAP analysis for DoS is routine but the feature rankings are plausible and consistent across both models.\n\nThe soft spots are real but mostly fixable. The random 80/20 split of the full NSL-KDD, rather than the standard KDDTrain+/KDDTest+ protocol, makes the accuracy non-comparable to the literature and inflates it because test samples come from the same distribution. The abstract's claimed superiority over traditional and black-box models is never tested — there are no baselines in the paper. The survey has 15 participants, all from one country, and the usability Cronbach's alpha is 0.60, below the usual 0.70 threshold; the authors acknowledge this. Minor inconsistencies (macro recall 0.84 vs 0.99 in the text, Bi-LSTM in the related work, a missing section number) suggest editing needs care but don't undermine the core.\n\nWho this is for: readers working on applied XAI for NIDS who want to see how a small user study can be bolted onto a standard model, and who might cite it as an example of human-centered evaluation. It does not deserve publication as is, but it's not a waste of referee time — with baselines, a corrected split, and reframed claims, it could be a modest conditional accept.\n\nMy recommendation: send it to review if the venue has a reasonable bar for empirical XAI work, but tell the authors to fix the split, add baselines, and retract the superiority claim. If they do that, I'd be fine with it as a workshop-level paper.","headline":"Routine CNN/LSTM on NSL-KDD plus SHAP and a small trust survey; the 99% accuracy is real under the authors' own random split, but the unsupported superiority claim and non-standard split keep it from being a serious contribution.","tokens_in":11992,"tokens_out":3791,"would_cite":false,"duration_ms":35568,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that an intrusion detection system combining CNN and LSTM networks can reach 99 percent accuracy on the NSL-KDD benchmark while remaining interpretable through SHAP explanations, and that security experts find the resultin","keywords":["intrusion detection","explainable AI","SHAP","CNN","LSTM","NSL-KDD","human-centered evaluation","trust calibration"],"falsifier":"Run the same CNN and LSTM architectures on the standard KDDTrain+ training set and KDDTest+ test set, and compare per-class accuracy and macro precision/recall/F1. If accuracy falls substantially below 99 percent, or if the class distribution in Section III-A does not match the actual NSL-KDD counts, the central performance claim is refuted.","tokens_in":10946,"feed_emoji":"🛡️","tokens_out":4059,"duration_ms":39013,"temperature":0.7,"pith_summary":"This paper tries to show that an intrusion detection system can be both accurate and explainable. The authors train a CNN and an LSTM on the NSL-KDD benchmark and report that both reach 99 percent overall accuracy, with the LSTM leading on macro-averaged precision, recall, and F1. They then apply SHAP to show which network features drive the models' decisions, finding that connection-error rates such as srv_serror_rate and serror_rate dominate DoS classification. Finally, they run a small expert survey through a custom UI to see whether security specialists trust and can use the explanations. The point is that high-performance deep learning can be paired with transparency in a way that supports real-world security deployment.","feed_headline":"Both CNN and LSTM hit 99% accuracy on NSL-KDD","feed_subtitle":"SHAP shows connection-error rates drive DoS detection; a 15-expert survey tests whether the explanations earn trust.","key_machinery":"The framework pairs two deep learning architectures — a 3-layer CNN that reads each 41-feature record as a one-dimensional sequence, and a 3-layer LSTM with dropout that treats the same features along a temporal dimension. Both classify traffic into five categories: Normal, DoS, Probe, R2L, and U2R. SHAP (SHapley Additive exPlanations) is the explainability engine: it computes feature attributions for the DoS class, with DeepExplainer and GradientExplainer used for the CNN and KernelExplainer for the LSTM. A custom UI presents predictions and SHAP explanations to cybersecurity experts, and survey instruments adapted from Mini-IPIP6, STIAS, TiA, and SUS capture trust, reliability, and usabili","core_discovery":"The central claim is that combining a CNN with an LSTM and adding SHAP explanations yields an intrusion detection system that is both highly accurate and interpretable. On NSL-KDD, both models reach 0.99 accuracy; SHAP identifies srv_serror_rate, dst_host_srv_serror_rate, and serror_rate as the most influential features for DoS detection, which the authors interpret as consistent with connection-failure patterns in denial-of-service attacks. A 15-person expert survey reports acceptable trust and reliability for the interface, though usability is marginal. The authors argue this closes the gap between black-box performance and human trust.","pith_inferences":["A direct test of the paper's core claim would be to rerun the identical models on the standard KDDTrain+/KDDTest+ partitions rather than a random 80/20 split; per-class results, especially R2L and U2R, would likely shift and the reported 99 percent accuracy may not survive because the paper's class counts in Section III-A appear to swap R2L and Probe.","The SHAP results are computed for only 100 test instances and one class (DoS); extending to other attack classes and larger samples would test whether the explanations are robust or class-specific artifacts.","The expert survey has only 15 participants from one country; the trust and reliability findings are promising but not generalizable until replicated with a larger, more diverse analyst pool.","If error-rate features remain top predictors across architectures, it suggests a cheap, simpler model trained on just those few features might achieve much of the same DoS performance — a testable extension."],"forward_implications":["If the 99 percent accuracy holds on standard benchmarks, deep learning IDS can match or exceed traditional signature-based systems while remaining interpretable.","SHAP's consistent identification of error-rate features across both architectures suggests a stable, physically meaningful signal for DoS detection that aligns with known attack patterns.","The expert survey results imply that explanations can make analysts trust a deep learning IDS enough for practical use, though usability needs work.","The framework's reliance on all 41 raw features, without dimensionality reduction, suggests that feature engineering can be deferred when using deep models."],"fun_headline_variants":["LSTM and CNN hit 99% accuracy on NSL-KDD","SHAP reveals top DoS features: serror rates dominate","Deep XAI framework: 99% IDS accuracy, SHAP explains","Interpretable IDS: CNN-LSTM hits 99%, SHAP flags serrors"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the authors' handling of NSL-KDD is valid: they split the full 125,973-row dataset randomly 80/20 instead of using the standard KDDTrain+/KDDTest+ partitions, and their stated class counts for R2L and Probe appear swapped. If the split or labels are wrong, the 99 percent accuracy and per-class comparisons are not meaningful against the published benchmark.","fun_headline_variants_meta":{"raw":{"variants":["LSTM and CNN hit 99% accuracy on NSL-KDD","SHAP reveals top DoS features: serror rates dominate","Deep XAI framework: 99% IDS accuracy, SHAP explains","Interpretable IDS: CNN-LSTM hits 99%, SHAP flags serrors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000228,"raw_usage":{"total_tokens":1335,"prompt_tokens":792,"completion_tokens":543,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":536,"completion_tokens_details":{"reasoning_tokens":462}},"tokens_in":536,"tokens_out":543,"duration_ms":5877,"temperature":1.0,"reasoning_tokens":462,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T04:22:17.430531+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same CNN and LSTM architectures on the standard KDDTrain+ training set and KDDTest+ test set, and compare per-class accuracy and macro precision/recall/F1. If accuracy falls substantially below 99 percent, or if the class distribution in Section III-A does not match the actual NSL-KDD counts, the central performance claim is refuted.","supporting_citations":[],"review_version":1}