{"id":"f9bca2c0-74e5-4f28-9674-41d2a6a77104","arxiv_id":"2502.06138","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A hybrid ANN-DCNN-BiLSTM-RNN ensemble is reported to detect botnet attacks with 99.76% accuracy, but the UNSW-NB15 dataset used has no botnet category.","lead":"The authors stack four standard neural networks and claim 99.76% accuracy for detecting botnet attacks in IoT networks using the UNSW-NB15 dataset. The claim is difficult to take at face value because this dataset does not contain a botnet class, so the title and abstract appear to overstate what the model actually detects.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"UNSW-NB15 contains no botnet class (Table II); the reported 99.76% is at best generic attack/Normal discrimination, so the botnet-detection claim is unsupported.","rationale":"The reader's weakest assumption is exactly the load-bearing point. I independently checked the manuscript: Section III-A says the dataset comprises 'Normal' plus nine attack types; Table II enumerates those nine (Fuzzers, Backdoors, DoS, Exploits, Generic, Reconnaissance, Analysis, Shellcode, Worms) plus Normal. UNSW-NB15, as described and as publicly documented, has no botnet category. Thus a model trained on this dataset cannot be said to detect botnets unless the authors define a botnet proxy, which they do not. The high aggregate metrics are consistent with common UNSW-NB15 results, where Normal-vs-attack or 10-class classification is easy to push to high accuracy; they provide no evidence specific to botnet traffic. I also note secondary reproducibility issues (no train/test split or code, inconsistent architecture naming ANN vs Bi-GRU vs RNN in abstract/Sections III–IV), but they are secondary. Because the central claim fails on dataset-label grounding, the reader's REJECT verdict is appropriate, and no adjustment is needed.","tokens_in":7441,"tokens_out":3018,"duration_ms":24587,"concrete_test":"Obtain the official UNSW-NB15 training and testing CSV files (or the labeled ground-truth partition) and count rows in the attack category column whose label is 'botnet' (or any botnet-derived label). If the count is zero, the dataset cannot support a botnet-detection experiment, and the paper's Table IV metrics should be reinterpreted as generic intrusion-detection performance. A complementary check: inspect any code/configuration for label remapping; none is provided.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III-A and Table II define the UNSW-NB15 label set as Normal, Fuzzers, Backdoors, DoS, Exploits, Generic, Reconnaissance, Analysis, Shellcode, and Worms. No 'botnet' class exists. Since the abstract and conclusion claim botnet-attack detection, every reported metric (99.76% accuracy, 98.46% precision, 97.10% recall, 98.66% F1, 99.18% ROC-AUC) is at best a measure of multiclass attack-vs-normal discrimination or attack-family classification. The paper provides no per-class confusion matrix, no mapping from any UNSW-NB15 attack label to 'botnet', and no separate botnet evaluation set. Therefore the central claim is unsupported by the data, independent of architecture quality. This is not a stylistic issue: it changes what the experiment demonstrates.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a stacked hybrid deep learning model combining ANN, DCNN, BiLSTM, and RNN for botnet attack detection in IoT environments, using the UNSW-NB15 dataset. The authors report a testing accuracy of 99.76%, precision of 98.46%, recall of 97.10%, F1-score of 98.66%, and ROC-AUC of 99.18%, and compare these results with several existing models. The central claim is that the proposed ensemble detects botnet attacks with state-of-the-art performance. However, the paper's own dataset description lists no botnet class, and the experimental setup (data split, preprocessing, balancing, and hyperparameter selection) is not sufficiently specified to support the reported claim.","tokens_in":7639,"tokens_out":3273,"duration_ms":29289,"significance":"If the central claim were supported, a 99.76% accuracy on botnet detection could be a meaningful incremental empirical contribution, since the proposed ensemble is not theoretically novel but might be practically useful. The paper's strengths include the use of a standard public dataset, reporting multiple evaluation metrics, and providing a comparison with prior work. These strengths do not compensate for the absence of a botnet label in the dataset used and the lack of reproducibility details. As presented, the results at best demonstrate attack-versus-normal discrimination or multiclass attack-family classification, not botnet detection, so the significance of the claimed finding is not established.","major_comments":[{"comment":"The UNSW-NB15 label set as presented in Table II has ten labels: Normal, Fuzzers, Backdoors, DoS, Exploits, Generic, Reconnaissance, Analysis, Shellcode, and Worms; there is no botnet category. Nevertheless, the abstract and conclusion claim that the model detects botnet attacks with 99.76% accuracy and 99.18% ROC-AUC. The manuscript provides no mapping from any UNSW-NB15 attack label to botnet behavior, no per-class confusion matrix, and no separate botnet evaluation set. Therefore every reported metric is at best a measure of attack-versus-normal discrimination or multiclass attack classification, not botnet detection, and the central claim is unsupported by the data.","section":"Section III-A, Table II; Abstract; Conclusion"},{"comment":"Hyperparameters appear to be selected on the same evaluation set used for the final results. Table III reports accuracy for 15 configurations of ANN, CNN, LSTM, RNN, and the proposed model, then selects the configuration with 99.76% accuracy; Table IV reports test-set metrics at five epochs. No validation split is described. The paper does not state how overfitting is controlled, whether the test set is the standard UNSW-NB15 test partition, or whether the reported test metrics are the ones used for configuration selection. Without this specification, the reported numbers are at risk of selection bias and cannot be independently reproduced.","section":"Section IV, Table III; Section V, Table IV"},{"comment":"The dataset description is internally inconsistent and does not anchor the classification task. Section III-A states that UNSW-NB15 comprises 82,332 records and that balancing attack types totals 82,332 cases, whereas the standard UNSW-NB15 release has a much larger full set with separate training and test partitions. Moreover, Table III describes the proposed model with a single output unit and sigmoid activation, which is a binary-output architecture, while Table V reports the proposed model as 'Multiclass class.' The manuscript never defines the task as binary Normal-versus-attack, binary botnet-versus-other, or multiclass attack-family classification, so the reported 99.76% accuracy is not attached to a well-defined problem.","section":"Section III-A; Section IV, Table III; Table V"}],"minor_comments":[{"comment":"The listed contributions mention Bi-GRU, but the proposed model name and all subsequent descriptions use ANN+DCNN+BiLSTM+RNN; clarify the exact architecture.","section":"Section IV"},{"comment":"Equations (1) through (5) appear as empty placeholders; the actual formulas for accuracy, precision, recall, F1-score, and ROC-AUC should be provided.","section":"Section V, Equations (1)-(5)"},{"comment":"The F1-score is reported as 98.66 at both epoch 15 and epoch 25, while the precision and recall values differ; please verify these entries for consistency.","section":"Table IV"},{"comment":"The comparative baseline results in Table V are listed as binary-class evaluations, while the proposed model is listed as multiclass, so the comparison is not based on a common protocol; state the evaluation setting for all models.","section":"Table V"},{"comment":"Several references contain formatting inconsistencies or missing DOI information (e.g., [13], [21], [25]); please standardize the reference list.","section":"References"}],"recommendation":"reject","confidential_remarks":"The central problem is that the paper claims botnet detection on a dataset that, by its own description, contains no botnet class. This is not a stylistic issue; it changes the scientific claim. The hyperparameter selection procedure also appears to use the test set for model choice, and no validation split is described. A resubmission with a botnet-specific dataset (e.g., Bot-IoT, N-BaIoT, or CTU-13) or with a clearly scoped and fully specified multiclass attack-detection claim could be reconsidered, but the present manuscript does not support its central conclusion."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi [Colleague],\n\nQuick take: the paper's headline claim doesn't survive contact with its own Table II. UNSW-NB15 has no botnet class; it has nine attack families. So the 99.76% accuracy and 99.18% AUC are, at best, evidence of strong generic attack-vs-normal discrimination. They are not evidence about botnet detection. The stack of ANN, DCNN, Bi-LSTM, and RNN is a routine ensemble; nothing new in architecture, loss, or feature engineering.\n\nWhat the paper does well: it runs a systematic hyperparameter sweep, reports a plausible ROC-AUC curve, and compares against several prior models on the same benchmark. If you read the results as \"multiclass attack classification on UNSW-NB15,\" the numbers are reasonable and the engineering is coherent.\n\nThe soft spots are serious. First and load-bearing: the dataset mismatch. Table II explicitly lists the classes—Normal, Fuzzers, Backdoors, DoS, Exploits, Generic, Reconnaissance, Analysis, Shellcode, Worms—no botnet. The paper never maps any of these to a botnet label, and there's no per-class confusion matrix. So the submitted claims are unsupported. Second, the architecture description is inconsistent: the abstract and contributions say Bi-GRU, but the method and Table III use RNN; the final model is called ANN+DCNN+BiLSTM+RNN. Third, the train/test split, preprocessing, and class balancing are unspecified, and no code or data are shared. Fourth, Table III appears to select hyperparameters by test-set accuracy, which is a mild selection bias; there are no error bars. These aren't minor quibbles—the first one alone is enough to invalidate the central claim.\n\nWho's this for? Readers who want a quick benchmark table for hybrid networks on UNSW-NB15 might find something, but anyone looking for botnet-specific detection gets nothing they can trust. With revision, the paper could be reframed as an ensemble benchmark for general NIDS, but as submitted it's not honest about what it measures.\n\nMy recommendation: desk reject. The mismatch between the claim and the dataset is fundamental, and the other omissions would require major revision. It's a useful cautionary example for a lab meeting, but it doesn't earn referee time.\n\n[Your name]","headline":"UNSW-NB15 has no botnet class, so the reported 99.76% is generic attack detection, not botnet detection; the architecture is a routine ensemble with serious reporting gaps.","tokens_in":8184,"tokens_out":3071,"would_cite":false,"duration_ms":26135,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper reports that a four-network stacked deep learning model detects botnet attacks in IoT with 99.76% accuracy on the UNSW-NB15 dataset.","keywords":["botnet detection","IoT security","deep learning","hybrid stacking","UNSW-NB15","artificial neural network","convolutional neural network","bidirectional LSTM"],"falsifier":"Inspect the actual class labels used to train the model: if none of the 82,332 records carries a 'botnet' label or a botnet-family name, the reported accuracy cannot be a botnet-detection score. Re-running the same stacked model on a dataset with explicit botnet labels, such as N-BaIoT or Bot-IoT, and comparing per-class accuracy would settle whether the botnet claim holds.","tokens_in":7260,"feed_emoji":"🛡️","tokens_out":9218,"duration_ms":73184,"temperature":0.7,"pith_summary":"The paper claims that a stacked hybrid of four deep-learning architectures—an artificial neural network, a deep convolutional network, a bidirectional long short-term memory network, and a recurrent neural network—can detect botnet attacks in an IoT environment. Using the UNSW-NB15 dataset, the model reports 99.76% testing accuracy, 98.46% precision, 97.10% recall, 98.66% F1-score, and 99.18% ROC-AUC. The authors argue that stacking these complementary learners captures both spatial patterns in packet data and temporal changes in attack behavior better than individual networks or earlier CNN-LSTM hybrids. A sympathetic reader would take this as evidence that hybrid stacking is a practical route to high-accuracy attack detection for connected devices.","feed_headline":"Four-network hybrid claims 99.76% accuracy for botnet detection","feed_subtitle":"A stack of ANN, DCNN, BiLSTM, and RNN beats single-model baselines on the UNSW-NB15 attack dataset.","key_machinery":"The load-bearing object is the stacking ensemble itself: an 18-layer architecture that joins an artificial neural network, a deep convolutional neural network, a bidirectional long short-term memory network, and a recurrent neural network, ending in a single sigmoid or tanh output unit. The model uses 64 and 32 hidden units, relu and sigmoid activations, and the adagrad optimizer over 25 epochs. The claimed mechanism is complementary specialization: the deep convolutional component extracts spatial regularities from raw traffic fields, the bidirectional long short-term memory and recurrent components track temporal dynamics as botnet behavior evolves, and the artificial neural network layer fuses their learned representations for the final decision. According to the paper, this division of labor is what lifts accuracy from the 80–96% range of individual configurations to 99.76% for the stack.","core_discovery":"The central discovery is that the named ANN+DCNN+BiLSTM+RNN stacking model, trained for 25 epochs on 82,332 balanced UNSW-NB15 records, classifies network traffic with 99.76% accuracy, 98.46% precision, 97.10% recall, 98.66% F1-score, and 99.18% ROC-AUC. The paper presents these figures as proof that the stack outperforms both the individual network configurations it fine-tuned, which ranged from 80.25% to 96.88% accuracy, and existing LSTM, CNN1D, and CNN-LSTM baselines, which ranged from 70.00% to 93.68% accuracy. This is the strongest result reported in the paper and the basis for the claim that hybrid stacking captures the complex, evolving patterns of botnet activity.","pith_inferences":["A reader should be cautious about reading the headline as pure botnet detection: the paper's own Table II lists nine UNSW-NB15 attack classes with no 'botnet' label, so the result is more precisely an attack-classification result unless the authors specify how those classes map to botnet activity.","The same stacking recipe would likely transfer to explicitly botnet-labelled datasets such as N-BaIoT or Bot-IoT, but the paper does not test that transfer, so it remains an open question.","The training-time column, ranging from 967 to 4644 seconds per run, suggests that real-time deployment would require inference optimization, which the paper does not address.","A natural next experiment is to retrain the stack on traffic captured from compromised IoT devices and compare detection latency and accuracy against the UNSW-NB15 numbers."],"forward_implications":["A stacked model of this type can be trained on UNSW-NB15-style flow features and applied to IoT gateway traffic to flag attack activity with high accuracy.","Because the reported precision and recall are both above 97%, the model would generate few false alarms per true detection, a practical requirement for security operations.","Individual networks in the paper's own hyperparameter table top out below 97% accuracy, so the claimed gain comes specifically from stacking rather than from any single component.","Future hybrid detectors can use the same four-way stacking recipe as a baseline for comparison on the UNSW-NB15 benchmark."],"supporting_citations":[{"why":"Supplies the UNSW-NB15 dataset on which every reported result is measured.","marker":"[22]"},{"why":"Defines the nine UNSW-NB15 attack classes plus normal traffic that the model is trained to classify.","marker":"[23]"},{"why":"Documents the UNSW-NB15 feature set and its use for intrusion detection, grounding the input representation.","marker":"[24]"},{"why":"Provides the LSTM baseline at 70.00% accuracy that the proposed model is compared against in Table V.","marker":"[25]"},{"why":"Provides the CNN-LSTM baseline at 93.68% accuracy in the same comparison.","marker":"[26]"},{"why":"Provides the CNN1D baseline at 89.80% accuracy in the same comparison.","marker":"[27]"},{"why":"Provides the CNN-LSTM baseline at 89.93% accuracy with precision and F1-score in the same comparison.","marker":"[28]"}],"fun_headline_variants":["Stacked deep nets hit 99.76% botnet detection","Hybrid AI stack detects IoT botnets at 99.76% accuracy","Four-network stack beats baselines for IoT botnet detection","Deep stacking model: 99.76% accurate botnet spotting","99.76% accuracy: stacked deep learning vs IoT botnets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the nine UNSW-NB15 attack classes listed in Table II can be treated as botnet attacks, even though no class is labelled 'botnet'; if that mapping is wrong, the 99.76% accuracy is evidence of general network-attack detection, not botnet detection.","fun_headline_variants_meta":{"raw":{"variants":["Stacked deep nets hit 99.76% botnet detection","Hybrid AI stack detects IoT botnets at 99.76% accuracy","Four-network stack beats baselines for IoT botnet detection","Deep stacking model: 99.76% accurate botnet spotting","99.76% accuracy: stacked deep learning vs IoT botnets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000209,"raw_usage":{"total_tokens":1421,"prompt_tokens":974,"completion_tokens":447,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":590,"completion_tokens_details":{"reasoning_tokens":356}},"tokens_in":590,"tokens_out":447,"duration_ms":4208,"temperature":1.0,"reasoning_tokens":356,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T16:37:15.815106+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Inspect the actual class labels used to train the model: if none of the 82,332 records carries a 'botnet' label or a botnet-family name, the reported accuracy cannot be a botnet-detection score. Re-running the same stacked model on a dataset with explicit botnet labels, such as N-BaIoT or Bot-IoT, and comparing per-class accuracy would settle whether the botnet claim holds.","supporting_citations":[{"cited_title":"Ahmad, Q","cited_arxiv_id":null,"evidence_quote":"Documents the UNSW-NB15 feature set and its use for intrusion detection, grounding the input representation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the LSTM baseline at 70.00% accuracy that the proposed model is compared against in Table V."},{"cited_title":"Lopez-Martin, B","cited_arxiv_id":null,"evidence_quote":"Provides the CNN-LSTM baseline at 93.68% accuracy in the same comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the CNN1D baseline at 89.80% accuracy in the same comparison."},{"cited_title":"Halbouni, T","cited_arxiv_id":null,"evidence_quote":"Provides the CNN-LSTM baseline at 89.93% accuracy with precision and F1-score in the same comparison."}],"review_version":1}