{"id":"ebdffca3-2417-4787-9241-8bb1a39b5696","arxiv_id":"2501.11618","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A curriculum-learning neural network with LIME feature un-learning and ensemble stacking reports 97 to 98 percent accuracy on Edge-IIoT, CIC-APT-IIoT-2024, and CIC-IoV-2024 intrusion detection benchmarks.","lead":"This paper proposes an IoT intrusion detection framework that trains a GRU/LSTM/attention network in curriculum stages, uses LIME to drop low-relevance features, and stacks Random Forest and XGBoost on top, reporting 97 to 98 percent accuracy on three benchmark datasets. The generalist interest is whether a lightweight, explainable model can secure edge devices without losing accuracy.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pre-split LDA/StandardScaler fitting leaks test information into training, so the reported 97–98% accuracies are not yet unbiased evidence for the central claim.","rationale":"The reader's weakest assumption focused on the 80-20 stratified random split for sequential IoT and CAN-bus data and on generalization of curriculum stages and the LIME threshold. I agree those are risks, but Section 4.1 exposes a more direct and concrete leakage: StandardScaler and LDA appear to be fitted before the split. LDA is a supervised projection, so fitting it on all data uses test labels. That makes the reported Table 1 numbers biased regardless of temporal autocorrelation. I also note secondary inconsistencies, such as the CIC-IoV F1-score reported as 99% in the Section 5 prose but 97% in Table 1, and the absence of error bars, seeds, or code; those reinforce the need for verification but are not the central load-bearing issue. The proposed test is a single, decisive check: rerun with preprocessing fit only on the training partition. I keep the reader's CONDITIONAL verdict because the flaw is concrete and fixable by re-evaluation, but the condition should explicitly require split-before-preprocessing. If the corrected accuracy falls, the central claim should be rejected.","tokens_in":10132,"tokens_out":7186,"duration_ms":86510,"concrete_test":"Ask the authors for the exact preprocessing code or rerun the pipeline from raw data with a strict protocol: (1) split CIC-IoV-2024 into 80/20 by time or by session before any scaling or LDA; (2) fit StandardScaler and LDA on the training split only; (3) apply the fitted transforms to the untouched test split; (4) run the same curriculum stages and ensemble and report accuracy and F1. If accuracy drops materially below the reported 98%, or if the ablation increments in Table 2 change, the central claim is unsupported as stated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim depends on the test set being truly held out. Section 4.1 states that feature scaling (StandardScaler) and supervised dimensionality reduction (LDA) are applied before the 80-20 split: 'Data reduction methods such as Linear Discriminant Analysis (LDA) reduce feature dimension... This processed data gets split up into training and testing data by an 80-20 split.' Fitting LDA on the full dataset uses test labels to construct the projection, and fitting the scaler on the full dataset uses test distribution statistics, so the accuracies in Table 1 are not unbiased estimates of generalization. This is a correctness risk independent of the temporal-leakage concern raised by the reader: even if a time-based split were used, the test set would still have influenced preprocessing. The paper supplies no code, seeds, or feature-level details that would rule out this reading. The hand-crafted curriculum stages and LIME threshold of -0.01 are secondary; they cannot rescue an evaluation whose held-out set has already informed the feature space.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a curriculum learning framework for IoT/IIoT intrusion detection, combining a GRU/LSTM/attention neural network with a LIME-based feature un-learning mechanism, edge-oriented quantization/pruning, and a stacking ensemble of the staged model with Random Forest and XGBoost. The authors evaluate the framework on Edge-IIoT, CIC-APT-IIoT-2024, and CIC-IoV-2024, reporting accuracies of 97%, 98%, and 98%, respectively, with a lightweight model of 94,051 parameters, and present an ablation study attributing accuracy gains to curriculum learning, XAI integration, and stacking. The central claim is that this combination yields accurate, transparent, and lightweight intrusion detection for diverse IoT networks.","tokens_in":10355,"tokens_out":3060,"duration_ms":35759,"significance":"If the reported results were sound, the paper would offer a practically relevant contribution: a compact temporal model with explicit parameter counts and deployment-oriented techniques, evaluated on three recent IoT/IIoT security datasets. The use of CIC-IoV-2024 and CIC-APT-IIoT-2024 is timely, and the explicit ablation breakdown in Table 2 is a useful organizing device. However, the empirical core is not currently reproducible or verifiable: no code, seeds, or hyperparameters are provided, and the preprocessing description in Section 4.1 indicates a supervised dimensionality-reduction step before the train/test split. The significance of the central accuracy claim therefore depends on fixes that are within the scope of a revision.","major_comments":[{"comment":"The preprocessing description states that StandardScaler and Linear Discriminant Analysis (LDA) are applied before the 80-20 split: 'This processed data gets split up into training and testing data by an 80-20 split.' Since LDA is a supervised projection that uses class labels and StandardScaler uses full-data statistics, the test set has already influenced the feature space, making the accuracies in Table 1 optimistically biased. Please move all preprocessing inside a cross-validation or nested-split pipeline, or explicitly state that the scaler and LDA were fit only on the training portion; providing code or detailed pseudo-code for the split would also allow verification.","section":"Section 4.1, Table 1"},{"comment":"The reported F1-score for CIC-IoV-2024 is internally inconsistent: the text states 'F1-Score 99%' while Table 1 reports 97%, and precision 100% with recall 97% implies F1 approximately 98.5%. These numbers cannot all be correct. Please recompute and harmonize all metrics, and report the exact formulas used for precision, recall, and F1.","section":"Section 5, Table 1"},{"comment":"The ablation study reports single-point accuracy increments (+6% for curriculum learning, +3% for XAI/LIME un-learning, +1% for stacking) without standard deviations, repeated seeds, or statistical tests. Moreover, the LIME-based un-learning removes features based on the model's own explanations, so the +3% could be an in-sample artifact if the LIME importance scores are computed on the full dataset or if the stage-wise retraining is not nested within the training split. Please clarify which data are used to compute LIME importances, and report mean and variance over multiple runs.","section":"Section 5, Table 2"},{"comment":"The paper does not compare the proposed method against any existing intrusion-detection baselines on the same datasets, even though Section 3 cites related work reporting high accuracies on Edge-IIoT (e.g., CNN-GRU and ensemble methods). Without such comparisons, the claim that the curriculum/XAI/ensemble framework 'further enhances generalization' is not empirically established. Please add comparisons to at least the strongest prior methods on each dataset, ideally from the references cited in Section 3.","section":"Section 5, Tables 1 and 2"}],"minor_comments":[{"comment":"The manuscript contains ACM template placeholders that should be removed before submission, including 'Make sure to enter the correct conference title from your rights confirmation email', 'XX, XX', and the placeholder DOI and ISBN lines.","section":"Title page and template metadata"},{"comment":"The text refers to 'LDA plots or Random Forest feature importance plots' and 'log scaling and binning' but none of these feature-engineering artifacts are shown or described with concrete settings; please either present them or remove the unsupported details.","section":"Section 4.1"},{"comment":"The hyperparameter-importance figure is presented without any description of how the importance values were obtained; please specify the sensitivity-analysis procedure, including the ranges of hyperparameters tested.","section":"Section 5, Figure 4"},{"comment":"The manuscript needs careful copyediting for grammar and style; for example, 'The motivation for this research is that the IoT devices are getting more and more mainstream' and 'it’s scalability' are unclear or incorrect. Also, table and figure references should be consistent (e.g., Table 2 is introduced after a paragraph that says 'the table 2 above').","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be a very early draft: the template placeholders, inconsistent table values, and absence of code/experimental detail make it difficult to assess. The data-leakage concern in Section 4.1 is serious because it directly affects the headline accuracies, but it is fixable by re-running the experiments with a proper split and making the pipeline available. The paper also needs realistic baseline comparisons to justify its novelty claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, quick take on arXiv:2501.11618. The paper's core idea is to train an intrusion detector with curriculum learning (simple attacks first), use LIME to identify and drop low-relevance features between stages, and stack a GRU/LSTM/attention net with RF and XGBoost. The ablation on CIC-IoV-2024 shows steady gains: +6% from curriculum, +3% from LIME-based feature un-learning, +1% from the ensemble, reaching 98% accuracy. That is a coherent story, and the lightweight model (94k parameters) is a plus. Credit where due: the staged learning setup is concretely described, the datasets are recent and relevant, and the ablation is a legitimate attempt to attribute gains.\n\nThe problems are in the evaluation. The biggest one is in Section 4.1: StandardScaler and LDA are applied before the 80-20 split. Fitting LDA on the full dataset uses test labels, and the scaler uses test statistics. That means the reported 97-98% accuracies are not unbiased estimates of generalization. This is load-bearing, not a quibble. Even a proper temporal split would not fix it as long as preprocessing is fit on the full set. The paper needs a re-run with preprocessing inside the training folds.\n\nSmaller issues: Table 1 reports F1 97% for CIC-IoV-2024 while the text says 99% (and precision 100/recall 97 implies ~98.5). No code, no seeds, no error bars, and no comparison to existing methods on the same datasets. The random 80-20 split for sequential IoT/CAN data is another vulnerability, though secondary.\n\nThe XAI contribution is an in-sample feature selection loop, which is fine as a modeling choice, but the claimed +3% is evidence only that dropping low-LIME-importance features helps on these benchmarks, not that the threshold generalizes to unseen attacks.\n\nWho is this for? A reader working on IoT intrusion detection might find the curriculum-plus-masking idea worth trying, but should not trust the headline numbers as they stand. The paper deserves a serious referee, but the first request will be a clean re-run with baselines. If those come back and the gains hold, this becomes a modest contribution. As is, it is a 'maybe' for a reading group and not something I would cite.\n\nRecommendation: send to peer review with heavy-revision expectations. The framework is not junk, but the evaluation protocol invalidates the central claim until fixed.","headline":"A plausible curriculum-learning pipeline for IoT intrusion detection, but the pre-split LDA/scaling makes the reported accuracies unreliable as evidence of generalization.","tokens_in":10878,"tokens_out":2519,"would_cite":false,"duration_ms":26496,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Curriculum learning with LIME-guided feature removal reaches 97–98% accuracy on IoT attack datasets.","keywords":["Curriculum Learning","Explainable AI (XAI)","LIME","Intrusion Detection System","IoT Security","Ensemble Stacking","Edge Deployment","Lightweight Neural Network"],"falsifier":"Train and test the framework on CIC-IoV-2024 using a time-ordered split—earliest 80% of messages for training, latest 20% for testing—instead of the paper's random 80-20 split, and compare the accuracy. If it drops well below 98%, the reported result depends on random splitting of sequential CAN-bus traffic and would not transfer to real deployments.","tokens_in":9955,"feed_emoji":"🛡️","tokens_out":7224,"duration_ms":62915,"temperature":0.7,"pith_summary":"The paper claims that a curriculum learning framework, where a lightweight neural network is trained first on easy normal and attack patterns and then on harder attacks, can detect intrusions in IoT networks at 97–98% accuracy across three recent benchmark datasets. It argues that adding LIME explanations to identify and remove low-relevance features at each stage, and then stacking the resulting model with Random Forest and XGBoost, contributes measurable accuracy gains over the base network. The reason this matters is that the whole model fits in about 367 KB (94,051 parameters), so the claimed performance is supposed to be achievable on resource-constrained edge devices while remaining interpretable. If the result holds, IoT security systems could get both high detection rates and human-readable explanations without needing large cloud infrastructure.","feed_headline":"Curriculum learning hits 97–98% on IoT attack detection","feed_subtitle":"Staged training, LIME feature un-learning, and stacking lift accuracy from 88% to 98%.","key_machinery":"The load-bearing mechanism is the staged curriculum loop: at each of several hand-designed stages (normal data, then simple, medium, and complex attacks), the shared neural network is trained, LIME is run on a fraction of the staged data, features whose importance falls below a threshold ($-0.01$) are removed, and the model is retrained on the remaining features. This 'feature un-learning' is what ties XAI to the curriculum, letting the model discard low-relevance inputs as it progresses to harder attacks. The neural backbone—GRU and LSTM layers plus self-attention and residual connections—captures both short- and long-term temporal dependencies in sequential IoT data, and the final stacking ensemble with Random Forest and XGBoost adds a small but consistent generalization gain. The architecture is kept edge-deployable by pruning and quantization, yielding 94,051 parameters.","core_discovery":"The paper's central claim is that a neural network built from GRU, LSTM, and attention layers, trained in curriculum stages ordered by attack complexity and refined by LIME-based feature un-learning, then combined into a stacking ensemble, achieves 98% accuracy on CIC-IoV-2024 and CIC-APT-IIoT-2024 and 97% on Edge-IIoT. The framework uses an adaptive feature mask layer, a dynamic convolutional layer for short-term dependencies, an attention-based temporal encoder for long-term dependencies, residual connections, and normalization, and is pruned and quantized to 94,051 parameters (367.39 KB). The paper reports an ablation on CIC-IoV-2024 in which the base network scores 88%; adding curriculum learning gives 94%, LIME integration and feature un-learning give 97%, and the stacking ensemble gives 98%. It also reports 99–100% precision, 95–97% recall, and F1-scores of 96–99% across the three datasets.","pith_inferences":["A natural stress test the authors did not run: replace the random 80-20 split with a time-ordered split on CIC-IoV-2024's CAN-bus stream. If accuracy drops, the random split may be letting the model see temporally adjacent attack traffic in both train and test, which would overstate real-world performance.","The LIME threshold of $-0.01$ is a fixed scalar chosen by inspection. An extension would be to make feature removal adaptive per stage, for example by keeping the top-k features by LIME weight, and to test whether the 3% XAI gain persists across thresholds.","Because curriculum stage groupings were hand-crafted per dataset (4 stages for Edge-IIoT, 2 for CIC-APT-IIoT, 3 for CIC-IoV-2024), a skeptic would want to see whether an automatic curriculum, with difficulty scored by model loss or confidence, reproduces the gains without hand-tuning."],"forward_implications":["If the reported accuracies hold, a single 367 KB model can serve as an intrusion detector on edge devices for vehicular (CAN-bus), general IoT, and industrial IoT traffic, with no cloud round-trip.","The ablation implies that each design choice—curriculum ordering, LIME-based feature removal, and ensemble stacking—pays a measurable accuracy dividend (6, 3, and 1 points respectively on CIC-IoV-2024), so the framework's performance is not attributable to one component alone.","Because LIME explanations are attached to each prediction, a security analyst can see which features (e.g., arp.opcode, dns.retransmission) drove an attack decision, which is the transparency the paper argues is missing from other IoT intrusion detection approaches.","The lightweight footprint suggests the framework could be further quantized or combined with federated learning, which the paper lists as future work for privacy-preserving distributed training."],"supporting_citations":[{"why":"Provides the CIC-IoV-2024 CAN-bus benchmark and its DoS and spoofing attack classes.","marker":"[28]"},{"why":"Supplies the Edge-IIoTset multi-layer IoT/IIoT dataset with 14 attack types used for the 97% result.","marker":"[29]"},{"why":"Provides the CIC-APT-IIoT-2024 provenance-based APT dataset for IIoT, behind the 98% result.","marker":"[30]"},{"why":"Defines the curriculum learning paradigm the framework builds on.","marker":"[10]"},{"why":"Introduces the original curriculum learning formulation that motivates staged training.","marker":"[11]"},{"why":"Supplies the LIME explainer used for feature un-learning and for the explanation example in Figure 5.","marker":"[18]"},{"why":"Frames the strengths and limitations of LIME that motivate its use here.","marker":"[17]"},{"why":"Provides the hybrid CNN-GRU baseline on Edge-IIoTset that the framework competes against.","marker":"[26]"}],"fun_headline_variants":["98% IoT attack detection with staged AI training","Curriculum learning and XAI push IoT security to 98% accuracy","Lightweight XAI model hits 97–98% accuracy on IoT attacks","Adaptive curriculum learning boosts IoT attack detection to 98%","Staged AI training with XAI lifts IoT detection accuracy to 98%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on assuming that randomly splitting each dataset into 80% train and 20% test keeps temporally ordered IoT and CAN-bus traffic from leaking between the two, and that the hand-chosen curriculum stages and a LIME cutoff of $-0.01$ will also work on attack types the model has not seen.","fun_headline_variants_meta":{"raw":{"variants":["98% IoT attack detection with staged AI training","Curriculum learning and XAI push IoT security to 98% accuracy","Lightweight XAI model hits 97–98% accuracy on IoT attacks","Adaptive curriculum learning boosts IoT attack detection to 98%","Staged AI training with XAI lifts IoT detection accuracy to 98%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000678,"raw_usage":{"total_tokens":3080,"prompt_tokens":943,"completion_tokens":2137,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":2047}},"tokens_in":559,"tokens_out":2137,"duration_ms":15776,"temperature":1.0,"reasoning_tokens":2047,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:02:19.828086+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train and test the framework on CIC-IoV-2024 using a time-ordered split—earliest 80% of messages for training, latest 20% for testing—instead of the paper's random 80-20 split, and compare the accuracy. If it drops well below 98%, the reported result depends on random splitting of sequential CAN-bus traffic and would not transfer to real deployments.","supporting_citations":[{"cited_title":"& Ghorbani, A","cited_arxiv_id":null,"evidence_quote":"Provides the CIC-IoV-2024 CAN-bus benchmark and its DoS and spoofing attack classes."},{"cited_title":"& Janicke, H","cited_arxiv_id":null,"evidence_quote":"Supplies the Edge-IIoTset multi-layer IoT/IIoT dataset with 14 attack types used for the 97% result."},{"cited_title":"& Sebe, N","cited_arxiv_id":null,"evidence_quote":"Defines the curriculum learning paradigm the framework builds on."},{"cited_title":"& Weston, J","cited_arxiv_id":null,"evidence_quote":"Introduces the original curriculum learning formulation that motivates staged training."},{"cited_title":"& Rajkumar, C","cited_arxiv_id":null,"evidence_quote":"Supplies the LIME explainer used for feature un-learning and for the explanation example in Figure 5."},{"cited_title":"& Aliouat, Z","cited_arxiv_id":null,"evidence_quote":"Provides the hybrid CNN-GRU baseline on Edge-IIoTset that the framework competes against."}],"review_version":1}