{"id":"6062bb2c-6902-4f7f-abb7-e60923b7a9c5","arxiv_id":"2509.20068","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A T+15 second anomaly prediction framework for IIoT, built on SDN telemetry and a digital twin, with reported LightGBM-GPU F2 score 0.822 on CICAPT-IIoT2024.","lead":"The paper proposes a digital twin for industrial IoT networks that predicts whether an attack will occur within the next 15 seconds, using SDN-collected traffic features and machine learning. The authors report that a GPU-accelerated LightGBM model performs best on two public datasets, though the test setup appears to leak future information.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Random stratified split leaks temporally adjacent near-duplicates into training, so the reported F2/AUC do not demonstrate T+15 forecasting for unseen future anomalies.","rationale":"The paper is transparent about its preprocessing and split, and the time-aware labeling idea is potentially reusable. However, the central empirical claim is a forecasting claim: the model should predict anomalies at T+15 on future data. A random stratified split on chronologically sorted records violates the temporal independence required for that claim. The reader's weakest_assumption identifies exactly this flaw, and the evidence in the paper supports it: the same attack flows appear on both sides of the split, so near-duplicate leakage is highly plausible. I do not see an internal inconsistency in the model architecture, but the evaluation protocol is the load-bearing support for the headline result. Since the current numbers do not establish generalization to unseen future anomalies, the reader's REJECT verdict is appropriate. No additional concern is needed; the rejection stands because the reported F2 and AUC cannot be interpreted as predictive performance without a temporal split.","tokens_in":8101,"tokens_out":5503,"duration_ms":44293,"concrete_test":"Re-run the LGBM-GPU pipeline on CICAPT-IIoT2024 with a strict chronological split: sort by timestamp, use the first 70% of records as training and the last 30% as test, with no shuffling. As a stronger check, discard the 15 seconds of training data immediately before the test start so no training label uses information from the test period. If F2 and AUC remain near 0.822/0.9982, the random split was not responsible; if they drop substantially, the reported numbers are leakage artifacts. Repeat the same procedure for CIC-IDS2017/RF+GS. Additionally, measure the minimum timestamp distance between each test sample and its nearest training sample under both split schemes to quantify temporal overlap.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III-B states that the CICAPT-IIoT2024 data was sorted chronologically by timestamp and then 'the dataset was split into training and testing subsets using stratified sampling to preserve class distributions.' Because label_t+15 is a forward-looking binary label, consecutive records from the same flow or attack session have nearly identical features and labels. A random stratified split interleaves these near-duplicates between training and test sets, so the model can effectively retrieve a test record's label from its temporally adjacent training neighbors. This inflates F2 and AUC and does not measure the ability to predict anomalies 15 seconds ahead on unseen future data. Table I's central numbers (LGBM-GPU F2=0.822, AUC=0.9982) are therefore not evidence of generalization. The same protocol is used for CIC-IDS2017. The abstract's claim of a 'real-time SD-TWIN deployment' is also not backed by deployment measurements; only offline training times are reported. This is a correctness risk in the evaluation protocol, not a flaw in the architecture or labeling idea itself.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents SD-TWIN, an SDN-based digital twin architecture for short-term anomaly prediction in IIoT. The authors generate a temporal label_t+15 for each network flow record, indicating whether an anomaly occurs within the next 15 seconds, and evaluate six machine/deep learning models on CICAPT-IIoT2024 and CIC-IDS2017. They report that a GPU-accelerated LightGBM achieves the best F2 (0.822) and AUC (0.9982) on the IIoT dataset, and that Random Forest with grid search performs well on CIC-IDS2017. The paper also describes a four-layer deployment with ONOS, Kafka, FastAPI, Redis Streams, and SHAP explanations, and discusses training-time trade-offs.","tokens_in":8314,"tokens_out":4714,"duration_ms":35101,"significance":"If the performance numbers were valid, the paper would make a useful contribution: it adapts standard intrusion datasets to a predictive T+15 formulation, connects the prediction to SDN-based mitigation actions, and provides an IIoT/enterprise comparison. The temporal labeling and recall-oriented evaluation are appropriate for early-warning applications. However, the headline empirical claim is currently undermined by the evaluation protocol (random split after temporal labeling), so the significance is conditional on a corrected temporal evaluation. The paper does not include code or deployment measurements, which limits reproducibility.","major_comments":[{"comment":"The evaluation protocol invalidates the headline forecasting numbers. The dataset is sorted chronologically and each row is assigned label_t+15 indicating an anomaly within the next 15 seconds. This makes consecutive rows within any 15-second window have identical or near-identical features and the same label. Splitting these rows with random stratified sampling places temporally adjacent near-duplicates in both training and test sets. Because a test record's label can be recovered from near-duplicate training records from the same attack/benign session, reported F2=0.822 and AUC=0.9982 for LGBM-GPU measure memorization of interleaved sessions, not generalization to unseen future anomalies. The same protocol is applied to CIC-IDS2017. A valid evaluation for T+15 prediction must use a temporal split (e.g., train on an early segment, test on a later segment with a gap of at least 15 s, or","section":"Section III-B, Table I"},{"comment":"The abstract and Section II claim a 'real-time SD-TWIN deployment' and real-time prediction service. However, no online evaluation is reported: there are no latency, throughput, end-to-end delay, or sustained-run measurements; Table I and Fig. 3 report only offline training times. The FastAPI/Redis/gRPC components are described architecturally but not benchmarked. This unsupported claim should be removed or substantiated with deployment experiments.","section":"Section II and abstract"},{"comment":"The preprocessing description is ambiguous about when Random UnderSampling (RUS) is applied. Section III-C.2 says RUS is applied 'before model fitting,' but if RUS is applied before the train/test split, the test set no longer reflects the original class distribution and test instances are selected by the same under-sampling procedure, which itself can leak information. Even if RUS is intended on the training set only, this needs to be stated and the split order clarified. The same clarification is needed for the decision-threshold optimization: thresholds are chosen from a precision-recall curve, but the text does not specify which partition is used.","section":"Section III-C.2"}],"minor_comments":[{"comment":"There are several typographical and grammatical errors, e.g., 'we've belived', 'data aggeragated', 'This challenges can be handle'. These should be corrected.","section":"Section I and II"},{"comment":"The AUC definition as an integral over TPR(FPR) is mathematically acceptable but omits the standard empirical estimator; consider using a more precise definition or referencing a standard implementation.","section":"Section IV, Eq. (4)"},{"comment":"AUC for LGBM-GPU is shown as 0.998 in the table but as 0.9982 in the text; use consistent rounding throughout.","section":"Table I"},{"comment":"The caption compares 'training time' but LGBM-GPU's 5631 s includes feature-selection heuristics and threshold scanning, not just model fitting. Please label this as optimization/training time or separate the stages.","section":"Fig. 3"}],"recommendation":"major_revision","confidential_remarks":"The temporal-leakage issue in the evaluation is fundamental to the paper's main claim, but it is fixable by re-running the experiments with a proper temporal split. I therefore recommend major revision rather than outright rejection. If the authors cannot provide temporal-split results that support the reported F2/AUC, the paper should be rejected in a later round. The architecture and temporal labeling idea are worth preserving."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is clean: take standard network-security datasets, shift the labels forward 15 seconds, and train a classifier to predict imminent anomalies inside an SDN/digital-twin pipeline. The labeling recipe is explicit and reproducible, the model comparison is broad, and the SHAP observation that IIoT anomalies are driven by duration-based features while enterprise attacks show up in packet-length features is a genuinely nice empirical contrast. The authors also honestly list the known limitations (binary-only anomalies, random under-sampling) rather than overselling. The soft spot is load-bearing and in the evaluation protocol. Section III-B says the dataset was sorted chronologically, then split into train/test via stratified sampling. With label_t+15 derived from the next 15 seconds, consecutive records are near-duplicates of each other. A random split interleaves those near-duplicates across train and test, so the test set is not temporally unseen. The model can effectively retrieve a test record's label from neighboring training rows. That means the reported LGBM-GPU F2 of 0.822 and AUC of 0.9982 on CICAPT-IIoT2024, and the CIC-IDS2017 numbers, do not measure the system's ability to forecast 15 seconds ahead on new attack sessions. A temporal split, training on earlier time windows and testing on strictly later ones or grouping by flow/session, is the minimum fix. A few secondary issues. The abstract promises a real-time SD-TWIN deployment, but the paper reports only offline training times and simulated architecture; no inference latency, throughput, or end-to-end control-loop measurements appear. Threshold optimization on the precision-recall curve is described without saying whether the threshold is chosen on validation data or the test set; if it is the latter, that is another leak. And for CIC-IDS2017, retaining IP addresses and port numbers as features risks letting the model memorize host identity rather than attack behavior. None of this makes the architecture or labeling idea worthless. The flaw is in the evaluation, not in the concept. A revision with temporally split evaluation and honest deployment numbers could be a solid applied paper. As submitted, though, the empirical claims do not hold, so I would not rely on the reported metrics. I would send it to peer review because the question is legitimate and the fix is straightforward, but I would expect rejection or major revision on the current evidence.","headline":"The T+15 labeling idea and SDN/digital-twin framing are worth a look, but the random stratified split on temporally adjacent records means the headline F2/AUC numbers do not demonstrate generalization to unseen future attacks.","tokens_in":668,"tokens_out":703,"would_cite":false,"duration_ms":84362,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SD-TWIN framework predicts IIoT attacks 15 seconds ahead with F2 score 0.822.","keywords":["digital twin","software-defined networking","anomaly prediction","IIoT security","LightGBM","time-aware labeling","T+15 forecasting","CICAPT-IIoT2024"],"falsifier":"Re-run the exact same training and evaluation procedure on CICAPT-IIoT2024 but split the data by time (e.g., train on the first 70% of timestamps, test on the last 30%, or split by attack session). If the GPU-LightGBM model's F2 score drops substantially below 0.822—say, below 0.6—while a chronological baseline with the same hyperparameters holds, the central empirical claim is falsified.","tokens_in":7977,"feed_emoji":"🔮","tokens_out":1633,"duration_ms":32420,"temperature":0.7,"pith_summary":"The paper proposes an SDN-based digital twin architecture, called SD-TWIN, that turns network telemetry into short-term anomaly predictions: each flow record is labeled with whether an attack occurs within the next 15 seconds, and supervised models are trained to forecast that label. On the CICAPT-IIoT2024 dataset, a GPU-accelerated LightGBM model achieves an F2 score of 0.822 and an ROC AUC of 0.9982, which the authors interpret as high recall alongside strong classification ability. The framework also includes a real-time pipeline with ONOS telemetry, Kafka streaming, Redis-based inference, and controller-triggered mitigation. A sympathetic reader would see the contribution as a concrete blueprint for proactive, twin-based IIoT security rather than just another detector.","feed_headline":"IIoT attacks predicted 15 seconds ahead with F2 of 0.822","feed_subtitle":"SDN-based digital twin with GPU-LightGBM turns flow telemetry into early-warning forecasts, but the split method needs scrutiny.","key_machinery":"The load-bearing mechanism is the temporal labeling strategy: for each record, a binary label `label_t+15` is generated by forward-index search on timestamps to mark whether any attack occurs within 15 seconds after the current flow. This converts a standard detection dataset into a forecasting problem. The SD-TWIN architecture then couples this predictor with an SDN controller (ONOS) that collects flow counters, streams them via Kafka, runs inference in a FastAPI micro-service, and can push mitigation rules back to the network.","core_discovery":"The central claim is that time-aware labeling—assigning each network flow a label indicating whether any anomaly occurs within the following 15 seconds—enables a digital-twin pipeline to predict imminent IIoT attacks, and that a GPU-accelerated LightGBM model is the best candidate for that prediction task. On CICAPT-IIoT2024, the model reports precision 0.615, recall 0.952, F2 0.822, and AUC 0.9982, which the authors describe as balancing high anomaly recall with strong classification performance. On the more conventional CIC-IDS2017 dataset, a grid-searched random forest reaches F2 0.912 and recall 0.920. The paper further shows that SHAP attributions differ by environment: packet-length fe","pith_inferences":["The random stratified train/test split is a serious threat to the reported numbers: because labels are derived from the next 15 seconds of the same session, temporally adjacent records are near-duplicates, so the model may memorize session patterns rather than generalize to unseen future anomalies. A chronological or session-held-out split would likely produce materially lower F2/AUC.","The paper labels anomalies only as binary (attack/benign) and uses random undersampling; extending the framework to attack-type-aware labels and more advanced imbalance handling (e.g., SMOTE or class weights) would clarify whether the recall-centric performance holds across specific APT techniques.","The SD-TWIN deployment is validated on emulated Mininet traffic, not on a physical IIoT testbed; the reported training/inference times may not translate to real constrained gateways, despite the mention of Raspberry Pi surrogate models.","The T+15 window is fixed and dataset-specific; testing different look-ahead windows (e.g., T+5, T+30) and measuring how F2 decays with horizon would give operators a practical sense of how far ahead the twin can reliably predict."],"forward_implications":["If the reported F2 and AUC hold under realistic evaluation, the framework gives operators a 15-second early-warning window to drop, rate-limit, or reroute traffic before an attack manifests.","The time-aware labeling recipe can be applied to other public security datasets, turning existing intrusion-detection benchmarks into predictive-forecasting benchmarks.","The contrast in SHAP features suggests that IIoT anomaly prediction should emphasize temporal persistence metrics, while enterprise detection should emphasize packet-shape features.","The architecture's modular layers (physical, telemetry, twin/analytics, visualization) provide a reference design for deploying ML-based predictions inside SDN-managed industrial networks.","The automatic retraining trigger and hot-swap mechanism, if operationalized, would let the twin adapt to evolving attack patterns without downtime."],"fun_headline_variants":["IIoT attacks forecast 15s out via SDN digital twin","GPU-LightGBM predicts IIoT threats 15s ahead, F2 0.822","Time-aware labeling foresees IIoT attacks 15s early","IIoT early warning: 15s lead time, but split method questioned","SDN twin plus LightGBM flags IIoT attacks 15s in advance"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The result depends on the assumption that a random stratified split of chronologically ordered records accurately measures the model's ability to predict future anomalies; if temporally adjacent training and test samples are near-duplicates, the reported F2 and AUC overstate real-world forecasting skill.","fun_headline_variants_meta":{"raw":{"variants":["IIoT attacks forecast 15s out via SDN digital twin","GPU-LightGBM predicts IIoT threats 15s ahead, F2 0.822","Time-aware labeling foresees IIoT attacks 15s early","IIoT early warning: 15s lead time, but split method questioned","SDN twin plus LightGBM flags IIoT attacks 15s in advance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000334,"raw_usage":{"total_tokens":1672,"prompt_tokens":705,"completion_tokens":967,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":449,"completion_tokens_details":{"reasoning_tokens":865}},"tokens_in":449,"tokens_out":967,"duration_ms":8096,"temperature":1.0,"reasoning_tokens":865,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T15:13:47.968379+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the exact same training and evaluation procedure on CICAPT-IIoT2024 but split the data by time (e.g., train on the first 70% of timestamps, test on the last 30%, or split by attack session). If the GPU-LightGBM model's F2 score drops substantially below 0.822—say, below 0.6—while a chronological baseline with the same hyperparameters holds, the central empirical claim is falsified.","supporting_citations":[],"review_version":1}