{"id":"5f88251e-980b-4a86-b11f-b9853d8cb315","arxiv_id":"2507.21193","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An LSTM trained on 5G key performance measurements detects DDoS attacks from user equipment with F1 above 0.96, while LIME, SHAP, and an LLM convert decisions into human-readable explanations and suggested mitigations.","lead":"This paper combines an LSTM detector for DDoS attacks on 5G radio measurements with LIME, SHAP, and a large language model that writes plain-language alerts and mitigation advice for network operators. It reports F1 scores above 0.96 on a public 5G testbed dataset, but the near-real-time deployment claim is architectural rather than measured end to end.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported F1>0.96 may be inflated because window size and past-data ratio are selected and reported on the same previous-day test sets (Figs. 6-7, Table V), with no held-out final evaluation.","rationale":"The central claim is that the proposed LSTM pipeline detects DDoS with F1>0.96 on real 5G KPMs and is suitable for near-real-time RIC deployment. The detection number is the foundation; interpretability and latency are secondary. The reader's weakest assumption concerns dataset representativeness and unmeasured end-to-end latency. I agree those are genuine limitations, but they are external-generalization concerns. The sharper, internal concern is that the evaluation protocol itself may inflate the headline number. The hyperparameter grid (window size, ratio) is evaluated on the same previous-day test sets that later appear in Table V as the final result, so the reported F1 is a selected maximum rather than an unbiased estimate. This is a correctness risk that can be resolved with a proper held-out day. The paper deserves credit for using a public real-5G testbed dataset, comparing with prior NCSRD results, and reporting inference FLOPs/latency; none of that rescues a possibly selected F1. This is why the verdict should remain conditional pending a held-out evaluation.","tokens_in":32764,"tokens_out":4357,"duration_ms":41358,"concrete_test":"Hold out Day 4 completely. Run the grid search (window size 1-10, ratio 0.1-0.5) on Days 1-3 only, using a proper temporal validation (e.g., train Day 1 -> validate Day 2; train Days 1-2 -> validate Day 3). Select the best configuration, retrain on Days 1-3 with that configuration, and evaluate once on Day 4. Compare the resulting F1 to the Day 4 entry in Table V (0.98 with ratio 0.3). If the held-out F1 is materially lower (e.g., below 0.90) or the chosen hyperparameters differ, the headline F1 claim is not robust. Repeat with several random seeds to obtain a confidence interval.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing quantitative claim is the average F1-score above 0.96 reported in the abstract and Table V. Section IV-B describes a grid search over window size (1-10) and past-data ratio (0.1-0.5) using the 'previous day test sets'; Figs. 6 and 7 report F1/FPR/FNR on those sets, and the text selects window size 3 and ratio 0.3 as optimal. Table V then reports F1 for the same protocol on the same previous-day test sets. Because the same data are used both to choose hyperparameters and to produce the final scores, the reported performance is subject to selection bias; there is no independent held-out evaluation. The abstract's 'F1-score > 0.96' therefore rests on an evaluation protocol that can overstate generalization. The paper itself defers operational deployment to future work (Section V), which is consistent with this concern being about the evidence, not the idea.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper combines a survey of XAI and LLM methods for intrusion detection in RANs with a proposed framework for DDoS detection in AI-RAN. The detection component is an LSTM trained on UE KPM time series extracted from E2 nodes, processed inside a Near-RT RIC xApp, followed by a separate rApp that applies LIME and SHAP and then uses LLMs to translate technical explanations into natural-language summaries. Experiments on the public NCSRD 5G testbed dataset report average F1 scores above 0.96, an inference time of 0.03 ms per sample on an Intel i7-10700 CPU, and readability scores for the LLM-generated explanations. The paper claims the framework is suitable for near-real-time deployment and provides interpretable, actionable outputs for non-expert operators.","tokens_in":32880,"tokens_out":3058,"duration_ms":33389,"significance":"If the claims were fully supported, the paper would be a useful contribution to AI-for-RAN security: it evaluates on a public, real-testbed dataset; it compares against prior NCSRD baselines; and it integrates detection, post-hoc XAI, and LLM-based summarization into a concrete O-RAN-style architecture. The survey portion is competently assembled, and the detection idea is plausible. However, the headline F1 claim currently rests on an evaluation protocol that selects hyperparameters on the same previous-day test sets used for final reporting, the near-real-time claim is based on a single component latency, and the interpretability evaluation measures readability rather than faithfulness. These issues are local and fixable, but they are load-bearing for the paper's central claims.","major_comments":[{"comment":"The grid search over window size and past-data ratio is evaluated on the same previous-day test sets that are then used to report the final F1 scores. The text selects window size 3 and ratio 0.3 based on performance on these sets, and Table V reports average F1 on the same previous-day test sets. Because the hyperparameters are chosen to optimize exactly the data used in the final table, the reported F1 > 0.96 is subject to selection bias and is not an unbiased estimate of generalization. A genuinely held-out evaluation, such as a completely unseen day or nested cross-validation, is required before the abstract's headline claim can be accepted.","section":"Section IV-B and IV-C, Figs. 6-7, Table V"},{"comment":"The near-real-time claim is supported only by the 0.03 ms LSTM inference time on an Intel i7-10700 CPU. The end-to-end pipeline also includes KPM preprocessing in the xApp, E2 transport, per-instance SHAP and LIME computation, LLM API calls, and rApp scheduling. Section III-A itself notes that LLM latency can exceed near-real-time constraints and places the XAI+LLM module in a non-real-time rApp. Without end-to-end latency measurements or a delay budget for each stage, the statement that the framework is 'well-suited for near real-time deployment' is not supported by the presented evidence.","section":"Section IV-C"},{"comment":"The interpretability evaluation uses only Flesch Reading Ease and Gunning Fog Index. These readability metrics do not measure whether the LLM-generated summaries are faithful to the LSTM's actual decision process, whether they correctly reflect the underlying SHAP/LIME attributions, or whether operators can act correctly on them. The paper needs an evaluation of explanation correctness, such as consistency checks with the XAI outputs, expert ratings, or task-based user studies, before claiming that the framework delivers 'interpretable outputs' as a validated contribution.","section":"Section IV-C and Table VII"},{"comment":"The paper reports inconsistent headline numbers for the proposed detector. Table V gives average F1 values of 0.96-0.99 with ratio 0.3, while Table VI lists 'Proposed DDoS Detection' as 0.98 with FPR 0.05% and FNR 6.31%. The relationship between these numbers is not explained, and it is unclear which evaluation setup corresponds to the abstract's F1 > 0.96 claim. The final F1 claim should be tied to one clearly defined evaluation protocol.","section":"Tables V and VI"}],"minor_comments":[{"comment":"The 'past data ratio' is never formally defined; please specify exactly how previous-day samples are mixed with current training data and how the ratio is applied during training.","section":"Section IV-B"},{"comment":"The dataset description does not state whether the 80-20 split is random over samples or structured by day and UE. A time-based split should be described to rule out temporal leakage between training and test instances.","section":"Section IV-A"},{"comment":"The FPR and FNR heatmaps are shown only for window sizes 1-7, while the grid search described in the text covers window sizes 1-10; please explain or extend the figure.","section":"Fig. 7"},{"comment":"The text says MinMax normalization was applied but does not state whether the normalization parameters were computed on the training data only and then applied to test data; please clarify.","section":"Section IV-B"},{"comment":"The Gunning Fog Index values are reported inconsistently, with some entries as numbers and others as 'college' or 'college graduate'; please use a consistent format.","section":"Table VII"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful survey component and a concrete detection pipeline evaluated on a public dataset; the central idea is worth pursuing. The main obstacle is evaluation rigor: the F1 claim needs a genuinely held-out evaluation, and the latency and interpretability claims need to be scoped to what was actually measured. These are fixable within the manuscript's scope, so I would not recommend rejection. No concerns about citation practices or novelty disclosure beyond the points already raised in the report."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a modest but useful system paper: LSTM-based DDoS detection on public 5G KPM data, LIME/SHAP, and an LLM that turns feature attributions into plain-language summaries, organized as an xApp/rApp split in O-RAN. The combination is new in the RAN-security literature, though it extends HuntGPT and 6G-XSec rather than breaking from them. The authors also compare against prior NCSRD baselines and are upfront that real deployment is future work.\n\nWhat I credit: the dataset (NCSRD) is public and genuinely from a 5G testbed; the description of preprocessing, windowing, and the prompt templates is detailed enough to reproduce; the xApp/rApp split is well-motivated given LLM latency; and the paper does not hide that the classical ML baselines (kNN, XGBoost) actually achieve F1=1.00 on the same data. That last point matters: it implies the detection task is not hard for non-temporal models, and the LSTM's advantage is claimed via temporal reasoning and latency, not raw accuracy.\n\nWhere it's soft: the headline F1>0.96 is not established. Section IV-B describes a grid search over window size and past-data ratio using the previous-day test sets; Figs. 6–7 show performance on those same sets; Table V then reports the chosen configuration's F1 on those same previous-day sets. With no held-out final evaluation, the number is a tuned maximum, not an honest out-of-sample score. The near-real-time claim rests on 0.03 ms for LSTM inference only; preprocessing, E2 delivery, XAI, and LLM calls are not measured. And the interpretability evaluation uses readability scores (Flesch, Gunning Fog), which say nothing about whether the LLM's summary is faithful to the model's actual reasoning. Minor: no confidence intervals, no released code, and the survey portion is broad but lists papers rather than synthesizing.\n\nThe central idea holds up. This is a reasonable architecture and a useful benchmark for RAN-edge DDoS detection with explanations. But the evaluation needs a replay with a genuinely held-out test set, end-to-end latency, and some faithfulness check (e.g., comparing LLM statements against the LIME/SHAP inputs) before I'd trust the headline numbers.\n\nRecommendation: send to peer review, yes. A good referee can force the held-out evaluation and the latency measurement. I wouldn't cite it in my own work until those numbers are fixed.","headline":"A worthwhile AI-RAN security system paper whose headline F1 is undermined by tuning on the test set; fix the evaluation and it's a solid contribution.","tokens_in":33505,"tokens_out":3233,"would_cite":false,"duration_ms":33950,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An LSTM trained on user-equipment 5G radio metrics can detect DDoS attacks with F1 above 0.96 and explain each alert in plain language.","keywords":["DDoS detection","LSTM","explainable AI","large language models","Open RAN","near-real-time RIC","key performance measures","anomaly detection"],"falsifier":"Measure the end-to-end detection loop on the same testbed or another 5G network—KPM collection, delivery to the controller, preprocessing, LSTM inference, and LLM explanation—and check whether average F1 stays above 0.96 and total latency stays within the near-real-time RIC budget; alternatively, rerun the training procedure on a fresh split or a different 5G dataset and see whether the 0.3 past-data ratio still yields F1 above 0.96.","tokens_in":32512,"feed_emoji":"📡","tokens_out":7057,"duration_ms":65892,"temperature":0.7,"pith_summary":"This paper argues that a small LSTM classifier reading three-step windows of per-user radio performance measures can detect DDoS attacks against a 5G testbed with F1-scores above 0.96, and that the same pipeline can explain each alert in plain language. The explanation chain uses LIME and SHAP to score feature contributions and a large language model to turn those scores into natural-language summaries and mitigation suggestions. The authors position this as a security application that runs inside the near-real-time RAN controller, with per-sample LSTM inference reported at 0.03 ms on a desktop CPU. If true, this gives mobile operators a detection component that is both accurate enough and interpretable enough to act on at the network edge.","feed_headline":"LSTM on 5G radio metrics flags DDoS attacks at F1 above 0.96","feed_subtitle":"Per-sample inference takes 0.03 ms, and SHAP, LIME, and an LLM turn each alert into plain-language advice","key_machinery":"The load-bearing object is an LSTM binary classifier over (3 time steps × 14 features) windows of user-equipment performance measures, trained with a past-data ratio of 0.3 to keep old attack signatures alive during updates. Around it, the pipeline chains three components: Kernel SHAP and LIME for post-hoc local attributions, a hand-built prompt that carries feature statistics, input sequence, model output, and explanation tables, and an LLM that produces a human-readable summary with suggested mitigations. The LSTM is what the detection claim rests on; the XAI-LLM chain is what turns that verdict into something an operator can verify.","core_discovery":"The paper's central discovery, on its own terms, is that temporal patterns in user-equipment performance measurements—sampled every five seconds and arranged into (3, 14) sequences—carry enough signal to separate DDoS traffic from normal traffic with average F1 above 0.96 across four test days. The key to maintaining that score over time is replaying a fraction (0.3) of past training data when updating the model, which prevents catastrophic forgetting; without it, the F1 for the last day drops to 0.36. The same LSTM is small enough that a single forward pass costs about 36K FLOPs, reported at 0.03 ms per sample on the CPU used in the experiments, which the paper takes as evidence the detector can fit near-real-time RIC operation. On the interpretability side, the paper shows that SHAP and LIME highlight consistent attack drivers such as uplink bitrate, uplink retransmissions, and downlink/uplink asymmetry, and that LLM-generated summaries turn those drivers into operator-readable text.","pith_inferences":["Beyond what the paper shows: the reported 0.03 ms covers only LSTM inference; the full loop through delivery, normalization, SHAP/LIME computation, and an LLM call will be much slower, so the near-real-time claim needs an end-to-end measurement before deployment.","Because the detector is trained on one testbed with a 1.7% attack rate, its high F1 could reflect dataset-specific artifacts; testing on other RAN traces or with different attack types would reveal whether the feature patterns generalize.","The LLM-generated mitigation advice is not validated for correctness; an operator who follows it without checking could act on a plausible-sounding but wrong suggestion, so the advisory role of the rApp is a sensible boundary.","If the same three-step LSTM were evaluated with online learning instead of day-wise retraining, the past-data ratio could be tested as a continual-learning mechanism against real drift."],"forward_implications":["A DDoS detector can run inside the near-real-time RAN controller as an xApp on per-UE telemetry, with the heavy explanation stage offloaded to an rApp outside the real-time loop.","Replaying 30% past data on each model update keeps detection stable across days, raising the worst-day F1 from 0.36 to 0.98.","On the dataset used here, the proposed LSTM (F1 0.98, FPR 0.05%, FNR 6.31%) beats the CNN and LSTM baselines (0.93 and 0.90) and remains competitive with kNN/XGBoost (1.00) while capturing temporal structure and lower inference cost.","Operators receive natural-language explanations with anomaly summary, misclassification likelihood, and mitigation steps, making automated RAN security review possible without a deep-learning expert.","Global SHAP importance shows attack detection relies on uplink/downlink asymmetry, uplink bitrate, and retransmissions, so these features can be prioritized in monitoring."],"supporting_citations":[{"why":"Defines SHAP values, the game-theoretic attribution used for local and global explanation of LSTM predictions.","marker":"[84]"},{"why":"Defines LIME, the local surrogate model used to produce human-readable feature rules for each prediction.","marker":"[85]"},{"why":"Supplies the 5G radio and core KPM dataset with benign and DDoS-labeled UE traffic used for training and evaluation.","marker":"[88]"},{"why":"Provides the kNN and XGBoost baselines on the same dataset that the paper compares against in detection performance.","marker":"[89]"},{"why":"Provides the CNN and LSTM baselines on the same dataset that the proposed LSTM must beat.","marker":"[90]"},{"why":"Specifies the E2 KPM service model, grounding the claim that per-UE KPMs are available to a near-real-time RIC xApp.","marker":"[11]"}],"fun_headline_variants":["5G DDoS detector hits F1>0.96 with LLM-readable alerts","LSTM on 5G metrics catches DDoS, explains via LLM","Interpretable DDoS detection in 5G RAN: F1>0.96","DDoS alerts from 5G radio metrics, explained by LLM, F1>0.96"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The dataset from one small 5G testbed, with three cells and nine UEs, is assumed to behave like real production RAN traffic, and the measured 0.03 ms LSTM inference is assumed to fit the near-real-time budget once delivery, preprocessing, XAI, and LLM steps are included.","fun_headline_variants_meta":{"raw":{"variants":["5G DDoS detector hits F1>0.96 with LLM-readable alerts","LSTM on 5G metrics catches DDoS, explains via LLM","Interpretable DDoS detection in 5G RAN: F1>0.96","DDoS alerts from 5G radio metrics, explained by LLM, F1>0.96"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000668,"raw_usage":{"total_tokens":3076,"prompt_tokens":1005,"completion_tokens":2071,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":621,"completion_tokens_details":{"reasoning_tokens":1977}},"tokens_in":621,"tokens_out":2071,"duration_ms":13788,"temperature":1.0,"reasoning_tokens":1977,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:43:18.748697+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the end-to-end detection loop on the same testbed or another 5G network—KPM collection, delivery to the controller, preprocessing, LSTM inference, and LLM explanation—and check whether average F1 stays above 0.96 and total latency stays within the near-real-time RIC budget; alternatively, rerun the training procedure on a fresh split or a different 5G dataset and see whether the 0.3 past-data ratio still yields F1 above 0.96.","supporting_citations":[{"cited_title":"A Unified Approach to Interpreting Model Predictions,","cited_arxiv_id":null,"evidence_quote":"Defines SHAP values, the game-theoretic attribution used for local and global explanation of LSTM predictions."},{"cited_title":"”Why Should I Trust You?","cited_arxiv_id":null,"evidence_quote":"Defines LIME, the local surrogate model used to produce human-readable feature rules for each prediction."},{"cited_title":"NCSRD-DS-5GDDoS: 5G Radio and Core metrics containing sporadic DDoS attacks,","cited_arxiv_id":null,"evidence_quote":"Supplies the 5G radio and core KPM dataset with benign and DDoS-labeled UE traffic used for training and evaluation."},{"cited_title":"User Terminals as Attackers: An Open Dataset Analysis of DDoS Attacks in 5G Networks,","cited_arxiv_id":null,"evidence_quote":"Provides the kNN and XGBoost baselines on the same dataset that the paper compares against in detection performance."},{"cited_title":"Advancing Predictive Security for Con- sumer Applications in Beyond 5G/6G Networks With Annotated Datasets,","cited_arxiv_id":null,"evidence_quote":"Provides the CNN and LSTM baselines on the same dataset that the proposed LSTM must beat."}],"review_version":2}