{"id":"32270c93-ec42-437e-a056-c5fec9860396","arxiv_id":"2505.06789","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"This paper presents an open-source NWDAF implementation with a standardized UPF event exposure service, MLflow-based model provisioning, and an SMF extension that automatically detects and mitigates abnormal UE behavior in 5G cores.","lead":"This paper builds and tests a real 5G core network system that automatically detects and blocks suspicious phones. It implements the 3GPP-standard data collection point in the network's user plane and closes the loop by having the session manager end connections of flagged devices.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Effectiveness claim lacks detection metrics: RF trained on CTU-13 is never validated on the Nmap attack used in §6.3, so measured mitigation latency is conditionally uninformative.","rationale":"The most load-bearing condition for the central claim is that the deployed detector actually detects the attack it is supposed to mitigate. The paper's strongest quantitative evidence is the latency chain in Figure 8, but this is an effectiveness measure only if detection is reliable. All components—UPF EES, NWDAF analytics, SMF enforcement—could work correctly and still fail as a security system if the model misses attacks or bans benign UEs. This is not a novelty or consensus dispute; it is an internal evidence gap: the section that reports the closed-loop result never reports the detector's output quality. I agree with the reader's weakest assumption, and would sharpen it as the dominant issue. The paper otherwise has credible engineering support: a public code artifact, a concrete 3GPP-compliant EES implementation, and controlled overhead experiments. Those parts justify a conditional accept, not rejection, but the effectiveness claim should be explicitly conditioned on detection metrics. Since the reader already assigned CONDITIONAL, I recommend no change to the verdict.","tokens_in":12753,"tokens_out":4842,"duration_ms":54059,"concrete_test":"As one decisive check: instrument the §6.3 testbed with ground-truth labels and compute the deployed model's classification metrics on the actual Nmap scan data and on benign UERANSIM traffic, using the same per-report windows (1s, 3s, 5s). Report precision, recall, F1, and false-positive rate per UE per reporting interval, and also the fraction of the 20 runs that led to a ban. If recall on the Nmap attack is high and false-positive rate on benign traffic is near zero, the closed-loop effectiveness claim stands; if not, the measured mitigation latency cannot support the claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Abstract and §6.3 claim closed-loop security effectiveness, but the only quantitative end-to-end result is attack-to-mitigation latency (Fig. 8). That latency is meaningful only if the Random Forest bot detector actually identifies the Nmap http-open-proxy scan as abnormal. The paper defines no detection accuracy metric: no precision, recall, F1, true/false positives, or detection-per-trial count. §5 trains on CTU-13 Scenarios 1,2,3,5 with node-level graph features (edges, weighted in/out degree, betweenness centrality); §6.3 tests on Nmap scanning of 20 servers, a behavior unseen in that training data. The model's transfer is assumed, not shown. Without per-report ground-truth evaluation on the testbed's own attack and benign traffic, Figure 8's t2/t3 numbers only measure the pipeline delay in cases where the UE was eventually banned; they give no evidence about missed attacks or false alarms. A false positive would cause SMF to release a legitimate UE's PDU session, a safety failure the current evaluation cannot detect. Section 6.3's trade-off discussion implicitly acknowledges detection quality matters but never quantifies either side.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an NWDAF implementation for 5G networks with three main engineering contributions: a 3GPP-compliant UPF Event Exposure Service for standardized real-time data collection, an ML Model Provisioning Service integrated with MLflow for model lifecycle management, and an SMF extension that subscribes to NWDAF abnormal-behavior analytics and automatically releases PDU sessions of flagged UEs. The system is integrated with Open5GS and OAI 5G Core, and evaluated on an Open5GS/UERANSIM testbed. The evaluation measures UPF latency and CPU overhead under increasing data rate and UE count, NWDAF module CPU usage at different reporting intervals, and end-to-end attack-to-mitigation latency for a bot-detection scenario using a Random Forest model trained on CTU-13 graph features and tested on an Nmap http-open-proxy scan. The paper claims the first implementation of the UPF Event Exposure Service and the first proof-of-concept closed-loop automation involving UPF, NWDAF, and a modified SMF.","tokens_in":12994,"tokens_out":5348,"duration_ms":56270,"significance":"If the implementation claims hold, this is a useful systems contribution: it addresses a real gap between 3GPP-specified NWDAF capabilities and available open-source implementations, and it makes the source code publicly available. The UPF overhead measurements with confidence intervals are a concrete data point for the cost of standardized event exposure, and the SMF-side closed-loop action is a natural and standards-aligned demonstration. The main weakness is that the closed-loop effectiveness claim is not supported by detection-quality metrics: the model is trained on CTU-13 but evaluated on a different attack type, and no accuracy, precision, recall, or false-positive rates are reported. This makes the end-to-end latency numbers conditional on unmeasured detection success and leaves safety-relevant false positives unexplored. The paper is therefore best viewed as a credible system prototype whose effectiveness claim needs substantially more evidence before it can be accepted as demonstrated.","major_comments":[{"comment":"The central effectiveness claim in the abstract and §6.3 is not supported by the reported measurements. Figure 8 reports t1/t2/t3 latencies averaged over 20 trials, but the paper gives no detection accuracy, precision, recall, F1, true-positive rate, or false-positive rate for the Random Forest bot detector on the testbed's Nmap http-open-proxy attack. The model is trained on CTU-13 Scenarios 1, 2, 3, and 5 with node-level graph features (§5), while §6.3 evaluates on a scanning behavior that is not present in that training data. The t2/t3 values are therefore conditional on the model having eventually flagged the UE; the paper does not report how often this happened, nor how often benign UEs were falsely flagged. Because a false positive triggers SMF to release a legitimate UE's PDU session via N4 (§3.4), the evaluation as designed cannot detect this safety-relevant failure mode. I request per-trial detection counts, ground-truth-labeled evaluation of both the testbed attack and benign traffic, and explicit TPR/FPR metrics, or the effectiveness claim should be revised to pipeline latency under successful detection.","section":"§6.3, Figure 8"},{"comment":"The training and evaluation data pipeline is underspecified, which blocks reproducibility of the detection claim. The model is trained on 'bot behavior data from Scenarios 1, 2, 3, and 5 of the CTU-13 dataset', but it is not stated whether benign examples used for training come from CTU-13 or from the 'different scripts inside UEs' mentioned in the same section. It is also not stated whether any CTU-13 data were held out for validation, and no validation accuracy for the Random Forest model is reported anywhere. The testbed's Nmap scan produces a graph (one UE, 20 servers, proxy-scan flows) that is structurally very different from CTU-13 botnet communication graphs, yet the paper does not discuss this domain shift. The authors should provide a validation experiment on the testbed's own labeled attack and benign traces, or explicitly limit the claim to a mechanism demonstration rather than a validated detector.","section":"§5, Bot Detection Model"},{"comment":"The scalability claim is broader than the evidence. The experiments use at most 7 simulated UEs and up to 100 Mbps in a UERANSIM-based testbed, and Figure 6c fixes the total data rate at 10 Mbit/s while increasing the number of UEs, so per-flow reporting is not stressed as the number of flows grows. The abstract and conclusion state scalability without these qualifications. I recommend either extending the UE-count experiments to ranges where per-flow notification generation is actually stressed, or qualifying the scalability claim to the tested regime.","section":"§6.1, Figure 6"}],"minor_comments":[{"comment":"The introduction says 'Section 5 demonstrates the operational workflow and evaluates the system against multiple metrics' and then 'Section 7 concludes', omitting Section 6, which is the Performance Evaluation. The section references should be corrected.","section":"§1, Section numbering"},{"comment":"The x-axis labels and legend use '1 UEs', '3 UEs', etc.; '1 UEs' should be '1 UE', and the legend entries should be made consistent with the plotted series.","section":"Figure 6c"},{"comment":"The sample notification contains a truncated IP address ('142..}') and the 'packFiltId' string appears cut off; the listing should show a complete, valid example.","section":"Listing 1"},{"comment":"The one-way delay is computed as half of the average RTT; this assumes symmetric network paths and should be stated as an approximation rather than a precise one-way measurement.","section":"§6.1, Latency measurement"},{"comment":"The ML Model Provision Service CPU share lumps together MLflow registry queries, model filtering, and inference serving, so the statement that this service dominates CPU usage is not decomposable. A per-submodule breakdown would make the scaling recommendation in §6.2 more actionable.","section":"§6.2, Figure 7"},{"comment":"Section 4 states that both Open5GS and OAI 5G Core support the entire closed-loop workflow, but Section 6 reports experiments only on Open5GS. The text should state explicitly whether the OAI integration was exercised in the closed-loop evaluation or only implemented.","section":"§4, Open5GS and OAI"}],"recommendation":"major_revision","confidential_remarks":"The novelty claims ('first implementation of the UPF Event Exposure Service', 'first proof-of-concept closed-loop automation') are plausible but rest on a limited literature survey; I did not find an independent basis to dispute them. The decisive issue for the editor: the paper's headline effectiveness claim is currently an engineering demonstration with unmeasured detection quality. This is repairable within the manuscript's scope by adding detection metrics and a validation dataset, so I recommend major revision rather than rejection. I also note that the paper's source-code availability and the concrete overhead measurements are genuine strengths that should be preserved in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know upfront. First, this is a genuine engineering contribution: the authors implement three pieces that didn't exist together in open-source 5G cores—a UPF Event Exposure Service, an ML model provisioning service built on MLflow, and an SMF extension that acts on NWDAF analytics to release PDU sessions. Second, the paper's claim that the system is \"effective\" for closed-loop security management is the soft spot. The evaluation measures pipeline latency but never measures detection accuracy or false positives.\n\nWhat's genuinely good: the UPF EES implementation is real, follows 3GPP message formats, and the overhead measurements (latency, CPU) use 95% CIs and show minimal impact. The NWDAF resource usage breakdown is useful, and the closed-loop workflow is clearly described. The code is public, which raises the bar for reproducibility. Table 1 gives a fair comparison of prior NWDAF work. The \"first\" claims are not independently verified, but they're plausible given the survey.\n\nThe central weakness is the security effectiveness. The Random Forest bot detector is trained on CTU-13 botnet flows and then tested on an Nmap http-open-proxy scan in the testbed. No precision, recall, or false positive rate is reported. That matters because a false positive would cause the SMF to release a legitimate UE's PDU session—a real safety failure. The latency numbers in Figure 8 are only meaningful for cases where the UE got banned; they say nothing about missed attacks or false alarms. The stress-test note put this correctly: the transfer of the CTU-13 model to a different attack is assumed, not shown. This isn't a fatal flaw for the engineering contribution, but it is a load-bearing gap for the \"effectiveness\" claim.\n\nMinor points: UERANSIM-simulated UEs and synthetic traffic limit generalization, but that's typical for this kind of systems work. The self-citation to Monarch is fine—it's used as motivation, not as evidence. The MLflow integration is straightforward but reasonable.\n\nWho should read this: operators and researchers wanting a working reference implementation of NWDAF data collection and closed-loop mitigation. It's a systems paper, not an ML/security evaluation paper. With a serious revision—adding detection metrics on the testbed's own traffic, or softening the effectiveness claim to \"latency of the pipeline\"—this could be a solid venue paper. I'd send it to peer review: the engineering deserves referee time, and the missing metrics are fixable.","headline":"Real open-source NWDAF engineering with a credible testbed, but the closed-loop security claim needs detection metrics before it lands.","tokens_in":13507,"tokens_out":2217,"would_cite":true,"duration_ms":21683,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A working 5G closed loop: UPF event exposure feeds NWDAF analytics that drive SMF session release, cutting attack response to under 10 seconds.","keywords":["NWDAF","5G","UPF event exposure service","closed-loop automation","anomaly detection","SMF policy enforcement","open-source 5G core","network automation"],"falsifier":"Run the deployed pipeline with labeled test traffic where the attack is a known botnet flow (e.g., a CTU-13 scenario) and measure true/false positive rates; if the model fails to flag the flow or flags benign traffic, the closed-loop mitigation claim collapses. Alternatively, disable the ML model and substitute a random or heuristic detector: if similar mitigation latency results, the ML component is not what carries the loop.","tokens_in":12589,"feed_emoji":"🛡️","tokens_out":4809,"duration_ms":39187,"temperature":0.7,"pith_summary":"This paper argues that 3GPP-standardized network automation for 5G is practically achievable today with open-source components. It presents the first implementation of the UPF Event Exposure Service, the standard interface for pulling user-plane telemetry, and wires it into an NWDAF that runs ML-based analysis and a modified SMF that terminates sessions of anomalous UEs. The result is a proof-of-concept closed loop that detects and mitigates a bot attack with sub-10-second latency and modest UPF overhead. Demonstrating the full feedback cycle—not just analytics in isolation—is the paper's central contribution.","feed_headline":"5G core auto-mitigates bot attacks in under 10 seconds","feed_subtitle":"First open-source UPF event-exposure service feeds NWDAF analytics that make SMF drop malicious sessions.","key_machinery":"The load-bearing object is the UPF Event Exposure Service implementation: five modules (encoder/decoder, server, data preparation, client) that convert raw packets into 3GPP-compliant JSON notifications on demand. It is what makes the rest of the loop possible, because it replaces ad hoc or indirect data collection with a standardized real-time feed from the user plane. Around it, the NWDAF's SBI/NBI split and the ML Model Provisioning Service (with MLflow as registry) supply the analysis, and the SMF extension supplies the action; together they complete the closed loop.","core_discovery":"The paper's central claim is the first end-to-end realization of the NWDAF closed loop defined by 3GPP: the UPF Event Exposure Service is implemented in C and embedded in Open5GS and OAI 5G cores, the NWDAF's southbound interface subscribes to per-flow volume reports in the standard JSON format, a Random-Forest bot detector ingests graph features computed from those reports, and an extended SMF consumes the resulting abnormal-behavior analytics and releases the offending PDU sessions via N4. The evaluation shows latency and CPU overhead of the EES are negligible against a baseline, NWDAF's data collection and notification modules are lightweight relative to ML provisioning, and the full attack-to-mitigation pipeline completes in under 10 seconds. The authors state this is the first implementation of the UPF EES and the first closed-loop proof-of-concept involving UPF, NWDAF, and SMF together.","pith_inferences":["The paper does not validate that the CTU-13-trained Random Forest generalizes to the Nmap scanning attack used in the testbed; if detection accuracy is low, the sub-10-second mitigation time is a best-case rather than typical figure.","The same EES-to-NWDAF-to-SMF architecture should extend to other 3GPP analytics IDs (e.g., QoS monitoring, mobility anomalies) and other enforcement points such as the PCF, which the authors list as future work.","The bottleneck identified—ML inference time—suggests that optimizing or pre-deploying detection models will directly improve closed-loop latency, possibly below the measured values.","A direct comparison of the UPF EES against the SMF-relay data collection used by prior work (e.g., via N4) would quantify the standardization benefit in latency and load."],"forward_implications":["Operators can collect user-plane telemetry through the standardized UPF EES rather than proprietary or SMF-relayed paths, improving interoperability and scalability.","Closed-loop security automation—detecting abnormal UE behavior and terminating sessions—can be deployed on open-source 5G cores without human intervention.","The NWDAF architecture scales by separating data collection and notification modules from the ML provisioning service, which dominates CPU usage and should be scaled independently.","Because the loop is driven by 3GPP-defined subscription parameters, shorter data collection intervals yield faster mitigation, at the cost of detection accuracy."],"supporting_citations":[{"why":"Defines the UPF Event Exposure Service format and semantics that the implementation follows.","marker":"[2]"},{"why":"Defines NWDAF analytics services, including UE abnormal behavior, that the closed loop consumes.","marker":"[3]"},{"why":"Specifies 5G procedures for PDU session management that the SMF extension invokes for session release.","marker":"[4]"},{"why":"Defines the 5G system architecture and role of UPF/SMF that the implementation integrates with.","marker":"[5]"},{"why":"Provides one of the open-source 5G core implementations (OAI 5GC) that the NWDAF is integrated with.","marker":"[7]"},{"why":"Supplies the model registry and serving infrastructure used by the ML Model Provisioning Service.","marker":"[10]"},{"why":"Provides the Open5GS core that hosts the UPF and SMF modifications.","marker":"[11]"},{"why":"Supplies the CTU-13 botnet dataset used to train the Random Forest detector.","marker":"[12]"},{"why":"Provides the UERANSIM UE/RAN simulator used in the testbed evaluation.","marker":"[13]"},{"why":"Represents prior monitoring work lacking standardized interfaces, motivating the EES implementation.","marker":"[17]"}],"fun_headline_variants":["First 5G closed loop: UPF to NWDAF to SMF in 10s","5G core closes the loop on botnets in under 10s","First NWDAF closed loop: 5G core self-heals bot attacks","Open-source 5G core auto-drops malicious sessions in 10s","5G core's first closed loop: NWDAF drops malicious sessions in 10s"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The closed-loop effectiveness claim rests on the untested assumption that the Random Forest model trained on CTU-13 botnet flows recognizes the Nmap-based scanning behavior used in the testbed; without validated detection accuracy, the measured mitigation latency may not reflect a genuinely successful detection.","fun_headline_variants_meta":{"raw":{"variants":["First 5G closed loop: UPF to NWDAF to SMF in 10s","5G core closes the loop on botnets in under 10s","First NWDAF closed loop: 5G core self-heals bot attacks","Open-source 5G core auto-drops malicious sessions in 10s","5G core's first closed loop: NWDAF drops malicious sessions in 10s"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000825,"raw_usage":{"total_tokens":3620,"prompt_tokens":968,"completion_tokens":2652,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":584,"completion_tokens_details":{"reasoning_tokens":2543}},"tokens_in":584,"tokens_out":2652,"duration_ms":15925,"temperature":1.0,"reasoning_tokens":2543,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:32:31.560538+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the deployed pipeline with labeled test traffic where the attack is a known botnet flow (e.g., a CTU-13 scenario) and measure true/false positive rates; if the model fails to flag the flow or flags benign traffic, the closed-loop mitigation claim collapses. Alternatively, disable the ML model and substitute a random or heuristic detector: if similar mitigation latency results, the ML component is not what carries the loop.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the UPF Event Exposure Service format and semantics that the implementation follows."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines NWDAF analytics services, including UE abnormal behavior, that the closed loop consumes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Specifies 5G procedures for PDU session management that the SMF extension invokes for session release."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the 5G system architecture and role of UPF/SMF that the implementation integrates with."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides one of the open-source 5G core implementations (OAI 5GC) that the NWDAF is integrated with."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the model registry and serving infrastructure used by the ML Model Provisioning Service."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Open5GS core that hosts the UPF and SMF modifications."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the CTU-13 botnet dataset used to train the Random Forest detector."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the UERANSIM UE/RAN simulator used in the testbed evaluation."},{"cited_title":"InGLOBECOM 2022-2022 IEEE Global Communications Conference","cited_arxiv_id":null,"evidence_quote":"Represents prior monitoring work lacking standardized interfaces, motivating the EES implementation."}],"review_version":1}