{"id":"80f46543-0b60-412d-95a1-d9a71daced17","arxiv_id":"2505.00240","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"Fine-tuned BERT-small reaches 99.75% test accuracy on combined IoT-23 and TON IoT attack classification, but the claimed real-time prevention advantage over traditional methods is not demonstrated.","lead":"This paper fine-tunes small BERT language models on two public IoT traffic datasets and reports about 99.7% accuracy for classifying attacks. The authors claim the resulting system enables real-time automated response on edge devices, but the paper does not compare against traditional security tools or evaluate the response component.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Temporal leakage from the record-level 60/20/20 split, not model quality, may explain the 99.75% test accuracy; the latency and prevention claims are also untested.","rationale":"The reader's REJECT verdict already rests on the same core problem: the paper's only quantitative evidence is a classification accuracy computed under an i.i.d. assumption that network-flow data likely violates. My independent read confirms this. Section IV's Table III and Figures 3-5 are the entire evaluation; there is no baseline comparison, no latency measurement, no resource comparison, and no execution of Algorithm 1. Section III.B describes the Docker setup and a prompt template, but no code, hyperparameters, or prompt details sufficient for reproduction. The contribution of a novel LLM-based framework is therefore supported only by a possibly leaked accuracy number. The stress-test pass does not change the reader's verdict; it sharpens the technical route by which the central claim could fail. The proposed temporal/session split is a direct test: if accuracy holds, the detection result should survive; if it drops, the paper's strongest quantitative support is invalid.","tokens_in":7457,"tokens_out":3596,"duration_ms":36238,"concrete_test":"Re-run the BERT-small evaluation with a temporal split: sort all records by timestamp, use the earliest 60% for training, the next 20% for validation, and the latest 20% for testing, keeping records from the same connection or attack session within one split. Recompute test accuracy, F1, precision, and recall from Table III. If accuracy drops materially below 99.75%, the original record-level split has temporal leakage and the central detection claim is overstated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing result is the 99.75% test accuracy in Table III, measured on a 60/20/20 split of individual flow records (Section IV: 'The dataset was split into 60% training, 20% validation, and 20% testing'). If the split is at record level, it assumes each flow is an i.i.d. sample. This is unlikely for IoT-23 and TON IoT: many rows describe the same TCP connection, botnet C&C session, or DDoS burst, so near-duplicate flows land on both sides of the split. The classifier can then memorize source IP/port/timestamp patterns from training copies and 'predict' the test copies without learning anything about unseen attacks. The reported accuracy is therefore not a valid estimate of generalization to new traffic. The paper provides no evidence that the split was grouped by connection, session, or time, and no per-dataset or per-attack results. Separately, the abstract's claims about response latency and resource efficiency over traditional methods are unsupported: no baseline IDS is evaluated, no latency is measured, and Algorithm 1, the decision-tree prevention logic, is never run or tested. The central claim rests on the 99.75% figure; if that figure is inflated by leakage, the paper's main quantitative support is invalid.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript proposes an LLM-based threat detection and prevention framework for IoT. The detection component fine-tunes lightweight BERT variants (BERT-Small, BERT-Mini, TinyBERT) on the IoT-23 and TON_IoT datasets after converting flow records into textual prompts; the prevention component is a decision-tree rule layer intended to trigger edge countermeasures, especially for DDoS. The authors report 99.75% test accuracy for BERT-Small, inference throughput around 287-294 req/s, and energy per request around 0.12-0.15 J/Req, and claim in the abstract that the system improves detection accuracy, response latency, and resource efficiency over traditional methods. The implementation is presented as a Docker-based edge-cloud simulation, but no code or artifact is supplied.","tokens_in":7597,"tokens_out":6374,"duration_ms":64381,"significance":"The idea of applying small fine-tuned language models to tabular network flow data via prompt encoding is worth exploring, and the use of public benchmark datasets (IoT-23, TON_IoT) is appropriate. If the experiments were valid, the reported near-perfect accuracy would be useful, although the practical relevance would still depend on the untested prevention and latency claims. However, the current evidence does not support the central contributions: the main accuracy figure may be inflated by an ungrouped random split, no baseline comparison exists for the headline improvement claim, and the prevention component is never evaluated. The paper's strengths are limited to a clear architecture description and a plausible deployability story; reproducibility is claimed but not demonstrated.","major_comments":[{"comment":"Section IV (evaluation split): the 60/20/20 split is described at the level of individual flow records, but both IoT-23 and TON_IoT contain many records belonging to the same TCP connections, botnet C&C sessions, or attack bursts. Without grouping by connection, session, or time window, near-duplicate flows will appear on both sides of the split, so the 99.75% test accuracy in Table III may reflect memorization rather than generalization to unseen attacks. The manuscript gives no evidence of a grouped split and no per-dataset or per-attack-class breakdown, so the central detection claim is not supported as stated.","section":"Section IV"},{"comment":"The abstract claims 'significant improvements in detection accuracy, response latency, and resource efficiency over traditional security methods,' but Section IV compares only the three BERT variants and reports no traditional IDS, ML baseline, or response-latency measurement. The 'Inference (Req/Sec)' column in Table III is throughput, not latency, and the 'Energy Consumption' values are not compared with any alternative. These headline claims are therefore unsupported.","section":"Abstract and Section IV"},{"comment":"Algorithm 1 is presented as the DDoS prevention mechanism, and Section III-A2 promises 'real-time, rule-based prevention optimized for resource-constrained IoT environments,' yet Section IV contains no implementation or evaluation of this algorithm. There are no results for the decisions it makes, no end-to-end edge latency, no false-positive/false-negative analysis of the mitigation actions, and no resource-usage measurements for the decision tree. As a result, the paper's conclusion that the system demonstrates effective automated response is not evidence-based.","section":"Algorithm 1 and Section III-A2"},{"comment":"Tables I and II show strong class imbalance, and Table III reports micro-averaged F1, precision, and recall. Micro-averages weight the majority classes heavily, so a 99.75% micro-F1 can accompany poor performance on rare attack types; indeed, Section IV concedes that all models 'struggle more with less frequent classes.' Per-class precision/recall or macro-averaged metrics are needed before the paper can claim detection of sophisticated or rare attacks.","section":"Tables I-III"}],"minor_comments":[{"comment":"Figure 1 appears as a floating figure before Section III with no in-text callout or explanatory caption, making the architecture figure difficult to interpret.","section":"Section II"},{"comment":"The prompt example gives one concrete record, but the manuscript does not specify how arbitrary records are converted into prompts (selected fields, normalization, tokenization, truncation length, prompt template variations), which prevents reproduction.","section":"Section III-B2"},{"comment":"The text uses 'Large Learning Models' where 'Large Language Models' is intended, and Section IV writes 'To IoT' instead of 'TON_IoT'; these should be corrected throughout.","section":"Introduction and Index Terms"},{"comment":"The label assignments are inconsistent: label '5' is used for both C&C-HeartBeat and C&C-FileDownload in Table II, and the text refers to 21 classes while the listed label ranges and duplicates do not make the mapping clear. The authors should provide a single unambiguous label table.","section":"Tables I and II"},{"comment":"The paper says the evaluation uses 'a combination of the IoT23 and To IoT datasets' but does not state how the two datasets are merged, re-labeled, or subsampled, nor how the class proportions in Tables I and II relate to the final training set.","section":"Section III-B2"},{"comment":"The conclusion claims robustness against 'evolving cyber threats' and mentions adversarial robustness, but no experiments on new, zero-day, or adversarial traffic are reported; the wording should be limited to the static benchmark that was actually evaluated.","section":"Section V"}],"recommendation":"reject","confidential_remarks":"The authors' own prior work (references [2] and [13]) is cited in passing, and reference [13] concerns LLMs and federated learning for IoT, which seems closely related to the future-work claims; the novelty overlap is not discussed. For a journal submission, the absence of any artifact release and the unvalidated prevention layer are, in my view, disqualifying at the present stage. If the editor prefers to allow the authors to run a properly grouped evaluation with baselines and prevention experiments, major revision would be the alternative framing, but my recommendation is reject as submitted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Alison—\n\nThe abstract promises significant improvements in detection accuracy, response latency, and resource efficiency over traditional security methods. The paper delivers none of those comparisons. That is the headline. The one quantitative result that might support the accuracy claim—99.75% test accuracy for BERT-small on IoT-23 plus TON_IoT—is undermined by how the split was done. A 60/20/20 random split of individual flow records is almost certainly leaking temporal and connection-level duplicates into the test set. IoT-23 in particular contains many rows from the same TCP connection or botnet session, so the model can memorize near-copies rather than learn to generalize. I think the stress-test note is right: this is a load-bearing flaw, not a quibble.\n\nTo be fair, the paper does some things well. It chooses a real problem, uses two standard public datasets, and describes a clean two-component design—fine-tuned lightweight BERT for detection, decision-tree rules for mitigation. The prompt template is concrete. The loss curves and confusion matrices indicate actual training runs, and the prose is readable. The authors also cite their own prior work in a relevant way; that is not a problem by itself.\n\nThe soft spots are, in order of severity: (1) no baselines at all, so 'over traditional methods' is an assertion, not a finding; (2) no latency or resource-usage measurements despite those being in the abstract; (3) the prevention component, Algorithm 1, is never executed; (4) no error bars, hyperparameters, code, or Docker artifacts, so the numbers are not reproducible; and (5) the leakage issue above. None of these are hidden flaws—they are visible from the text itself.\n\nIf the authors re-ran with a grouped split (by connection or time), added a couple of classical baselines, actually ran the decision-tree prevention logic, and reported per-attack performance, the paper could become a modest but useful applied contribution. As submitted, I would not accept it. I would, however, send it out rather than desk-reject: the architecture is coherent, the datasets are appropriate, and the evaluation gaps are concrete and fixable. A competent referee should have no trouble writing the revision checklist.\n\nFor you: probably not worth citing in your own work in its current form, but if you teach or read about evaluation pitfalls, it is a good example of temporal leakage destroying a high-accuracy claim.","headline":"The 99.75% accuracy figure is likely a temporal-leakage artifact, and the abstract's latency and prevention claims are untested; the paper is coherent but poorly evaluated.","tokens_in":8210,"tokens_out":3141,"would_cite":false,"duration_ms":32481,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A fine-tuned BERT-small LLM detects IoT cyber threats at 99.75% accuracy and triggers automated edge countermeasures in real time.","keywords":["IoT security","large language models","intrusion detection","intrusion prevention","BERT fine-tuning","DDoS mitigation","IoT-23 dataset","TON_IoT dataset"],"falsifier":"Re-run the experiment with a time-ordered split—train on the earliest 60% of each dataset's flow records and test on the latest 40%, using the records' timestamps—and compare accuracy and F1. A large drop would show the reported 99.75% comes from overlapping connections or time windows rather than from detecting unseen attack patterns.","tokens_in":7140,"feed_emoji":"🛡️","tokens_out":10590,"duration_ms":95603,"temperature":0.7,"pith_summary":"The paper claims that a two-component security system—an LLM-based detector plus a decision-tree preventer—can keep IoT networks safe in real time without human intervention. The detector is a lightweight BERT variant fine-tuned on the IoT-23 and TON_IoT datasets, whose flow records are converted into natural-language prompts. In a simulated edge-cloud test, the authors report 99.75 percent test accuracy and F1 for BERT-small, with roughly 288 inferences per second and 0.14 joules per request. The practical significance, if the claim holds, is that LLM-driven security can run on constrained devices and act immediately rather than waiting for cloud instructions.","feed_headline":"Lightweight AI catches IoT attacks at 99.75% accuracy","feed_subtitle":"An LLM detector plus decision-tree edge response promises real-time, autonomous IoT security without cloud round-trips","key_machinery":"The load-bearing mechanism is the pairing of a fine-tuned BERT-small classifier with a decision-tree prevention policy. The classifier treats each network connection as a text prompt summarizing ports, protocol, duration, bytes, packets, and connection state, and outputs one of 21 labels via softmax over logits with cross-entropy training. The decision tree then maps a DDoS label to concrete countermeasures using attack intensity, source-IP count, system load, and attack duration. The container-based deployment is what lets the authors simulate edge and cloud tiers and measure accuracy, latency, and resource use in a reproducible way.","core_discovery":"The central claim is that lightweight language models, fine-tuned on IoT traffic data and fed with prompt-structured flow summaries, can both detect and help mitigate cyber threats at the edge. The paper reports that BERT-small reaches 99.75 percent test accuracy, precision, recall, and F1 across 21 classes, with the lowest training and validation loss among the three BERT variants considered (BERT-small, BERT-mini, TinyBERT). The prevention half uses a decision tree that takes the LLM's DDoS label and applies context-dependent actions—rate limiting for moderate intensity, IP blocking and traffic redirection for extreme intensity, CAPTCHA for many source IPs, and honeypot redirection for long attacks. The authors interpret these results as evidence that an LLM-driven framework can overcome the limitations of traditional rule-based and cloud-centered IoT security.","pith_inferences":["A natural extension beyond the paper is to feed the LLM's confidence or attention weights into the decision-tree policy, so low-confidence detections could trigger more conservative actions.","If the reported accuracy survives a time-ordered split, the same prompt-plus-classifier recipe could transfer to other telemetry domains, such as industrial control or 5G edge slices, by swapping the fine-tuning data and prompt template.","Because the reported metrics are per-flow classification, a deployment study should also measure how often benign traffic is rate-limited or blocked, capturing the real cost of false positives."],"forward_implications":["A BERT-small flow classifier can run at about 288 requests per second while using roughly 0.14 joules per request, making on-device LLM detection plausible for resource-constrained IoT gateways.","Once the LLM labels a flow as DDoS, the decision-tree layer can immediately select rate limiting, IP blocking, CAPTCHA, or honeypot redirection, so prevention does not wait for a cloud round-trip.","Fine-tuning on IoT-23 and TON_IoT gives the detector coverage of 21 traffic classes spanning DDoS, scanning, botnet C&C, XSS, backdoor, injection, ransomware, and MITM.","Because the deployment is modular and containerized, the detector and preventer can be evaluated and swapped independently across simulated edge and cloud environments."],"supporting_citations":[{"why":"Motivates domain-specific LLM fine-tuning for security by applying it to zero-day vulnerability detection.","marker":"[3]"},{"why":"Supplies the prompt-engineering approach for context-aware IoT traffic analysis that the detection prompt builds on.","marker":"[12]"},{"why":"Provides the IoT-23 labeled flow records used to fine-tune the detector, including botnet, DDoS, and scanning classes.","marker":"[14]"},{"why":"Provides the TON_IoT multi-source telemetry and attack-type distributions used alongside IoT-23 for training.","marker":"[15]"},{"why":"Defines TinyBERT, the comparison model that anchors the lightweight-model evaluation.","marker":"[16]"}],"fun_headline_variants":["Lightweight BERT detects IoT attacks with 99.75% accuracy","LLM edge defense: 99.75% accuracy, real-time mitigation","Tiny BERT models slash IoT threat response times","Edge LLM stops DDoS with decision tree actions","LLM-based IoT security: autonomous, accurate, fast"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that a random 60/20/20 split of network flow records creates independent training and test samples; if flows from the same connection or time window appear in both splits, the reported 99.75% accuracy may reflect temporal leakage rather than generalization to new attacks.","fun_headline_variants_meta":{"raw":{"variants":["Lightweight BERT detects IoT attacks with 99.75% accuracy","LLM edge defense: 99.75% accuracy, real-time mitigation","Tiny BERT models slash IoT threat response times","Edge LLM stops DDoS with decision tree actions","LLM-based IoT security: autonomous, accurate, fast"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000292,"raw_usage":{"total_tokens":1645,"prompt_tokens":828,"completion_tokens":817,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":444,"completion_tokens_details":{"reasoning_tokens":731}},"tokens_in":444,"tokens_out":817,"duration_ms":6054,"temperature":1.0,"reasoning_tokens":731,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:46:36.284966+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the experiment with a time-ordered split—train on the earliest 60% of each dataset's flow records and test on the latest 40%, using the records' timestamps—and compare accuracy and F1. A large drop would show the reported 99.75% comes from overlapping connections or time windows rather than from detecting unseen attack patterns.","supporting_citations":[{"cited_title":"LLMs for Cyber Security: New Opportunities","cited_arxiv_id":"2404.11338","evidence_quote":"Motivates domain-specific LLM fine-tuning for security by applying it to zero-day vulnerability detection."},{"cited_title":"A Unified Framework for Context-Aware IoT Management and State-of-the-Art IoT Traffic Anomaly Detection","cited_arxiv_id":"2412.19830","evidence_quote":"Supplies the prompt-engineering approach for context-aware IoT traffic analysis that the detection prompt builds on."},{"cited_title":"Iot-23: A labeled dataset with malicious and benign iot network traffic,","cited_arxiv_id":null,"evidence_quote":"Provides the IoT-23 labeled flow records used to fine-tune the detector, including botnet, DDoS, and scanning classes."},{"cited_title":"A new distributed architecture for evaluating ai-based security systems at the edge: Network ton iot datasets,","cited_arxiv_id":null,"evidence_quote":"Provides the TON_IoT multi-source telemetry and attack-type distributions used alongside IoT-23 for training."}],"review_version":1}