{"id":"fc4990c0-3746-47f0-ad54-3687848a0531","arxiv_id":"2605.26166","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Targeted fixes to AOC-IDS using balanced sampling, pseudo-label filtering, mixup, and a lighter autoencoder raise accuracy to 95.45% with XGBoost and 90.88% with deep learning on UNSW-NB15 while cutting parameters 55%.","lead":"This paper replicates and then improves an existing autonomous online intrusion detection system for IoT devices by balancing class data, filtering unreliable labels, and shrinking the model size. A smart generalist might read it to understand practical tweaks that could make security tools more accurate and runnable on small connected devices.","discovery_kind":"extension","skeptic_critique":{"model":"grok-4.3","headline":"All gains shown only on static UNSW-NB15 splits; no online/streaming or IoT-device evaluation performed","rationale":"The load-bearing assumption identified by the reader (static-benchmark gains translating to live evolving IoT) is exactly the weakest link for the headline claim. The replication match is already shown to be close and is not the primary risk. No other internal contradictions (e.g., in equations or ablations) are visible from the supplied information.","tokens_in":1793,"tokens_out":327,"duration_ms":40861,"concrete_test":"Re-implement the combined DL pipeline and run it in an incremental/online protocol: train on first 60 % of UNSW-NB15, then feed remaining samples sequentially while updating the decision module; report final accuracy and adaptation latency versus the replicated AOC-IDS baseline. If the relative gain disappears or becomes statistically insignificant, the online-IoT claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the proposed changes (XGBoost-BalSamp and PseudoFilter+MixupAug+LiteAE) improve AOC-IDS for autonomous online IoT use. All reported numbers (89.39 % replication, 95.45 % XGBoost, 90.88 % best-run DL) come from the standard offline train/test partition of UNSW-NB15. No experiment uses sequential data arrival, concept drift, pseudo-label updates over time, or measures latency/memory on actual edge hardware. Parameter reduction alone does not establish online deployability.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper replicates the AOC-IDS method (Autoencoder with CRC loss and Gaussian decision module) on UNSW-NB15 to 89.39% accuracy (close to the published 89.19%), then proposes XGBoost-BalSamp to reach 95.45% accuracy (+6.26%) and a combined DL pipeline (PseudoFilter + MixupAug + LiteAE) to 90.88% accuracy / 91.45% F1 with 55% fewer parameters. It claims these targeted fixes address class imbalance, unreliable pseudo-labels, limited generalization, and IoT computational overhead for autonomous online IDS.","tokens_in":1917,"tokens_out":533,"duration_ms":24251,"significance":"If the accuracy and parameter reductions hold under proper statistical controls and online evaluation, the work would offer practical, deployable improvements to AOC-IDS for resource-constrained IoT settings. The close replication of the base method is a clear strength. However, because all numbers come from a single static offline split, the significance for the paper's stated goal of autonomous online operation remains limited.","major_comments":[{"comment":"Abstract: The central claim that the proposed methods improve AOC-IDS for 'autonomous online' IoT deployment is not supported by the reported experiments; all accuracy figures (89.39%, 95.45%, 90.88%) are obtained on the standard offline train/test partition of UNSW-NB15 with no sequential arrival, concept-drift, or real-time pseudo-label update protocols.","section":"Abstract"},{"comment":"Abstract: The DL pipeline reports a 'best-run' accuracy of 90.88% without error bars, number of runs, variance, or statistical significance tests against the replicated baseline, making it impossible to judge whether the 1.49% gain is reliable or due to random variation.","section":"Abstract"},{"comment":"Abstract: No ablation isolates the contribution of each proposed component (BalSamp, PseudoFilter, MixupAug, LiteAE) nor compares them against standard alternatives (e.g., other balancing techniques or lightweight autoencoders), so the necessity of the specific combination cannot be assessed.","section":"Abstract"}],"minor_comments":[{"comment":"Abstract: The sentence beginning 'These results demonstrate...' lacks a preceding period or space after the preceding sentence.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback. We address each major comment point-by-point below, with plans for targeted revisions where appropriate.","responses":[{"response":"We agree that the reported experiments use the standard static train/test split of UNSW-NB15, matching the evaluation protocol in the original AOC-IDS paper. The proposed components (PseudoFilter, MixupAug, LiteAE) are explicitly motivated by challenges in autonomous online settings such as unreliable pseudo-labels and the need for generalization under evolving threats. However, we acknowledge that the abstract and claims overstate the direct support for online operation without sequential or drift-aware experiments. In revision we will tone down the abstract and introduction to state that results are obtained on the benchmark split while the methods target online deployment needs.","revision_made":"partial","referee_comment":"[Abstract] Abstract: The central claim that the proposed methods improve AOC-IDS for 'autonomous online' IoT deployment is not supported by the reported experiments; all accuracy figures (89.39%, 95.45%, 90.88%) are obtained on the standard offline train/test partition of UNSW-NB15 with no sequential arrival, concept-drift, or real-time pseudo-label update protocols."},{"response":"This observation is correct and we will revise the manuscript to report mean performance and standard deviation over multiple runs (minimum of five), along with statistical significance testing (e.g., paired t-test) against the replicated AOC-IDS baseline.","revision_made":"yes","referee_comment":"[Abstract] Abstract: The DL pipeline reports a 'best-run' accuracy of 90.88% without error bars, number of runs, variance, or statistical significance tests against the replicated baseline, making it impossible to judge whether the 1.49% gain is reliable or due to random variation."},{"response":"We will add an ablation study section that isolates the effect of each component individually and in combination, and that benchmarks against common alternatives such as SMOTE for class balancing and other lightweight autoencoder designs.","revision_made":"yes","referee_comment":"[Abstract] Abstract: No ablation isolates the contribution of each proposed component (BalSamp, PseudoFilter, MixupAug, LiteAE) nor compares them against standard alternatives (e.g., other balancing techniques or lightweight autoencoders), so the necessity of the specific combination cannot be assessed."}],"tokens_in":1525,"tokens_out":533,"duration_ms":30495,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The paper first replicates the AOC-IDS baseline to 89.39% accuracy, close to the published 89.19%. It then applies balanced sampling to an XGBoost variant for 95.45% accuracy and combines pseudo-label filtering, mixup, and a compressed autoencoder for 90.88% accuracy with 55% fewer parameters.\n\nThe replication is the most useful part because it lets readers trust the starting point for the comparisons. The changes themselves are established techniques applied to one existing system rather than a new method.\n\nThe evaluation stays limited to the standard offline train/test split of UNSW-NB15. No experiments use streaming data, concept drift, or repeated pseudo-label updates over time, and there are no latency or memory measurements on actual IoT hardware. The title and abstract emphasize autonomous online detection for evolving threats, yet the numbers come only from static benchmark runs. That gap makes it hard to judge whether the reported gains would hold in the setting the work claims to target.\n\nThe work is transparent about its methods and reports the replication numbers directly. It will mainly interest researchers already working on this specific IDS line or similar benchmark-driven IoT security papers. Readers seeking new frameworks or real deployment evidence will find the contribution incremental.\n\nI would send it to peer review at an applied security venue. Reviewers will likely press on generalization and online testing, but the replication and clear reporting give it enough grounding to be worth referee time.","headline":"Replicates AOC-IDS cleanly on UNSW-NB15 then layers standard ML fixes for accuracy and size gains, but all tests stay on static splits with no online or device runs.","tokens_in":2385,"tokens_out":382,"would_cite":false,"duration_ms":22358,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Targeted fixes for class imbalance and pseudo-label reliability in AOC-IDS deliver 95.45 percent accuracy on UNSW-NB15 while shrinking the deep model by 55 percent.","keywords":["intrusion detection","IoT security","autoencoder","class imbalance","pseudo-labeling","XGBoost","lightweight models"],"falsifier":"Running the improved system on a live IoT network that introduces previously unseen attack types and observing whether the reported accuracy advantage disappears.","tokens_in":2710,"feed_emoji":"🛡️","tokens_out":658,"duration_ms":24657,"temperature":0.7,"pith_summary":"The paper first replicates the published AOC-IDS system on the UNSW-NB15 benchmark and obtains nearly identical accuracy. It then isolates four concrete obstacles to IoT deployment: class imbalance, unreliable pseudo-labels, weak generalization, and high parameter counts. Separate remedies are introduced for each: balanced sampling with an XGBoost classifier on one path and a filtered pseudo-label plus mixup augmentation plus a lighter autoencoder on the deep-learning path. These produce measurable accuracy and F1 gains together with a large reduction in model size. A sympathetic reader would care because resource-limited IoT devices need detectors that remain effective against changing threats without constant manual tuning or extra hardware.","feed_headline":"Targeted fixes lift AOC-IDS accuracy to 95.45 percent","feed_subtitle":"Balanced sampling and a 55 percent smaller autoencoder also raise F1 to 91.45 percent for IoT deployment","key_machinery":"The combination of PseudoFilter for reliable pseudo-labels, MixupAug for better generalization, and LiteAE for reduced computational cost, together with BalSamp for handling class imbalance in the tree-based path.","core_discovery":"By applying balanced sampling to an XGBoost classifier the authors reach 95.45 percent accuracy, a 6.26 point gain over the replicated baseline. Combining PseudoFilter, MixupAug, and LiteAE in the deep-learning pipeline yields 90.88 percent accuracy and 91.45 percent F1 while using 55 percent fewer parameters than the original autoencoder architecture.","pith_inferences":["These techniques might generalize to other online learning setups beyond intrusion detection if the core mechanisms are isolated.","Testing on additional IoT-specific datasets could reveal whether the gains persist under different traffic distributions.","The replication effort suggests that future papers should publish exact implementation details to enable fair comparisons."],"forward_implications":["If the accuracy gains hold, IoT edge devices can run higher-performing detectors without extra hardware.","The autonomous online nature of the original system is preserved while the identified limitations are addressed.","Reduced parameter count directly improves deployability on resource-constrained IoT devices.","The replication step shows that small, targeted changes can be evaluated against a published baseline."],"fun_headline_variants":["Balanced sampling achieves 95.45 percent AOC-IDS accuracy","LiteAE with PseudoFilter and MixupAug achieves 90.88 percent accuracy","55 percent smaller autoencoder achieves 90.88 percent accuracy","Deep learning approach achieves 91.45 percent F1 on AOC-IDS"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"Accuracy gains measured on the static UNSW-NB15 benchmark will translate to live IoT environments with evolving attack patterns.","fun_headline_variants_meta":{"raw":{"variants":["Balanced sampling achieves 95.45 percent AOC-IDS accuracy","LiteAE with PseudoFilter and MixupAug achieves 90.88 percent accuracy","55 percent smaller autoencoder achieves 90.88 percent accuracy","Deep learning approach achieves 91.45 percent F1 on AOC-IDS"]},"model":"grok-4.3","cost_usd":0.015006,"raw_usage":{"total_tokens":6463,"prompt_tokens":706,"num_sources_used":0,"completion_tokens":78,"cost_in_usd_ticks":150062000,"prompt_tokens_details":{"text_tokens":706,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":5679,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":706,"tokens_out":78,"duration_ms":47970,"temperature":1.0,"reasoning_tokens":5679,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-29T23:29:43.844939+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Running the improved system on a live IoT network that introduces previously unseen attack types and observing whether the reported accuracy advantage disappears.","supporting_citations":[],"review_version":1}