{"id":"66716547-d56b-4370-b7b8-e9c348e7925e","arxiv_id":"2606.00155","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"RWKV model pretrained on benign L3/L4 metadata achieves PR-AUC 0.93 for zero-shot attack detection via perplexity, outperforming LSTM baselines.","lead":"The paper trains a RWKV language model on unlabeled benign network flow metadata (packet lengths, timings, flags) to learn a 'grammar' of normal traffic, then flags attacks via high perplexity. A smart generalist might read it because it claims to detect intrusions in fully encrypted traffic without needing payload access or attack examples during pretraining.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"Monday-only pretraining leaves open whether non-Monday benign flows also produce high perplexity, undermining attack-specific separation.","rationale":"The reader's weakest_assumption pinpoints exactly this representativeness gap, which is load-bearing for any unsupervised perplexity detector. The LSTM comparison addresses only architectural inductive bias, not the data-distribution assumption. Full-text details on test-set composition or day-wise splits are not visible in the supplied abstract, so the concern stands.","tokens_in":1861,"tokens_out":331,"duration_ms":22524,"concrete_test":"Partition the full dataset by day; after training on Monday benign flows only, compute perplexity on all Tuesday–Friday benign flows and on attack flows. Recompute PR-AUC treating non-Monday benign as the sole negative class; if it falls below 0.80 at the original operating threshold, the reported 0.93 separation is not attack-specific.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Claim (2) asserts that per-flow perplexity from a model trained solely on 344,232 Monday benign flows separates attacks with PR-AUC 0.93. This requires that the Monday distribution captures the full support of benign traffic and that attacks are the only source of elevated perplexity. If benign flows on other days exhibit different but still learnable patterns (different port distributions, timing statistics, or TTL behaviors), they would also receive high perplexity, making the score reflect day-of-week shift rather than grammar violation. The abstract supplies no perplexity statistics or ROC/PR curves on held-out benign flows from Tuesday–Friday.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper proposes PLM-NIDS, a RWKV-4 state-space model trained only on unlabelled benign network flows (344,232 Monday flows) using L3/L4 metadata sequences (packet length, inter-arrival time, TTL, flags, hashed ports) as tokens. It claims three results: (1) benign traffic has learnable structure, shown by causal LM validation loss of 0.204; (2) attack flows violate this grammar, yielding per-flow perplexity separation with PR-AUC 0.93 in a zero-label setting; (3) the separation is architecturally nontrivial, as an LSTM on the same sequences yields ROC-AUC ~0.50. Supervised fine-tuning raises PR-AUC to 0.94 and ROC-AUC to 0.75; the O(T) recurrent inference supports streaming detection without DPI or flow buffering.","tokens_in":2039,"tokens_out":582,"duration_ms":15385,"significance":"If the central claims hold after addressing the representativeness issue, the work would be significant for encryption-agnostic NIDS: it demonstrates that next-token prediction on metadata alone can produce a useful anomaly score without attack labels, with a concrete LSTM contrast showing the value of the RWKV inductive bias. The streaming capability and zero-label framing are operationally relevant strengths.","major_comments":[{"comment":"Abstract, claim (2): The reported PR-AUC = 0.93 for attack separation by perplexity is load-bearing for the zero-label contribution, yet the manuscript provides no perplexity statistics, ROC/PR curves, or quantitative comparison on held-out benign flows from Tuesday–Friday. Without this, it is impossible to rule out that elevated perplexity reflects day-of-week distribution shift rather than attack-specific grammar violation.","section":"Abstract"},{"comment":"Dataset and experimental setup (assumed §3–4): The abstract states validation loss 0.204, PR-AUC 0.93, and LSTM baseline 0.50 but supplies no description of the full dataset source, train/test split procedure, tokenization vocabulary construction, sequence length handling, or statistical significance testing (e.g., confidence intervals on PR-AUC). These omissions make the numerical claims impossible to reproduce or assess for robustness.","section":"Dataset and experimental setup"},{"comment":"Abstract, fine-tuning paragraph: The claim that supervised fine-tuning raises PR-AUC from 0.93 to 0.94 is presented without an ablation isolating the contribution of the zero-shot perplexity score versus the added labels, leaving unclear whether the core unsupervised result already suffices or whether labels are required for operational performance.","section":"Abstract"}],"minor_comments":[],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for their careful reading and constructive comments on reproducibility and the need for additional controls. We address each point below and will revise the manuscript accordingly.","responses":[{"response":"We agree this analysis is necessary. The revised manuscript will add perplexity histograms, ROC/PR curves, and quantitative tables comparing held-out benign flows from Tuesday–Friday (which exhibit low perplexity matching the Monday validation set) against attack flows. This will demonstrate that the separation arises from grammar violation rather than temporal shift.","revision_made":"yes","referee_comment":"[Abstract] Abstract, claim (2): The reported PR-AUC = 0.93 for attack separation by perplexity is load-bearing for the zero-label contribution, yet the manuscript provides no perplexity statistics, ROC/PR curves, or quantitative comparison on held-out benign flows from Tuesday–Friday. Without this, it is impossible to rule out that elevated perplexity reflects day-of-week distribution shift rather than attack-specific grammar violation."},{"response":"We acknowledge the current manuscript lacks these details. The revision will expand the experimental setup section to specify the CIC-IDS2017 source, the exact train/test split (Monday benign flows only for pre-training), vocabulary construction from L3/L4 metadata, sequence length handling, and bootstrap confidence intervals on all reported metrics.","revision_made":"yes","referee_comment":"[Dataset and experimental setup] Dataset and experimental setup (assumed §3–4): The abstract states validation loss 0.204, PR-AUC 0.93, and LSTM baseline 0.50 but supplies no description of the full dataset source, train/test split procedure, tokenization vocabulary construction, sequence length handling, or statistical significance testing (e.g., confidence intervals on PR-AUC). These omissions make the numerical claims impossible to reproduce or assess for robustness."},{"response":"We will add an explicit ablation comparing zero-shot perplexity scoring against the fine-tuned model. The revision will show that the unsupervised perplexity already reaches PR-AUC 0.93, with fine-tuning contributing only a marginal 0.01 gain, confirming the core zero-label result is sufficient for strong performance.","revision_made":"yes","referee_comment":"[Abstract] Abstract, fine-tuning paragraph: The claim that supervised fine-tuning raises PR-AUC from 0.93 to 0.94 is presented without an ablation isolating the contribution of the zero-shot perplexity score versus the added labels, leaving unclear whether the core unsupervised result already suffices or whether labels are required for operational performance."}],"tokens_in":1695,"tokens_out":534,"duration_ms":24360,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The paper trains an RWKV-4 model only on 344k Monday benign flows and reports that per-flow perplexity separates attacks at PR-AUC 0.93 with no attack labels used. An LSTM on the same sequences collapses to a majority predictor. That contrast is the clearest new piece.\n\nThe approach is encryption-agnostic by design and the O(T) inference is operationally relevant. Treating L3/L4 metadata as tokens and showing a state-space model can learn the structure is a reasonable extension of existing language-modeling ideas to this domain.\n\nThe main gap is the Monday-only pretraining. If benign flows on Tuesday or later have different but still predictable patterns in ports, timing, or flags, they would also score high perplexity and the metric would reflect day shift rather than attack grammar. The abstract gives no perplexity numbers or curves on held-out benign flows from other days, so the separation could be narrower than claimed.\n\nDataset details, tokenization, train/test splits, and any statistical tests are also absent, which makes the 0.93 number hard to interpret. The supervised fine-tuning step is noted but not ablated against the zero-shot case in detail.\n\nThis is worth sending to review for the NIDS community. The inductive-bias result is worth checking and the practical constraints are real, but the generalization claim needs the missing benign-day experiments before it holds up.","headline":"RWKV on Monday benign flows gives a clean perplexity separation from attacks but the day-specific training leaves the attack-specific claim unproven without checks on other benign days.","tokens_in":2513,"tokens_out":367,"would_cite":false,"duration_ms":12532,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"A language model trained solely on benign network flows detects attacks through elevated perplexity on their metadata sequences.","keywords":["network intrusion detection","language modeling","perplexity scoring","flow metadata","state space model","encryption agnostic detection","unsupervised anomaly detection"],"falsifier":"A collection of attack flows that receive perplexity scores comparable to the benign validation set under the trained model.","tokens_in":2756,"feed_emoji":"🛡️","tokens_out":581,"duration_ms":14742,"temperature":0.7,"pith_summary":"The paper shows that packet metadata sequences from normal traffic form a learnable grammar that a state-space model can capture from unlabeled examples alone. Attack flows then stand out because they produce higher perplexity under the same model, yielding strong separation without any attack examples seen during pretraining. The separation depends on the model's recurrent pretraining, since an LSTM baseline on identical data collapses to a trivial predictor. Because only header fields are used, the method applies directly to encrypted protocols.","feed_headline":"Benign flows alone train a model that flags attacks by sequence surprise","feed_subtitle":"RWKV trained on Monday metadata separates attacks at 0.93 PR-AUC with zero attack labels and no payload inspection.","key_machinery":"RWKV-4 state-space model trained as a causal language model on tokenized sequences of packet length, inter-arrival time, TTL, flags and hashed ports, with per-flow perplexity used as the detection score.","core_discovery":"Benign network flows possess a statistically consistent structure in L3/L4 metadata that a causal language model can learn, and attack flows reliably violate that structure enough for per-flow perplexity to serve as an effective unsupervised detector.","pith_inferences":["Similar unsupervised pretraining on normal data could apply to anomaly detection in other sequential domains such as system logs or sensor streams.","The approach suggests that many signature-based intrusion systems might be replaceable by grammar-violation scores when representative normal traffic is available.","Real-time streaming becomes feasible in high-speed environments where flow reassembly is impractical."],"forward_implications":["Detection runs at line rate because the model supports O(T) recurrent inference without buffering entire flows.","The detector remains effective on TLS 1.3 and QUIC traffic since it never examines payloads.","Adding a small amount of labeled data through supervised fine-tuning further improves precision and recall.","Architectures without the causal pretraining bias fail to produce the separation on the same metadata sequences."],"fun_headline_variants":["Benign flows train RWKV to flag attacks by flow perplexity","Language model on network metadata detects intrusions via perplexity","Causal pretraining on benign traffic enables attack detection by surprise","Metadata sequence model spots attacks without payload or labels"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"Attack flows will produce reliably higher perplexity than benign flows rather than simply different but still predictable patterns.","fun_headline_variants_meta":{"raw":{"variants":["Benign flows train RWKV to flag attacks by flow perplexity","Language model on network metadata detects intrusions via perplexity","Causal pretraining on benign traffic enables attack detection by surprise","Metadata sequence model spots attacks without payload or labels"]},"model":"grok-4.3","cost_usd":0.006156,"raw_usage":{"total_tokens":2958,"prompt_tokens":776,"num_sources_used":0,"completion_tokens":65,"cost_in_usd_ticks":61562000,"prompt_tokens_details":{"text_tokens":776,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2117,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":776,"tokens_out":65,"duration_ms":14050,"temperature":1.0,"reasoning_tokens":2117,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-28T22:15:27.084346+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A collection of attack flows that receive perplexity scores comparable to the benign validation set under the trained model.","supporting_citations":[],"review_version":1}