{"id":"0d17cbb1-0086-44af-8e46-a4646a816e61","arxiv_id":"2608.00895","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A multi-LLM consensus framework rates five NIDS benchmark datasets against 68 banking-relevant, network-observable MITRE ATT&CK techniques, finding UNSW-NB15 most useful and CIC-DDoS2019 largely blind to banking attacks.","lead":"The authors used four large language models to decide which cyberattack techniques a passive network sensor can detect, then rated five public intrusion detection datasets on coverage of banking-sector attacks. UNSW-NB15 got the top weighted coverage score at 82.2%, while CIC-DDoS2019 missed about 90% of banking-relevant behaviors.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LLM coverage labels are unvalidated: UNSW-NB15's 82.2% lead rests on indirect labels never checked against actual dataset traffic, so the central ranking may be an artifact of dataset profiles.","rationale":"The paper's strongest quantitative claims are that UNSW-NB15 is the best banking-sector NIDS benchmark among the five and that CIC-DDoS2019 has an 89.9% blind spot. Both are computed from Eq. (2) using LLM-assigned Direct/Indirect/None labels, and those labels are never checked against the datasets themselves. This is load-bearing because all downstream numbers are affine functions of the labels; a few mislabeled high-priority techniques will change percentages and potentially the ordering. The reader's flagged concern about the hand-selected 40/40/20 priority weights is real but secondary: weights rescale the same binary labels, whereas mapping errors can add or remove entire techniques from the numerator. I therefore focus the test on mapping validity rather than weights. The paper deserves credit for honest limitations, published prompt templates, reproducible artifacts, and a conservative dispute-resolution policy; these justify keeping the conditional acceptance rather than moving to rejection. The proposed check is feasible if the artifact repository includes the LLM outputs and dataset profiles, and it would tell us whether the central ranking reflects dataset content or an artifact of the multi-LLM mapping procedure.","tokens_in":11515,"tokens_out":6159,"duration_ms":59000,"concrete_test":"Take a stratified sample of 20 technique-dataset pairs from the published artifacts, including UNSW-NB15 Direct/Indirect labels and CIC-DDoS2019 None labels. Have an independent analyst with access to the actual raw PCAP or flow records, not the text profiles, label each pair using the paper's Direct/Indirect/None definitions; compute agreement and recompute Eq. (2) with the human labels. If UNSW-NB15 loses first place or CIC-DDoS2019 coverage rises above about 20%, the reported ranking is an artifact of profile-driven LLM judgment. As a cheaper auxiliary check, recompute coverage with Indirect weighted 0.5 instead of 1.0; if UNSW-NB15 drops below another dataset, the 82.2% headline depends on the direct/indirect conflation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central ranking is computed from Eq. (2) using Stage-2 labels (Direct/Indirect/None) that four LLMs assign from text dataset profiles, not from the datasets themselves. No independent check verifies that a 'Direct' label corresponds to traffic actually present in UNSW-NB15 or that a 'None' label is absent from CIC-DDoS2019. This is load-bearing because the coverage numerator is a weighted sum over those labels: a handful of false Direct/Indirect calls on high-priority techniques can move UNSW-NB15 by several points and potentially change the ordering. The paper's own numbers make the risk concrete: UNSW's 82.2% rests on 63.8% indirect evidence, and Eq. (2) gives Indirect the same unit credit as Direct (c_i=1 for both). If, say, half of those indirect labels reflect plausible prerequisites rather than artifacts observable in the captured traffic, UNSW's score falls toward its 18.4% direct-only figure. CIC-DDoS2019's 10.1% could also be an underestimate if its PCAPs contain C2 or scanning traffic that the profile omits. LLM consensus reduces inter-model disagreement but not systematic error, since all models read the same profile and share its blind spots. The 75% threshold and exclusion of 12 disputed techniques are conservative, but they validate agreement among LLMs, not correctness against ground truth.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a three-stage framework for evaluating how well network intrusion detection benchmark datasets cover MITRE ATT&CK techniques relevant to banking. The authors filter 210 banking-related techniques through NIST SP 800-94 passive sensor constraints using a four-LLM consensus engine (75% agreement), obtaining 68 network-observable techniques; then map those techniques to five datasets using the same LLM consensus procedure over text dataset profiles. A hand-weighted Priority Score (Eq. 1, weights 40/40/20) and a weighted coverage metric (Eq. 2, counting Direct and Indirect equally) yield the headline results: UNSW-NB15 achieves 82.2% weighted coverage (18.4% direct) and CIC-DDoS2019 achieves only 10.1%.","tokens_in":11814,"tokens_out":8154,"duration_ms":58919,"significance":"If the coverage estimates were validated against actual packet contents, this would be a valuable contribution to sector-aware NIDS evaluation: the NIST-constrained baseline is a sensible way to avoid overclaiming detectability, the consensus prompt design is clearly documented, and the artifacts (GitHub repository, prompt templates) support reproducibility. The paper also makes falsifiable predictions (e.g., no dataset directly covers T1071; Initial Access and Impact are underrepresented) that could be tested by future work. However, the current quantitative conclusions rest on two unvalidated choices: LLM labels without ground-truth checking, and hand-picked weights without sensitivity analysis.","major_comments":[{"comment":"Section 3.3, Eq. (2): The Direct/Indirect/None mapping labels are produced entirely by four LLMs reading text dataset profiles; no label is verified against the actual packets, flows, or labels of any dataset. Because Eq. (2) sums these labels with equal credit for Direct and Indirect (c_i = 1 for both), a handful of mislabeled high-priority techniques can materially change the reported coverage. For instance, UNSW-NB15's 82.2% coverage contains 63.8% indirect evidence; if half of those indirect labels are wrong, its score falls toward the 18.4% direct-only figure. The authors should either (a) validate a sample of labels by inspecting actual dataset artifacts, or (b) reframe the percentages as 'LLM-profile-based coverage estimates' and temper the dataset-ranking claims accordingly.","section":null},{"comment":"Section 3.3, Eq. (1): The Priority Score weights (40/40/20 for Entity Usage, Tactic Importance, Group Diversity) are asserted without derivation, calibration, or sensitivity analysis. These weights directly determine the weighted coverage in Eq. (2), so the headline conclusion that UNSW-NB15 is the best dataset is conditional on this hand-picked choice. The paper should provide a sensitivity analysis (e.g., vary the weights over a plausible range and report whether the ranking persists) or derive the weights from an explicit, reproducible model. Without this, the main quantitative claim is not robust.","section":null},{"comment":"Sections 3.2 and 4.1: The 68-technique baseline is itself an LLM consensus output, with 21 of 210 techniques requiring manual adjudication after 2-2 or 2-1-1 splits. No ground truth (e.g., expert labeling or actual attack traces) checks whether the Stage-1 YES/PARTIAL/NO labels are correct. Since the coverage percentages in Section 4.3 are ratios over this baseline, any error in the baseline changes both numerator and denominator. Please report the per-model Stage-1 classifications for all 210 techniques, describe the manual adjudication protocol, and quantify how the 68-technique count and the Section 4.3 coverage scores change when the 75% consensus threshold is varied.","section":null}],"minor_comments":[{"comment":"References: Reference [10] (Hasan et al., 'Retrieval of surface reflectance from NOAA-AVHRR satellite data') appears unrelated to the ATT&CK/NIDS context in which it is cited (Section 1, paragraph 4); please correct or remove this citation.","section":null},{"comment":"Section 3.3: The priority tier definitions (High/Common/Medium/Rare) are introduced but never used in the coverage calculation or results; clarify their role or remove them.","section":null},{"comment":"Section 3.2: The manual adjudication of the 21 Stage-1 disputes is mentioned but not described (who performed it, what criteria were used, what the outcomes were for each disputed technique). Provide this detail for reproducibility.","section":null},{"comment":"Section 4.3: The text reports selected weighted coverage numbers (UNSW-NB15 82.2%, CTU-13, CIC-DDoS2019 10.1%) but does not provide a table with the full breakdown (total, direct, indirect, excluded count) for all five datasets; adding such a table would let readers verify the ranking.","section":null},{"comment":"Section 5: The claim that a 50% consensus threshold 'would expand the detected techniques by 12%' and a unanimous threshold 'would have excluded 35% of the baseline' is not supported by any table or calculation in the paper; either present the underlying data or soften the quantitative claim.","section":null}],"recommendation":"major_revision","confidential_remarks":"The central issue is that the empirical claims are not yet anchored to ground truth. The LLM-consensus approach is a reasonable heuristic for pre-screening coverage, but the authors frame the results as 'coverage' of the datasets themselves. If the authors can add even a small validation sample (e.g., manual inspection of 20-30 techniques across datasets) and a weight-sensitivity analysis, the paper would be much stronger. As is, the paper is more of a framework proposal with illustrative results than a validated measurement. Given the venue (ML for Cyber Security), this may be acceptable as a short paper, but for a journal the lack of validation and sensitivity analysis is a blocking issue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is a useful framework wrapped around unverified numbers. The idea that actually matters — filtering the 210 banking-relevant ATT&CK techniques through NIST SP 800-94 passive-sensor constraints down to 68 network-observable ones before scoring dataset coverage — is sound, and it is a real step beyond the prior energy-sector gap analysis by Tory et al. The banking baseline, the conservative dispute-resolution rules, and the honest reporting of direct (18.4%) versus indirect (63.8%) coverage all work in the paper's favor. The authors also ship prompts and artifacts on GitHub, so the pipeline is reproducible in structure.\n\nThe stress-test note holds up on reading. Every headline number is the output of four LLMs reading text profiles of the datasets; nobody checked any Direct or Indirect label against the actual PCAPs or labels. Consensus reduces inter-model disagreement, not systematic error — all four models read the same profile text and share its blind spots. Equation (2) gives Direct and Indirect the same credit, so UNSW's 82.2% is mostly the 63.8% indirect evidence that was never verified. If half of those indirect calls are wrong, the score drops toward the 18.4% direct-only figure. The Priority Score weights (40/40/20) are asserted, not derived, and there is no sensitivity analysis; a different defensible weighting could plausibly reorder the middle of the pack. I would not call this fatal — the authors flag most of it in the limitations section — but the percentages are illustrative estimates, not measurements.\n\nOne minor thing: refs 10 and 11 (satellite remote sensing, GNSS time synchronization) are off-topic self-citations from the co-author group. That looks like padding and an editor should ask about it.\n\nWho this is for: people who evaluate or procure NIDS datasets and want a structured way to check whether a benchmark reflects their sector's threat model. It deserves a serious referee. My recommendation: send it out, and push in review for either a partial ground-truth validation — verify a dozen Direct labels against actual dataset traffic — or a sensitivity analysis over the priority weights and the 75% threshold. Ideally both.","headline":"Useful framework (NIST-constrained ATT&CK filtering plus a banking baseline) whose headline coverage numbers are unvalidated LLM consensus over dataset text profiles — worth refereeing, but the percentages should be read as estimates.","tokens_in":12358,"tokens_out":6073,"would_cite":true,"duration_ms":49940,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper establishes a priority-weighted, NIST-constrained scoring method for NIDS datasets against banking-relevant MITRE ATT&CK techniques, and finds UNSW-NB15 highest at 82.2% weighted coverage with CIC-DDoS2019 at 10.1%.","keywords":["Network Intrusion Detection System","Banking Security","MITRE ATT&CK","Large Language Models","NIST SP 800-94","Dataset Coverage Evaluation","Consensus Evaluation","Priority Score"],"falsifier":"Recompute all weighted coverage scores under alternative Priority Score weightings, such as equal weights on entity usage, tactic importance, and group diversity, or weights fitted to real banking intrusion reports, and test whether UNSW-NB15 still ranks first and CIC-DDoS2019 still ranks last; the headline result stands only if the ordering survives plausible reweighting.","tokens_in":11320,"feed_emoji":"🏦","tokens_out":13000,"duration_ms":98458,"temperature":0.7,"pith_summary":"The paper develops a sector-aware way to score NIDS benchmark datasets for banking environments, arguing that raw accuracy on generic datasets overstates operational readiness. It filters 210 banking-specific MITRE ATT&CK techniques through NIST SP 800-94 passive-sensor limits (no TLS decryption, no endpoint visibility), keeping 68 network-observable behaviors, and maps those onto five public datasets using a four-LLM consensus engine. The headline result is that UNSW-NB15 has the highest weighted coverage at 82.2%, but only 18.4% of that is direct technique-level evidence, while CIC-DDoS2019 leaves 89.9% of the baseline uncovered. A sympathetic reader should care because a NIDS model can score well on a generic benchmark yet miss the banking-specific behaviors—SWIFT manipulation, ATM jackpotting, credential abuse—that determine whether the system helps in practice.","feed_headline":"UNSW-NB15 tops 82.2% banking coverage; CIC-DDoS2019 reaches 10.1%","feed_subtitle":"Most of that coverage is indirect, so high benchmark scores may not equal banking-ready detection.","key_machinery":"The load-bearing mechanism is a three-stage evaluation pipeline. Stage one extracts banking-relevant ATT&CK techniques from the knowledge base. Stage two applies NIST SP 800-94 passive-sensor constraints, labeling each technique YES, PARTIAL, or NO and retaining the 68 that are network-observable. Stage three maps those 68 techniques to each dataset as Direct, Indirect, or None, using four independently prompted large language models with a 75% agreement threshold and conservative tie-break rules, then scores datasets with the weighted coverage formula $\\frac{\\sum_i w_i c_i}{\\sum_i w_i} \\times 100$ where $w_i$ comes from the hand-set Priority Score $PS = U_e + I_t + D_g$ (weights 40/40/20). The 75% consensus threshold and the Indirect-versus-None exclusion rules are what convert noisy model judgments into a stable coverage number.","core_discovery":"The central claim is that existing public NIDS benchmarks give uneven, mostly indirect coverage of the adversary behaviors that matter in core banking, and that this can be measured in a reproducible way. Under the paper's NIST-constrained definition of passive NIDS visibility, only 68 of 210 banking-relevant ATT&CK techniques are observable in network traffic at all. Mapping those 68 techniques to five benchmarks, UNSW-NB15 is the most useful at 82.2% weighted coverage (18.4% direct, 63.8% indirect), CTU-13 is dominated by indirect coverage (40.4% indirect versus 9.0% direct), and CIC-DDoS2019 covers only 10.1%, an 89.9% blind spot for core banking behaviors. At tactic level, Discovery and Exfiltration are best covered while Initial Access and Impact are systematically underrepresented, and no dataset provides direct coverage of T1071 (Application Layer Protocol), a key command-and-control technique. The paper concludes that high benchmark accuracy can create false assurance and that banking-native datasets with explicit ATT&CK labels are needed.","pith_inferences":["If the Priority Score weights were recalibrated (for example, made equal across entity usage, tactic importance, and group diversity, or fitted to real banking intrusion reports), the dataset ordering might change; the 40/40/20 choice is a modelling decision, not a measured quantity.","Because the underlying LLMs are accessed through non-deterministic consumer interfaces, a second research group rerunning the same prompts could obtain slightly different YES/PARTIAL/NO boundaries or mapping labels; the published dispute rules make the procedure reproducible in structure but not guaranteed in exact output.","The same pipeline could be applied to host-based telemetry (process execution, file and registry changes) to build a complementary endpoint-observable baseline, which would turn the paper's NIDS-only blind spots into quantifiable comparisons across sensor types.","The universal absence of direct T1071 coverage suggests a concrete acceptance test for any future banking-native dataset: include labelled application-layer command-and-control traffic and check whether the coverage score moves from Indirect to Direct."],"forward_implications":["UNSW-NB15 should be read as an anomaly-detection benchmark for banking, not a technique-classification benchmark, because 63.8% of its 82.2% score comes from indirect artifacts that do not identify specific tradecraft such as SWIFT message manipulation or ATM jackpotting.","CIC-DDoS2019 alone cannot validate a banking NIDS: its 10.1% weighted coverage means models trained on it have not seen 89.9% of the network-observable banking attack baseline.","Passive NIDS will systematically under-cover Initial Access and Impact tactics, so closing the validation gap requires host-based telemetry and TLS inspection in addition to better datasets.","The published 68-technique baseline gives dataset builders a concrete checklist: a banking-native benchmark should add explicit SWIFT and ATM traffic with ATT&CK labels for currently missing techniques.","The same NIST-constrained consensus procedure can be rerun as ATT&CK evolves or as new datasets appear, making the coverage audit repeatable rather than a one-time result."],"supporting_citations":[{"why":"Supplies the ATT&CK v18 knowledge base and STIX/JSON repository from which the 210 banking-sector techniques are extracted.","marker":"[15]"},{"why":"Defines the NIST SP 800-94 passive sensor constraints (no TLS decryption, passive-only, no endpoint visibility) that filter the baseline down to 68 network-observable techniques.","marker":"[21]"},{"why":"UNSW-NB15 is the benchmark that achieves the highest 82.2% weighted coverage in the evaluation.","marker":"[17]"},{"why":"CIC-DDoS2019 is the benchmark with the 89.9% coverage blind spot, anchoring the warning about single-vector datasets.","marker":"[22]"},{"why":"Provides the earlier ATT&CK-to-NIDS-dataset evaluation framework and keyword-based sector filtering that this work extends with NIST constraints and a banking focus.","marker":"[19]"},{"why":"The closest prior consensus-based NIDS dataset gap analysis (energy sector) that the paper extends to banking with stricter passive-sensor filtering.","marker":"[24]"},{"why":"Motivates the 75% consensus threshold used to aggregate the four LLM classifications and mappings.","marker":"[18]"},{"why":"Describes the construction of UWF-ZeekData22, one of the five mapped benchmarks, with ATT&CK-aligned labeling.","marker":"[3]"},{"why":"CICIDS2017 is one of the five evaluated benchmarks and contributes to the coverage and dispute statistics.","marker":"[6]"},{"why":"CTU-13 is one of the five evaluated benchmarks; its campaign-level labels drive the 50% disputed-mapping result.","marker":"[9]"}],"fun_headline_variants":["UNSW-NB15 top at 82.2% but only 18.4% direct ATT&CK coverage","Banking NIDS blind spot: CIC-DDoS2019 misses 89.9% of key techniques","Only 68 of 210 banking ATT&CK techniques are even network-visible","CIC-DDoS2019: 89.9% blind on banking attacks","Benchmark scores mislead: banking ATT&CK coverage is mostly indirect"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking rests on the hand-selected 40/40/20 weights in the Priority Score formula; if those weights were changed, the claim that UNSW-NB15 is the best banking-sector benchmark and the size of every coverage gap could change.","fun_headline_variants_meta":{"raw":{"variants":["UNSW-NB15 top at 82.2% but only 18.4% direct ATT&CK coverage","Banking NIDS blind spot: CIC-DDoS2019 misses 89.9% of key techniques","Only 68 of 210 banking ATT&CK techniques are even network-visible","CIC-DDoS2019: 89.9% blind on banking attacks","Benchmark scores mislead: banking ATT&CK coverage is mostly indirect"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001161,"raw_usage":{"total_tokens":4870,"prompt_tokens":1072,"completion_tokens":3798,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":688,"completion_tokens_details":{"reasoning_tokens":3680}},"tokens_in":688,"tokens_out":3798,"duration_ms":24024,"temperature":1.0,"reasoning_tokens":3680,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:14:39.983298+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute all weighted coverage scores under alternative Priority Score weightings, such as equal weights on entity usage, tactic importance, and group diversity, or weights fitted to real banking intrusion reports, and test whether UNSW-NB15 still ranks first and CIC-DDoS2019 still ranks last; the headline result stands only if the ordering survives plausible reweighting.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the ATT&CK v18 knowledge base and STIX/JSON repository from which the 210 banking-sector techniques are extracted."},{"cited_title":"In: 2015 military communications and information systems conference (MilCIS)","cited_arxiv_id":null,"evidence_quote":"UNSW-NB15 is the benchmark that achieves the highest 82.2% weighted coverage in the evaluation."},{"cited_title":"In: 2019 Inter - national Carnahan Conference on Security Technology (ICCST)","cited_arxiv_id":null,"evidence_quote":"CIC-DDoS2019 is the benchmark with the 89.9% coverage blind spot, anchoring the warning about single-vector datasets."},{"cited_title":"Computers & Security 153, 104777 (2025), https://doi.org/10.1016/j.cose.2025.104777","cited_arxiv_id":null,"evidence_quote":"Provides the earlier ATT&CK-to-NIDS-dataset evaluation framework and keyword-based sector filtering that this work extends with NIST constraints and a banking focus."},{"cited_title":"arXiv preprint arXiv:2511.00360 (2025), https://doi.org/10.48550/arXiv.2511.00360","cited_arxiv_id":null,"evidence_quote":"The closest prior consensus-based NIDS dataset gap analysis (energy sector) that the paper extends to banking with stricter passive-sensor filtering."},{"cited_title":"Data 8 (1), 18 (2023), https://www.mdpi.com/ 2306-5729/8/1/18","cited_arxiv_id":null,"evidence_quote":"Describes the construction of UWF-ZeekData22, one of the five mapped benchmarks, with ATT&CK-aligned labeling."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CICIDS2017 is one of the five evaluated benchmarks and contributes to the coverage and dispute statistics."},{"cited_title":"Computers & Security 45, 100–123 (2014)","cited_arxiv_id":null,"evidence_quote":"CTU-13 is one of the five evaluated benchmarks; its campaign-level labels drive the 50% disputed-mapping result."}],"review_version":2}