{"id":"867651fb-56c3-4b3c-81be-40f363722bf2","arxiv_id":"2505.08006","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Feature Alignment Precision, Recall, and F1 compare SHAP explanations from intrusion detection models against MITRE/D3FEND-derived feature sets, showing deep models align better than random forest on CICIDS2017.","lead":"This paper proposes three new scores that check whether the features an AI intrusion detection system highlights match the features security experts say matter for each attack. It applies them to three models on a public network traffic dataset and finds that deep learning models align better with domain knowledge than a random forest.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The empirical claim rests on the unpublished, unvalidated domain-informed feature sets F_c; if those mappings are wrong or unreproducible, every FAP/FAR/FAF1 value in the evaluation measures arbitrary overlap rather than explanation quality.","rationale":"The reader's weakest_assumption is exactly the load-bearing point: F_c is the ground truth against which explanation quality is measured, and it is neither published nor validated. I see no reason to move the verdict: the metric definitions are formally clear and could be useful if paired with a credible reference set, so CONDITIONAL is appropriate. The paper's own admissions about bot and infiltration classes and about the need for fine-tuning support the concern rather than defusing it. The concrete test of releasing the mapping and measuring inter-rater agreement would settle whether the empirical demonstration survives.","tokens_in":12807,"tokens_out":4113,"duration_ms":43291,"concrete_test":"Publish the complete F_c mapping for every attack class: attack class to MITRE ATT&CK technique IDs, technique to D3FEND detect tactic IDs, and tactic to exact CICIDS2017 feature names. Then ask two independent security analysts, blinded to the authors' mapping, to reconstruct F_c from the same knowledge-base version and dataset feature list, and compute inter-annotator agreement (e.g., Fleiss' kappa or per-class Jaccard overlap). If agreement is low (κ < 0.6) or the mapping cannot be reconstructed from the cited sources, Table 1 and Figures 2-4 cannot be interpreted as measuring domain alignment; if agreement is high, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 states F_c were constructed by mapping each CICIDS2017 attack class to MITRE ATT&CK techniques, then to D3FEND detect tactics, then to dataset feature names, but no mapping table, feature list, or validation is given. ATT&CK/D3FEND describe high-level adversarial behaviors and defensive countermeasures, while CICIDS2017 features are low-level flow statistics; the intermediate link is underdetermined. The authors implicitly concede fragility in Section 4.2, where bot and infiltration classes have empty predefined feature sets, and in Section 4.3, where they note the sets 'may require additional fine-tuning.' If F_c is incomplete or misaligned with actual feature names, then FAP is just the fraction of top-k features that happen to be in an arbitrary list, and FAR is undefined or zero for empty sets. The central claim that the metrics 'effectively distinguish explanation quality across X-IDSs' is therefore not supported by the reported experiments until F_c is released and shown to be reproducible. This is not a flaw in the metric formulas themselves; it is a construct-validity failure in the empirical demonstration.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper defines three metrics—Feature Alignment Precision (FAP), Feature Alignment Recall (FAR), and Feature Alignment F1 (FAF1)—that compare the top-k features of a post hoc explanation (SHAP) for an X-IDS against class-specific domain-informed feature sets F_c derived from MITRE ATT&CK and D3FEND. The metrics are defined at instance, class, and dataset levels (Eqs. 1–9) and evaluated on CICIDS2017 with Random Forest, DNN, and CNN-BiLSTM classifiers. The authors report that the deep models' explanations align better with their reference sets, especially at small k, and argue the metrics can select among X-IDSs and guide the choice of the top-k cutoff.","tokens_in":12967,"tokens_out":5148,"duration_ms":50655,"significance":"If the empirical demonstration is sound, the metrics provide a simple, parameter-free, domain-grounded scorecard for explanation content, filling a real gap because most XAI evaluation focuses on fidelity and simplicity rather than domain relevance. The formal definitions are clean, the three-level aggregation is useful, and the metrics involve no fitted constants, with the reference sets external to the models, which mitigates circularity concerns. The central risk is construct validity: every reported score inherits the quality of the manually constructed F_c, and the current paper neither releases nor independently validates these sets, so the load-bearing empirical claims remain unsupported until that is addressed.","major_comments":[{"comment":"The empirical claims in Table 1 and Figures 2–4 rest entirely on the class-specific feature sets F_c, but the paper does not provide the mapping table, the resulting feature lists, or any validation that they align with CICIDS2017's feature names. The authors state that the sets were derived by mapping each attack class to MITRE ATT&CK techniques, then to D3FEND detect tactics, then to dataset feature names, but the intermediate link is underdetermined and the mappings are not shown. This is load-bearing because FAP, FAR, and FAF1 are defined as overlap with these sets; if the sets are wrong, incomplete, or unreproducible, the reported scores measure overlap with an arbitrary list. The authors' own admission in Section 4.3 that the sets 'may require additional fine-tuning' and that bot and infiltration classes have empty sets confirms the fragility. Please release the full mapping, justify each feature selection against D3FEND artifacts, and show sensitivity of the results to alternative plausible mappings.","section":"Section 4.1 (F_c construction)"},{"comment":"The class-level and dataset-level metrics average only over test instances that are 'correctly predicted as the class c', while the text in Section 3.2 earlier defines X_test_c as test instances that share the same label c. This inconsistency changes the evaluation population across models with different accuracy and excludes all false predictions, so comparisons such as 'DNN vs RF at k=5' may reflect differences in which instances survive the correctness filter rather than differences in explanation alignment. Moreover, explanations for incorrect predictions are relevant in practice and are simply dropped. Please define a single fixed evaluation population (for example, all test instances of class c, or separate correct/incorrect analyses) and report accuracies so the reader can assess the population overlap for each model and class.","section":"Section 3.2, Eqs. (2), (5), (8)"},{"comment":"For bot and infiltration, the paper states that the predefined domain-informed feature set contains no features, but the instance-level FAR in Eq. (4) divides by |F_c|, which is then zero, and the FAF1 in Eq. (7) degenerates. Reporting these classes as zero FAR and including them in the dataset-level averages in Eqs. (3), (6), and (9) silently mixes undefined or degenerate values into the aggregate scores and makes the dataset-level numbers class-composition dependent. Please specify how empty reference sets are handled (exclude the class, mark as not applicable, or use a separate convention) and recompute all aggregates accordingly.","section":"Section 4.2 and Eq. (4)"},{"comment":"The claim that the metrics 'effectively differentiate between X-IDSs' is supported only by descriptive point estimates; no confidence intervals, significance tests, or random baseline (for example, overlap of top-k features with a random or frequency-matched feature set) are reported. At several k values the differences between DNN and CNN-BiLSTM are small (for example, FAF1 0.24 vs 0.16 at k=10 and FAP 0.17 vs 0.20 at k=40), so without uncertainty quantification the differentiation claim is overstated. Please add per-class error bars or bootstrapped intervals and a baseline to show that the metrics separate signal from chance.","section":"Section 4.3 and Table 1"}],"minor_comments":[{"comment":"In the caption of Figure 4(b), 'FPR' should be 'FAR', since the text and axis labels refer to the FAP–FAR trade-off.","section":"Figure 4(caption)"},{"comment":"There are typos: 'DDoD/DoS' for 'DDoS/DoS' in Section 4.2, and 'T op-k' in the Table 1 header.","section":"Section 4.2 and Table 1"},{"comment":"The sentence 'A high FAP means that most of the features selected by the model belong to the set of domain-informed features Ex' uses the wrong symbol: it should be F_c, not Ex.","section":"Section 3.3"},{"comment":"The averaging notation in Eqs. (3), (6), and (9) should make explicit that the sum runs only over classes with non-empty F_c and with at least one evaluated instance; otherwise the dataset-level aggregates are undefined when such classes are included.","section":"Eq. (3) and related notation"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable incremental contribution, but the empirical demonstration is the entire added value beyond the straightforward set-overlap formulas. The weakest link is the undocumented manual construction of F_c, which must be addressed before the paper can be accepted. The novelty relative to existing feature-overlap evaluations such as [3] and [10] should also be sharpened, since the paper's claim of novelty largely rests on the domain-knowledge grounding of the reference sets."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper defines FAP, FAR, and FAF1, which are precision, recall, and F1 computed over the top-k explanation features versus a reference set. That's not new math—the formulas are textbook set overlap. What's slightly fresh is the packaging: using MITRE ATT&CK and D3FEND to build the reference sets, so the metrics measure alignment with domain knowledge rather than just generic faithfulness. For a security team choosing among X-IDS models, that's a reasonable scorecard idea.\n\nCredit where it's due: the formal definitions are clean, the three evaluation levels (instance, class, dataset) are sensible, and the paper is unusually honest about its own limitations. It explicitly says bot and infiltration have empty reference sets, admits the feature sets may require fine-tuning, and flags that arithmetic mean aggregation is outlier-sensitive. That transparency suggests the authors know where their weak points are.\n\nThe soft spots are real and load-bearing. The biggest one: the domain-informed feature sets F_c are never released. There is no mapping table from ATT&CK techniques to D3FEND tactics to CICIDS2017 feature names. The whole empirical evaluation depends on these hand-built sets, and without them every FAP/FAR/FAF1 value in Table 1 and Figures 2–4 is unverifiable. The stress-test note is right that the link from high-level attack behaviors to low-level flow statistics is underdetermined. The paper implicitly concedes this in Section 4.3, but the concession doesn't fix the absence of the artifact.\n\nNext issue: class- and dataset-level metrics are averaged only over correctly predicted instances (Eqs. 2, 5, 8). If models have different accuracy, you are comparing averages over different populations. That's a genuine confound. Also, FAR for empty F_c is undefined, yet reported as zero; the paper never states the convention. And there are no error bars or repeated runs, so the observed DNN-vs-RF differences could be noise.\n\nThese are fixable. Publish the feature sets, give the mapping, compute metrics on all instances or justify the restriction, handle empty sets explicitly, and add variance estimates. With those changes, this becomes a legitimate incremental contribution to XAI evaluation for IDS.\n\nMy recommendation: send it to peer review, but mark it as major revision. The idea is useful, the writing is clear, and the authors seem honest. A serious referee could push them to make the reference sets available and clean up the evaluation. As it stands, the empirical claim that the metrics 'effectively distinguish' models is not supported—not because the metrics are wrong, but because the reference sets are absent.","headline":"Three set-based alignment metrics, cleanly defined but not new; the useful part is the domain-reference idea, and the empirical claims need the reference sets published before they can be checked.","tokens_in":13576,"tokens_out":1856,"would_cite":false,"duration_ms":20451,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"New metrics grade intrusion-detection explanations by domain knowledge.","keywords":["explainable AI","intrusion detection","explanation evaluation","feature alignment metrics","SHAP","domain-informed features","MITRE ATT&CK","CICIDS2017"],"falsifier":"Replace each $F_c$ with a randomly drawn feature set of the same size and recompute FAP/FAR/FAF1 for the three models; if the same model ordering (deep models above Random Forest) persists under random references, the metrics are capturing model behaviour rather than domain alignment. A more targeted check: the paper's own results show bot and infiltration classes have empty $F_c$ and hence zero FAR at every k; hand-construct reference sets for those two classes and see whether the reported rankings and conclusions change.","tokens_in":12543,"feed_emoji":"🛡️","tokens_out":5676,"duration_ms":49927,"temperature":0.7,"pith_summary":"The paper proposes three metrics—Feature Alignment Precision (FAP), Feature Alignment Recall (FAR), and Feature Alignment F1 (FAF1)—that score how well an intrusion-detection system's explanation lines up with the features that security domain knowledge says should matter for each attack. The metrics compare the top-k features a post-hoc explainer like SHAP highlights against predefined domain-informed feature sets derived from the ATT&CK and D3FEND knowledge bases. Applied to three models on the CICIDS2017 dataset, the metrics separate the models: the deep models produce explanations in closer agreement with domain knowledge than Random Forest, especially at small k. If the metrics are right, security analysts get an actionable, quantitative way to choose among explainable IDSs and to decide how many features to show.","feed_headline":"Three metrics grade intrusion-detection explanations by domain knowledge","feed_subtitle":"FAP, FAR, and FAF1 compare an explainer's top features to security indicators, ranking models by relevance.","key_machinery":"The load-bearing object is the domain-informed feature set $F_c$ for each attack class, manually constructed by mapping each CICIDS2017 attack to ATT&CK technique(s), then to D3FEND detect tactics, with enrichment from contextual resources. The metrics are set-containment ratios: FAP = $\\frac{|E_x(k) \\cap F_c|}{|E_x(k)|}$, FAR = $\\frac{|E_x(k) \\cap F_c|}{|F_c|}$, and FAF1 the harmonic mean of the two, where $E_x(k)$ is the explainer's ordered top-k features. The reference set $F_c$ is what turns a generic \"top features\" list into a domain-grounded score; its quality determines the validity of everything downstream.","core_discovery":"The central claim is that explanation quality in an X-IDS can be quantified as alignment between the explainer's top-k features and a reference set of domain-informed features. For each attack class $c$, the paper defines $F_c$, a set of features derived from mapping the attack to ATT&CK techniques and D3FEND detect tactics. FAP then measures the fraction of the top-k features that fall in $F_c$, FAR measures the fraction of $F_c$ covered by the top-k, and FAF1 is their harmonic mean; each metric is computed at the instance, class, and dataset level. Evaluated on CICIDS2017 with SHAP explanations from Random Forest, DNN, and CNN-BiLSTM models, the metrics show that the deep models align with the domain reference much better than Random Forest at small k, and that classes differ widely, with bot and infiltration showing zero alignment because their reference sets are empty. The paper argues these scores give analysts a way to pick models and to set the top-k cutoff.","pith_inferences":["One could validate the metrics by user studies: ask analysts whether explanations with higher FAF1 actually speed up accurate triage; the paper does not run this test.","FAP and FAR over k resemble a precision-recall curve, so an area-under-curve summary could rank models with a single number; the paper stops at per-k values.","The empty bot and infiltration sets suggest the construction of $F_c$ is itself a scientific deliverable; the metrics could be inverted to score knowledge-base completeness rather than model quality.","Because the reference sets are hand-built, inter-annotator agreement on $F_c$ would be a natural reliability check the paper does not report."],"forward_implications":["Analysts can compare competing explainable IDS models on a domain-grounded scale rather than on fidelity or simplicity alone.","The trade-off curves identify a recommended k: the point where FAF1 peaks, or where FAP and FAR cross, tells how many features to show an analyst.","Consistently low scores for an attack class flag that the domain-informed reference set, not only the model, needs revision.","The same metric definitions apply to other datasets and other post-hoc explainers, since they depend only on top-k lists and reference sets."],"supporting_citations":[{"why":"Supplies the Random Forest and DNN models and the approach of mapping CICIDS2017 attack classes to ATT&CK techniques.","marker":"[20]"},{"why":"Provides the CNN-BiLSTM architecture used as the third X-IDS in the evaluation.","marker":"[21]"},{"why":"Defines SHAP, the post-hoc explanation method whose top-k features the metrics evaluate.","marker":"[22]"},{"why":"Introduces the CICIDS2017 dataset on which all models are trained and evaluated.","marker":"[19]"},{"why":"Is the ATT&CK knowledge base used to map each attack class to offensive techniques.","marker":"[24]"},{"why":"Introduces D3FEND, the defensive-countermeasure knowledge base used to derive the detect-tactic features.","marker":"[13]"},{"why":"Closest prior evaluation, comparing XAI features against a ground-truth set with a precision-recall-like score; the paper extends it by separating correctness and completeness.","marker":"[10]"}],"fun_headline_variants":["Domain-aligned metrics grade X-IDS explanation quality","Feature-alignment metrics benchmark X-IDS explainers","New X-IDS metrics score explanations against domain knowledge","FAP, FAR, FAF1: domain-grounded X-IDS explanation metrics","Scoring intrusion-detection explanations with domain-aligned metrics"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that the manually built feature sets $F_c$ accurately and completely capture the features that should matter for each attack class in the dataset; if those sets are wrong or incomplete, every FAP, FAR, and FAF1 score measures noise rather than explanation quality.","fun_headline_variants_meta":{"raw":{"variants":["Domain-aligned metrics grade X-IDS explanation quality","Feature-alignment metrics benchmark X-IDS explainers","New X-IDS metrics score explanations against domain knowledge","FAP, FAR, FAF1: domain-grounded X-IDS explanation metrics","Scoring intrusion-detection explanations with domain-aligned metrics"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000653,"raw_usage":{"total_tokens":2998,"prompt_tokens":953,"completion_tokens":2045,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":569,"completion_tokens_details":{"reasoning_tokens":1961}},"tokens_in":569,"tokens_out":2045,"duration_ms":16011,"temperature":1.0,"reasoning_tokens":1961,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:05:48.862613+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace each $F_c$ with a randomly drawn feature set of the same size and recompute FAP/FAR/FAF1 for the three models; if the same model ordering (deep models above Random Forest) persists under random references, the metrics are capturing model behaviour rather than domain alignment. A more targeted check: the paper's own results show bot and infiltration classes have empty $F_c$ and hence zero FAR at every k; hand-construct reference sets for those two classes and see whether the reported rankings and conclusions change.","supporting_citations":[{"cited_title":"Applied Sciences 14(10), 4170 (2024)","cited_arxiv_id":null,"evidence_quote":"Supplies the Random Forest and DNN models and the approach of mapping CICIDS2017 attack classes to ATT&CK techniques."},{"cited_title":"In:2020 3rd International Conference on Artificial Intelligence and Pattern Recognition, pp","cited_arxiv_id":null,"evidence_quote":"Provides the CNN-BiLSTM architecture used as the third X-IDS in the evaluation."},{"cited_title":"In: Advances in Neural Information Processing Systems, vol","cited_arxiv_id":null,"evidence_quote":"Defines SHAP, the post-hoc explanation method whose top-k features the metrics evaluate."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the CICIDS2017 dataset on which all models are trained and evaluated."},{"cited_title":"https://attack.mitre.org/, last accessed 2025/04/19 Evaluating Explanation Quality in X-IDS 19","cited_arxiv_id":null,"evidence_quote":"Is the ATT&CK knowledge base used to map each attack class to offensive techniques."},{"cited_title":"The MITRE Corporation, vol","cited_arxiv_id":null,"evidence_quote":"Introduces D3FEND, the defensive-countermeasure knowledge base used to derive the detect-tactic features."},{"cited_title":"In: Proceed- ings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining (KDD 2021), pp","cited_arxiv_id":null,"evidence_quote":"Closest prior evaluation, comparing XAI features against a ground-truth set with a precision-recall-like score; the paper extends it by separating correctness and completeness."}],"review_version":1}