{"id":"615df8e8-2640-47b8-a01e-7df680e87bc1","arxiv_id":"2412.02084","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"EBM and XGBoost achieve comparable phishing detection accuracy across 12 datasets, with EBM showing advantages in explanation stability, accuracy, and actionability based on qualitative SHAP analysis.","lead":"This paper compares a white-box model (EBM) with a black-box model (XGBoost) for phishing detection on 12 public datasets. It finds similar predictive accuracy, while EBM provides clearer and more actionable explanations.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 4's visual SHAP-based ratings are not reproducible; the claimed EBM advantage in stability, explanation accuracy, and actionability needs a quantitative test.","rationale":"The reader's weakest assumption—that the explainability ratings are based on subjective visual interpretation of SHAP plots—is exactly the load-bearing weakness of the paper. The predictive accuracy tables are straightforward, but the comparative claim about explainability depends on Table 4, which is not derived from any quantitative metric. The paper even contains an internal inconsistency: Table 3 rates both models 'High' on stability, while Section V claims EBM is more stable. This supports the reader's CONDITIONAL verdict: the claim may hold, but it cannot be accepted as stated. No change to the verdict is needed; it should remain CONDITIONAL pending a reproducible quantitative evaluation.","tokens_in":13226,"tokens_out":2854,"duration_ms":30472,"concrete_test":"On ds_129K112 and ds_100K20, compute SHAP values for EBM and XGBoost on a fixed test set. Quantify stability as the mean absolute SHAP-value change after adding Gaussian noise to each instance (10 repeats); quantify explanation accuracy as the rank correlation between SHAP feature importance and leave-one-out accuracy drops; quantify actionability as the Jaccard overlap of the top-5 SHAP features with known phishing-indicative features (e.g., URL length, domain age). Compare models and test differences with a paired Wilcoxon test. If EBM does not significantly beat XGBoost on stability and explanation accuracy, Table 4's ratings and the conclusion are unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central conclusion ('EBM outperforms XGBoost in terms of actionability, explanation accuracy, and stability') rests on Table 4, which rates models based on the authors' visual interpretation of SHAP value distributions, guided only by ad hoc statements in Section IV.B (e.g., 'narrow spreads indicate high stability'). No thresholds, blinding, inter-rater reliability, or quantitative operationalization is provided, and no code or dataset links are given. Section III.B promises statistical tests to validate observed differences, but none are reported. Moreover, Table 3 already rates both models as 'High (except Moderate for noisy datasets)' on stability, directly conflicting with the later claim that EBM is more stable. Thus, the paper's primary comparative claim about explainability is unsupported by a reproducible, quantitative procedure.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a comparative analysis of black-box (XGBoost) and white-box (Explainable Boosting Machine, EBM) models for phishing detection. The study evaluates both models on 12 public datasets in terms of predictive accuracy (accuracy, precision, recall, FP rate, ROC AUC, runtime) and explainability metrics (fidelity, simplicity, comprehensiveness, consistency, stability, accuracy of explanations, interpretability, actionability). The predictive performance comparison is based on straightforward experiments. The explainability comparison, however, relies on the authors' qualitative interpretation of SHAP value distributions, guided by ad hoc criteria listed in Section IV.B. The paper concludes that EBM outperforms XGBoost in actionability, explanation accuracy, and stability, and is generally better suited for applications requiring explainability and actionable insights. The abstract and conclusions also note that both models are comparable in interpretability and consistency, with room for improvement on diverse datasets.","tokens_in":13350,"tokens_out":2969,"duration_ms":29284,"significance":"If the central conclusion were supported, the paper would provide practical guidance for practitioners choosing between a black-box and a white-box model for phishing detection, where explainability is often a critical requirement. The study covers a wide range of public datasets, which is a strength, and the accuracy results are transparently reported in tabular form. The machine-checked nature of the accuracy experiments (though without code or data links) gives some confidence in the performance comparisons. However, the paper's main contribution — the explainability advantage of EBM — is not established by the presented evidence, because the ratings in Table 4 are subjective visual assessments. The paper also promises statistical tests that are not reported, and it contains internal inconsistencies in the stability ratings. These issues undermine the paper's central claim and, as currently written, the paper does not meet the standards of a rigorous comparative study in explainable AI.","major_comments":[{"comment":"The central claim that 'EBM outperforms XGBoost in terms of actionability, explanation accuracy, and stability' rests entirely on Table 4, which rates the models based on the authors' visual interpretation of SHAP value distributions following the ad hoc guidance in points (a)–(e) of Section IV.B. No thresholds, blinded evaluation, inter-rater reliability, or quantitative operationalization is provided. For instance, 'narrow spreads indicate high stability' is not a measurable criterion, and the same guidance could plausibly lead different analysts to different ratings. To support the claim, the authors need to provide quantitative metrics—for example, the variance of SHAP values across perturbed inputs for stability, cosine similarity or rank correlation between explanations for consistency, and feature-ranking agreement with domain knowledge for explanation accuracy—or alternatively they must explicitly reframe the comparisons as qualitative observations and temper the conclusions accordingly.","section":"Section IV.B / Table 4"},{"comment":"Section III.B promises that 'statistical tests will validate the significance of observed differences between models' and that 'robustness analysis will be performed,' but no statistical tests, confidence intervals, or error bars are reported anywhere in Section IV. The accuracy tables (Tables 1 and 2) present single-point estimates without variance, so the reader cannot determine whether the small differences between XGBoost and EBM (e.g., 89.68% vs 88.97% on ds_100K20) are meaningful. The paper should either include paired statistical tests across the 12 datasets (e.g., Wilcoxon signed-rank test) or explicitly state that no such tests were conducted and adjust the strength of the comparative claims.","section":"Section III.B"},{"comment":"There is an internal inconsistency in the stability assessment. Table 3 rates both EBM and XGBoost as 'High (except Moderate for noisy datasets)' on the stability dimension, but Section V and the Conclusions state that EBM is better than XGBoost in stability, with Section IV.B's Table 4 rating EBM as 'Moderate to High' and XGBoost as 'Moderate.' The authors need to reconcile these two assessments; as written, the quantitative-looking Table 3 directly contradicts the qualitative conclusion, and the reader cannot tell which evidence is authoritative.","section":"Table 3 vs. Section V/Conclusions"},{"comment":"The experiments are described without sufficient detail for reproduction: no code is provided, no hyperparameter settings are given (only 'optimized hyperparameters' is mentioned in Section III.B), and the specific dataset sources and preprocessing steps are not described. Dataset names such as 'ds_100K20' and 'ds_600K11_rev.csv' are not mapped to publicly identifiable resources, and the paper states only that datasets were 'taken from Kaggle.com and data.mendeley.com.' Without a reproducibility statement, the accuracy numbers cannot be verified, and the runtime comparisons are meaningless without hardware and software environment details. The authors should provide a data availability statement and, at minimum, an appendix with hyperparameters and the exact train-validation-test split methodology.","section":"Reproducibility"}],"minor_comments":[{"comment":"Figure 2's caption reads 'EDM and XGB in ds_11055 Explanation Plot' but the text refers to ds_10K18; Figure 5's caption says 'EBM and XGB in ds_600K11 Explanation Plot' while the text refers to ds_11K89. The captions should be corrected, and 'EDM' should be 'EBM.'","section":"Figure captions"},{"comment":"The abstract states 'Experimental metrics covers 3 measurements such as predictive accuracy and explainability metrics,' but only two categories are listed; the sentence should be clarified to indicate exactly what the three measurements are.","section":"Abstract"},{"comment":"The article history contains placeholder dates ('Received 5 April 20XX') that should be removed or filled, and the keywords contain typos ('Detction', 'Analisys', 'Explaiability') that must be corrected.","section":"Article history and keywords"},{"comment":"Several references are incomplete or malformed, including entries with 'undefined' as the author (e.g., [16], [23]) and missing page numbers or venue details. The reference list should be checked against a standard format.","section":"References"},{"comment":"The explainability metrics defined in Section II.C (fidelity, simplicity, comprehensiveness, etc.) are not operationalized in Section IV.B; for example, the 'Accuracy of Explanation' metric is assessed by 'validating feature ranking against domain knowledge' without any systematic scoring procedure. The authors should either define concrete measures for each metric or clearly label the assessments as subjective.","section":"Section II.C"}],"recommendation":"major_revision","confidential_remarks":"The paper is currently not ready for publication because its central claim about explainability is based on subjective visual ratings rather than a reproducible quantitative methodology. The accuracy comparison is simple and might be publishable as a benchmark, but the paper's contribution is framed as an explainability comparison. I would consider a revised version that (1) adds quantitative explainability metrics, (2) reports statistical tests or disclaims their absence, (3) resolves the Table 3/Table 4 stability contradiction, and (4) includes a reproducibility statement. If these issues are not addressed, the contribution would fall below the threshold for a serious journal. The scope is appropriate for a cybersecurity or applied ML venue, but the current presentation is more of a draft than a finished manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe genuinely new piece here is the accuracy benchmark: XGBoost and EBM run on twelve public phishing datasets, with accuracy, precision, recall, FPR, AUC, and runtime. That table is useful if you need a quick sense of how these two models compare on realistic data. The rest of the paper is mostly a qualitative literature summary plus a conclusion about explainability that does not have real evidence behind it.\n\nThe paper's only original explanatory contribution is Table 4, which rates EBM and XGBoost on stability, consistency, explanation accuracy, interpretability, and actionability. Those ratings come from the authors looking at SHAP-value distributions and applying ad hoc rules like 'narrow spreads indicate high stability.' There are no thresholds, no blinded rating, no inter-rater reliability, no code, and no dataset links, so nobody can reproduce or verify the ratings. Section III.B promises statistical tests; none are reported. The paper also contains a suspicious inconsistency: Table 3 rates both models High on stability, then Table 4 says EBM is better. It may be a matter of different metrics, but the paper never explains the distinction.\n\nTo be fair, the predictive accuracy results are straightforward and plausible. EBM usually trails XGBoost slightly in accuracy, which is expected for a more interpretable model. The runtime figures are striking—EBM takes ~3,500 seconds on one dataset where XGBoost takes 22 seconds—but there's no discussion of variance or experimental setup. The writing is rough, with typos, inconsistent table numbering, and at least one mismatched figure caption. These issues don't invalidate the benchmark, but they make the paper feel unfinished.\n\nThe central claim—that EBM is better for actionability, explanation accuracy, and stability—is unsupported as presented. The empirical benchmark is real, though. If the authors could add a quantitative explainability evaluation, release code and data, and run the promised statistical tests, the paper would be worth reading. As it stands, it's a useful data point with an overreach.\n\nMy take: send it to peer review if you want to force the authors to do the quantitative work, but don't expect the current version to survive. I'd cite the accuracy table if I needed it, but not the explainability conclusion. For a reading group, it's a decent example of why visual inspection of XAI plots is not a measurement.","headline":"Useful accuracy benchmark of XGBoost vs EBM on 12 phishing datasets, but the explainability conclusion rests on unreproducible visual inspection of SHAP plots.","tokens_in":13862,"tokens_out":3160,"would_cite":false,"duration_ms":29901,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that an Explainable Boosting Machine matches XGBoost's phishing detection accuracy while producing more stable, accurate, and actionable explanations.","keywords":["phishing detection","explainable AI","white-box model","black-box model","Explainable Boosting Machine","XGBoost","SHAP","model interpretability"],"falsifier":"One concrete test is to train both models multiple times with different random seeds on the same twelve datasets, then measure the variance of SHAP values across seeds and across small input perturbations; if XGBoost's variance is not larger than EBM's, the claimed stability advantage fails. A user study where security analysts are shown EBM and XGBoost explanations and asked to identify the phishing-triggering feature would likewise settle whether the claimed actionability difference is real.","tokens_in":13030,"feed_emoji":"🎣","tokens_out":5522,"duration_ms":53692,"temperature":0.7,"pith_summary":"The paper aims to settle which type of machine learning model, a black-box gradient-boosted tree model or a white-box explainable boosting machine, should be used for phishing detection when explanations matter. After running both models on twelve public phishing datasets, it finds that XGBoost and EBM achieve nearly matching accuracy, precision, recall, false-positive rate, and ROC AUC. The difference appears in explainability: the paper claims EBM delivers more stable, more accurate, and more actionable feature explanations than XGBoost, while the two are comparable on consistency and interpretability. The practical conclusion is that organizations needing transparent, actionable phishing alerts should prefer EBM, and that model choice should depend on whether the goal is raw performance or human-understandable insight.","feed_headline":"White-box EBM tops XGBoost for actionable phishing alerts","feed_subtitle":"The interpretable model matches black-box accuracy while giving more stable and useful explanations.","key_machinery":"The central object is the Explainable Boosting Machine (EBM), a white-box model whose prediction is a sum of one-dimensional shape functions over individual features plus optional pairwise interactions, so each feature's contribution to a phishing score can be read directly. The comparison mechanism is SHAP: for both EBM and XGBoost, the paper visualizes SHAP value distributions on selected datasets and interprets their spread, direction, and ordering using a five-part rubric covering stability, consistency, accuracy of explanation, interpretability, and actionability. That rubric, applied to the plots, carries the claim that EBM explains better than XGBoost.","core_discovery":"The paper's central claim is that in phishing detection, a white-box Explainable Boosting Machine can match the predictive performance of XGBoost, a black-box gradient boosting model, while providing better explanations for its decisions. Across twelve public datasets, both models scored close on accuracy, precision, recall, false-positive rate, and ROC AUC, with EBM roughly 80 to 100 percent accuracy and XGBoost roughly 82 to 100 percent. The paper then rates explainability by inspecting SHAP value distributions and concludes that EBM's feature contributions are more stable, its explanations align better with domain knowledge, and its top features support more actionable security decisions. The recommendation is that applications needing explainability and actionable insight are better served by EBM.","pith_inferences":["A direct user study, in which security analysts or end users act on EBM versus XGBoost explanations, would test the actionability claim more rigorously than the paper's visual SHAP inspection, which is only a proxy.","Because both models are explained with SHAP, the measured differences in explanation stability and accuracy may partly reflect how SHAP attributes additive models versus tree ensembles; comparing multiple attribution methods would separate model effects from explanation-method effects.","The near-parity in accuracy suggests that interpretable models should be considered a strong default for phishing detection rather than a fallback, although the reported runtimes indicate that EBM's computational cost on very large datasets is a real constraint."],"forward_implications":["Phishing detection systems can adopt EBM without sacrificing much detection accuracy, since EBM is reported to match XGBoost on accuracy, precision, recall, false-positive rate, and AUC across all twelve datasets.","Security teams using EBM should be able to see which individual URL or content features pushed a site toward a phishing verdict, and turn that directly into a blocking rule or user warning.","XGBoost retains an advantage in scalability and training runtime, particularly on very large datasets where EBM's runtime grows sharply, such as the 129-thousand-instance and 662-thousand-instance datasets.","The paper's conclusion implies that model choice should be driven by deployment objective: EBM when explainability and actionability matter most, XGBoost when raw scalability and flexibility dominate."],"supporting_citations":[{"why":"Supplies the Explainable Boosting Machine implementation used as the white-box model in the experiments.","marker":"[23]"},{"why":"Provides the LIME method and the general framing of explaining any classifier, part of the explanation toolkit the paper maps to black-box models.","marker":"[6]"},{"why":"Supplies the fidelity metric and the idea of faithful explanations used in the quantitative interpretability comparison.","marker":"[31]"},{"why":"Defines actionability as a metric and frames the phishing-research context that supports the paper's recommendation of EBM for actionable insights.","marker":"[35]"},{"why":"Provides the interpretable machine learning reference used for the white-box explainability concepts and metric definitions.","marker":"[42]"},{"why":"Frames the black-box versus white-box explainability comparison within computer security applications, which motivates the paper's analytical review.","marker":"[5]"}],"fun_headline_variants":["Interpretable EBM matches XGBoost accuracy in phishing","White-box EBM offers clearer phishing alerts than XGBoost","Phishing detection: EBM rivals XGBoost with better explanations","For phishing, a white-box model matches black-box performance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that visually inspecting SHAP value distributions using the paper's ad hoc rating guidance is a valid and objective measure of explanation stability, consistency, accuracy, interpretability, and actionability; if that measurement is not valid, the claim that EBM explains better than XGBoost is unsupported.","fun_headline_variants_meta":{"raw":{"variants":["Interpretable EBM matches XGBoost accuracy in phishing","White-box EBM offers clearer phishing alerts than XGBoost","Phishing detection: EBM rivals XGBoost with better explanations","For phishing, a white-box model matches black-box performance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000381,"raw_usage":{"total_tokens":1981,"prompt_tokens":861,"completion_tokens":1120,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":477,"completion_tokens_details":{"reasoning_tokens":1047}},"tokens_in":477,"tokens_out":1120,"duration_ms":8190,"temperature":1.0,"reasoning_tokens":1047,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:50:06.494948+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"One concrete test is to train both models multiple times with different random seeds on the same twelve datasets, then measure the variance of SHAP values across seeds and across small input perturbations; if XGBoost's variance is not larger than EBM's, the claimed stability advantage fails. A user study where security analysts are shown EBM and XGBoost explanations and asked to identify the phishing-triggering feature would likewise settle whether the claimed actionability difference is real.","supporting_citations":[{"cited_title":"InterpretML","cited_arxiv_id":null,"evidence_quote":"Supplies the Explainable Boosting Machine implementation used as the white-box model in the experiments."},{"cited_title":"SoK: A Comprehensive Reexamination of Phishing Research From the Security Perspective,","cited_arxiv_id":null,"evidence_quote":"Defines actionability as a metric and frames the phishing-research context that supports the paper's recommendation of EBM for actionable insights."},{"cited_title":"Molnar, Interpretable Machine Learning: A Guide for Making Black Box Models Explainable , 2nd ed","cited_arxiv_id":null,"evidence_quote":"Provides the interpretable machine learning reference used for the white-box explainability concepts and metric definitions."},{"cited_title":"Sok: Explainable machine learning for computer security applications,","cited_arxiv_id":null,"evidence_quote":"Frames the black-box versus white-box explainability comparison within computer security applications, which motivates the paper's analytical review."}],"review_version":1}