REVIEW 3 major objections 5 minor 40 references
XAI and Android Malware Models
T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Applying five XAI techniques to Android malware classifiers shows they reveal feature-level drivers, but the explainers disagree enough that conclusions depend on the method chosen.
desk verdict Competent XAI survey-plus-experiments on Android malware; the inconsistency finding is real but the SHAP-specific claim rests on a feature-selection confound. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The work is carried by a battery of feature-attribution techniques applied to trained classifiers. SHAP (Shapley Additive exPlanations) attributes each prediction to features by averaging cooperative-game contributions—Shapley values—and supplies global importance plots, dependence plots, and force plots; TreeSHAP, KernelSHAP, and DeepSHAP are used for the random forest, SVM/k-NN, and MLP respectively. LIME builds a locally faithful sparse linear surrogate around individual samples; ELI5 measures permutation importance by shuffling one feature at a time; PDP plots visualize the marginal effect of a feature while averaging out others; and Grad-CAM localizes the pixels a CNN relies on. A shared experimental device is also load-bearing: because KernelSHAP and DeepSHAP are expensive, the authors use random-forest-based recursive feature elimination to fix a common set of the top ten features, then compute SHAP on that subset for every model.
What would settle it
Re-run the global SHAP analysis with a feature subset chosen by each model's own feature importance, such as its permutation importance or LIME rankings, and check whether dangerous and total perm remain the top two features across all models; if they do not, the paper's central SHAP conclusion is an artifact of the random-forest-based recursive feature elimination.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that XAI techniques can be informative when applied to Android malware classifiers, but they do not tell a single consistent story. Using a ten-family subset of the KronoDroid dataset, the authors train six classifiers and interrogate them with five explanation families. SHAP global explanations are singled out as particularly informative: across the RBF-SVM, random forest, k-NN, and MLP models, the features dangerous and total perm rank highest, and PDP plots show interpretable relationships such as an approximately linear rise in SHAP values as dangerous increases for the RBF-SVM. Yet ELI5's permutation importance for random forest and the random forest's own feature weights put different features at the top, and Grad-CAM indicates the CNN relies on different criteria than the random forest. The paper therefore establishes both a positive result—explanations can be extracted and visualized—and a cautionary one: conclusions about which features drive Android malware decisions depend on the explainer chosen.
Load-bearing premise
The load-bearing premise is that the ten features chosen by random-forest feature selection are sufficient to represent how every model makes decisions, so if a different feature subset were used, the reported SHAP rankings of dangerous and total perm could change.
Editorial extensions
If this is right
- SHAP can be used to produce global feature-importance rankings for Android malware classifiers, and on this dataset dangerous and total perm emerge as the top two features across RBF-SVM, random forest, k-NN, and MLP.
- Local explanations are available for individual samples: LIME assigns probabilities and per-feature contributions, and SHAP force plots show which features push a sample toward a given malware family.
- The explanation methods disagree: only four of the random forest's own top-ten features appear among ELI5's top twenty, and Grad-CAM shows the CNN emphasizes different regions than the random forest, so no single explainer should be treated as authoritative.
- All six classifiers score within roughly four percentage points of each other, with random forest best, so explanations can be sought without giving up much accuracy.
- PDP plots expose interpretable feature relationships, such as the near-linear effect of dangerous on the RBF-SVM's predictions, which can be checked against domain knowledge.
Reading between the lines
- A testable extension is to recompute the SHAP global rankings using feature subsets chosen by each model's own importance signal; if dangerous and total perm remain top-ranked under those subsets, the paper's SHAP result is robust rather than an artifact of the random-forest selector.
- The observed cross-technique disagreement suggests that XAI-based feature pruning for Android malware should keep only features that rank highly across multiple explainers, otherwise downstream detectors inherit the explainer's bias.
- If explanation inconsistency is as widespread as the paper and prior work suggest, the disagreement between explainers could itself be a diagnostic signal: samples where LIME and SHAP strongly diverge may lie near decision boundaries or be adversarial inputs.
- The Grad-CAM result implies the CNN is not merely re-weighting the same features the random forest uses, which raises the possibility that different model families learn genuinely different decision rules on the same data; that would motivate choosing models by explanation similarity as well as accuracy.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper applies five XAI techniques (LIME, SHAP, PDP plots, ELI5, and Grad-CAM) to SVM, Random Forest, k-NN, MLP, and CNN classifiers trained on a 10-family subset of the KronoDroid Android malware dataset with 468 features. It reports model accuracies, generates global and local explanations, and compares the feature-importance rankings produced by different methods. The authors conclude that all tested XAI techniques provide useful information, that SHAP was particularly informative, and that different XAI techniques do not always agree. The paper also includes a literature review of XAI work applied to Android malware.
Significance. If the experimental conclusions are supported, the paper provides a useful comparative benchmark of post-hoc explanation methods on a realistic malware classification task, with concrete observations about inter-method disagreement. The manuscript is largely reproducible: the dataset, model hyperparameters, and train/test setup are specified, and the authors report runtime comparisons for the SHAP variants. The claim that feature-importance conclusions depend on the chosen XAI method is credible and consistent with prior work. However, the strongest claim, that SHAP was 'particularly informative' and that dangerous and total perm are the top features across models, rests on an under-specified and potentially circular 10-feature SHAP protocol, and the CNN/Grad-CAM comparison is confounded by the Random-Forest-based image construction. These issues are fixable but require changes to the experimental reporting or to the strength of the conclusions.
major comments (3)
- [Section 4.3.5 (Figures 8-12); Section 5] The SHAP global analyses are computed on a 10-feature subset selected by Random-Forest-based RFE, but the manuscript never states whether SVM, k-NN, MLP, and Random Forest were retrained on those 10 features for the SHAP experiments or whether SHAP was computed with only 10 sampled features of the full 468-feature models. Under the first reading, the SHAP variable importance plots, beeswarm plot, PDP plots, and force plot describe reduced classifiers whose accuracies are not reported and are not the models of Table 3. Under the second reading, the values are not Shapley values for the full models: with 458 features omitted, the efficiency property stated in Section 2.4.5 cannot hold, and the importances are conditional on the RF-selected coordinate set. Therefore, the claim in Section 5 that 'SHAP explanations were particularly informative' and the cross-model agreement on dangerous and total perm are not supported by the evidence as presented. The authors should either retrain each model on the selected 10 features and report those reduced models' accuracy, or compute SHAP on the full models with a sampling scheme that respects the model under explanation.
- [Section 4.3.4 (Figure 6)] The CNN input images are generated by ranking all 468 features according to Random Forest feature importance and then reshaping the ordered feature vector into a 22x22 image. This construction encodes the RF ranking into the spatial layout of the image, so the comparison between Grad-CAM and RF feature importance is not an independent comparison of two models on the same representational footing. Moreover, the 22x22 layout creates arbitrary adjacencies between features that have no domain meaning, and convolutional filters will exploit those adjacencies. Consequently, the conclusion that 'the Random Forest and CNN models appear to be using much different criteria' is not established. The experiment should be repeated with a feature ordering that does not depend on the model being compared, or the claim should be restricted to the chosen ordering.
- [Table 3] The manuscript states that stratified 5-fold cross-validation is used, but Table 3 reports only single accuracy, precision, recall, and F1 values per model with no variance and no statement of whether these are averaged over folds or taken from one fold. Since all models lie within approximately 4% in accuracy, the ranking of Random Forest as best and the relative ordering of the other models cannot be assessed without error bars. Reporting mean and standard deviation (or per-fold results) across the five folds is needed to support the performance comparisons used in the paper.
minor comments (5)
- [Section 4.3, first paragraph] 'KernalSHAP' should be 'KernelSHAP'.
- [Section 4.3.1] 'Random Forrest' should be 'Random Forest'.
- [Section 4.1, hyperparameter list] 'n estimator = 100' should be 'n_estimators = 100'; '300 hidden layers' should presumably read '300 hidden units'; and 'ReLu' should be 'ReLU'.
- [Section 4.3.2] The comparison of ELI5 and Figure 5 counts overlap between a top-20 list and a top-10 list, so the statement that only four of the top 10 features appear among the top 20 should be computed at the same cutoff or accompanied by the full ELI5 ranking for verification.
- [Section 4.1 and Figure 2] The pie chart in Figure 3 gives percentages that do not sum to 100 (25+16+13+12+9+8+6+4+4+3 = 100, but rounding makes individual labels appear inconsistent); the x-axis of Figure 2 lacks descriptive label text, though the caption explains the quantity.
Circularity Check
No circularity: the XAI comparison is an empirical study with no derivation that reduces to its own inputs; feature-selection choices create validity limitations, not circular reasoning.
full rationale
This paper is an empirical comparison of XAI techniques applied to trained Android malware classifiers. It contains no derivation chain in which a claimed result is equivalent by construction to an input or fitted parameter. The Shapley values are defined in Eq. (1) and Eq. (3) directly from the model output, and the subsequent SHAP plots, PDP plots, and force plots are computed from those definitions; the conclusions about feature importance are reported as observations about the computed explanations, not as predictions forced by the definitions. The use of Random-Forest-based RFE to select ten features for SHAP (Section 4.3.5) and the use of Random Forest feature ordering to construct CNN input images (Section 4.3.4) are methodological choices that may bias the scope of the explanations, but they do not make the central claims circular: the SHAP rankings are still computed from each explained model's behavior on the selected features, and the Grad-CAM analysis explicitly compares its output with the Random Forest ordering rather than assuming equality. The self-citations [3, 33, 34] are background references for malware analysis techniques and machine learning concepts; they are not load-bearing evidence for any XAI result or for the paper's conclusions about explainer consistency. Therefore no circular step is present, and the appropriate score is 0.
Assumptions & free parameters
free parameters (7)
- SVM kernel =
RBF
- Random Forest n_estimators =
100
- k-NN k =
5
- MLP hidden layers =
300 (text says '300 hidden layers', likely 300 neurons per layer)
- MLP learning rate =
0.0001
- CNN filters, filter size, dropout =
32; 2x2; 0.25
- Number of features sampled for SHAP =
10
assumptions (4)
- standard math Shapley values and their efficiency, symmetry, linearity, and null properties are valid for the models considered.
- domain assumption KronoDroid ground-truth labels and extracted static and dynamic features are correct and representative of real Android malware families.
- ad hoc to paper Ordering 468 tabular features by Random Forest importance and reshaping into a 22x22 image preserves information meaningful to a CNN.
- ad hoc to paper The top-10 features from Random-Forest RFE are sufficient input for SHAP explanations of all models.
Cite this review
Pith. "Pith review of XAI and Android Malware Models." pith.science (2026). https://pith.science/paper/PZRRKJ64
@misc{pith2026241116817,
author = {Pith},
title = {Pith review of: XAI and Android Malware Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/PZRRKJ64}},
note = {Machine review of arXiv:2411.16817}
}
abstract
Android malware detection based on machine learning (ML) and deep learning (DL) models is widely used for mobile device security. Such models offer benefits in terms of detection accuracy and efficiency, but it is often difficult to understand how such learning models make decisions. As a result, these popular malware detection strategies are generally treated as black boxes, which can result in a lack of trust in the decisions made, as well as making adversarial attacks more difficult to detect. The field of eXplainable Artificial Intelligence (XAI) attempts to shed light on such black box models. In this paper, we apply XAI techniques to ML and DL models that have been trained on a challenging Android malware classification problem. Specifically, the classic ML models considered are Support Vector Machines (SVM), Random Forest, and $k$-Nearest Neighbors ($k$-NN), while the DL models we consider are Multi-Layer Perceptrons (MLP) and Convolutional Neural Networks (CNN). The state-of-the-art XAI techniques that we apply to these trained models are Local Interpretable Model-agnostic Explanations (LIME), Shapley Additive exPlanations (SHAP), PDP plots, ELI5, and Class Activation Mapping (CAM). We obtain global and local explanation results, and we discuss the utility of XAI techniques in this problem domain. We also provide a literature review of XAI work related to Android malware.
Figures
Figures from the paper (8 more)
Reference graph
Works this paper leans on
-
[1]
Explainable artificial intelligence for cybersecurity: A lit- erature survey
Fabien Charmet, Harry Tanuwidjaja, Solayman Ayoubi, Pierre-Fran ccois Gimenez, Yufei Han, Houda Jmila, Gregory Blanc, Takeshi Takahashi, and Zonghua Zhang. Explainable artificial intelligence for cybersecurity: A lit- erature survey. Annals of Telecommunications, 77, 2022
work page 2022
-
[2]
DENAS: Automated rule generation by knowledge extraction from neural networks
Simin Chen, Soroush Bateni, Sampath Grandhi, Xiaodi Li, Cong Liu, and Wei Yang. DENAS: Automated rule generation by knowledge extraction from neural networks. In Proceedings of ESEC/FSE 2020 , pages 813–825, 2020
work page 2020
-
[3]
Austin, and Mark Stamp
Anusha Damodaran, Fabio Di Troia, Corrado Aaron Visaggio, Thomas H. Austin, and Mark Stamp. A comparison of static, dynamic, and hybrid analysis for malware detection. Journal of Computer Virology and Hacking Techniques, 13(1):1–12, 2017
2017
-
[4]
Can We Trust Your Explanations? Sanity Checks for Interpreters in Android Malware Analysis
Ming Fan, Wenying Wei, Xiaofei Xie, Yang Liu, Xiaohong Guan, and Ting Liu. Can we trust your explanations? sanity checks for interpreters in Android malware analysis. https://arxiv.org/abs/2008.05895
work page Pith review arXiv 2008
-
[5]
Understanding privacy awareness in Android app descriptions using deep learning
Johannes Feichtner and Stefan Gruber. Understanding privacy awareness in Android app descriptions using deep learning. In Proceedings of the Tenth ACM Conference on Data and Application Security and Privacy , pages 203– 214, 2020
work page 2020
-
[6]
G DATA blog: Cyber attacks on Android devices on the rise. https://www.gdatasoftware.com/blog/2018/11/31255-cyber- attacks-on-android-devices-on-the-rise , 2018. 25
work page 2018
-
[7]
Google. AI explanations whitepaper. https://storage.googleapis.com/ cloud-ai-whitepapers/AI%20Explainability%20Whitepaper.pdf
-
[8]
Alejandro Guerra-Manzanares, Hayretdin Bahsi, and Sven N˜ omm. KronoDroid: Time-based hybrid-featured dataset for effective Android mal- ware detection and characterization. Computers & Security , 110(C):8–14, 2021
work page 2021
Show all 40 references
-
[9]
XAI: Explainable artificial intelligence
David Gunning, Mark Stefik, Jaesik Choi, Timothy Miller, Simone Stumpf, and Guang-Zhong Yang. XAI: Explainable artificial intelligence. Science Robotics, 4(37), 2019
2019
-
[10]
An Introduction to Machine Learning In- terpretability
Patrick Hall and Navdeep Gill. An Introduction to Machine Learning In- terpretability. O’Reilly Media and H2O, second edition, 2019
2019
-
[11]
Towards an interpretable deep learning model for mobile malware detection and family identification
Giacomo Iadarola, Fabio Martinelli, Francesco Mercaldo, and Antonella Santone. Towards an interpretable deep learning model for mobile malware detection and family identification. Computers & Security , 105:102–198, 2021
2021
-
[12]
What is random forest? https://www.ibm.com/topics/random- forest
IBM. What is random forest? https://www.ibm.com/topics/random- forest
-
[13]
Zeus virus
Kaspersky. Zeus virus. https://www.kaspersky.com/resource-center/ threats/zeus-virus, 2023
2023
-
[14]
To- wards explainable CNNs for Android malware detection.Procedia Computer Science, 184:959–965, 2021
Martin Kinkead, Stuart Millar, Niall McLaughlin, and Philip O’Kane. To- wards explainable CNNs for Android malware detection.Procedia Computer Science, 184:959–965, 2021
2021
-
[15]
Mikhail Korobov and Konstantin Lopuhin. ELI5. https://eli5. readthedocs.io/en/latest/index.html, 2016
2016
-
[16]
ELI5: Permutation importance
Mikhail Korobov and Konstantin Lopuhin. ELI5: Permutation importance. https://eli5.readthedocs.io/en/latest/blackbox/permutation_ importance.html, 2016
2016
-
[17]
https://github.com/marcotcr/lime
LIME. https://github.com/marcotcr/lime
-
[18]
Ex- plainable AI: A review of machine learning interpretability methods
Pantelis Linardatos, Vasilis Papastefanopoulos, and Sotiris Kotsiantis. Ex- plainable AI: A review of machine learning interpretability methods. En- tropy, 23(1), 2021
2021
-
[19]
Deep learn- ing for Android malware defenses: A systematic literature review
Yue Liu, Chakkrit Tantithamthavorn, Li Li, and Yepang Liu. Deep learn- ing for Android malware defenses: A systematic literature review. ACM Computing Surveys , 55(8):1–36, 2022
2022
-
[20]
Explainable AI for Android malware detection: Towards understanding why the models perform so well? https://arxiv.org/abs/2209.00812, 2022
Yue Liu, Chakkrit Tantithamthavorn, Li Li, and Yepang Liu. Explainable AI for Android malware detection: Towards understanding why the models perform so well? https://arxiv.org/abs/2209.00812, 2022
2022 arXiv
-
[21]
A unified approach to interpreting model predictions
Scott Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in Neural Information Processing Systems , 30:4765– 4774, 2017. http://arxiv.org/abs/1705.07874. 26
2017 arXiv
-
[22]
https://developers.google.com/android/play- protect/phacategories, 2023
Malware categories. https://developers.google.com/android/play- protect/phacategories, 2023
2023
-
[23]
Kimmel, Mahmoud Abdelsalam, and Maanak Gupta
Harikha Manthena, Jeffrey C. Kimmel, Mahmoud Abdelsalam, and Maanak Gupta. Analyzing and explaining black-box models for online malware de- tection. IEEE Access, 11:25237–25252, 2023
2023
-
[24]
What is interpretability? https://it.mathworks.com/ discovery/interpretability.html
MathWorks. What is interpretability? https://it.mathworks.com/ discovery/interpretability.html
-
[25]
Win32/winwebsec
Microsoft Blog. Win32/winwebsec. https://www.microsoft.com/en- us/wdsi/threats/malware-encyclopedia-description?Name=Win32/ Winwebsec, 2017
2017
-
[26]
Model Explainability and Interpretability
Pradeepta Mishra. Model Explainability and Interpretability . Apress Pub- lishers, 2022
2022
-
[27]
Interpretable Machine Learning
Christoph Molnar. Interpretable Machine Learning . Independently published, second edition, 2022. https://christophm.github.io/ interpretable-ml-book
2022
-
[28]
https://paperswithcode.com/ method/cam
PapersWithCode: Class-activation map. https://paperswithcode.com/ method/cam
-
[29]
Malware detection using opcodes statistical features
Saeid Rezaei, Ali Afraz, Fereidoon Rezaei, and Mohammad Reza Shamani. Malware detection using opcodes statistical features. In 8th International Symposium on Telecommunications, IST, pages 151–155, 2016
2016
-
[30]
Exploring LIME explanations and the mathematics behind it
Denis Rothman. Exploring LIME explanations and the mathematics behind it. https://www.codemotion.com/magazine/ai-ml/lime-explainable- ai/, 2020
2020
-
[31]
https://scikit-learn.org/stable/modules/generated/ sklearn.svm.SVC.html
scikit-learn. https://scikit-learn.org/stable/modules/generated/ sklearn.svm.SVC.html
-
[32]
Explanation- guided backdoor poisoning attacks against malware classifiers
Giorgio Severi, Jim Meyer, Scott Coull, and Alina Oprea. Explanation- guided backdoor poisoning attacks against malware classifiers. In 30th USENIX Security Symposium, USENIX Security 21, pages 1487–1504, 2021
2021
-
[33]
Information Security: Principles and Practice
Mark Stamp. Information Security: Principles and Practice . Wiley, third edition, 2021
2021
-
[34]
Introduction to Machine Learning with Applications in Infor- mation Security
Mark Stamp. Introduction to Machine Learning with Applications in Infor- mation Security. Chapman and Hall/CRC Press, second edition, 2023
2023
-
[35]
Explainable malware de- tection system using transformers-based transfer learning and multi-model visual representation
Farhan Ullah, Amjad Alsirhani, Mohammed Mujib Alshahrani, Abdullah Alomari, Hamad Naeem, and Syed Aziz Shah. Explainable malware de- tection system using transformers-based transfer learning and multi-model visual representation. Sensors, 22(18), 2022
2022
-
[36]
Evaluating explanation methods for deep learning in security
Alexander Warnecke, Daniel Arp, Christian Wressnegger, and Konrad Rieck. Evaluating explanation methods for deep learning in security. https://arxiv.org/abs/1906.02108, 2020. 27
1906 arXiv
-
[37]
Bozhi Wu, Sen Chen, Cuiyun Gao, Lingling Fan, Yang Liu, Weiping Wen, and Michael R. Lyu. Why an Android app is classified as malware? towards malware classification interpretation. https://arxiv.org/abs/ 2004.11516, 2020
2004 arXiv
-
[38]
Ex- plainable machine learning in cybersecurity: A survey.International Journal of Intelligent Systems , 37(12):12305–12334, 2022
Feixue Yan, Sheng Wen, Surya Nepal, Cecile Paris, and Yang Xiang. Ex- plainable machine learning in cybersecurity: A survey.International Journal of Intelligent Systems , 37(12):12305–12334, 2022
2022
-
[39]
Fast TreeSHAP: Accelerating SHAP value computation for trees
Jilei Yang. Fast TreeSHAP: Accelerating SHAP value computation for trees. https://arxiv.org/abs/2109.09847v3, 2021
2021 arXiv
-
[40]
CADE: Detecting and explaining concept drift samples for security applications
Limin Yang, Wenbo Guo, Qingying Hao, Arridhana Ciptadi, Ali Ah- madzadeh, Xinyu Xing, and Gang Wang. CADE: Detecting and explaining concept drift samples for security applications. In 30th USENIX Security Symposium, USENIX Security 21, pages 2327–2344, 2021. 28 Appendix (a) RB...
2021
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.