REVIEW 3 major objections 8 minor 39 references
On Defending Against Label Flipping Attacks on Malware Detection Systems
T0 review · 3 major / 8 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Semi-supervised defenses LSD and CSD can correct labels flipped by a silhouette-based poisoning attack and beat KSSD, with CSD about 19% more accurate and 15% lower FPR.
desk verdict The defense algorithms as written cannot execute, so the claimed 19% accuracy gain over KSSD is untestable. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing objects are the SCLFA attack and the two defenses. SCLFA runs K-means with two clusters on the training data, computes each sample's silhouette value, and flips labels wherever the silhouette value is at most zero, meaning the sample looks as though it belongs in the other cluster. LSD is an ensemble: validation data trains label propagation, label spreading, and a three-layer Conv-1D CNN, and each flipped training label is replaced by the majority vote of those three predictions and the original poisoned label. CSD trains the CNN on validation data, computes the four clustering metrics on the validation partition, then adds each training sample one by one and accepts its CNN-predicted label only if the sum of absolute metric changes stays within 0.1. The CNN itself is a fixed architecture with three Conv-1D layers of 16, 32, and 64 filters, kernel and stride of 2, max-pooling, and a dense sigmoid output.
What would settle it
Run SCLFA on any of the three Android datasets, then flip a small fraction of validation labels before applying CSD; if CSD's accuracy drops below KSSD's or below the no-defense baseline, the defense's clean-validation assumption is load-bearing. A direct reproducibility check is also possible: on Drebin API features with 300 random-forest-selected features, CSD should reproduce roughly the reported 19% accuracy gain and 15% FPR reduction over KSSD.
Extended reading notes
Core claim
The paper's central claim is that a poisoning attack can be built by clustering training samples and flipping the labels of those with non-positive silhouette values, and that two CNN-based semi-supervised defenses can undo most of the damage. LSD trains label propagation, label spreading, and the CNN on a clean validation set and re-labels each training sample by majority vote among these three predictions and the poisoned label. CSD instead uses four clustering-quality metrics (Rand index, mutual information, homogeneity, Fowlkes-Mallows index) computed on validation data as a baseline, and keeps each training sample's label only if adding that sample to the validation set changes the summed metrics by at most 0.1. The paper reports that across the Drebin, Contagio, and Genome datasets and API, intent, and permission feature sets, both defenses outperform KSSD and GANX, with CSD the strongest and roughly 19% more accurate than KSSD while lowering FPR by about 15%.
Load-bearing premise
The defenses assume the validation data has clean, trustworthy labels and is representative enough to train the propagators and the CNN and to serve as a clustering baseline; if an attacker can flip labels in the validation set, the relabeling and the 0.1 threshold comparisons are corrupted.
Editorial extensions
If this is right
- A defender facing a silhouette-based label-flip attack can recover most of the classifier's accuracy without discarding the poisoned training set.
- CSD offers a practical accuracy-versus-speed trade: it is slower than KSSD but much more accurate, and clearly faster than LSD while maintaining the higher accuracy.
- Random-forest feature selection down to 300 features preserves or improves the defenses' advantage while cutting runtime sharply in many configurations.
- The defenses hold across all three Android datasets and three feature types, so the relabeling mechanism is not tied to a single data distribution.
Reading between the lines
- The paper treats the validation set as safe; the natural next attack is to poison validation labels, which would likely break both defenses and force a defense that does not trust any fixed reference partition.
- The CSD acceptance threshold of 0.1 is a hand-set constant; varying it should produce a trade-off between correcting too many good labels and leaving flipped labels in place, and a data-dependent threshold might do better.
- Because the features are binary and sparse, the clustering-metric shortcut is cheap; for dense or continuous features the same four metrics may not scale, so the defense's practical reach depends on feature representation.
- SCLFA flips only samples that silhouette analysis already regards as mis-clustered; an attacker who flips high-confidence, correctly clustered samples may stress the defenses differently and would be a stronger test.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a silhouette-clustering-based label flipping attack (SCLFA) against Android malware classifiers, and two defenses built on semi-supervised learning: LSD (label-based, using label propagation, label spreading, and a CNN with a voting step) and CSD (clustering-based, using four clustering metrics and validation data to decide whether each training sample is properly labeled). The authors report that LSD and CSD improve accuracy and lower false-positive rate relative to the KSSD and GANX baselines across three Android datasets (Drebin, Contagio, Genome) and three feature types (API, Intent, Permission), claiming up to 19% higher accuracy and about 15% lower FPR for CSD. The paper also includes computational-complexity comparisons. The central empirical claim is that the two proposed defenses are effective and superior to prior defenses, but the algorithms as specified are not executable, which calls the reported results into question.
Significance. The threat model (label flipping with a silhouette-based selection of samples to flip) is reasonable and the intended approach of using validation data to correct poisoned labels is a direction worth exploring. The paper also makes a concrete baseline comparison and provides runtime measurements. However, the claims of superiority are not assessable because the two central algorithms contain shape mismatches and an undefined voting function, so the reported accuracy and FPR numbers in Figures 4-6 and Tables 2-3 cannot be reproduced from the manuscript. The authors provide a GitHub link for source code, but no code is available in the manuscript and the link in reference [31] appears malformed. There are no machine-checked proofs or theoretical guarantees; the contribution is purely empirical, and the empirical evidence is undermined by the algorithmic inconsistencies.
major comments (3)
- [Section 4.3.2, Algorithm 3] The CSD algorithm is not executable as written. Line 1 sets X to X_validation (which has n_validation rows), while line 3 sets YCorrected to the labels predicted for X_train (which has n_train rows). Lines 4-7 then compute Rand Index, Mutual Information, Homogeneity, and Fowlkes-Mallows Index from X and YCorrected; all four metrics require the two input clusterings to have the same number of elements, but n_validation and n_train are generally different. The loop in lines 8-16 repeats the mismatch: X_temp has n_validation+1 rows, yet YCorrected still contains n_train entries, so the comparison at line 11 is undefined. Consequently, the accuracy and FPR values reported for CSD in Fig. 6 and Table 2 cannot be attributed to the algorithm as specified in the paper.
- [Section 4.3.1, Algorithm 2] The LSD algorithm is also not executable as specified. Line 1 sets X to X_train and line 2 sets Y to Y_validation, and lines 3-10 fit label spreading, label propagation, and a CNN on (X, Y); this is only meaningful when n_train equals n_validation, and even then it pairs training features with validation labels, which is semantically incorrect. Additionally, line 11 calls an undefined Voting function with an argument YCorrected that has never been assigned a value on any previous line, and the input Poisoned_Y_train is never used. Thus the voting step that is essential to the LSD method is not defined, and the empirical comparison for LSD is unsupported.
- [Section 7, Conclusion] The headline claim that CSD 'has approximately 19% higher Accuracy than the KSSD and has about 15% lower FPR compared to the KSSD' is not supported by any statistical analysis. Figures 4-6 and Tables 2-3 report point estimates without error bars, confidence intervals, or significance tests, and no number of independent runs is stated. Given the variability typical of malware-detection datasets, the comparison may not be robust, and the absence of variance reporting is a load-bearing gap for a paper whose contribution is empirical.
minor comments (8)
- [Section 4.3.2, Algorithm 3] The threshold comparison S <= 0.1 in line 11 is a hand-chosen parameter with no sensitivity analysis; the manuscript should state how this threshold was selected and how the results depend on it.
- [Section 4.3.1, Algorithm 2] The description of the voting step in the text (Section 4.3.1) says the ensemble votes among 'LP output, Label Spreading, CNN predicted labels and poisoned labels,' but Algorithm 2 passes YCorrected (uninitialized) and none of the inputs explicitly as poisoned labels; the intended voting procedure should be formalized.
- [Algorithm 4, line 1] The loop header 'for i = 1≤m' should read 'for i = 1 to m'.
- [Table 2] The header of Table 2 is garbled, with repeated 'FNR' and 'AUC' labels; the column structure should be cleaned so that each WoFS/WFS pair is clearly labeled for each dataset.
- [Fig. 6] The x-axis labels DS1, DS2, DS3, -F, and -S are not defined in the caption; the caption says DS1=Drebin, DS2=Contagio, DS3=Genome, but this should be stated in the figure caption itself.
- [Reference [31]] The GitHub URL contains spaces ('mshojafar / sourcecodes') and is likely broken; please provide a working link and verify that the source code is publicly accessible.
- [Section 3.2] The phrase 'flip the labels by adding some perturbation of existing labels' is confusing; label flipping changes the class value, not a feature perturbation, and should be phrased accordingly.
- [Section 4.2, Definition 1] The statement 'we assign an interval [-1,1] for each sample' is imprecise; the silhouette value is computed, not assigned, and the definition should say that the silhouette value is used as a criterion for flipping.
Circularity Check
No significant circularity: the empirical comparison is self-contained; the only self-citation (GANX baseline) is not load-bearing.
full rationale
The paper's central claims are empirical performance comparisons, not derivations from fitted inputs. SCLFA is an attack constructed from Silhouette/K-means, and LSD/CSD are evaluated on held-out test partitions against KSSD [26] and GANX [30] using standard metrics (Accuracy, FPR, AUC, FNR), so the reported 'approximately 19% higher Accuracy than the KSSD' (Section 7) is a measured comparison with an external baseline, not a quantity forced by construction. The GANX baseline is a self-citation (Ref. [30] overlaps with the present authors), but it is used only as a comparison algorithm in Table 2 and Figures 4-6; the superiority claim does not reduce to that baseline, and the main claimed margin is against KSSD, which is independent work. The clean-validation assumption (validation data trains LP/LS/CNN in LSD and supplies baseline cluster metrics in CSD) is a threat-model limitation, not circularity: the defense's output labels are for the poisoned training set, while validation labels are an external input. The undefined 'Voting' step in Algorithm 2 and the row-count mismatches in Algorithms 2-3 are reproducibility/correctness defects that would make the experiments hard to reproduce, but they are not instances of a prediction reducing to its own input by definition. No equation in the paper defines an output in terms of the claimed result, and no fitted parameter is renamed as a prediction. Hence no circular step can be exhibited.
Assumptions & free parameters
free parameters (4)
- CSD threshold =
0.1
- Number of selected features =
300
- Number of clusters in attack =
2
- CNN architecture =
16/32/64 filters, kernel size 2, stride 2
assumptions (4)
- domain assumption Validation data is clean and unaffected by the attacker
- domain assumption Euclidean distance is appropriate for binary feature vectors
- domain assumption The four clustering metrics (RI, MI, HM, FMI) reliably indicate label quality
- ad hoc to paper Silhouette values from K-means clusters identify good flip candidates
Cite this review
Pith. "Pith review of On Defending Against Label Flipping Attacks on Malware Detection Systems." pith.science (2026). https://pith.science/paper/4K5ZOQRY
@misc{pith2026190804473,
author = {Pith},
title = {Pith review of: On Defending Against Label Flipping Attacks on Malware Detection Systems},
year = {2026},
howpublished = {\url{https://pith.science/paper/4K5ZOQRY}},
note = {Machine review of arXiv:1908.04473}
}
abstract
Label manipulation attacks are a subclass of data poisoning attacks in adversarial machine learning used against different applications, such as malware detection. These types of attacks represent a serious threat to detection systems in environments having high noise rate or uncertainty, such as complex networks and Internet of Thing (IoT). Recent work in the literature has suggested using the $K$-Nearest Neighboring (KNN) algorithm to defend against such attacks. However, such an approach can suffer from low to wrong detection accuracy. In this paper, we design an architecture to tackle the Android malware detection problem in IoT systems. We develop an attack mechanism based on Silhouette clustering method, modified for mobile Android platforms. We proposed two Convolutional Neural Network (CNN)-type deep learning algorithms against this \emph{Silhouette Clustering-based Label Flipping Attack (SCLFA)}. We show the effectiveness of these two defense algorithms - \emph{Label-based Semi-supervised Defense (LSD)} and \emph{clustering-based Semi-supervised Defense (CSD)} - in correcting labels being attacked. We evaluate the performance of the proposed algorithms by varying the various machine learning parameters on three Android datasets: Drebin, Contagio, and Genome and three types of features: API, intent, and permission. Our evaluation shows that using random forest feature selection and varying ratios of features can result in an improvement of up to 19\% accuracy when compared with the state-of-the-art method in the literature.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[31]
Taheri, R., Shojafar, M.: source code of label flipping attack/defenses on android data. https://github.com/ mshojafar / sourcecodes / blob / master / Taheri % 20et % 20al- NCAA2020.zip (2020). [Online; accessed 22- February-2020]
work page 2020
-
[1]
http://contagiominidump.blogspot
Contagio dataset. http://contagiominidump.blogspot. com/ (2020). [Online; accessed 22-February-2020]
work page 2020
-
[2]
https://scikit- learn.org/stable/ modules/label propagation.html (2020)
Label propagation. https://scikit- learn.org/stable/ modules/label propagation.html (2020). [Online; ac- cessed 22-February-2020]
work page 2020
-
[3]
https://nlp.stanford.edu/IR-book/ html/htmledition/mutual- information- 1.html (2020)
Mutual information. https://nlp.stanford.edu/IR-book/ html/htmledition/mutual- information- 1.html (2020). [Online; accessed 22-February-2020]
work page 2020
-
[4]
Arp, D., Spreitzenbarth, M., Hubner, M., Gascon, H., Rieck, K., Siemens, C.: Drebin: Effective and explainable detection of android malware in your pocket. In: Ndss, vol. 14, pp. 23–26 (2014)
work page 2014
-
[5]
Aviles-Rivero, A.I., Papadakis, N., Li, R., Alsaleh, S.M., Tan, R.T., Schonlieb, C.B.: Beyond supervised classifi- cation: Extreme minimal supervision with the graph 1- laplacian. arXiv preprint arXiv:1906.08635 (2019)
work page Pith review arXiv 2019
-
[6]
In: 2018 IEEE International Congress on Internet of Things (ICIOT), pp
Baracaldo, N., Chen, B., Ludwig, H., Safavi, A., Zhang, R.: Detecting poisoning attacks on machine learning in iot environments. In: 2018 IEEE International Congress on Internet of Things (ICIOT), pp. 57–64. IEEE (2018)
work page 2018
-
[7]
arXiv preprint arXiv:1704.02654 (2017)
Bhagoji, A.N., Cullina, D., Mittal, P.: Dimensionality re- duction as a defense against evasion attacks on machine learning classifiers. arXiv preprint arXiv:1704.02654 (2017)
arXiv 2017
Show all 39 references
-
[8]
Neuro- computing 192, 61–71 (2016)
Bootkrajang, J.: A generalised label noise model for clas- sification in the presence of annotation errors. Neuro- computing 192, 61–71 (2016)
2016
-
[9]
In: Joint European con- ference on machine learning and knowledge discovery in databases, pp
Bootkrajang, J., Kab´ an, A.: Label-noise robust logistic regression and its applications. In: Joint European con- ference on machine learning and knowledge discovery in databases, pp. 143–158. Springer (2012) 18 Rahim Taheri et al
2012
-
[10]
Pattern Recognition 47(11), 3641–3655 (2014)
Bootkrajang, J., Kab´ an, A.: Learning kernel logistic re- gression in the presence of class label noise. Pattern Recognition 47(11), 3641–3655 (2014)
2014
-
[11]
In: Proceedings of Twenty- Seventh International Joint Conference on Artificial In- telligence, pp
Dong-DongChen, W., WeiGao, Z.H.: Tri-net for semi- supervised deep learning. In: Proceedings of Twenty- Seventh International Joint Conference on Artificial In- telligence, pp. 2014–2020 (2018)
2018
-
[12]
In: ESANN, pp
Fr´ enay, B., Kab´ an, A., et al.: A comprehensive introduc- tion to label noise. In: ESANN, pp. 667–676 (2014)
2014
-
[13]
The Journal of Machine Learning Research 17(1), 2096–2030 (2016)
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., Lempitsky, V.: Domain-adversarial training of neural networks. The Journal of Machine Learning Research 17(1), 2096–2030 (2016)
2016
-
[14]
In: 2019 13th European Conference on Antennas and Propagation (EuCAP), pp
Guo, B., Tian, L., Zhang, J., Zhang, Y., Yu, L., Zhang, J., Liu, Z.: A clustering algorithm based on joint ker- nel density for millimeter wave radio channels. In: 2019 13th European Conference on Antennas and Propagation (EuCAP), pp. 1–5. IEEE (2019)
2019
-
[15]
In: Proceedings of the IEEE con- ference on computer vision and pattern recognition, pp
He, K., Sun, J.: Convolutional neural networks at con- strained time cost. In: Proceedings of the IEEE con- ference on computer vision and pattern recognition, pp. 5353–5360 (2015)
2015
-
[16]
IEEE Transactions on Image Processing 14(3), 360–369 (2005)
Hirakawa, K., Parks, T.W.: Adaptive homogeneity- directed demosaicing algorithm. IEEE Transactions on Image Processing 14(3), 360–369 (2005)
2005
-
[17]
In: Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pp
Iscen, A., Tolias, G., Avrithis, Y., Chum, O.: Label prop- agation for deep semi-supervised learning. In: Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5070–5079 (2019)
2019
-
[18]
In: Proc
Jiang, X., Zhou, Y.: Dissecting android malware: Char- acterization and evolution. In: Proc. of IEEE S&P, pp. 95–109 (2012)
2012
-
[19]
arXiv preprint arXiv:1511.08228 (2015)
Kaiser, L., Sutskever, I.: Neural gpus learn algorithms. arXiv preprint arXiv:1511.08228 (2015)
2015 arXiv
-
[20]
arXiv preprint arXiv:1606.01584 (2016)
Laishram, R., Phoha, V.V.: Curie: A method for protect- ing svm classifier from poisoning attack. arXiv preprint arXiv:1606.01584 (2016)
2016 arXiv
-
[21]
IEEE transactions on pattern analysis and machine intelligence 37(1), 175–188 (2014)
Li, Y.F., Zhou, Z.H.: Towards making unlabeled data never hurt. IEEE transactions on pattern analysis and machine intelligence 37(1), 175–188 (2014)
2014
-
[22]
In: International Conference on Machine Learning, pp
Maclaurin, D., Duvenaud, D., Adams, R.: Gradient-based hyperparameter optimization through reversible learn- ing. In: International Conference on Machine Learning, pp. 2113–2122 (2015)
2015
-
[23]
In: Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pp
Mu˜ noz-Gonz´ alez, L., Biggio, B., Demontis, A., Paudice, A., Wongrassamee, V., Lupu, E.C., Roli, F.: Towards poi- soning of deep learning algorithms with back-gradient op- timization. In: Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pp. 27–38....
2017
-
[24]
In: Advances in neural information processing systems, pp
Natarajan, N., Dhillon, I.S., Ravikumar, P.K., Tewari, A.: Learning with noisy labels. In: Advances in neural information processing systems, pp. 1196–1204 (2013)
2013
-
[25]
In: Proc
Papernot, N., et al.: Distillation as a defense to adversar- ial perturbations against deep neural networks. In: Proc. of IEEE S&P, pp. 582–597 (2016)
2016
-
[26]
In: Joint European Conference on Machine Learning and Knowl- edge Discovery in Databases, pp
Paudice, A., Mu˜ noz-Gonz´ alez, L., Lupu, E.C.: Label sani- tization against label flipping poisoning attacks. In: Joint European Conference on Machine Learning and Knowl- edge Discovery in Databases, pp. 5–15. Springer (2018)
2018
-
[27]
Journal of the American Statistical as- sociation 66(336), 846–850 (1971)
Rand, W.M.: Objective criteria for the evaluation of clus- tering methods. Journal of the American Statistical as- sociation 66(336), 846–850 (1971)
1971
-
[28]
arXiv preprint arXiv:1803.09050 (2018)
Ren, M., Zeng, W., Yang, B., Urtasun, R.: Learning to reweight examples for robust deep learning. arXiv preprint arXiv:1803.09050 (2018)
2018 arXiv
-
[29]
In: Advances in Neural Information Processing Systems, pp
Shafahi, A., Huang, W.R., Najibi, M., Suciu, O., Studer, C., Dumitras, T., Goldstein, T.: Poison frogs! targeted clean-label poisoning attacks on neural networks. In: Advances in Neural Information Processing Systems, pp. 6103–6113 (2018)
2018
-
[30]
arXiv preprint arXiv:1904.09433 (2019)
Taheri, R., Javidan, R., Shojafar, M., Conti, M., et al.: Can machine learning model with static features be fooled: an adversarial machine learning approach. arXiv preprint arXiv:1904.09433 (2019)
2019 arXiv
-
[32]
arXiv preprint arXiv:1808.08994 (2018)
Wang, Y., Chaudhuri, K.: Data poisoning attacks against online learning. arXiv preprint arXiv:1808.08994 (2018)
2018 arXiv
-
[33]
arXiv preprint arXiv:1811.12506 (2018)
Xia, Y., Liu, F., Yang, D., Cai, J., Yu, L., Zhu, Z., Xu, D., Yuille, A., Roth, H.: 3d semi-supervised learning with uncertainty-aware multi-view co-training. arXiv preprint arXiv:1811.12506 (2018)
2018 arXiv
-
[34]
1689–1698 (2015)
Xiao, H., Biggio, B., Brown, G., Fumera, G., Eckert, C., Roli, F.: Is feature selection secure against training data poisoning? In: International Conference on Machine Learning, pp. 1689–1698 (2015)
2015
-
[35]
Neurocomputing 160, 53–62 (2015)
Xiao, H., Biggio, B., Nelson, B., Xiao, H., Eckert, C., Roli, F.: Support vector machines under adversarial label contamination. Neurocomputing 160, 53–62 (2015)
2015
-
[36]
arXiv preprint arXiv:1703.01340 (2017)
Yang, C., Wu, Q., Li, H., Chen, Y.: Generative poisoning attack method against neural networks. arXiv preprint arXiv:1703.01340 (2017)
2017 arXiv
-
[37]
IEEE transactions on cybernetics 46(3), 766–777 (2016)
Zhang, F., Chan, P.P., Biggio, B., Yeung, D.S., Roli, F.: Adversarial feature selection against evasion attacks. IEEE transactions on cybernetics 46(3), 766–777 (2016)
2016
-
[38]
In: Pro- ceedings of the 18th ACM SIGKDD international confer- ence on Knowledge discovery and data mining, pp
Zhou, Y., Kantarcioglu, M., Thuraisingham, B., Xi, B.: Adversarial support vector machine learning. In: Pro- ceedings of the 18th ACM SIGKDD international confer- ence on Knowledge discovery and data mining, pp. 1059–
-
[1067]
SUPERFLUIDITY
ACM (2012) On Defending Against Label Flipping Attacks on Malware Detection Systems 19 Biographies Rahim Taheri received his B.Sc. degree of Computer engineering from Bahonar Technical College of Shiraz and M.Sc. degree of computer networks at the Shiraz University of Technolo...
2012
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.