REVIEW 3 major objections 4 minor 34 references
Detection of Backdoors in Trained Classifiers Without Access to the Training Set
T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Backdoors in trained DNNs are found using only held-out clean images.
desk verdict A genuinely useful post-training backdoor detector with a plausible but experimentally supported core premise; the optimization heuristic deserves scrutiny, but the paper warrants peer review. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the group-misclassification perturbation problem: minimize $\|v\|$ subject to at least fraction $\pi$ of clean examples from class $s$ being classified as class $t$ after adding $v$, solved for every ordered pair $(s,t)$. Since the indicator constraint is not differentiable, the paper minimizes a surrogate, the negative average posterior probability of $t$ over the class-$s$ images, via gradient descent until the $\pi$ fraction is reached. The resulting sizes are turned into reciprocal statistics $r_{st}=1/d(v_{st}^\ast)$; detection is a robust hypothesis test in which the $(K-1)^2$ smallest reciprocals estimate a conditional Gamma null distribution and the maximum reciprocal is declared an attack if its order-statistic $p$-value falls below a threshold. A polynomial correction compensates for pairs that start with naturally high confusion.
What would settle it
On a clean classifier trained with two nearly identical classes, find an innocent pair $(s,t)$ whose optimized common perturbation has a norm as small as a planted backdoor pair's; if its reciprocal $r_{st}$ exceeds the detection threshold, the anomaly test cannot separate them, matching the failure mode Section 6 predicts.
Extended reading notes
Core claim
The central claim is that an imperceptible backdoor leaves a measurable trace in the decision boundary: the backdoor pattern itself is a tiny common perturbation that sends most images of the source class $s^\ast$ to the target class $t^\ast$, so after optimizing a group-misclassification perturbation for every class pair, the pair $(s^\ast,t^\ast)$ stands out as a statistical outlier in perturbation size. The paper tests this by learning a minimal $L^2$ (or $L^1$) perturbation for each pair, converting sizes into reciprocal statistics, fitting a conditional Gamma null distribution to the $(K-1)^2$ least extreme values, and using an order-statistic $p$-value to decide whether the largest reciprocal indicates an attack. In experiments with CIFAR-10 and a ResNet-20 architecture, it detects sparse pixel and global watermark backdoors with high accuracy, correctly identifies source and target in most cases, and produces pattern estimates that resemble the true backdoor even when the planted pattern is larger than necessary.
Load-bearing premise
The load-bearing premise is that the backdoor pair's minimal flip perturbation is always anomalously smaller than every innocent pair's, an empirical gap the paper demonstrates experimentally rather than derives, and the authors concede it can fail when two naturally similar classes already have very low confusion.
Editorial extensions
If this is right
- A defender with only clean labeled images can screen a downloaded or outsourced DNN before deployment, and reject it if a backdoor is flagged.
- The detector works for single-source and multiple-source attacks and for both sparse pixel and global imperceptible patterns, and a generalized internal-layer variant covers multiplicative and patch-replacement embeddings.
- The estimated pattern can be used to mitigate the attack by fine-tuning with clean images augmented by the pattern, or as a basis for in-flight rejection of suspicious inputs.
- The detection threshold is a single user-set hyperparameter that approximately fixes the false-positive rate, and performance is stable over a wide range of the required misclassification fraction $\pi$.
Reading between the lines
- The same pair-wise minimal-perturbation statistic is a measure of decision-boundary fragility; it could double as an interpretability tool that surfaces hidden confusions unrelated to backdoors, a direction the paper itself notes.
- A determined attacker aware of this defense could try to make the backdoor pattern mimic the minimal common perturbation of a naturally confusable pair, or use a perceptible innocuous object, which is explicitly outside the paper's assumptions.
- The method's detection gap should be tested across architectures and datasets with naturally near-duplicate classes, such as fine-grained breeds or species, where the weakest assumption predicts false alarms.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper addresses post-training detection of backdoor attacks in DNN image classifiers when the defender has no access to the poisoned training set, only the trained classifier and a clean labeled set. The proposed method estimates, for every ordered class pair (s,t), a perturbation that induces a prescribed fraction π of group misclassifications from s to t (Algorithm 1), takes the reciprocal of its norm as a detection statistic, fits a Gamma null distribution to the (K−1)^2 smallest reciprocals, and computes an order-statistic p-value for the largest reciprocal (Section 3.1.3). If the p-value is below threshold θ, the classifier is declared attacked and the class pair attaining the maximum statistic is inferred as source/target, with the optimized perturbation used as the estimated backdoor pattern. Experiments on CIFAR-10 with ResNet-20 compare several objective variants with Neural Cleanse and report high detection rates for single- and multiple-source attacks, low false positives on clean classifiers, and successful estimation of backdoor patterns, plus extensions to multiplicative and feature-space perturbations.
Significance. If the central separation premise holds, this is a useful post-training defense: it requires no poisoned training set, makes no assumptions on pattern shape or number of source classes (beyond one target), and goes beyond detection to source/target inference and pattern estimation. The experimental design is careful in separating attacker and defender knowledge, and the comparison with Neural Cleanse is informative, especially the single-source failure mode. The paper also demonstrates a generalized feature-space variant that handles non-additive embedding mechanisms, and shows that the detector does not require the true backdoor pattern to be norm-minimal (Section 5.3.4). The main caveat is that the separation premise is supported only by heuristic optimization and a limited set of 25-realization experiments; the statistical inference additionally assumes an i.i.d. Gamma null for dependent statistics, and the acknowledged similar-class failure mode is not addressed.
major comments (3)
- [§3.1.2, Algorithm 1 and Eq. (1)] The load-bearing statistic d(v*_st) is not the minimum perturbation norm defined by problem (1). Algorithm 1 performs unconstrained gradient descent on the surrogate J_st(v) and terminates at the first iteration where the group misclassification fraction reaches π; Section 3.1.2 explicitly notes that a large step size can terminate with a perturbation much larger than needed. Thus the observed separation between the backdoor pair and non-backdoor pairs may reflect the optimization trajectory and step-size choice rather than a genuine property of the classifier. Because every downstream inference step (null fitting, order-statistic p-value, source/target inference) uses the ordering of these statistics, this gap between the stated objective and the implemented algorithm is central. I ask the authors to either solve a norm-constrained version of (1) (e.g., by adding a penalty on ||v|| or performing line-search backtracking to the minimal successful norm) or to demonstrate empirically that the anomaly ordering is invariant over a range of step sizes and initialization choices. Without such evidence, the central premise of Section 3.1.1 is not established beyond a heuristic.
- [§3.1.3, Eq. (4)] The order-statistic p-value in Eq. (4) assumes the K(K−1) reciprocal statistics are independent and identically distributed under the null. These statistics are not independent: they are computed on the same classifier, from overlapping clean images, and class pairs share source or target classes. The Gamma null density is also assumed rather than derived. The paper's own clean-classifier results show the calibration is off: the mean order-statistic p-value is 0.590 for the Clean group (Section 5.2.2), and false detection rates are much lower than the nominal θ. This means θ does not fix the false positive rate as claimed. The authors should either justify the i.i.d. assumption, use a calibration/permutation-based null, or report empirical false positive rates for the thresholds they recommend rather than relying on the theoretical uniform p-value claim.
- [§6 and §3.3] The detection premise is that non-attacked class pairs require large perturbations, but Section 6 concedes that two very similar classes with low confusion can require a perturbation as small as a true backdoor pair. The confusion-correction in Section 3.3 only addresses pairs with abnormally high initial misclassification fractions; it does not cover the low-confusion similar-class case, and no mechanism is proposed to distinguish such false pairs from true backdoors. Since this is a known false-positive mode for a method that claims user-controlled false positive rates, it should be quantified (e.g., with a dataset containing near-duplicate classes) and either mitigated or clearly stated as a scope restriction in the main text, not only in the conclusions.
minor comments (4)
- [§5.2.3] The decomposition "f1 ◦ f2(·)" has a naming error: both the feature map and the classifier head are called f1; the second should be f2. This makes the generalized objective in Eq. (10) harder to follow.
- [Algorithm 1] The pseudocode omits the upper-bound termination and line-search rule described in the text; including them would improve reproducibility and make the actual stopping criterion unambiguous.
- [§3.3, Eq. (7)] The notation for the polynomial fit is ambiguous: d(0)_st is defined as an argmin over d0 of an inner minimization over {a_m}, but the dependence of the inner minimizer on d0 should be made explicit, and the text should state how T is chosen for the polynomial fit.
- [§2.4] The defender is said to possess a labeled clean set, yet the method is described as "purely unsupervised" in several places; clarifying that "unsupervised" means no labels of attack status, rather than no labeled clean data, would prevent confusion.
Circularity Check
No significant circularity: the central separation premise is an explicit empirical hypothesis tested on externally crafted attacks, and the inference pipeline does not reduce to its inputs.
full rationale
The paper's derivation chain is: for each source-target pair, approximately solve the constrained minimization (1) via gradient descent on the surrogate J_st(v) in Algorithm 1; convert the resulting perturbation norms to reciprocal statistics; learn a Gamma null from the (K-1)^2 smallest reciprocals; and test the largest reciprocal with the order-statistic p-value (4). No step in this chain is defined in terms of the ground-truth attack labels or of the detection outcome. The key premise, stated in Section 3.1.1, is that a true backdoor pair requires an anomalously small common perturbation; this is an empirical domain assumption, not a consequence of the method's definitions, and the paper explicitly concedes in Section 6 that very similar classes with low confusion or perceptible innocuous-object backdoors can violate it. Algorithm 1's surrogate objective and first-hit termination rule are acknowledged approximations (Section 3.1.2), but they do not encode the attacked pair or force the ordering of the reciprocal statistics; they are correctness concerns, not circularity. The class-confusion correction (Section 3.3) fits a polynomial to each pair's own trajectory without using attack labels. Self-citations [30], [31], and [32] are not load-bearing: [30] is noted as a different inference approach, [31] addresses perceptible backdoors, and [32] is used as benchmark evidence outside the present fitted values. Experimental validation uses held-out classifiers and externally devised attacks with no parameter fitted to the reported detection results. The derivation is therefore self-contained with respect to circularity.
Assumptions & free parameters
free parameters (3)
- pi (target misclassification fraction) =
0.8 (default; user-set)
- theta (detection threshold) =
0.05 (default; 0.01 and 0.2 also tested)
- M (polynomial order for confusion correction) =
3 (2, 4, and 5 give similar results)
assumptions (4)
- domain assumption Gamma distribution for null reciprocal statistics, with K(K-1) statistics treated as i.i.d.
- domain assumption Attacker uses a single target class and an imperceptible additive perturbation; defender has clean labeled data from all classes
- domain assumption Perturbation-size separation: non-attacked pairs require large perturbations, while the attacked pair requires a small one
- domain assumption Gradient descent on the surrogate objective J_st discovers sufficiently small (near-minimal) perturbations
Cite this review
Pith. "Pith review of Detection of Backdoors in Trained Classifiers Without Access to the Training Set." pith.science (2026). https://pith.science/paper/37DDXD2M
@misc{pith2026190810498,
author = {Pith},
title = {Pith review of: Detection of Backdoors in Trained Classifiers Without Access to the Training Set},
year = {2026},
howpublished = {\url{https://pith.science/paper/37DDXD2M}},
note = {Machine review of arXiv:1908.10498}
}
read the original abstract
Recently, a special type of data poisoning (DP) attack targeting Deep Neural Network (DNN) classifiers, known as a backdoor, was proposed. These attacks do not seek to degrade classification accuracy, but rather to have the classifier learn to classify to a target class whenever the backdoor pattern is present in a test example. Launching backdoor attacks does not require knowledge of the classifier or its training process - it only needs the ability to poison the training set with (a sufficient number of) exemplars containing a sufficiently strong backdoor pattern (labeled with the target class). Here we address post-training detection of backdoor attacks in DNN image classifiers, seldom considered in existing works, wherein the defender does not have access to the poisoned training set, but only to the trained classifier itself, as well as to clean examples from the classification domain. This is an important scenario because a trained classifier may be the basis of e.g. a phone app that will be shared with many users. Detecting backdoors post-training may thus reveal a widespread attack. We propose a purely unsupervised anomaly detection (AD) defense against imperceptible backdoor attacks that: i) detects whether the trained DNN has been backdoor-attacked; ii) infers the source and target classes involved in a detected attack; iii) we even demonstrate it is possible to accurately estimate the backdoor pattern. We test our AD approach, in comparison with alternative defenses, for several backdoor patterns, data sets, and attack settings and demonstrate its favorability. Our defense essentially requires setting a single hyperparameter (the detection threshold), which can e.g. be chosen to fix the system's false positive rate.
Figures
Figures from the paper (10 more)
Reference graph
Works this paper leans on
-
[1]
H. Pirsiavash A. Saha, A. Subramanya. Hidden trigger backdoor attacks. In AAAI, 2020
work page 2020
-
[2]
B. Chen, W. Carvalho, N. Baracaldo, H. Ludwig, B. Edwards, T. Lee, I. Molloy, and B. Srivastava. Detecting Backdoor Attacks on Deep Neural Networks by Activation 40 Clustering. http://arxiv.org/abs/1811.03728, Nov. 8, 2018
arXiv 2018
-
[3]
X. Chen, C. Liu, B. Li, K. Lu, and D. Song. Targeted backdoor attacks on deep learning systems using data poisoning. https://arxiv.org/abs/1712.05526v1, 2017
arXiv 2017
-
[4]
J. Ba D. P. Kingma. Adam: A method for stochastic optimization. In ICLR, 2015
work page 2015
-
[5]
A backdoor attack against LSTM-based text classification systems
J. Dai and C. Chen. A backdoor attack against lstm-based text classification systems. https://arxiv.org/abs/1905.12457, 2019
work page Pith review arXiv 1905
- [6]
-
[7]
I. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial exam- ples. In Proc. ICLR, 2015
work page 2015
-
[8]
T. Gu, B. Dolan-Gavitt, and S. Garg. BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain. https://arxiv.org/abs/1708.06733, Mar. 2019
arXiv 2019
Show all 34 references
-
[9]
W. Guo, L. Wang, X. Xing, M. Du, and D. Song. TABOR: A Highly Ac- curate Approach to Inspecting and Restoring Trojan Backdoors in AI Systems. https://arxiv.org/abs/1908.01763, 2019
1908 arXiv
-
[10]
F. R. Hampel. The influence curve and its role in robust estimation. Journal of the American Statistical Association, 69, 1974
1974
-
[11]
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proc. CVPR, 2016
2016
-
[12]
Huang, A.D
L. Huang, A.D. Joseph, B. Nelson, B.I.P. Rubinstein, and J.D. Tygar. Adversarial machine learning. In Proc. 4th ACM Workshop on Artificial Intelligence and Security (AISec), 2011
2011
-
[13]
Juang and S
B.-H. Juang and S. Katigiri. Discriminative learning for minimum error classification. IEEE Trans. on Signal Processing, 1992
1992
-
[14]
C. Liao, H. Zhong, A. Squicciarini, S. Zhu, and D.J. Miller. Backdoor embedding in convolutional neural network models via invisible perturbation. CoRR, 2018. 41
2018
-
[15]
K. Liu, B. Doan-Gavitt, and S. Garg. Fine-Pruning: Defending Against Backdoor Attacks on Deep Neural Networks. In Proc. RAID, 2018
2018
-
[16]
Y. Liu, S. Ma, Y. Aafer, W.-C. Lee, and Zhai J. Trojaning attack on neural networks. In Proc. NDSS, San Diego, CA, Feb. 2018
2018
-
[17]
Miller, Y
D.J. Miller, Y. Wang, and G. Kesidis. When Not to Classify: Anomaly Detection of Attacks (ADA) on DNN classifiers at test time. Neural Computation, 31, 2019
2019
-
[18]
Miller, Z
D.J. Miller, Z. Xiang, and G. Kesidis. Adversarial learning in statistical classification: A comprehensive review of defenses against attacks. https://arxiv.org/abs/1904.06292, 2019
1904 arXiv
-
[19]
Mooosavi-Dezfooli, A
S. Mooosavi-Dezfooli, A. Fawzi, O. Fawzi, and P. Frossard. Universal adversarial per- turbations. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition , 2017
2017
-
[20]
Moosavi-Dezfooli, A
S.-M. Moosavi-Dezfooli, A. Fawzi, and P. Frossard. DeepFool: a simple and accurate method to fool deep neural networks, 2016
2016
-
[21]
Papernot, P
N. Papernot, P. McDaniel, I. Goodfellow, S. Jha, Z. Celik, and A. Swami. Practical black box attacks against machine learning. In Proc. Asia CCS, 2017
2017
-
[22]
Papernot, P
N. Papernot, P. McDaniel, S. Jha, M. Fredrikson, Z.B. Celik, and A. Swami. The limitations of deep learning in adversarial settings. In Proc. 1st IEEE European Symp. on Security and Privacy , 2016
2016
-
[23]
G. Schwarz. Estimating the dimension of a model. The Annals of Statistics , 6(2):461– 464, 1978
1978
-
[24]
Szegedy, W
C. Szegedy, W. Zaremba, I Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intriguing properties of neural networks. In Proc. ICLR, 2014
2014
-
[25]
Tamer, F
F. Tamer, F. Zhang, A. Juels, M. Reiter, and T. Ristenpart. Stealing machine learning models via prediction apis. In Proc. USENIX Security Symposium, 2016
2016
-
[26]
B. Tran, J. Li, and A. Madry. Spectral signatures in backdoor attacks. In Proc. NIPS, 2018. 42
2018
-
[27]
https://www.iarpa.gov/index.php/ research-programs/trojai
Trojans in artificial intelligence (TrojAI). https://www.iarpa.gov/index.php/ research-programs/trojai
-
[28]
B. Wang, Y. Yao, S. Shan, H. Li, B. Viswanath, H. Zheng, and B.Y. Zhao. Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks. In Proc. IEEE Symposium on Security and Privacy , 2019
2019
-
[29]
Wang, D.J
Y. Wang, D.J. Miller, and G. Kesidis. When Not to Classify: Detection of Reverse Engineering Attacks on DNN Image Classifiers. In Proc. IEEE ICASSP, Brighton, UK, May 2019
2019
-
[30]
Xiang, D
Z. Xiang, D. J. Miller, and G. Kesidis. Revealing backdoors, post-training, in dnn clas- sifiers via novel inference on optimized perturbations inducing group misclassification. In ICASSP, pages 3827–3831, 2020
2020
-
[31]
Xiang, D
Z. Xiang, D. J. Miller, and G. Kesidis. Revealing Perceptible Backdoors, without the Training Set, via the Maximum Achievable Misclassification Fraction Statistic. In Proc. IEEE MLSP, Oct. 2020
2020
-
[32]
Xiang, D.J
Z. Xiang, D.J. Miller, and G. Kesidis. A Benchmark Study of Backdoor Data Poisoning Defenses for Deep Neural Network Classifiers and A Novel Defense Only Legitimate Samples. In Proc. IEEE MLSP, Pittsburgh, Oct. 2019
2019
-
[33]
H. Xiao, B. Biggio, B. Nelson, H. Xiao, C. Eckert, and F. Roli. Support vector machines under adversarial label contamination. Neurocomputing, 160(C):53–62, July 2015
2015
-
[34]
X. Yuan, P. He, Q. Zhu, and X. Li. Adversarial examples: Attacks and defenses for deep learning. IEEE Transactions on Neural Networks and Learning Systems , 30(9):2805– 2824, Sep. 2019. 43
2019
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.