REVIEW 5 major objections 6 minor 28 references
Robust Knowledge Distillation in Federated Learning: Counteracting Backdoor Attacks
T0 review · 5 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read A federated-learning defense called RKD filters poisoned client updates by their angle to the global model, keeping backdoor attack success below 17% while preserving accuracy above 80% under non-IID data and up to 60% malicious clients.
desk verdict RKD is a promising recombination of known defense pieces, but its trust anchor — the highest-cosine cluster is benign — is contradicted by the paper's own TSBA description and needs evidence, not just assertion. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the automated clustering step: each client update $\theta_i^r$ is scored by cosine similarity $s_i = (\theta_i^r)^\top M_{\text{global}}^r / (\|\theta_i^r\|\|M_{\text{global}}^r\|)$ to the current global model, and HDBSCAN clusters these scalar scores; the cluster with the highest mean score is declared benign. Model selection then computes the element-wise median of the benign models and keeps only models within $\epsilon = \mu_d + k\sigma_d$ of it in $\ell^1$ distance. Knowledge distillation averages the selected models' logits on an unlabeled public set, forms soft pseudo-labels with temperature $T$, trains a distilled student by KL divergence, and averages weights with stochastic weight averaging to produce the next global model. This three-stage pipeline is what carries the claim: cosine similarity makes clustering tractable in high-dimensional parameter space, median selection removes residual outliers, and distillation suppresses both backdoor triggers and non-IID variance.
What would settle it
Run RKD with 30 clients and 60% malicious clients using TSBA or A3FL, but have each attacker send the current global model vector plus a small perturbation aligned with the backdoor trigger, chosen so its cosine similarity to the global model is at least as high as the benign clients' average; if the global model's attack success rate exceeds 17% while clean accuracy stays high, the benign-cluster anchor has failed. A sharper test: poison the global model for one round with a large scaled malicious update, then check whether the next round's honest clients are clustered as outliers and excluded while the attackers are admitted.
Extended reading notes
Core claim
On the paper's own terms, RKD's central discovery is that a reliable benign ensemble can be recovered from heterogeneous client updates by projecting each update to a one-dimensional cosine-similarity score against the global model, clustering those scores, and trusting only the highest-similarity cluster. The approach is designed so that angular deviation from the global model, not raw parameter distance, is the signal separating poisoned from honest updates, which is why the authors claim it survives non-IID data and attacker majorities of up to 60%. The selected near-median models form an ensemble whose averaged logits, with temperature-smoothed softmax targets, are distilled into the global model via KL divergence; this step both transfers collective knowledge and smooths over non-IID variation. The paper's headline evidence is that this pipeline keeps clean-task accuracy above 80% while driving attack success rate below 17% for A3FL, F3BA, DBA, ADBA, and TSBA attacks.
Load-bearing premise
The load-bearing assumption is that the group of client updates with the highest average cosine similarity to the current global model is the benign group, so the global model's direction must itself be trustworthy.
Editorial extensions
If this is right
- If RKD holds up, federated learning deployments under non-IID data can defend backdoors without requiring the common IID-data or small-malicious-fraction assumptions.
- With up to 60% malicious clients, RKD keeps main-task accuracy above 80% and attack success below 17% across CIFAR-10, EMNIST, and Fashion-MNIST for A3FL, F3BA, DBA, ADBA, and TSBA attacks.
- The exclusion strategy stops flagged attackers from receiving the updated global model, and the perturbed-global variant (RKD-PGM) hides their status with noise of norm about $10^{-4}$ while retaining similar accuracy.
- Ablations show each of the three components matters: without clustering, malicious updates enter the ensemble; without median selection, residual outliers raise attack success; without knowledge distillation, accuracy drops by about 20% in non-IID settings.
Reading between the lines
- The defense inherits a vulnerability the paper does not analyze: if the global model is already poisoned, or if attackers craft updates more aligned with its direction than benign updates, the 'benign cluster' label flips and the filter would exclude honest clients while admitting attackers.
- Because the dynamic minimum cluster size $Q = \max(2, \lceil 0.2N - r\rceil)$ decays to 2 after a few rounds, small colluding groups could eventually form their own high-similarity cluster and be accepted as benign; this is a testable boundary condition, not a claim in the paper.
- The 16% unlabeled public dataset needed for distillation is a practical cost and a new trust assumption; how distillation behaves with smaller, skewed, or distributionally different public data is left open.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Robust Knowledge Distillation (RKD), a server-side defense for federated learning. At each round, the server computes the cosine similarity of each client update to the current global model, clusters these scalar scores with HDBSCAN, declares the cluster with the highest mean similarity to be benign, selects models near the element-wise median of that cluster, and distills the selected ensemble into the next global model using an unlabeled server dataset. Clients flagged as malicious are either excluded from receiving the updated global model or receive a perturbed version. The authors evaluate RKD against five backdoor attacks (A3FL, F3BA, DBA, ADBA, TSBA) on CIFAR-10, EMNIST, and Fashion-MNIST, with 20%, 40%, and 60% malicious clients under Non-IID partitions, and claim MTA above 80% with ASR below 17%, outperforming FedAvg, FLAME, FedDF, FedRAD, FedBE, RLR, and FoolsGold.
Significance. If the claims hold, RKD would be a practically valuable defense because it does not require IID data or a small attacker fraction, and it is computationally cheaper than FedDF and FedBE. The paper ships code, includes an ablation study, proposes a dynamic cluster-size schedule, and evaluates several recent attacks, which are strengths. The main weakness is that the central trust anchor—the max-cosine cluster rule—is not justified theoretically or diagnosed empirically, and the experimental reporting is incomplete: the claimed confidence intervals appear in no figure, and no final numerical tables are provided. These gaps currently prevent the reader from verifying the headline numbers.
major comments (5)
- [Algorithm 2, lines 8-10] The defense assumes that the HDBSCAN cluster with the highest mean cosine similarity to the current global model is the benign cluster. This is the load-bearing step, and the paper gives no formal or empirical justification for it. An adversary can submit an update of the form θ_global + γΔ with small γ, whose cosine similarity to θ_global tends to 1; this is consistent with the train-and-scale TSBA attack described in Section V.C.e. Such an update would be placed in the "benign" cluster and would enter the distillation ensemble. Once the global model is itself poisoned, benign updates that disagree with the poisoned direction will have lower similarity and be filtered out, so the error can self-reinforce. The paper reports no cluster-assignment accuracy or per-round diagnosis showing that the selected cluster is actually the benign one. Please provide a correctness condition under which the max-cosine cluster is benign, add cluster-membership diagnostics to the experiments, and evaluate RKD against a TSBA variant that explicitly aligns the scaled update with the global direction.
- [Section V.F] The text states that all experiments were repeated five times and that confidence intervals are reported at a significance level of ρ = 0.01, but Figures 1-15 show no error bars or confidence bands, and no table reports final MTA/ASR values with intervals. The abstract's quantitative claims (MTA > 80%, ASR < 17%) therefore cannot be checked or reproduced from the manuscript. Please add a table of final means and intervals for every attack/dataset/attacker-fraction configuration, and either add confidence bands to the figures or explicitly state that the plotted curves are means only.
- [Section V.F] The evaluation matrix does not cover the claimed "various scenarios" uniformly. ADBA is tested only on CIFAR-10 (Figure 7), A3FL is not tested on EMNIST, and different attacks are evaluated under different heterogeneity levels (α = 0.3 for A3FL, α = 0.5 for F3BA, ADBA, and TSBA, α = 0.9 for DBA). This confounds attack difficulty with dataset and heterogeneity, so the stated superiority over baselines is only established for the particular configurations shown. Please either fill in the full cross-product of attacks, datasets, and α values, or explicitly restrict the claims to the tested configurations.
- [Sections IV.C and V.F.7] Only the HDBSCAN minimum-cluster-size schedule Q is given a sensitivity analysis. Other free parameters—the model-selection threshold ε (defined only as "one may set" in Section IV.C), distillation temperature T, KD epochs and learning rate, and the size of the server distillation dataset—are fixed without a sensitivity study or even a reported default value for ε. Because these are tuned on the same three benchmarks, the claim that RKD works "without relying on restrictive assumptions" needs at least a robustness check over these parameters or a statement of the ranges for which the results hold.
- [Algorithm 1, lines 6-16] The exclusion strategy is unclear. Line 9 sends the "current local model M^r_i" to a malicious client before the server collects the round-r models in line 16, and the server cannot know M^r_i at that point. The prose in Section IV.A says excluded clients continue training on their current local model, but the algorithm appears to send the stale model back. Please clarify what exactly is sent, from where the server obtains it, and when, since this affects the attack dynamics for adaptive adversaries and the reproducibility of the exclusion strategy.
minor comments (6)
- [References] Reference [12] contains a typo: "University of Tront" should be "University of Toronto."
- [Section II.A] The symbol α is used both for the Dirichlet heterogeneity parameter and for the scaling factor in the model-poisoning description; please rename one of them to avoid confusion.
- [Section V.B] The attack setup does not specify the trigger pattern, the target class, or the number of poisoned samples per client; please provide these details for reproducibility.
- [Section V.F.7 and Section V.C.e] The TSBA scaling factor γ is never given a range or concrete value, so it is unclear which scaling regime the reported results cover; please report the γ values used.
- [Table I] Defense-time measurements lack hardware/software context and variance estimates, and the RLR value of 0.020 seconds is unusually small; please describe the evaluation environment and whether times are averaged over rounds or runs.
- [Figures 1 and 2] Figure 1 includes the RKD(PGM) variant but Figure 2 does not; please clarify whether PGM was evaluated on Fashion-MNIST and, if so, why it is omitted.
Circularity Check
No circular derivation: RKD's effectiveness claim is grounded in held-out test-set evaluation, and the sole self-citation is non-load-bearing.
full rationale
The paper's central claim is that RKD maintains MTA above 80% and ASR below 17% under Non-IID data with up to 60% malicious clients. This is an empirical result computed on held-out clean and poisoned test sets, not a quantity that appears as a fitted constant or constraint in the derivation. The clustering rule in Algorithm 2 (declare benign the cluster with highest mean cosine similarity to the global model) is a design heuristic; it could be vulnerable to attacks that align with the global model, but a possible failure mode is an empirical soundness issue, not circularity. The dynamic minimum cluster size Q and the selection threshold epsilon are tuned on the same benchmarks, which raises generalization concerns but does not make the prediction equivalent to the fitted inputs by construction. The only self-citation is [1] (the authors' earlier RFCL work), mentioned in Related Work and explicitly described as having diminished effectiveness against adaptive backdoor attacks; RKD's own mechanism is specified and evaluated in the paper rather than resting on that citation. Thus no load-bearing step reduces, by definition or by self-citation chain, to its own inputs. Score 2 reflects the presence of one minor non-load-bearing self-citation, with no significant circularity in the central derivation.
Assumptions & free parameters
free parameters (5)
- HDBSCAN minimum cluster size schedule Q =
Q = max(2, ceil(0.2N - r))
- Model selection threshold epsilon =
epsilon = mu_d + k sigma_d with k unspecified
- Distillation temperature T =
not reported
- Server distillation dataset fraction =
16% of total training data
- KD epochs and learning rate =
EKD and eta not fully reported
assumptions (4)
- ad hoc to paper Cluster with highest mean cosine similarity to the global model is benign
- domain assumption Server has access to an unlabeled dataset Dval disjoint from client training data
- domain assumption HDBSCAN on one-dimensional cosine similarities produces reliable separation of benign and malicious updates
- domain assumption Knowledge distillation using KL divergence and SWA transfers ensemble knowledge without reintroducing backdoors
Cite this review
Pith. "Pith review of Robust Knowledge Distillation in Federated Learning: Counteracting Backdoor Attacks." pith.science (2026). https://pith.science/paper/5HFMW6CZ
@misc{pith2026250200587,
author = {Pith},
title = {Pith review of: Robust Knowledge Distillation in Federated Learning: Counteracting Backdoor Attacks},
year = {2026},
howpublished = {\url{https://pith.science/paper/5HFMW6CZ}},
note = {Machine review of arXiv:2502.00587}
}
read the original abstract
Federated Learning (FL) enables collaborative model training across multiple devices while preserving data privacy. However, it remains susceptible to backdoor attacks, where malicious participants can compromise the global model. Existing defence methods are limited by strict assumptions on data heterogeneity (Non-Independent and Identically Distributed data) and the proportion of malicious clients, reducing their practicality and effectiveness. To overcome these limitations, we propose Robust Knowledge Distillation (RKD), a novel defence mechanism that enhances model integrity without relying on restrictive assumptions. RKD integrates clustering and model selection techniques to identify and filter out malicious updates, forming a reliable ensemble of models. It then employs knowledge distillation to transfer the collective insights from this ensemble to a global model. Extensive evaluations demonstrate that RKD effectively mitigates backdoor threats while maintaining high model performance, outperforming current state-of-the-art defence methods across various scenarios.
Figures
Figures from the paper (8 more)
Reference graph
Works this paper leans on
-
[1]
E. Alharbi, L. S. Marcolino, A. Gouglidis, and Q. Ni. Robust federated learning method against data and model poisoning attacks with hetero- geneous data distribution. In ECAI 2023, pages 85–92. IOS Press, 2023 0 20 40 60 80 100 Rounds 0.0 0.2 0.4 0.6 0.8 1.0Accuracy Rate(%) RKD FedAvg FLAME FedDF FedRAD FedBE RLR FG 0 20 40 60 80 100 Rounds 0.0 0.2 0.4 0...
work page 2023
-
[2]
S. Andreina, G. A. Marson, H. M ¨ollering, and G. Karame. Baffle: Backdoor detection via feedback-based federated learning. In 2021 IEEE 41st International Conference on Distributed Computing Systems (ICDCS), pages 852–863. IEEE, 2021
work page 2021
-
[3]
Bagdasaryan, A
E. Bagdasaryan, A. Veit, Y . Hua, D. Estrin, and V . Shmatikov. How to backdoor federated learning. In International conference on artificial intelligence and statistics, pages 2938–2948. PMLR, 2020
2020
-
[4]
X. Cao, M. Fang, J. Liu, and N. Z. Gong. Fltrust: Byzantine- robust fed- erated learning via trust bootstrapping. arXiv preprint arXiv:2012.13995, 2020
arXiv 2012
-
[5]
H.-Y . Chen and W.-L. Chao. Fedbe: Making the Bayesian model ensem- ble applicable to federated learning. arXiv preprint arXiv:2009.01974, 2020
arXiv 2009
- [6]
-
[7]
P. Fang and J. Chen. On the vulnerability of backdoor defences for federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 11800–11808, 2023
work page 2023
-
[8]
C. Fung, C. J. Yoon, and I. Beschastnikh. The limitations of federated learning in sybil settings. In 23rd International Symposium on Research in Attacks, Intrusions and Defences (RAID 2020), pages 301–316, 2020
work page 2020
Show all 28 references
-
[9]
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on Computer Vision and pattern recognition, pages 770–778, 2016
2016
-
[10]
K. Khan, S. U. Rehman, K. Aziz, S. Fong, and S. Sarasvady. DBSCAN: Past, present and future. In The fifth international conference on the applications of digital information and web technologies (ICADIWT 2014), pages 232–238. IEEE, 2014
2014
-
[11]
and Zagoruyko, S., 2017, June
Komodakis, N. and Zagoruyko, S., 2017, June. Paying more attention to attention: improving the performance of convolutional neural networks via attention transfer. In ICLR
2017
-
[12]
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A., 2009. Learning Multiple Layers of Features from Tiny Images. Master’s thesis, University of Tront
2009
-
[13]
T. Lin, L. Kong, S. U. Stich, and M. Jaggi. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems, 33:2351–2363, 2020
2020
-
[14]
McMahan, E
B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273–1282. PMLR, 2017
2017
-
[15]
T. D. Nguyen, P. Rieger, M. Miettinen, and A.-R. Sadeghi. Poisoning attacks on federated learning-based iot intrusion detection system. In Proc. Workshop Decentralized IoT Syst. Secure.(DISS), pages 1–7, 2020
2020
-
[16]
T. D. Nguyen, P. Rieger, R. De Viti, H. Chen, B. B. Brandenburg, H. Yalame, H. M ¨ollering, H. Fereidooni, S. Marchal, M. Miettinen, et al. FLAME: Taming backdoors in federated learning. In 31st USENIX Security Symposium (USENIX Security 22), pages 1415–1432, 2022
2022
-
[17]
M. S. Ozdayi, M. Kantarcioglu, and Y . R. Gel. Defending against backdoors in federated learning with robust learning rate. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 9268– 9276, 2021
2021
-
[18]
S. P. Sturluson, S. Trew, L. Mu ˜noz-Gonz´alez, M. Grama, J. Passerat- Palmbach, D. Rueckert, and A. Alansary. FEDRAD: Federated robust adaptive distillation. arXiv preprint arXiv:2112.01405, 2021
2021 arXiv
-
[19]
H. Wang, K. Sreenivasan, S. Rajput, H. Vishwakarma, S. Agarwal, J.- y. Sohn, K. Lee, and D. Papailiopoulos. Attack of the tails: Yes, you really can backdoor federated learning. Advances in Neural Information Processing Systems, 2020
2020
-
[20]
C. Xie, K. Huang, P. Y . Chen, and B. Li. DBA: Distributed backdoor attacks against federated learning. International Conference on Learning Representations, 2020
2020
-
[21]
Yurochkin, M
M. Yurochkin, M. Agarwal, S. Ghosh, K. Greenewald, N. Hoang, and Y . Khazaeni. Bayesian nonparametric federated learning of neural networks. International conference on machine learning, pages 7252–
-
[22]
A3FL: Adversarially adaptive backdoor attacks to federated learning
Zhang H, Jia J, Chen J, Lin L, Wu D. A3FL: Adversarially adaptive backdoor attacks to federated learning. Advances in Neural Information Processing Systems, 2024
2024
-
[23]
Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms
Xiao, Han, Kashif Rasul, and Roland V ollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 (2017)
2017 arXiv
-
[24]
Y ., and Brinton, C
Yuan, L., Wang, Z., Sun, L., Philip, S. Y ., and Brinton, C. G. (2024). De- centralized federated learning: A survey and perspective. IEEE Internet of Things Journal
2024
-
[25]
and Fernandes, N.C., 2023
Neto, H.N.C., Hribar, J., Dusparic, I., Mattos, D.M.F. and Fernandes, N.C., 2023. A survey on securing federated learning: Analysis of applications, attacks, challenges, and trends. IEEE Access, 11, pp.41928- 41953
2023
-
[26]
”Anti-distillation backdoor attacks: Backdoors can really survive in knowledge distillation.” In Proceedings of the 29th ACM International Conference on Multimedia, pp
Ge, Yunjie, Qian Wang, Baolin Zheng, Xinlu Zhuang, Qi Li, Chao Shen, and Cong Wang. ”Anti-distillation backdoor attacks: Backdoors can really survive in knowledge distillation.” In Proceedings of the 29th ACM International Conference on Multimedia, pp. 826-834. 2021
2021
-
[27]
Peva Blanchard, El Mahdi El Mhamdi, Rachid Guerraoui, and Julien Stainer, ‘Machine learning with adversaries: Byzantine tolerant gradi- ent descent’, Advances in Neural Information Processing Systems, 30, (2017)
2017
-
[28]
5650– 5659
Dong Yin, Yudong Chen, Ramchandran Kannan, and Peter Bartlett, ‘Byzantine-robust distributed learning: Towards optimal statistical rates’, in International Conference on Machine Learning, pp. 5650– 5659. PMLR, (2018)
2018
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.