REVIEW 4 major objections 6 minor 35 references
Heterogeneity-Oblivious Robust Federated Learning
T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Horus claims the first robust federated learning framework for hyper-heterogeneous clients, using only LoRA-A spectral features for poisoning detection and projection-guided LoRA aggregation.
desk verdict Promising LoRA-A stability insight, but the detection rule as written cannot remove the stated 20% attackers with 10 clients; needs revision before the results can be fully trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the LoRA adapter, a low-rank update $\Delta W = BA$ wrapped around a frozen backbone layer, with $A$ the input projection and $B$ the output projection. LoRA functions as an architecture-agnostic plug-in, so clients with different backbones, such as CNN versus RNN, can exchange updates of consistent low-rank form. Horus inserts LoRAs only in the first feature layer and classifier, the two layers whose top-$k$ singular value energy ratio is empirically stable across clients. Detection uses only LoRA-A's singular values via HOPS, a shape-agnostic score combining spectral entropy and top-$k$ energy ratio with round-wise normalization; aggregation uses masked zero-padding to align shapes and projection-guided weighting by the inner product of each client's leading right singular vector with the previous global direction.
What would settle it
Run Horus with an adaptive adversary that optimizes poisoned clients' LoRA-A matrices so their spectral entropy and top-k energy ratio match the benign round-wise statistics; if HOPS scores then fail to separate poisoned from benign clients, the detection core collapses. A simpler check: measure the top-5 energy ratio of LoRA-A across many heterogeneous clients and attacks; if benign-client variance overlaps the poisoned-client shift on a new architecture or attack family, the method's detection margin is not general.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that the input projection of a LoRA adapter, LoRA-A, is markedly more stable than the output projection LoRA-B under hyper-heterogeneity and poisoning, and its singular-value spectrum separates poisoned from benign clients cleanly enough to drive both detection and aggregation. Based on this, Horus builds a Heterogeneity-Oblivious Poisoning Score (HOPS) from spectral entropy and top-k energy ratio of LoRA-A singular values, flags clients whose scores deviate from round-wise percentiles, and then aggregates only LoRA updates from benign clients, masked and zero-padded to a common shape and weighted by projection onto the global dominant direction. The paper positions this as the first robust FL framework for hyper-heterogeneous settings and supports it with experiments where Horus attains the best global accuracy in 49 of 54 dataset-attack-heterogeneity configurations, with the remaining five as runner-up.
Load-bearing premise
The paper's method stands or falls on the empirical claim that the input-side part of the low-rank adapter stays stable and tells poisoned clients apart from benign ones across all the heterogeneity the framework allows, a claim illustrated with a few example curves rather than proved or stress-tested.
Editorial extensions
If this is right
- Robust federated learning can work across clients with fundamentally different architectures, such as CNN versus RNN, and imbalanced communication, because only low-rank LoRA updates are exchanged.
- Poisoning detection becomes architecture- and shape-agnostic: HOPS uses only singular values of LoRA-A, so clients with different layer shapes can be compared in one score.
- Aggregation can suppress adversarial drift while preserving benign diversity by reweighting updates by their projection onto the previous global direction.
- Per-round communication payload drops to the sub-megabyte range, with up to a 420x reduction versus full-model heterogeneous baselines, without sacrificing accuracy.
- Using LoRA-A alone for detection outperforms using LoRA-B or both, because the less stable B introduces false positives.
Reading between the lines
- The paper leaves open whether LoRA-A stability transfers to transformer-based clients or cross-modal architectures; that is a testable claim, not one the experiments cover.
- Because HOPS normalizes against the current round's client statistics, a coordinated set of attackers that shifts the round-wise mean could anchor the score; the paper does not analyze this collusion regime.
- Projection-guided weighting relies on the previous global direction; if that direction is corrupted early, the weighting could amplify rather than suppress drift, a failure mode the paper does not address.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Horus, a federated learning framework for hyper-heterogeneous clients (varying data distributions, architectures, and communication capabilities) that is intended to be robust to poisoning attacks. Clients insert LoRAs into two empirically stable layers, share only the LoRA-A and LoRA-B matrices, and the server computes a Heterogeneity-Oblivious Poisoning Score (HOPS) from spectral features of LoRA-A to filter poisoned clients. Remaining updates are aligned via zero-padding and aggregated with projection-guided weights based on alignment with global principal directions. The authors report that Horus outperforms state-of-the-art robust and heterogeneous FL baselines in 49 of 54 dataset/attack/heterogeneity settings.
Significance. If the claims hold, the paper addresses a real gap: robust FL under simultaneous data, architecture, and communication heterogeneity. The core design—decoupling LoRA-A from LoRA-B and using shape-agnostic spectral features for detection—is novel and potentially useful in practical federated systems. The ablations are informative and support the value of layer selection, LoRA-A-only detection, and projection-guided weighting. However, the central empirical premise (stability and discriminative power of LoRA-A) is supported mainly by example curves, and the quantitative comparisons lack error bars, so the strength of the claimed advantage is not yet established.
major comments (4)
- [§V-B2, Algorithm 1, and §VI-A] The stated detection rule cannot remove the stated attack fraction. The text sets the threshold as θ_t = Percentile_p({S_c}) with p=95, and §VI-A states that 20% of clients are poisoned each round; with N=10, that is two poisoned clients. For any 10 scores, the 95th percentile lies strictly between the 9th and 10th order statistics, so the condition S_c > θ_t flags at most the unique maximum score. Therefore one of the two poisoned clients survives every round, contradicting the claim that HOPS filters poisoned clients and undermining the 49/54 robustness result. Please specify the exact percentile or removal rule used in the experiments, and reconcile the pseudocode with the reported 20% attacker setting.
- [§III, Observation 2, Fig. 2] The load-bearing premise that LoRA-A remains stable and discriminative between benign and poisoned clients is supported only by a small set of example curves from one setting (CIFAR-10, LIE, α=0.5, one benign and two poisoned clients). No quantitative aggregate evidence is provided, such as the distribution of HOPS scores for benign versus poisoned clients, detection ROC/AUC, or false-positive/true-positive rates averaged over datasets, attacks, and architectures. Because the entire HOPS detector and projection-guided aggregation rest on this observation, the paper should provide a quantitative validation of the separation, or substantially temper the universality of the claim.
- [§VI-G, §VI-H, Tables I and II] The two hyperparameters that control the detection score, λ in Eq. (4) and the LoRA rank r, are tuned on the same benchmarks used for the headline comparisons, and all reported accuracy numbers appear to be single runs without error bars or significance tests. Figure 7b shows that the optimal λ differs by attack type (0.3 for dispersive, 0.7 for directional), which makes the fixed-λ results in Table I hard to interpret. Please report mean±std over multiple seeds, select λ and r on a validation split, and state the percentile p used in the robustness tables so that the claimed consistent advantage can be assessed.
- [§VI-B, §VI-C] The comparison protocol applies zero-padding dimensional alignment to all baselines before aggregation, but robust aggregation methods such as Krum, Median, and Trimmed Mean are defined for a shared parameter space and may be systematically disadvantaged when the parameter space contains many padded, meaningless dimensions. The paper should clarify whether the baseline numbers are obtained in their native homogeneous setting or after the padding protocol, and justify that the comparison does not bias the results against the baselines.
minor comments (6)
- [Algorithm 1] The pseudocode contains two nested 'for round t=1 to T do' loops (lines 1 and 8) that should be a single loop over rounds; as written, the control flow is ambiguous.
- [Table I] Several entries in the FMNIST block run together without separators (e.g., '60.8451.68' and '47.9753.66'), making the table hard to read; please insert proper delimiters.
- [§VI-A] The sentence '20% of clients are selected in each round to perform poisoning attacks' is ambiguous: it should state whether 20% of the 10 participating clients (i.e., 2 clients) are poisoned in every round.
- [§V-B2 and §VI-H] The paper says the choice of k=5 'follows the paper [9]', but reference [9] is the Eckart-Young theorem, which does not specify a value of k; please either correct the citation or justify k=5 directly.
- [Throughout] There are several naming inconsistencies, including 'FLDetecter' vs 'FLDetector', 'Dnc' vs 'DnC', and 'FedHeLLo' vs 'FedHello'; please standardize.
- [Fig. 7b] The caption of Fig. 7b does not identify which curve corresponds to the dispersive attack and which to the directional attack, and the text refers to 'CIFAR10-LIE-min-max' as a single example; please make the figure legend and text consistent.
Circularity Check
No significant circularity: Horus's claims are benchmark-empirical designs, not definitional reductions.
full rationale
No circular derivation is present. Horus is an empirical framework: the LoRA-A stability premise (Sec. III, Obs. 2) is asserted from motivating experiments, and the HOPS score (Eq. 4) plus projection-guided aggregation (Eqs. 8-9) are then validated against external benchmarks (three datasets, six attacks, eight robust-FL baselines) in Table I; the headline 49/54 result is not obtained by substituting the paper's definitions into one another. The paper contains no load-bearing self-citations: references [15], [9], [24], etc. are external and not by the present authors, and no 'uniqueness' result is imported from the authors' prior work. Parameter choices (r=8 via Fig. 7a, lambda via Sec. VI-H, k=5 from [9]) are hyperparameter selections reported with sensitivity analysis, not fitted quantities that make the target accuracy an identity; notably the paper explicitly says Horus 'remains consistently above baselines' across lambda and leaves adaptive lambda to future work. The reviewer's percentile observation (Algorithm 1, p=95, N=10, 20% attackers) is a logical mismatch between the specified threshold and the claimed two-client removal; that is a correctness/implementation issue, not a circularity pattern, so it does not raise the score. Hence no step of the paper's argument reduces by construction to its own inputs.
Assumptions & free parameters
free parameters (4)
- HOPS weight lambda =
0.3 for dispersive attacks, 0.7 for directional attacks
- LoRA rank r =
8 (best at 16, 0.35% higher)
- top-k for energy ratio, k =
5
- Detection percentile p =
95
assumptions (5)
- domain assumption LoRA incremental updates Delta W = BA with small rank r preserve the collaborative learning signal.
- domain assumption LoRA-A singular-value spectrum is stable under heterogeneity and poisoning and separates benign from poisoned clients.
- domain assumption The first feature layer and the classifier are the stable, semantically universal layers across heterogeneous architectures.
- domain assumption Zero-padding to the maximum shape and masked averaging preserves update semantics across architectures.
- domain assumption The first right singular vector of a padded LoRA is a meaningful global direction for reweighting.
Cite this review
Pith. "Pith review of Heterogeneity-Oblivious Robust Federated Learning." pith.science (2026). https://pith.science/paper/LNNDYNSZ
@misc{pith2026250803579,
author = {Pith},
title = {Pith review of: Heterogeneity-Oblivious Robust Federated Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/LNNDYNSZ}},
note = {Machine review of arXiv:2508.03579}
}
read the original abstract
Federated Learning (FL) remains highly vulnerable to poisoning attacks, especially under real-world hyper-heterogeneity, where clients differ significantly in data distributions, communication capabilities, and model architectures. Such heterogeneity not only undermines the effectiveness of aggregation strategies but also makes attacks more difficult to detect. Furthermore, high-dimensional models expand the attack surface. To address these challenges, we propose Horus, a heterogeneity-oblivious robust FL framework centered on low-rank adaptations (LoRAs). Rather than aggregating full model parameters, Horus inserts LoRAs into empirically stable layers and aggregates only LoRAs to reduce the attack uncover a key empirical observation that the input projection (LoRA-A) is markedly more stable than the output projection (LoRA-B) under heterogeneity and poisoning. Leveraging this, we design a Heterogeneity-Oblivious Poisoning Score using the features from LoRA-A to filter poisoned clients. For the remaining benign clients, we propose projection-aware aggregation mechanism to preserve collaborative signals while suppressing drifts, which reweights client updates by consistency with the global directions. Extensive experiments across diverse datasets, model architectures, and attacks demonstrate that Horus consistently outperforms state-of-the-art baselines in both robustness and accuracy.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
S. Alam, L. Liu, M. Yan, and M. Zhang. Fedrolex: Model-heterogeneous federated learning with rolling sub-model extraction. InAdvances in Neural Information Processing Systems, 2022
work page 2022
-
[2]
D. Alistarh, D. Grubic, J. Li, R. Tomioka, and M. V ojnovic. QSGD: communication-efficient SGD via gradient quantization and encoding. In I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V . N. Vishwanathan, and R. Garnett, editors,Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Sys...
work page 2017
-
[3]
E. Bagdasaryan, A. Veit, Y . Hua, D. Estrin, and V . Shmatikov. How to backdoor federated learning. InInternational conference on artificial intelligence and statistics, pages 2938–2948. PMLR, 2020
work page 2020
- [4]
-
[5]
H. S. Behl, A. G. Baydin, and P. H. S. Torr. Alpha MAML: adaptive model-agnostic meta-learning.CoRR, abs/1905.07435, 2019
arXiv 1905
-
[6]
P. Blanchard, E. M. El Mhamdi, R. Guerraoui, and J. Stainer. Machine learning with adversaries: Byzantine tolerant gradient descent. In NeurIPS, 2017
work page 2017
-
[7]
Z. Chen et al. Robustfl: Robust federated learning via subspace aggregation.IEEE Transactions on Neural Networks and Learning Systems, 2023
work page 2023
-
[8]
E. Diao, J. Ding, and V . Tarokh. Heterofl: Computation and com- munication efficient federated learning for heterogeneous clients. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021
work page 2021
Show all 35 references
-
[9]
Eckart and G
C. Eckart and G. Young. The approximation of one matrix by another of lower rank.Psychometrika, 1(3):211–218, 1936
1936
-
[10]
E. M. El Mhamdi, R. Guerraoui, and S. Rouault. The hidden vulnera- bility of distributed learning in byzantium. InInternational Conference on Machine Learning (ICML), pages 3521–3530, 2018
2018
-
[11]
M. Fang, X. Cao, J. Jia, and N. Gong. Local model poisoning attacks to{Byzantine-Robust}federated learning. In29th USENIX security symposium (USENIX Security 20), pages 1605–1622, 2020
2020
-
[12]
Z. Gao, Z. Zhang, Y . Guo, and Y . Gong. Federated adaptive fine- tuning of large language models with heterogeneous quantization and lora. InIEEE INFOCOM 2025 - IEEE Conference on Computer Communications, London, United Kingdom, May 19-22, 2025, pages 1–10. IEEE, 2025
2025
-
[13]
Gijsbers, F
P. Gijsbers, F. Pfisterer, J. N. van Rijn, B. Bischl, and J. Vanschoren. Meta-learning for symbolic hyperparameter defaults. In K. Krawiec, editor,GECCO ’21: Genetic and Evolutionary Computation Conference, Companion Volume, Lille, France, July 10-14, 2021, pages 151–152. ACM, 2021
2021
-
[14]
Horv ´ath, S
S. Horv ´ath, S. Laskaridis, M. Almeida, I. Leontiadis, S. I. Venieris, and N. D. Lane. Fjord: Fair and accurate federated learning under heterogeneous targets with ordered dropout. In M. Ranzato, A. Beygelz- imer, Y . N. Dauphin, P. Liang, and J. W. Vaughan, editors,Advances ...
2021
-
[15]
E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, W. Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022
2022
-
[16]
Ivkin, D
N. Ivkin, D. Rothchild, E. Ullah, V . Braverman, I. Stoica, and R. Arora. Communication-efficient distributed SGD with sketching. In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alch´e-Buc, E. B. Fox, and R. Garnett, editors,Advances in Neural Information Processing Syst...
2019
-
[17]
Krizhevsky
A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009
2009
-
[18]
Li and J
D. Li and J. Wang. Fedmd: Heterogenous federated learning via model distillation.CoRR, abs/1910.03581, 2019
1910 arXiv
-
[19]
Q. Li, X. Wen, N. Z. Gong, et al. Federated learning with adaptive model aggregation. InNeurIPS, 2021
2021
-
[20]
T. Li, A. K. Sahu, A. Talwalkar, and V . Smith. Federated learning: Challenges, methods, and future directions.IEEE Signal Processing Magazine, 37(3):50–60, 2020
2020
-
[21]
Y . Liu, J. Zhao, Q. Liu, Y . Dou, and X. Hu. Fml: Federated learning with flexible model layering. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 10736–10744, 2024
2024
-
[22]
McMahan, E
B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas. Communication-efficient learning of deep networks from decentralized data. InArtificial intelligence and statistics, pages 1273–1282. PMLR, 2017
2017
-
[23]
Mitliagkas, C
I. Mitliagkas, C. Zhang, S. Hadjis, and C. R ´e. Asynchrony begets mo- mentum, with an application to deep learning. In54th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2016, Monticello, IL, USA, September 27-30, 2016, pages 997–1004. IEEE, 2016
2016
-
[24]
Shejwalkar and A
V . Shejwalkar and A. Houmansadr. Manipulating the byzantine: Opti- mizing model poisoning attacks and defenses for federated learning. In NDSS, 2021
2021
-
[25]
Steinhardt, P
J. Steinhardt, P. W. Koh, and P. Liang. Certified defenses for data poisoning attacks. InNeurIPS, 2017
2017
-
[26]
Visin, K
F. Visin, K. Kastner, K. Cho, M. Matteucci, A. Courville, and Y . Bengio. Renet: A recurrent neural network based alternative to convolutional networks.arXiv preprint arXiv:1505.00393, 2015
2015 arXiv
-
[27]
G. Wan, W. Huang, and M. Ye. Federated graph learning under domain shift with generalizable prototypes. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 15429–15437, 2024
2024
-
[28]
H. Xiao, K. Rasul, and R. V ollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms.arXiv preprint arXiv:1708.07747, 2017
2017 arXiv
-
[29]
J. Xu, Z. Zhang, and R. Hu. Achieving byzantine-resilient federated learning via layer-adaptive sparsified model aggregation. InIEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2025, Tucson, AZ, USA, February 26 - March 6, 2025, pages 1508–1517. IEEE, 2025
2025
-
[30]
J. Xu, Z. Zhang, and R. Hu. Achieving byzantine-resilient federated learning via layer-adaptive sparsified model aggregation. InProceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1508–1517, 2025
2025
-
[31]
D. Yin, Y . Chen, K. Ramchandran, and P. Bartlett. Byzantine-robust distributed learning: Towards optimal statistical rates. InICML, 2018
2018
-
[32]
Yu and P
R. Yu and P. Li. Toward resource-efficient federated learning in mobile edge computing.IEEE Netw., 35(1):148–155, 2021
2021
-
[33]
Zhang, Y
J. Zhang, Y . Hua, H. Wang, T. Song, Z. Xue, R. Ma, and H. Guan. Fedala: Adaptive local aggregation for personalized federated learning. InProceedings of the AAAI conference on artificial intelligence, vol- ume 37, pages 11237–11244, 2023
2023
-
[34]
Zhang, X
Z. Zhang, X. Cao, J. Jia, and N. Z. Gong. Fldetector: Defending federated learning against model poisoning attacks via detecting ma- licious clients. InProceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining, pages 2545–2555, 2022
2022
-
[35]
Zhang, P
Z. Zhang, P. Liu, J. Xu, and R. Hu. Fed-hello: Efficient federated foundation model fine-tuning with heterogeneous lora allocation.CoRR, abs/2506.12213, 2025
2025 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.