Pith. sign in

REVIEW 5 major objections 4 minor 106 references

Accelerated Training of Federated Learning via Second-Order Methods

T0 review · 5 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that second-order optimization—using Hessian curvature—generally accelerates federated learning, reducing communication rounds and improving accuracy, and organizes the field into six method categories.

desk verdict A useful survey map of second-order FL methods, but the blanket 'generally improve' claim is unsupported by the survey's own evidence. read the letter →

arxiv 2505.23588 v1 pith:4EBXRANI submitted 2025-05-29 cs.LG cs.DC

Mrinmay Sen , Sidhant R Nair , C Krishna Mohan This is my paper
classification cs.LGcs.DC
keywords federatedlearningsecond-orderoptimizationHessianmatrixNewtonmethodcommunicationefficiencynon-IIDdatasurvey
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Federated learning's main practical bottleneck is slow training of the global model: first-order methods such as FedAvg need many communication rounds, especially when client data is heterogeneous. This survey claims that second-order methods, which scale updates by curvature information from the Hessian rather than by a fixed learning rate, generally converge faster and reach higher test accuracy in fewer rounds. To make that claim usable, the paper proposes a taxonomy of six categories based on how the Hessian is obtained or approximated, and compares their computation, memory, transmission, and convergence profiles. The paper's conclusion is that curvature information is highly beneficial for FL training, provided the Hessian and its inverse are handled efficiently, and it positions the taxonomy as a selection guide for researchers.

What carries the argument

The load-bearing object is the Hessian matrix $H = \nabla^2 F(x)$ and the Newton update $x \leftarrow x - H^{-1}g$, which uses curvature to choose both direction and step size. Because computing, storing, inverting, and transmitting the full Hessian is prohibitive—$O(d^2)$ computation and memory per client and $O(d^3)$ for inversion—and because averaging local inverse Hessians does not equal the inverse of the global Hessian, the survey's taxonomy classifies methods by their approximation strategy. Each category is a different answer to the question 'where does curvature come from and who computes it': Hessian-vector products, compressed full Hessians, server-side BFGS, diagonal vectors, Nyström sketches, or rank-one matrices. The categories carry the argument because the survey claims that performance differences track these choices of curvature approximation and placement.

What would settle it

Run one controlled benchmark with identical data partitions, model architectures, and per-method tuning budgets on non-IID, partial-participation tasks, comparing FedAvg against representatives from each category (for example DANE, FedNL, FedSSO, and FAGH); if FedAvg matches or beats them in communication rounds and final test accuracy, the survey's central conclusion fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that moving from the first-order update $x \leftarrow x - \eta \nabla F(x)$ to a Newton-type update $x \leftarrow x - H^{-1}\nabla F(x)$ lets the global model follow the curvature of the objective and reach the optimum with quadratic convergence. In federated settings this translates into fewer communication rounds or less wall-clock time to hit a target loss or accuracy. The survey groups current methods into six categories by how they obtain curvature: Hessian-free methods such as DANE, GIANT, and DONE; full local Hessian computation with compression (FedNL, SHED, FedNS); quasi-Newton methods (FedSSO, DQN-Fed); Hessian diagonal approximations (FopLAHD, LTDA, HWA, Fed-Sophia); Nyström-based approximations (FONN, Nys-FL); and one-rank approximations (FReNG, FAGH). Across the empirical results collected from these methods' papers, the survey observes that second-order methods generally reduce classification error, training loss, and test loss and increase test accuracy faster than first-order baselines, with the known exception of FedDANE, which underperforms under non-IID data and partial participation.

Load-bearing premise

The empirical conclusion assumes that 'method X beats method Y' results measured on different datasets, partitions, models, and baselines can be pooled into a general statement that second-order methods are better.

Editorial extensions

If this is right

  • If the survey's conclusion is right, a practitioner facing non-IID data and partial client participation can narrow options to the diagonal, Nyström, and one-rank categories, which keep computation close to FedAvg and still accelerate convergence.
  • The full-local-Hessian and quasi-Newton families become attractive mainly when a powerful server exists, since they shift the heavy $O(d^2)$–$O(d^3)$ work to the server side.
  • The Hessian-free family's four transmissions per round may erase its convergence advantage in high-latency networks, making two-transmission methods like LocalNewton and FedSSO better fits for bandwidth-constrained settings.
  • A convergence-rate hierarchy emerges from the tables: super-linear claims for full-Hessian methods, linear-quadratic claims for GIANT-style and Nyström/one-rank methods, and sub-linear claims for FedSSO, all stated for convex objectives.
  • The taxonomy itself is the practical output: it lets researchers pick an algorithm by matching constraints rather than reading every method paper.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: the survey's pooled comparison likely overstates the generality of 'second-order beats first-order,' because each method was tested on different datasets, partitions, and baselines; a direct head-to-head benchmark is needed before the superiority claim can be trusted.
  • Inference: the theory in the surveyed papers suggests the gap between first- and second-order methods should grow as the problem's condition number worsens, so a natural controlled experiment would vary conditioning while holding data and model fixed.
  • Inference: the pattern points toward a research target the survey leaves implicit: a method that combines server-side curvature approximation with FedAvg-level transmission and partial-participation support would dominate the design space.
  • Inference: because all empirical comparisons report only target-accuracy or final-loss results at a chosen round count, the field would gain from reporting full convergence curves and per-round wall-clock time, which would make the 'fewer rounds or less time' claim testable across methods.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. This survey reviews second-order optimization methods for federated learning (FL). It groups 22 methods into six categories (Hessian-free, full local Hessian, quasi-Newton, Hessian diagonal, Nyström, and one-rank approximations), describes each method's algorithm, reported empirical behavior, and complexity, and summarizes the results in four tables. The paper's central claim, stated in Section IV-B, is that second-order methods generally improve FL training by reducing classification error, training loss, and test loss and by increasing test accuracy in fewer communication rounds or less time compared with first-order methods. The survey is positioned as filling a gap in FL survey literature that has focused on heterogeneity, privacy, and security rather than on optimization speed.

Significance. If the central claim were adequately supported, the paper would provide a useful organizational reference for researchers choosing among second-order FL methods. Its strengths are the breadth of coverage, the grouping of methods into a clear taxonomy, and the systematic reporting of local and server computation, memory, transmission, convergence, and applicability conditions in Tables I, II, and IV. The paper also usefully collects empirical claims from the original papers into a single table. However, the survey does not include new experiments or controlled comparisons, and its main empirical conclusion is drawn from heterogeneous, qualitative self-reported results. The taxonomy and tables are valuable as a catalogue, but the generalized superiority claim in Section IV-B is not supported by the evidence presented in the paper itself, and several internal inconsistencies in the method descriptions and tables need correction.

major comments (5)
  1. [Section IV-B, Tables III and IV] The central claim that 'second-order methods generally improve FL training' is contradicted by the survey's own evidence. Table III records FedDANE as underperforming FedAvg and FedProx in reducing training loss, and Table IV states that the entire Hessian-free category, which includes 7 of the 22 surveyed methods, is 'not suitable for Non-IID or partial client scenarios.' The generalization should be restricted to the specific categories and settings for which the reported evidence is consistent, or reformulated as a conditional conclusion.
  2. [Section IV-B, first paragraph] The empirical comparison is explicitly based on 'experimental information from the respective papers,' and Table III pools results across different datasets, data partitions, models, loss functions, baselines, numbers of communication rounds, and metrics. The table entries are qualitative verdicts such as 'faster reduction' with no shared unit of measurement. Such heterogeneous self-reported results cannot support a general superiority claim about second-order methods relative to first-order methods; at most they support a statement that several second-order algorithms were reported to perform well in their original papers.
  3. [Section III-A-6 (DONE)] There is an internal contradiction in the description of DONE. The main text states that DONE 'can be applicable to the non-iid data across clients,' but the Cons paragraph states that DONE is designed for a strongly convex global objective with 'homogeneous data across clients.' Table II also labels DONE as non-iid. This contradiction affects the reliability of the taxonomy and should be resolved, and the same check should be applied to other rows where the design conditions in Table II differ from the empirical conditions in Table III (for example, FedNL is listed as non-iid with FCP+PCP in Table II but as iid with FCP in Table III).
  4. [Section II-A-2, Eq. (4), and Section IV-A] The paper repeatedly attributes 'quadratic convergence' to second-order optimization, but Table II shows that most surveyed methods achieve only linear, sub-linear, or linear-quadratic convergence, and several methods (FopLAHD, LTDA, HWA, Fed-Sophia, FedNew) are listed without a stated convergence rate. The convergence discussion should distinguish the idealized Newton method from the practical approximations used in FL and should not overstate the rates that the survey itself documents.
  5. [Section III-B-1 (FedNL) versus Table I] The complexity text for FedNL reports client-to-server transmission cost O(r+d) and server-to-client transmission cost O(r), while Table I lists O(2d) and O(d), respectively. These numbers should be reconciled or the discrepancy explained, since the communication cost is one of the paper's main comparison criteria.
minor comments (4)
  1. [Throughout] There are numerous typographical and grammatical errors, including 'Exiting' for 'Existing' in Sections III-B, III-C, and III-D, 'demostrate' in Section III-A-6, 'phishig' in Section III-B-5, 'long with strong' in Section III-A-3, and 'FPC' for 'FCP' in the Table III header. A careful proofread is needed.
  2. [Title and header] The manuscript header still reads 'Manuscript received April 19, 2021; revised August 16, 2021,' which conflicts with the arXiv submission date of May 2025 and should be updated.
  3. [Reference [40]] Reference [40] is incomplete: it gives the title but no publication venue, date, or page information.
  4. [Section III-D-1] The method name is spelled inconsistently as FopLAHD, FopLADH, and FOPLAHD in different places; one standard spelling should be used throughout.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the survey aggregates independent published results; the presence of self-citations does not make the empirical claim circular.

full rationale

The paper is a survey, not a derivation: there are no fitted parameters, no theorem derived from the paper's own definitions, and no prediction generated from a model built on the target data. The central claim in Section IV-B ('second-order methods generally improve FL training...') is an inductive generalization from the surveyed papers' reported experiments, as the paper itself states: 'The empirical comparisons are made by using experimental information from the respective papers of the methods.' Several surveyed methods (FOPLAHD, LTDA, FReNG, FAGH, Nys-FL, FONN) are authored by the survey authors, which creates a mild self-interest signal, but the survey does not invoke a uniqueness theorem, an ansatz, or a definitional identity to force that conclusion. Each cited method is a separate publication with its own external benchmarks, and the survey does not use its own conclusions to validate those methods. The 'generally improve' claim is better evaluated as an evidence-quality or correctness risk: Table III includes FedDANE underperforming, and Table IV restricts the entire Hessian-free category to IID/full-participation settings, so the generalization is stronger than the pooled evidence supports. That is an empirical over-claim, not a circular reduction. Accordingly, no circular step is identified.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The survey introduces no free parameters or invented entities. Its conclusions rest on the reported results of the cited papers and on the validity of its six-way categorization of second-order FL methods.

assumptions (2)
  • domain assumption The convergence rates and empirical results attributed to each surveyed method are accurately reported in the original papers.
    The survey provides no new experiments or proofs; its conclusions depend entirely on the claims of the cited papers (Sections III-A to III-F).
  • domain assumption The six taxonomy categories (Hessian-free, full local Hessian, quasi-Newton, diagonal, Nyström, one-rank) form a valid and useful partition of second-order FL methods.
    The paper does not justify why these six categories are exhaustive or mutually exclusive, and some methods could arguably fit multiple categories.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Accelerated Training of Federated Learning via Second-Order Methods." pith.science (2026). https://pith.science/paper/4EBXRANI

@misc{pith2026250523588,
  author       = {Pith},
  title        = {Pith review of: Accelerated Training of Federated Learning via Second-Order Methods},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4EBXRANI}},
  note         = {Machine review of arXiv:2505.23588}
}
read the original abstract

This paper explores second-order optimization methods in Federated Learning (FL), addressing the critical challenges of slow convergence and the excessive communication rounds required to achieve optimal performance from the global model. While existing surveys in FL primarily focus on challenges related to statistical and device label heterogeneity, as well as privacy and security concerns in first-order FL methods, less attention has been given to the issue of slow model training. This slow training often leads to the need for excessive communication rounds or increased communication costs, particularly when data across clients are highly heterogeneous. In this paper, we examine various FL methods that leverage second-order optimization to accelerate the training process. We provide a comprehensive categorization of state-of-the-art second-order FL methods and compare their performance based on convergence speed, computational cost, memory usage, transmission overhead, and generalization of the global model. Our findings show the potential of incorporating Hessian curvature through second-order optimization into FL and highlight key challenges, such as the efficient utilization of Hessian and its inverse in FL. This work lays the groundwork for future research aimed at developing scalable and efficient federated optimization methods for improving the training of the global model in FL.

Figures

Figures reproduced from arXiv: 2505.23588 by the authors.

Figure 1
Figure 1. This figure compares first-order and second-order optimization [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

106 extracted references · 46 canonical work pages

  1. [46]

    On second- order optimization methods for federated learning,

    S. Bischoff, S. G ¨unnemann, M. Jaggi, and S. U. Stich, “On second- order optimization methods for federated learning,” arXiv preprint arXiv:2109.02388, 2021

    arXiv 2021

  2. [1]

    Federated learning of deep networks using model averaging,

    H. B. McMahan, E. Moore, D. Ramage, and B. A. y Arcas, “Federated learning of deep networks using model averaging,” arXiv preprint arXiv:1602.05629, vol. 2, no. 2, 2016

    arXiv 2016

  3. [2]

    Fedprox: Fedsplit algorithm based federated learning for statistical and system heterogeneity in medical data com- munication,

    C. Mathew and P. Asha, “Fedprox: Fedsplit algorithm based federated learning for statistical and system heterogeneity in medical data com- munication,” J. Internet Serv. Inf. Secur. , vol. 14, no. 3, pp. 353–370, 2024

    2024

  4. [3]

    SCAFFOLD: stochastic controlled averaging for federated learning,

    S. P. Karimireddy, S. Kale, M. Mohri, S. J. Reddi, S. U. Stich, and A. T. Suresh, “SCAFFOLD: stochastic controlled averaging for federated learning,” in Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , ser. Proceedings of Machine Learning Research, vol. 119. PMLR, 2020, pp. 5132–5143

    2020

  5. [4]

    Model-contrastive federated learning,

    Q. Li, B. He, and D. Song, “Model-contrastive federated learning,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 10 713–10 722

    2021

  6. [5]

    Federated learning based on dynamic regulariza- tion,

    D. A. E. Acar, Y . Zhao, R. M. Navarro, M. Mattina, P. N. Whatmough, and V . Saligrama, “Federated learning based on dynamic regulariza- tion,” in 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021

    2021

  7. [6]

    Implicit gradient alignment in distributed and federated learning,

    L. Barba, M. Jaggi, and Y . Dandi, “Implicit gradient alignment in distributed and federated learning,” in AAAI Conference on Artificial Intelligence, AAAI, vol. 22, 2021

    2021

  8. [7]

    Handling data heterogeneity in federated learning with global data distribution

    C. Nagaraju, M. Sen, C. K. Mohan, F. Imai, C. Distante, and S. Battiato, “Handling data heterogeneity in federated learning with global data distribution.” in IMPROVE, 2023, pp. 121–125. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 18

    2023

Show all 106 references
  1. [8]

    With a little help from my friend: Server-aided federated learning with partial client participation,

    H. Yang, P. Qiu, P. Khanduri, and J. Liu, “With a little help from my friend: Server-aided federated learning with partial client participation,” 2023

    2023

  2. [9]

    Achieving linear speedup with partial worker participation in non-iid federated learning,

    H. Yang, M. Fang, and J. Liu, “Achieving linear speedup with partial worker participation in non-iid federated learning,” Proceedings of ICLR, 2021

    2021

  3. [10]

    Fast federated learning in the presence of arbitrary device unavailability,

    X. Gu, K. Huang, J. Zhang, and L. Huang, “Fast federated learning in the presence of arbitrary device unavailability,” Advances in Neural Information Processing Systems , vol. 34, pp. 12 052–12 064, 2021

    2021

  4. [11]

    Anchor sampling for federated learning with partial client participation,

    F. Wu, S. Guo, Z. Qu, S. He, Z. Liu, and J. Gao, “Anchor sampling for federated learning with partial client participation,” in International Conference on Machine Learning . PMLR, 2023, pp. 37 379–37 416

    2023

  5. [12]

    Fedvarp: Tackling the variance due to partial client participation in federated learning,

    D. Jhunjhunwala, P. Sharma, A. Nagarkatti, and G. Joshi, “Fedvarp: Tackling the variance due to partial client participation in federated learning,” in Uncertainty in Artificial Intelligence . PMLR, 2022, pp. 906–916

    2022

  6. [13]

    Federated learning with differential privacy: Algorithms and performance analysis,

    K. Wei, J. Li, M. Ding, C. Ma, H. H. Yang, F. Farokhi, S. Jin, T. Q. Quek, and H. V . Poor, “Federated learning with differential privacy: Algorithms and performance analysis,” IEEE transactions on information forensics and security , vol. 15, pp. 3454–3469, 2020

    2020

  7. [14]

    Ldp-fed: Federated learning with local differential privacy,

    S. Truex, L. Liu, K.-H. Chow, M. E. Gursoy, and W. Wei, “Ldp-fed: Federated learning with local differential privacy,” in Proceedings of the third ACM international workshop on edge systems, analytics and networking, 2020, pp. 61–66

    2020

  8. [15]

    Federated learning and differential privacy for medical image analysis,

    M. Adnan, S. Kalra, J. C. Cresswell, G. W. Taylor, and H. R. Tizhoosh, “Federated learning and differential privacy for medical image analysis,” Scientific reports, vol. 12, no. 1, p. 1953, 2022

    1953

  9. [16]

    Federated learning with bayesian differ- ential privacy,

    A. Triastcyn and B. Faltings, “Federated learning with bayesian differ- ential privacy,” in 2019 IEEE International Conference on Big Data (Big Data). IEEE, 2019, pp. 2587–2596

    2019

  10. [17]

    Differential privacy meets federated learning under communication constraints,

    N. Mohammadi, J. Bai, Q. Fan, Y . Song, Y . Yi, and L. Liu, “Differential privacy meets federated learning under communication constraints,” IEEE Internet of Things Journal , vol. 9, no. 22, pp. 22 204–22 219, 2021

    2021

  11. [18]

    Attack of the tails: Yes, you really can backdoor federated learning,

    H. Wang, K. Sreenivasan, S. Rajput, H. Vishwakarma, S. Agarwal, J.-y. Sohn, K. Lee, and D. Papailiopoulos, “Attack of the tails: Yes, you really can backdoor federated learning,” Advances in neural information processing systems , vol. 33, pp. 16 070–16 084, 2020

    2020

  12. [19]

    Exploring adversarial attacks in federated learning for medical imaging,

    E. Darzi, F. Dubost, N. M. Sijtsema, and P. M. van Ooijen, “Exploring adversarial attacks in federated learning for medical imaging,” IEEE Transactions on Industrial Informatics , 2024

    2024

  13. [20]

    Analyzing user-level privacy attack against federated learning,

    M. Song, Z. Wang, Z. Zhang, Y . Song, Q. Wang, J. Ren, and H. Qi, “Analyzing user-level privacy attack against federated learning,” IEEE Journal on Selected Areas in Communications , vol. 38, no. 10, pp. 2430–2444, 2020

    2020

  14. [21]

    Toward federated learning models resistant to adversarial attacks,

    F. Hu, W. Zhou, K. Liao, H. Li, and D. Tong, “Toward federated learning models resistant to adversarial attacks,” IEEE Internet of Things Journal, vol. 10, no. 19, pp. 16 917–16 930, 2023

    2023

  15. [22]

    Zero knowledge clustering based adversarial mitigation in heterogeneous federated learning,

    Z. Chen, P. Tian, W. Liao, and W. Yu, “Zero knowledge clustering based adversarial mitigation in heterogeneous federated learning,”IEEE Transactions on Network Science and Engineering , vol. 8, no. 2, pp. 1070–1083, 2020

    2020

  16. [23]

    Giant: Globally improved approximate newton method for distributed optimization,

    S. Wang, F. Roosta, P. Xu, and M. W. Mahoney, “Giant: Globally improved approximate newton method for distributed optimization,” Advances in Neural Information Processing Systems , vol. 31, 2018

    2018

  17. [24]

    Localnewton: Reducing communication bottleneck for distributed learning,

    V . Gupta, A. Ghosh, M. Derezinski, R. Khanna, K. Ramchandran, and M. Mahoney, “Localnewton: Reducing communication bottleneck for distributed learning,” arXiv preprint arXiv:2105.07320 , 2021

    2021 arXiv

  18. [25]

    Fedsso: A federated server-side second-order optimization algorithm,

    X. Ma, R. Bao, J. Jiang, Y . Liu, A. Jiang, J. Yan, X. Liu, and Z. Pan, “Fedsso: A federated server-side second-order optimization algorithm,” 2022. [Online]. Available: https://arxiv.org/abs/2206.09576

    2022 arXiv

  19. [26]

    Communication-efficient dis- tributed optimization using an approximate newton-type method,

    O. Shamir, N. Srebro, and T. Zhang, “Communication-efficient dis- tributed optimization using an approximate newton-type method,” in Proceedings of the 31th International Conference on Machine Learning, ICML 2014, Beijing, China, 21-26 June 2014 , ser. JMLR Workshop and Confer...

    2014

  20. [27]

    Fednl: Making newton-type methods applicable to federated learning,

    M. Safaryan, R. Islamov, X. Qian, and P. Richt ´arik, “Fednl: Making newton-type methods applicable to federated learning,” in International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , ser. Proceedings of Machine Learning Research, K....

    2022

  21. [28]

    Nys-fl: A communication efficient federated learning with nystr ¨om approximated global newton direc- tion,

    M. Sen, C. K. Mohan, and K. Qin, “Nys-fl: A communication efficient federated learning with nystr ¨om approximated global newton direc- tion,” in 2023 IEEE International Conference on High Performance Computing & Communications, Data Science & Systems, Smart City & Dependabili...

    2023

  22. [29]

    Fonn: Federated optimization with nys-newton,

    C. Nagaraju, M. Sen, and C. K. Mohan, “Fonn: Federated optimization with nys-newton,” in TENCON 2023-2023 IEEE Region 10 Conference (TENCON). IEEE, 2023, pp. 530–534

    2023

  23. [30]

    Over-the-air federated learning via second-order optimization,

    P. Yang, Y . Jiang, T. Wang, Y . Zhou, Y . Shi, and C. N. Jones, “Over-the-air federated learning via second-order optimization,” IEEE transactions on wireless communications , vol. 21, no. 12, pp. 10 560– 10 575, 2022

    2022

  24. [31]

    Foplahd: Federated optimization using locally approximated hessian diagonal,

    M. Sen and C. K. Mohan, “Foplahd: Federated optimization using locally approximated hessian diagonal,” in International Conference on Big Data Analytics . Springer, 2023, pp. 235–245

    2023

  25. [32]

    Done: distributed approximate newton-type method for federated edge learning,

    C. T. Dinh, N. H. Tran, T. D. Nguyen, W. Bao, A. R. Balef, B. B. Zhou, and A. Y . Zomaya, “Done: distributed approximate newton-type method for federated edge learning,” IEEE Transactions on Parallel and Distributed Systems , vol. 33, no. 11, pp. 2648–2660, 2022

    2022

  26. [33]

    Freng: Federated optimization by using regularized natural gradient descent,

    M. Sen and C. Gayatri, “Freng: Federated optimization by using regularized natural gradient descent,” in 2023 International Conference on Machine Learning and Applications (ICMLA) . IEEE, 2023, pp. 1889–1894

    2023

  27. [34]

    Federated learning review: Fundamentals, enabling technologies, and future applications,

    S. Banabilah, M. Aloqaily, E. Alsayed, N. Malik, and Y . Jararweh, “Federated learning review: Fundamentals, enabling technologies, and future applications,” Inf. Process. Manag. , vol. 59, no. 6, p. 103061, 2022

    2022

  28. [35]

    The impact of adversarial attacks on federated learning: A survey,

    K. N. Kumar, C. K. Mohan, and L. R. Cenkeramaddi, “The impact of adversarial attacks on federated learning: A survey,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 5, pp. 2672– 2691, 2023

    2023

  29. [36]

    A systematic survey for differential privacy techniques in federated learning,

    Y . Zhang, Y . Lu, and F. Liu, “A systematic survey for differential privacy techniques in federated learning,” Journal of Information Security, vol. 14, no. 2, pp. 111–135, 2023

    2023

  30. [37]

    Differential privacy federated learning: A comprehensive review

    F. Shan, S. Mao, Y . Lu, and S. Li, “Differential privacy federated learning: A comprehensive review.” International Journal of Advanced Computer Science & Applications , vol. 15, no. 7, 2024

    2024

  31. [38]

    Differentially private federated learning: A systematic review,

    J. Fu, Y . Hong, X. Ling, L. Wang, X. Ran, Z. Sun, W. H. Wang, Z. Chen, and Y . Cao, “Differentially private federated learning: A systematic review,” arXiv preprint arXiv:2405.08299 , 2024

    2024

  32. [39]

    A survey of security threats in federated learning,

    Y . Feng, Y . Guo, Y . Hou, Y . Wu, M. Lao, T. Yu, and G. Liu, “A survey of security threats in federated learning,” Complex & Intelligent Systems, vol. 11, no. 2, pp. 1–26, 2025

    2025

  33. [40]

    Challenges, applications and design aspects of federated learning: A survey

    N. AKHTER, M. HASAN, R. AMIN, and K. E. AZIZ, “Challenges, applications and design aspects of federated learning: A survey.”

  34. [41]

    Federated learning on non-iid data: A survey,

    H. Zhu, J. Xu, S. Liu, and Y . Jin, “Federated learning on non-iid data: A survey,” Neurocomputing, vol. 465, pp. 371–390, 2021

    2021

  35. [42]

    Federated learning with non-iid data: A survey,

    Z. Lu, H. Pan, Y . Dai, X. Si, and Y . Zhang, “Federated learning with non-iid data: A survey,” IEEE Internet of Things Journal , 2024

    2024

  36. [43]

    A survey of federated learning on non-iid data,

    X. Han, M. Gao, L. Wang, Z. He, and Y . Wang, “A survey of federated learning on non-iid data,” ZTE Communications, vol. 20, no. 3, p. 17, 2022

    2022

  37. [44]

    Federated learning for generalization, robustness, fairness: A survey and benchmark,

    W. Huang, M. Ye, Z. Shi, G. Wan, H. Li, B. Du, and Q. Yang, “Federated learning for generalization, robustness, fairness: A survey and benchmark,” IEEE Trans. Pattern Anal. Mach. Intell. , vol. 46, no. 12, pp. 9387–9406, 2024

    2024

  38. [45]

    Federated learning with non-iid data,

    Y . Zhao, M. Li, L. Lai, N. Suda, D. Civin, and V . Chandra, “Federated learning with non-iid data,” arXiv preprint arXiv:1806.00582 , 2018

    2018 arXiv

  39. [47]

    Review of second-order optimization techniques in artificial neural networks backpropagation,

    H. H. Tan and K. H. Lim, “Review of second-order optimization techniques in artificial neural networks backpropagation,” in IOP conference series: materials science and engineering , vol. 495, no. 1. IOP Publishing, 2019, p. 012003

    2019

  40. [48]

    A survey of deep learning optimizers–first and second order methods,

    R. Kashyap, “A survey of deep learning optimizers–first and second order methods,” arXiv preprint arXiv:2211.15596 , 2022

    2022 arXiv

  41. [49]

    A state-of-the-art survey on solving non-iid data in federated learning,

    X. Ma, J. Zhu, Z. Lin, S. Chen, and Y . Qin, “A state-of-the-art survey on solving non-iid data in federated learning,” Future Gener. Comput. Syst., vol. 135, pp. 244–258, 2022

    2022

  42. [50]

    Stochastic gradient descent,

    N. Ketkar and N. Ketkar, “Stochastic gradient descent,” Deep learning with Python: A hands-on introduction , pp. 113–132, 2017

    2017

  43. [51]

    Nys-newton: Nystr \

    D. Singh, H. Tankaria, and M. Yamada, “Nys-newton: Nystr \” om- approximated curvature for stochastic optimization,” arXiv preprint arXiv:2110.08577, 2021

    2021 arXiv

  44. [52]

    Second-order stochastic optimization for machine learning in linear time,

    N. Agarwal, B. Bullins, and E. Hazan, “Second-order stochastic optimization for machine learning in linear time,” Journal of Machine Learning Research, vol. 18, no. 116, pp. 1–40, 2017. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 19

    2017

  45. [53]

    Distributed estimation of the inverse hessian by determinantal averaging,

    M. Derezinski and M. W. Mahoney, “Distributed estimation of the inverse hessian by determinantal averaging,” in Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8- 14, 2019, Vancouver,...

    2019

  46. [54]

    Deep learning via hessian-free optimization

    J. Martens et al., “Deep learning via hessian-free optimization.” in Icml, vol. 27, 2010, pp. 735–742

    2010

  47. [55]

    Hessian-free optimization for learning deep multidimensional recurrent neural networks,

    M. Cho, C. Dhir, and J. Lee, “Hessian-free optimization for learning deep multidimensional recurrent neural networks,” Advances in Neural Information Processing Systems , vol. 28, 2015

    2015

  48. [56]

    Communication-efficient dis- tributed optimization using an approximate newton-type method,

    O. Shamir, N. Srebro, and T. Zhang, “Communication-efficient dis- tributed optimization using an approximate newton-type method,” in International conference on machine learning . PMLR, 2014, pp. 1000–1008

    2014

  49. [57]

    Disco: Distributed optimization for self- concordant empirical loss,

    Y . Zhang and X. Lin, “Disco: Distributed optimization for self- concordant empirical loss,” in International conference on machine learning. PMLR, 2015, pp. 362–370

    2015

  50. [58]

    Aide: Fast and communication efficient distributed optimization,

    S. J. Reddi, J. Kone ˇcn`y, P. Richt ´arik, B. P ´ocz´os, and A. Smola, “Aide: Fast and communication efficient distributed optimization,” arXiv preprint arXiv:1608.06879 , 2016

    2016 arXiv

  51. [60]

    Stochastic dual coordinate ascent methods for regularized loss,

    S. Shalev-Shwartz and T. Zhang, “Stochastic dual coordinate ascent methods for regularized loss,” The Journal of Machine Learning Research, vol. 14, no. 1, pp. 567–599, 2013

    2013

  52. [61]

    Gradient-based learning applied to document recognition,

    Y . LeCun, L. Bottou, Y . Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE , vol. 86, no. 11, pp. 2278–2324, 1998

    1998

  53. [62]

    Parallelized stochastic gradient descent,

    M. Zinkevich, M. Weimer, L. Li, and A. Smola, “Parallelized stochastic gradient descent,” Advances in neural information processing systems , vol. 23, 2010

    2010

  54. [63]

    Distributed optimization and statistical learning via the alternating direction method of multipliers,

    S. Boyd, N. Parikh, E. Chu, B. Peleato, J. Eckstein et al., “Distributed optimization and statistical learning via the alternating direction method of multipliers,” Foundations and Trends® in Machine learning , vol. 3, no. 1, pp. 1–122, 2011

    2011

  55. [64]

    Quartz: Randomized dual coordi- nate ascent with arbitrary sampling,

    Z. Qu, P. Richt ´arik, and T. Zhang, “Quartz: Randomized dual coordi- nate ascent with arbitrary sampling,” Advances in neural information processing systems, vol. 28, 2015

    2015

  56. [65]

    A universal catalyst for first-order optimization,

    H. Lin, J. Mairal, and Z. Harchaoui, “A universal catalyst for first-order optimization,” Advances in neural information processing systems , vol. 28, 2015

    2015

  57. [66]

    Libsvm: a library for support vector machines,

    C.-C. Chang and C.-J. Lin, “Libsvm: a library for support vector machines,” ACM transactions on intelligent systems and technology (TIST), vol. 2, no. 3, pp. 1–27, 2011

    2011

  58. [67]

    Distributed optimization with arbitrary local solvers: Co- coa+ and beyond,

    M. Tak ´ac, “Distributed optimization with arbitrary local solvers: Co- coa+ and beyond,” 2016

    2016

  59. [68]

    Leaf: A benchmark for federated settings,

    S. Caldas, S. M. K. Duddu, P. Wu, T. Li, J. Kone ˇcn`y, H. B. McMahan, V . Smith, and A. Talwalkar, “Leaf: A benchmark for federated settings,” arXiv preprint arXiv:1812.01097 , 2018

    2018 arXiv

  60. [69]

    Federated optimization in heterogeneous networks,

    T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V . Smith, “Federated optimization in heterogeneous networks,” Proceedings of Machine learning and systems , vol. 2, pp. 429–450, 2020

    2020

  61. [70]

    G. H. Golub and C. F. Van Loan, Matrix computations. JHU press, 2013

    2013

  62. [71]

    Nesterov, Introductory lectures on convex optimization: A basic course

    Y . Nesterov, Introductory lectures on convex optimization: A basic course. Springer Science & Business Media, 2013, vol. 87

    2013

  63. [72]

    On the limited memory bfgs method for large scale optimization,

    D. C. Liu and J. Nocedal, “On the limited memory bfgs method for large scale optimization,” Mathematical programming, vol. 45, no. 1, pp. 503–528, 1989

    1989

  64. [73]

    Conjugate gradient method,

    J. L. Nazareth, “Conjugate gradient method,” Wiley Interdisciplinary Reviews: Computational Statistics , vol. 1, no. 3, pp. 348–353, 2009

    2009

  65. [74]

    Classical iterative methods for linear systems,

    W. C. Rheinboldt, “Classical iterative methods for linear systems,” Tech. Univ. Munich, 2009

    2009

  66. [75]

    Emnist: Extending mnist to handwritten letters,

    G. Cohen, S. Afshar, J. Tapson, and A. Van Schaik, “Emnist: Extending mnist to handwritten letters,” in 2017 international joint conference on neural networks (IJCNN) . IEEE, 2017, pp. 2921–2926

    2017

  67. [76]

    A public domain dataset for human activity recognition using smart- phones

    D. Anguita, A. Ghio, L. Oneto, X. Parra, J. L. Reyes-Ortiz et al. , “A public domain dataset for human activity recognition using smart- phones.” in Esann, vol. 3, no. 1, 2013, pp. 3–4

    2013

  68. [77]

    Basis matters: Better communication-efficient second order methods for federated learning,

    X. Qian, R. Islamov, M. Safaryan, and P. Richt ´arik, “Basis matters: Better communication-efficient second order methods for federated learning,” in International Conference on Artificial Intelligence and Statistics, AISTATS 2022, 28-30 March 2022, Virtual Event , ser. Pro- c...

    2022

  69. [78]

    Fednew: A communication-efficient and privacy-preserving newton-type method for federated learning,

    A. Elgabli, C. B. Issaid, A. S. Bedi, K. Rajawat, M. Bennis, and V . Ag- garwal, “Fednew: A communication-efficient and privacy-preserving newton-type method for federated learning,” in International confer- ence on machine learning . PMLR, 2022, pp. 5861–5877

    2022

  70. [79]

    Shed: A newton- type algorithm for federated learning based on incremental hessian eigenvector sharing,

    N. Dal Fabbro, S. Dey, M. Rossi, and L. Schenato, “Shed: A newton- type algorithm for federated learning based on incremental hessian eigenvector sharing,” Automatica, vol. 160, p. 111460, 2024

    2024

  71. [80]

    Fedns: A fast sketching newton- type algorithm for federated learning,

    J. Li, Y . Liu, and W. Wang, “Fedns: A fast sketching newton- type algorithm for federated learning,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 12, 2024, pp. 13 509– 13 517

    2024

  72. [81]

    Distributed second order meth- ods with fast rates and compressed communication,

    R. Islamov, X. Qian, and P. Richt ´arik, “Distributed second order meth- ods with fast rates and compressed communication,” in International conference on machine learning . PMLR, 2021, pp. 4617–4628

    2021

  73. [82]

    Distributed learning with compressed gradient differences,

    K. Mishchenko, E. Gorbunov, M. Tak ´aˇc, and P. Richt´arik, “Distributed learning with compressed gradient differences,” Optimization Methods and Software, pp. 1–16, 2024

    2024

  74. [83]

    Acceleration for compressed gradient descent in distributed and federated optimization,

    Z. Li, D. Kovalev, X. Qian, and P. Richt ´arik, “Acceleration for compressed gradient descent in distributed and federated optimization,” arXiv preprint arXiv:2002.11364 , 2020

    2002 arXiv

  75. [84]

    Local sgd: Unified theory and new efficient methods,

    E. Gorbunov, F. Hanzely, and P. Richt ´arik, “Local sgd: Unified theory and new efficient methods,” in International Conference on Artificial Intelligence and Statistics . PMLR, 2021, pp. 3556–3564

    2021

  76. [85]

    Dingo: Distributed newton-type method for gradient-norm optimization,

    R. Crane and F. Roosta, “Dingo: Distributed newton-type method for gradient-norm optimization,” Advances in neural information process- ing systems, vol. 32, 2019

    2019

  77. [86]

    Stephen j,

    J. Nocedal, “Stephen j,” Wright. Numerical optimization. Springer Science+ Business Media , 2006

    2006

  78. [87]

    Distributed quasi-newton method for fair and fast federated learning,

    S. M. Hamidi and L. Ye, “Distributed quasi-newton method for fair and fast federated learning,” arXiv preprint arXiv:2501.10877 , 2025

    2025 arXiv

  79. [88]

    Learning multiple layers of features from tiny images,

    A. Krizhevsky, G. Hinton et al., “Learning multiple layers of features from tiny images,” 2009

    2009

  80. [89]

    Federated accelerated stochastic gradient descent,

    H. Yuan and T. Ma, “Federated accelerated stochastic gradient descent,” Advances in Neural Information Processing Systems, vol. 33, pp. 5332– 5344, 2020

    2020

  81. [90]

    Adaptive federated optimization,

    S. Reddi, Z. Charles, M. Zaheer, Z. Garrett, K. Rush, J. Kone ˇcn`y, S. Kumar, and H. B. McMahan, “Adaptive federated optimization,” arXiv preprint arXiv:2003.00295 , 2020

    2003 arXiv

  82. [91]

    Feddane: A federated newton-type method,

    T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V . Smithy, “Feddane: A federated newton-type method,” in 2019 53rd Asilomar Conference on Signals, Systems, and Computers . IEEE, 2019, pp. 1227–1231

    2019

  83. [92]

    Algorithms for multicriterion optimization,

    H. Mukai, “Algorithms for multicriterion optimization,” IEEE transac- tions on automatic control , vol. 25, no. 2, pp. 177–186, 2003

    2003

  84. [93]

    Tiny imagenet visual recognition challenge,

    Y . Le and X. Yang, “Tiny imagenet visual recognition challenge,” CS 231N, vol. 7, no. 7, p. 3, 2015

    2015

  85. [94]

    Cinic-10 is not imagenet or cifar-10,

    L. N. Darlow, E. J. Crowley, A. Antoniou, and A. J. Storkey, “Cinic-10 is not imagenet or cifar-10,” arXiv preprint arXiv:1810.03505 , 2018

    2018 arXiv

  86. [95]

    Federated learning with fair averaging,

    Z. Wang, X. Fan, J. Qi, C. Wen, C. Wang, and R. Yu, “Federated learning with fair averaging,” arXiv preprint arXiv:2104.14937 , 2021

    2021 arXiv

  87. [96]

    Federated learning with matched averaging,

    H. Wang, M. Yurochkin, Y . Sun, D. Papailiopoulos, and Y . Khaz- aeni, “Federated learning with matched averaging,” arXiv preprint arXiv:2002.06440, 2020

    2002 arXiv

  88. [97]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 770–778

    2016

  89. [98]

    Federated optimization with linear-time approximated hessian diagonal,

    M. Sen, C. K. Mohan, and A. K. Qin, “Federated optimization with linear-time approximated hessian diagonal,” in International Confer- ence on Pattern Recognition and Machine Intelligence. Springer, 2023, pp. 106–113

    2023

  90. [99]

    Robust federated learning under statistical heterogeneity via hessian-weighted aggregation,

    A. Ahmad, W. Luo, and A. Robles-Kelly, “Robust federated learning under statistical heterogeneity via hessian-weighted aggregation,” Ma- chine Learning, vol. 112, no. 2, pp. 633–654, 2023

    2023

  91. [100]

    Fed-sophia: A communication-efficient second-order federated learning algorithm,

    A. Elbakary, C. B. Issaid, M. Shehab, K. Seddik, T. ElBatt, and M. Ben- nis, “Fed-sophia: A communication-efficient second-order federated learning algorithm,” in ICC 2024-IEEE International Conference on Communications. IEEE, 2024, pp. 950–955

    2024

  92. [101]

    An estimator for the diagonal of a matrix,

    C. Bekas, E. Kokiopoulou, and Y . Saad, “An estimator for the diagonal of a matrix,” Applied numerical mathematics , vol. 57, no. 11-12, pp. 1214–1229, 2007

    2007

  93. [102]

    Overcoming forgetting in federated learning on non-iid data,

    N. Shoham, T. Avidor, A. Keren, N. Israel, D. Benditkis, L. Mor-Yosef, and I. Zeitak, “Overcoming forgetting in federated learning on non-iid data,” arXiv preprint arXiv:1910.07796 , 2019. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 20

    1910 arXiv

  94. [103]

    Sophia: A scalable stochastic second-order optimizer for language model pre-training,

    H. Liu, Z. Li, D. Hall, P. Liang, and T. Ma, “Sophia: A scalable stochastic second-order optimizer for language model pre-training,” arXiv preprint arXiv:2305.14342 , 2023

    2023 arXiv

  95. [104]

    Fagh: Accelerating federated learning with approxi- mated global hessian,

    M. Sen et al. , “Fagh: Accelerating federated learning with approxi- mated global hessian,” arXiv preprint arXiv:2403.11041 , 2024

    2024 arXiv

  96. [105]

    Understanding how consistency works in federated learning via stage-wise relaxed initialization,

    Y . Sun, L. Shen, and D. Tao, “Understanding how consistency works in federated learning via stage-wise relaxed initialization,” Advances in Neural Information Processing Systems , vol. 36, pp. 80 543–80 574, 2023

    2023

  97. [106]

    Implicit gradient alignment in dis- tributed and federated learning,

    Y . Dandi, L. Barba, and M. Jaggi, “Implicit gradient alignment in dis- tributed and federated learning,” inProceedings of the AAAI Conference on Artificial Intelligence , vol. 36, no. 6, 2022, pp. 6454–6462

    2022

  98. [107]

    Fedexp: Speeding up federated averaging via extrapolation,

    D. Jhunjhunwala, S. Wang, and G. Joshi, “Fedexp: Speeding up federated averaging via extrapolation,” in International Conference on Learning Representations, 2023. Mrinmay Sen is a joint research scholar at the De- partment of Artificial Intelligence, Indian Institute of Techn...

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.