Pith. sign in

REVIEW 2 major objections 5 minor 16 references

FedDuA: Doubly Adaptive Federated Learning

T0 review · 2 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Pith's one-line read One formula chooses the global learning rate in federated learning and is minimax optimal under an approximate projection condition.

desk verdict Useful extension of FedExP to the preconditioned FedOpt family, with a clean mirror-descent derivation and a correct one-step minimax argument for the non-momentum case, but the momentum variant's minimax claim is unsupported and needs qualification. read the letter →

arxiv 2505.11126 v1 pith:Q7B437BX submitted 2025-05-16 cs.LG stat.ML

classification cs.LGstat.ML
keywords federatedlearningadaptiveoptimizationmirrordescentglobalrateclientheterogeneitygradientminimaxoptimalityFedDuA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FedDuA claims that the server-side global learning rate in federated learning can be chosen from the local updates themselves rather than tuned by hand, and that the right choice is $\eta_t^g = m_t / (\lVert v_t \rVert^2_{G_t^{-1}} + \epsilon_g)$ where $m_t$ is half the average squared norm of client updates and $G_t$ is the coordinate-wise preconditioner borrowed from FedAdagrad/FedAdam. The paper derives this rule as a lower bound on the step size that minimizes Bregman distance to an optimum, and proves it is minimax optimal under an approximate projection condition. It also proves convergence for convex smooth objectives with full-batch local SGD and full participation. The reason this matters is that it unifies two previously separate kinds of adaptivity, coordinate-wise and inter-client, inside a single formula that costs the server nothing extra.

What carries the argument

The machinery is the mirror-descent reformulation of the server update: the distance-generating function $\psi_t(x)=\tfrac12 x^\top G_t x$ induces a Bregman divergence $D_{\psi_t}(w^*|w)$, and the dual update $\theta^{t+1}=\theta^t+\eta_t^g v_t$ lets the step size be chosen in the dual space. The approximate projection condition (A.P.C.) says local training does not move the average client model farther from $w^*$; under it, the optimal step size is lower-bounded by $m_t/\lVert v_t\rVert^2_{G_t^{-1}}$, and Theorem 4.1 shows that taking exactly this lower bound minimizes the worst-case distance over all optima consistent with A.P.C. With momentum, $m_t$ is replaced by its exponentially weighted version, so the same formula covers both FedDuAdagrad and FedDuAdam.

What would settle it

Test Theorem 4.1 directly: in a convex overparameterized problem where A.P.C. holds, compute both FedDuA's update and the exact minimizer of $\sup_{w^*\in H} [D_{\psi_t}(w^*|w)-D_{\psi_t}(w^*|w^t)]$; any disagreement refutes the minimax claim. Separately, on a real task, measure per round whether $(1/M)\sum_i \lVert w^t+\Delta_i^t-w^*\rVert^2 \le \lVert w^t-w^*\rVert^2$; if it typically fails while FedDuA still converges, the empirical success is not explained by the minimax theorem.

Watch

Extended reading notes

Core claim

The paper's central claim is that the FedDuA update $w^{t+1} = w^t + \eta_t^g G_t^{-1} v_t$, with $\eta_t^g = m_t / (\lVert v_t \rVert^2_{G_t^{-1}} + \epsilon_g)$, is the unique minimizer of the worst-case Bregman distance to any optimum consistent with the approximate projection condition (Theorem 4.1). Under that condition, partially adaptive rules, coordinate-wise only like FedOpt or client-heterogeneity only like FedExP, are suboptimal because the minimizer is unique. For convex $L$-smooth losses with full-batch local SGD and full participation, the averaged iterate satisfies $F(\bar w_T)-F(w^*) = O\bigl(D_{\psi_0}(w^*|w^0) / \sum_t \eta_t^g \eta_l \tau\bigr) + O(\eta_l \tau \sigma_*^2) + O(\eta_l^2 \tau(\tau-1) L \sigma_*^2)$ (Theorem 4.3), where the bias terms are independent of the mirror map.

Load-bearing premise

The load-bearing premise is that after each round of local training, the client models are on average no farther from some global optimum than before the round; with real minibatch SGD and partial participation this condition can fail.

Editorial extensions

If this is right

  • FedDuA changes only the server's aggregation step, and the quantities it needs, norms of the already-received local updates and the preconditioner, are server-side, so no extra client cost arises.
  • The formula is hyperparameter-free at $\epsilon_g=0$, and the experiments show that $\epsilon_g=0$ works, so the method removes the need to tune a global learning rate.
  • On anisotropic problems, FedDuAdagrad's initialization error term is dimension-free, unlike FedExP and FedAvg, so the advantage should grow with model dimension.
  • FedDuA composes with local-training modifications: combining it with SCAFFOLD-type local updates outperforms vanilla SCAFFOLD on CIFAR-100.
  • Under the stated assumptions, minimax optimality is strict: any update rule differing from FedDuA is suboptimal for some optimum consistent with the approximate projection condition.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the approximate projection condition is violated in practice by minibatch SGD or partial participation, the minimax theorem no longer pins down the correct step size; one can test how often A.P.C. holds on real FL runs and whether the empirical gains persist when it fails.
  • The mirror-descent derivation is generic: replacing $G_t$ with another strongly convex preconditioner should yield a new doubly adaptive rule by the same lower-bound argument.
  • The dimension-free bound hints that FedDuA-type rates could be most valuable in very wide or transformer-like models, consistent with the paper's ViT experiment.
  • Extending the convergence analysis to partial participation would require treating $\eta_t^g$ as stochastic, and the minimax optimality may or may not survive in expectation; that is a concrete open question.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes FedDuA, a server-side federated learning update rule that combines coordinate-wise preconditioning (via FedAdagrad/FedAdam style matrices G_t) with a heterogeneity-adaptive global learning rate eta_t^g = m_t / (||v_t||^2_{G^{-1}_t} + epsilon_g), where m_t measures the average squared norm of local updates. The update is derived from a mirror-descent formulation of the server step: lower bounds on the optimal step size are proven under an approximate projection condition (Theorems 3.2 and 3.3), a minimax optimality claim is made (Theorem 4.1), and a convex convergence bound is proven under full-batch, full-participation assumptions (Theorem 4.3), with a dimension-dependent rate analysis for FedDuAdagrad (Corollary 4.4 and Appendix F). Experiments on synthetic data, CIFAR-10/100, FEMNIST, and Shakespeare compare FedDuA with FedAvg, FedExP, FedOpt, and their momentum variants.

Significance. The paper has several genuine strengths: the mirror-descent perspective provides a clean way to unify coordinate-wise adaptivity and client-heterogeneity adaptivity without any extra client-side cost; the lower-bound derivations are careful; the non-momentum convergence analysis is nontrivial and the experimental study is extensive, including robustness checks and combination with SCAFFOLD. The conditional minimax result for the non-momentum case v_t = Delta_bar_t is a credible contribution beyond FedExP. However, the headline claim in the abstract and Section 4.1 that the doubly adaptive step-size rule is minimax optimal is not actually proven for the momentum variant FedDuAdam, which is the main experimental workhorse. Because the proofs of Theorems 4.1 and 4.3 both assume v_t = Delta_bar_t, the manuscript currently overclaims its theoretical scope. The issue is fixable by restricting the claims or by supplying a genuinely different argument, but it is load-bearing for the paper's central message.

major comments (2)
  1. [Section 4.1, Theorem 4.1 and Appendix C] The theorem as stated covers FedDuA defined in Eq. (3) with a general v_t, and the abstract's 'minimax optimal' claim is unqualified. The proof, however, only treats v_t = Delta_bar_t: the dual-feasibility condition theta_t - theta + lambda Delta_bar_t = 0 leads to w = grad phi(theta_t + lambda Delta_bar_t), and the lambda-optimality condition uses h_t(lambda) with Delta_bar_t. For FedDuAdam in Algorithm 1, v_t = beta_1 v_{t-1} + (1 - beta_1) Delta_bar_t is generally not parallel to Delta_bar_t. With H defined by the A.P.C. inequality, which is the half-space <Delta_bar_t, w* - w^t> >= (1/(2M)) sum_i ||Delta_i^t||^2, the worst-case distance difference V(w) has linear coefficient theta_t - theta_w = -eta_t^g v_t in w*. For the supremum over H to be finite, this coefficient must lie in the cone generated by -Delta_bar_t; when v_t has a component outside span(Delta_bar_t), V(w) = +infinity for every eta_t^g > 0 and no finite minimizer exists. The lower bound in Theorem 3.3 does not repair this gap, because a lower bound on a minimizer does not establish minimax optimality. The abstract and Section 4.1 must restrict the minimax claim to v_t = Delta_bar_t (i.e., FedDuAdagrad) or provide a proof for the momentum rule under a different, explicitly stated uncertainty set.
  2. [Section 4.2, Theorem 4.3 and Appendix D] The theorem statement says 'generated by FedDuA' without restricting v_t, but the proof again assumes v_t = Delta_bar_t: the first display writes phi_t(theta_{t+1}) - phi_t(theta_t) - eta_t^g <w*, Delta_bar_t> and defines H_t with Delta_bar_t. Thus the convergence guarantee has not been established for the FedDuAdam variant used extensively in the experiments, and the 'with and without momentum' discussion in Section 3 is not backed by this theorem. Please either state Theorem 4.3 for FedDuAdagrad only, extend the proof to the momentum case with appropriate additional assumptions, or explicitly state that the momentum analysis in Section 4 is limited to the lower bound in Theorem 3.3.
minor comments (5)
  1. [Section 3, Theorem 3.3] The assumption says 'w_s is updated as Eq. (3) with eta_s^g <= h^{-1}_s(m_s)', but Eq. (3) already defines eta_s^g := h^{-1}_s(m_s); the inequality should be stated as an explicit separate assumption or the equality in Eq. (3) should be relaxed.
  2. [Appendix D, proof of Theorem 4.3] In the displayed line 'D_t(w*|w^{t+1}) - D_t(w*|w^{t+1}) <= ...', the second occurrence of D_t(w*|w^{t+1}) should be D_t(w*|w^t); as printed the left-hand side is identically zero and the inequality is vacuous.
  3. [Appendix E, proof of Corollary 4.4] The displayed sum sum_{t=0}^{T-1} (D_t(w*|w^t) - D_{t-1}(w*|w^t)) does not match the numerator from Theorem 4.3, which is D_0(w*|w^0) + sum_{t=1}^{T-1} (D_t(w*|w^t) - D_{t-1}(w*|w^t)); the final D^2 tr(G_{T-1}) bound is plausible, but the displayed algebra should be corrected.
  4. [Section 4.1, Theorem 4.1] The symbol 'w*_{t+1}' is used for the minimizer of V(w), which is not the global optimum w*; renaming it (for example, widetilde{w}_{t+1}) would avoid confusion.
  5. [Section 5, Table 2] Table 2 reports 'last iterate' accuracy while the text says the last two iterates are averaged; please clarify which quantity is actually reported.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: FedDuA's step size is computed from observed local updates, and the minimax and convergence theorems are independent derivations rather than restatements of the inputs.

full rationale

The paper's central step-size rule is derived, not fitted: η_t^g is set to h_t^{-1}(m_t), where m_t is defined directly from the norms of the clients' local updates and h_t is defined from the mirror-descent dual objective. Theorem 3.2 derives a lower bound on the optimal step size under the approximate projection condition, and FedDuA adopts that lower bound as its step size; this is an explicit design choice followed by a proof, not a parameter fitted to the quantity later called a prediction. Theorem 4.1 solves a genuine minimax problem: for the set H of optima satisfying A.P.C., it characterizes the unique minimizer of the worst-case Bregman distance difference and shows that this minimizer equals the FedDuA update for v_t = Δ̄_t. The Lagrangian argument identifies the condition θ_t − θ + λΔ̄_t = 0 for finiteness of the supremum, and then minimizes over λ; this is a real variational characterization, not a tautology. The convergence analysis in Theorem 4.3 uses standard L-smoothness and convexity assumptions and only uses the definition of η_t^g through the identity h_t(η_t^g) = m_t; that identity is algebraic, not an assumed conclusion. The A.P.C. condition is borrowed transparently from external prior work (Jhunjhunwala et al. 2023) as an explicit assumption, not smuggled in as an ansatz, and the authors do not rely on a self-citation chain to force the result. The momentum-variant minimax claim is not actually established by the Appendix C proof, which treats v_t = Δ̄_t rather than the FedDuAdam momentum vector; however, that is a proof gap or correctness concern, not an input-output circularity. No step in the paper reduces, by definition or by self-citation, to its own inputs.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

No numbers are fitted to data in the derivation. The algorithm's hyperparameters (eta_l, tau, beta_1, beta_2, epsilon, epsilon_g) are tuned in experiments, but the theory states conditions under which convergence holds rather than fitting constants. The main unproven inputs are the A.P.C. conditions, convexity/smoothness, full-batch/full-participation, and the anisotropic decay model used for the dimension-free illustration.

assumptions (6)
  • domain assumption Approximate Projection Condition (Assumption 3.1): (1/M) sum_i ||w^t + Delta_i^t - w*||^2 <= ||w^t - w*||^2
    Borrowed from FedExP; used to prove Theorems 3.2, 3.3 and 4.1. May be violated by real local updates, especially with minibatch SGD and partial participation.
  • domain assumption Strong A.P.C.: <Delta_bar^t, w* - w^t> >= (1/M) sum_i ||Delta_i^t||^2
    Used for the momentum lower bound in Theorem 3.3 and the induction argument in Appendix B.
  • domain assumption Local loss functions F_i are convex and L-smooth, and (1/M) sum_i ||grad F_i(w*)||^2 <= sigma_*^2
    Assumption 4.2 for Theorem 4.3; standard convex-setting assumptions that exclude non-convex deep learning, which the experiments nonetheless cover.
  • ad hoc to paper Full-batch SGD and full client participation in the convergence analysis
    Stated in Theorem 4.3 and defended as a simplification in Section I; the experimental regime uses minibatches of size 50 and 20 participating clients, which the theory does not cover.
  • ad hoc to paper Anisotropic decay model [Delta_bar^t]_k = Theta(a_t k^{-beta}) with beta > 1 and a_t decreasing
    Used in Section 4.2.1 and Appendix F to derive the dimension-independent T1 = O(D^2 / (eta_l tau T)) claim; this is an illustrative structural assumption, not a theorem for all tasks.
  • domain assumption H, the set of w* satisfying A.P.C., is nonempty
    Theorem 4.1 requires H nonempty; if the local updates are inconsistent, the worst-case optimality argument does not apply.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FedDuA: Doubly Adaptive Federated Learning." pith.science (2026). https://pith.science/paper/Q7B437BX

@misc{pith2026250511126,
  author       = {Pith},
  title        = {Pith review of: FedDuA: Doubly Adaptive Federated Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q7B437BX}},
  note         = {Machine review of arXiv:2505.11126}
}
read the original abstract

Federated learning is a distributed learning framework where clients collaboratively train a global model without sharing their raw data. FedAvg is a popular algorithm for federated learning, but it often suffers from slow convergence due to the heterogeneity of local datasets and anisotropy in the parameter space. In this work, we formalize the central server optimization procedure through the lens of mirror descent and propose a novel framework, called FedDuA, which adaptively selects the global learning rate based on both inter-client and coordinate-wise heterogeneity in the local updates. We prove that our proposed doubly adaptive step-size rule is minimax optimal and provide a convergence analysis for convex objectives. Although the proposed method does not require additional communication or computational cost on clients, extensive numerical experiments show that our proposed framework outperforms baselines in various settings and is robust to the choice of hyperparameters.

Figures

Figures reproduced from arXiv: 2505.11126 by the authors.

Figure 1
Figure 1. Test accuracy for FedDuA and baselines without server momentum (upper) and with server momentum (lower). Our proposed methods (green dashdot) consistently outperform baselines. and NLP tasks for real-world datasets. We compare FedDuA with the following baselines: FedAvg, FedExP, FedOpt (FedAdagrad and FedAdam), and their momentum variants (FedAvgM, FedExPM). These algorithms do not require additional computational o… view at source ↗
Figure 2
Figure 2. Test accuracy averaged over the last 5 iterates with different [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 4
Figure 4. Training of ViT on CIFAR-100. Coordinate-wise adaptivity is essential in training of Transformers As discussed in previous works [Zhang et al., 2024, Tomihari and Sato, 2025], adaptive methods such as Adam outperform SGD in centralized training of Transformers. To see the effect of adaptivity in FL, we train a Vision Transformer (ViT) [Dosovitskiy et al., 2021] on CIFAR-100 [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Long term behavior of each algorithm. NLP dataset For the NLP task, we tune ηl over {10−2 , 10−3/2 , 10−1 , 10−1/2 , 100}. The grid of ηg is {10−1 , 10−1/2 , 100 , 101/2 , 101} for FedAvg(M) and SCAF￾FOLD, and {10−3 , 10−5/2 , 10−2 , 10−3/2 , 10−1} for FedOPT. The grid…
Figure 6
Figure 6. Figure 6: Comparison with FedProx. In this section, we provide a comparison with FedProx [Li et al., 2020]. This is an algorithm that modifies the local objective and thus can be combined with the FedDuA framework. For the FedProx-type local training procedure, we tune the addit…
Figure 7
Figure 7. Figure 7: Comparison of global learning rates ηg for different methods. 0 100 200 Communication Rounds 10−10 10−7 10−4 10−1 Training Loss Synthetic 0 200 400 Communication Rounds 0.00 0.25 0.50 0.75 1.00 Training Loss CIFAR10 0 200 400 Communication Rounds 0 1 2 3 Training Loss …
Figure 8
Figure 8. Figure 8: Comparison of training loss for different methods. [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 6 canonical work pages

  1. [1]

    Here, we used the induction hypothesis for the first inequality and strong A.P.C

    1 M MX i=1 ∆t i 2 +β 1mt−1 = 2mt. Here, we used the induction hypothesis for the first inequality and strong A.P.C. for the second inequality. Then, we obtain the result by induction. C Proof for Theorem 4.1 Givenw∈R d, the worst-case distance difference is defined as V(w) = sup w∗∈H V(w,w ∗) := sup w∗∈H Dψt(w∗|w)−D ψt(w∗|w t). From the definition of the ...

  2. [3]

    For ViT experiments, we use the architecture in omiita [2024]

    We also provide the architecture of LSTM and CNN in Table 4 and Table 5, respectively. For ViT experiments, we use the architecture in omiita [2024]. Table 3: Datasets and models used in our experiments Dataset Task Model # of Classes License Synthetic dataset Regression Linear N/A N/A CIFAR-10 Image classification ResNet-18 10 MIT License CIFAR-100 Image...

  3. [4]

    S. P. Karimireddy, M. Jaggi, S. Kale, M. Mohri, S. J. Reddi, S. U. Stich, and A. T. Suresh. Mime: Mim- icking centralized stochastic algorithms in federated learning.arXiv preprint arXiv:2008.03606, 2020a. S. P. Karimireddy, S. Kale, M. Mohri, S. Reddi, S. Stich, and A. T. Suresh. Scaffold: Stochastic controlled averaging for federated learning. InInterna...

  4. [5]

    We see that FedDuA consistently outperforms other methods in terms of convergence speed and final accuracy. H.2 Comparison with FedProx 0 100 200 300 400 500 Communication Rounds 30 40 50 60 70Test Accuracy (%) CIFAR100 FedProx FedDuAdaGrad + FedProx FedAvg FedDuAdaGrad Figure 6: Comparison with FedProx. In this section, we provide a comparison with FedPr...

  5. [6]

    Following Jhunjhunwala et al

    Other hyperparameters are kept the same across all methods. Following Jhunjhunwala et al. [2023], we use weight decay of 10−4, learning rate decay of0.998, and gradient clipping to stabilize the training for image classification tasks. Synthetic datasetFor the synthetic dataset, We tune ηl over{10−3,10−5/2,10−2,10−3/2,10−1}, andηg over{10−1,10−1/2,10 0,10...

  6. [8]

    URL https://openreview.net/ forum?id=aBO5SvgSt1. A. Tomihari and I. Sato. Understanding Why Adam Outperforms SGD: Gradient Heterogeneity in Transformers.arXiv preprint arXiv:2502.00213,

  7. [15]

    The grid of ηg is{10−1,10−1/2,10 0,10 1/2,10 1} for Fe- dAvg(M) and SCAFFOLD, and{10−4,10−7/2,10−3,10−5/2,10−2} for FedOPT

    Image classification datasetsFor the image classification tasks, we tune ηl over {10−2,10−3/2,10−1,10−1/2,10 0}. The grid of ηg is{10−1,10−1/2,10 0,10 1/2,10 1} for Fe- dAvg(M) and SCAFFOLD, and{10−4,10−7/2,10−3,10−5/2,10−2} for FedOPT. The grid of ϵg is{10−3,10−5/2,10−2,10−3/2,10−1} for FedDuA, and{10−4,10−7/2,10−3,10−5/2,10−2} for FedExP(M). We fixϵ= 10...

  8. [2011]

    Faghri, D

    F. Faghri, D. Duvenaud, D. J. Fleet, and J. Ba. A study of gradient variance in deep learning.arXiv preprint arXiv:2007.04532,

Show all 16 references
  1. [2015]

    Koneˇcn`y, H

    J. Koneˇcn`y, H. B. McMahan, F. X. Yu, P. Richtárik, A. T. Suresh, and D. Bacon. Federated learning: Strategies for improving communication efficiency.arXiv preprint arXiv:1610.05492,

  2. [2016]

    S. H. Lee, S. Sharma, M. Zaheer, and T. Li. Efficient Adaptive Federated Optimization.arXiv preprint arXiv:2410.18117,

  3. [2019]

    Caldas, S

    S. Caldas, S. M. K. Duddu, P. Wu, T. Li, J. Koneˇcn`y, H. B. McMahan, V . Smith, and A. Talwalkar. Leaf: A benchmark for federated settings.arXiv preprint arXiv:1812.01097,

  4. [2020]

    Haddadpour and M

    F. Haddadpour and M. Mahdavi. On the convergence of local descent methods in federated learning. arXiv preprint arXiv:1910.14425,

  5. [2021]

    C. Xie, O. Koyejo, I. Gupta, and H. Lin. Local adaalter: Communication-efficient stochastic gradient descent with adaptive learning rates.arXiv preprint arXiv:1911.09030,

  6. [2022]

    Nielsen, J.-D

    F. Nielsen, J.-D. Boissonnat, and R. Nock. Bregman voronoi diagrams: Properties, algorithms and applications.arXiv preprint arXiv:0709.2196,

  7. [2023]

    ForR 1, we have R1 = 1 M MX i=1 ⟨wt−w∗,∆t i⟩ = ηl M MX i=1 τ−1X k=0 ⟨wt−w∗,∇Fi(wt i,k)⟩

    for the last inequality. ForR 1, we have R1 = 1 M MX i=1 ⟨wt−w∗,∆t i⟩ = ηl M MX i=1 τ−1X k=0 ⟨wt−w∗,∇Fi(wt i,k)⟩. As shown in the proof of Theorem 1 in Jhunjhunwala et al. [2023], the right-hand side can be bounded as R1≥η lτ(F(wt)−F(w ∗))− ηlL 2M MX i=1 τ−1X k=0 wt i,k−wt 2 ....

  8. [2025]

    J. Wang, Z. Xu, Z. Garrett, Z. Charles, L. Liu, and G. Joshi. Local adaptivity in federated learning: Convergence and consistency.arXiv preprint arXiv:2106.02305,

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.