REVIEW 2 major objections 5 minor 16 references
FedDuA: Doubly Adaptive Federated Learning
T0 review · 2 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read One formula chooses the global learning rate in federated learning and is minimax optimal under an approximate projection condition.
desk verdict Useful extension of FedExP to the preconditioned FedOpt family, with a clean mirror-descent derivation and a correct one-step minimax argument for the non-momentum case, but the momentum variant's minimax claim is unsupported and needs qualification. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is the mirror-descent reformulation of the server update: the distance-generating function $\psi_t(x)=\tfrac12 x^\top G_t x$ induces a Bregman divergence $D_{\psi_t}(w^*|w)$, and the dual update $\theta^{t+1}=\theta^t+\eta_t^g v_t$ lets the step size be chosen in the dual space. The approximate projection condition (A.P.C.) says local training does not move the average client model farther from $w^*$; under it, the optimal step size is lower-bounded by $m_t/\lVert v_t\rVert^2_{G_t^{-1}}$, and Theorem 4.1 shows that taking exactly this lower bound minimizes the worst-case distance over all optima consistent with A.P.C. With momentum, $m_t$ is replaced by its exponentially weighted version, so the same formula covers both FedDuAdagrad and FedDuAdam.
What would settle it
Test Theorem 4.1 directly: in a convex overparameterized problem where A.P.C. holds, compute both FedDuA's update and the exact minimizer of $\sup_{w^*\in H} [D_{\psi_t}(w^*|w)-D_{\psi_t}(w^*|w^t)]$; any disagreement refutes the minimax claim. Separately, on a real task, measure per round whether $(1/M)\sum_i \lVert w^t+\Delta_i^t-w^*\rVert^2 \le \lVert w^t-w^*\rVert^2$; if it typically fails while FedDuA still converges, the empirical success is not explained by the minimax theorem.
Extended reading notes
Core claim
The paper's central claim is that the FedDuA update $w^{t+1} = w^t + \eta_t^g G_t^{-1} v_t$, with $\eta_t^g = m_t / (\lVert v_t \rVert^2_{G_t^{-1}} + \epsilon_g)$, is the unique minimizer of the worst-case Bregman distance to any optimum consistent with the approximate projection condition (Theorem 4.1). Under that condition, partially adaptive rules, coordinate-wise only like FedOpt or client-heterogeneity only like FedExP, are suboptimal because the minimizer is unique. For convex $L$-smooth losses with full-batch local SGD and full participation, the averaged iterate satisfies $F(\bar w_T)-F(w^*) = O\bigl(D_{\psi_0}(w^*|w^0) / \sum_t \eta_t^g \eta_l \tau\bigr) + O(\eta_l \tau \sigma_*^2) + O(\eta_l^2 \tau(\tau-1) L \sigma_*^2)$ (Theorem 4.3), where the bias terms are independent of the mirror map.
Load-bearing premise
The load-bearing premise is that after each round of local training, the client models are on average no farther from some global optimum than before the round; with real minibatch SGD and partial participation this condition can fail.
Editorial extensions
If this is right
- FedDuA changes only the server's aggregation step, and the quantities it needs, norms of the already-received local updates and the preconditioner, are server-side, so no extra client cost arises.
- The formula is hyperparameter-free at $\epsilon_g=0$, and the experiments show that $\epsilon_g=0$ works, so the method removes the need to tune a global learning rate.
- On anisotropic problems, FedDuAdagrad's initialization error term is dimension-free, unlike FedExP and FedAvg, so the advantage should grow with model dimension.
- FedDuA composes with local-training modifications: combining it with SCAFFOLD-type local updates outperforms vanilla SCAFFOLD on CIFAR-100.
- Under the stated assumptions, minimax optimality is strict: any update rule differing from FedDuA is suboptimal for some optimum consistent with the approximate projection condition.
Reading between the lines
- If the approximate projection condition is violated in practice by minibatch SGD or partial participation, the minimax theorem no longer pins down the correct step size; one can test how often A.P.C. holds on real FL runs and whether the empirical gains persist when it fails.
- The mirror-descent derivation is generic: replacing $G_t$ with another strongly convex preconditioner should yield a new doubly adaptive rule by the same lower-bound argument.
- The dimension-free bound hints that FedDuA-type rates could be most valuable in very wide or transformer-like models, consistent with the paper's ViT experiment.
- Extending the convergence analysis to partial participation would require treating $\eta_t^g$ as stochastic, and the minimax optimality may or may not survive in expectation; that is a concrete open question.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes FedDuA, a server-side federated learning update rule that combines coordinate-wise preconditioning (via FedAdagrad/FedAdam style matrices G_t) with a heterogeneity-adaptive global learning rate eta_t^g = m_t / (||v_t||^2_{G^{-1}_t} + epsilon_g), where m_t measures the average squared norm of local updates. The update is derived from a mirror-descent formulation of the server step: lower bounds on the optimal step size are proven under an approximate projection condition (Theorems 3.2 and 3.3), a minimax optimality claim is made (Theorem 4.1), and a convex convergence bound is proven under full-batch, full-participation assumptions (Theorem 4.3), with a dimension-dependent rate analysis for FedDuAdagrad (Corollary 4.4 and Appendix F). Experiments on synthetic data, CIFAR-10/100, FEMNIST, and Shakespeare compare FedDuA with FedAvg, FedExP, FedOpt, and their momentum variants.
Significance. The paper has several genuine strengths: the mirror-descent perspective provides a clean way to unify coordinate-wise adaptivity and client-heterogeneity adaptivity without any extra client-side cost; the lower-bound derivations are careful; the non-momentum convergence analysis is nontrivial and the experimental study is extensive, including robustness checks and combination with SCAFFOLD. The conditional minimax result for the non-momentum case v_t = Delta_bar_t is a credible contribution beyond FedExP. However, the headline claim in the abstract and Section 4.1 that the doubly adaptive step-size rule is minimax optimal is not actually proven for the momentum variant FedDuAdam, which is the main experimental workhorse. Because the proofs of Theorems 4.1 and 4.3 both assume v_t = Delta_bar_t, the manuscript currently overclaims its theoretical scope. The issue is fixable by restricting the claims or by supplying a genuinely different argument, but it is load-bearing for the paper's central message.
major comments (2)
- [Section 4.1, Theorem 4.1 and Appendix C] The theorem as stated covers FedDuA defined in Eq. (3) with a general v_t, and the abstract's 'minimax optimal' claim is unqualified. The proof, however, only treats v_t = Delta_bar_t: the dual-feasibility condition theta_t - theta + lambda Delta_bar_t = 0 leads to w = grad phi(theta_t + lambda Delta_bar_t), and the lambda-optimality condition uses h_t(lambda) with Delta_bar_t. For FedDuAdam in Algorithm 1, v_t = beta_1 v_{t-1} + (1 - beta_1) Delta_bar_t is generally not parallel to Delta_bar_t. With H defined by the A.P.C. inequality, which is the half-space <Delta_bar_t, w* - w^t> >= (1/(2M)) sum_i ||Delta_i^t||^2, the worst-case distance difference V(w) has linear coefficient theta_t - theta_w = -eta_t^g v_t in w*. For the supremum over H to be finite, this coefficient must lie in the cone generated by -Delta_bar_t; when v_t has a component outside span(Delta_bar_t), V(w) = +infinity for every eta_t^g > 0 and no finite minimizer exists. The lower bound in Theorem 3.3 does not repair this gap, because a lower bound on a minimizer does not establish minimax optimality. The abstract and Section 4.1 must restrict the minimax claim to v_t = Delta_bar_t (i.e., FedDuAdagrad) or provide a proof for the momentum rule under a different, explicitly stated uncertainty set.
- [Section 4.2, Theorem 4.3 and Appendix D] The theorem statement says 'generated by FedDuA' without restricting v_t, but the proof again assumes v_t = Delta_bar_t: the first display writes phi_t(theta_{t+1}) - phi_t(theta_t) - eta_t^g <w*, Delta_bar_t> and defines H_t with Delta_bar_t. Thus the convergence guarantee has not been established for the FedDuAdam variant used extensively in the experiments, and the 'with and without momentum' discussion in Section 3 is not backed by this theorem. Please either state Theorem 4.3 for FedDuAdagrad only, extend the proof to the momentum case with appropriate additional assumptions, or explicitly state that the momentum analysis in Section 4 is limited to the lower bound in Theorem 3.3.
minor comments (5)
- [Section 3, Theorem 3.3] The assumption says 'w_s is updated as Eq. (3) with eta_s^g <= h^{-1}_s(m_s)', but Eq. (3) already defines eta_s^g := h^{-1}_s(m_s); the inequality should be stated as an explicit separate assumption or the equality in Eq. (3) should be relaxed.
- [Appendix D, proof of Theorem 4.3] In the displayed line 'D_t(w*|w^{t+1}) - D_t(w*|w^{t+1}) <= ...', the second occurrence of D_t(w*|w^{t+1}) should be D_t(w*|w^t); as printed the left-hand side is identically zero and the inequality is vacuous.
- [Appendix E, proof of Corollary 4.4] The displayed sum sum_{t=0}^{T-1} (D_t(w*|w^t) - D_{t-1}(w*|w^t)) does not match the numerator from Theorem 4.3, which is D_0(w*|w^0) + sum_{t=1}^{T-1} (D_t(w*|w^t) - D_{t-1}(w*|w^t)); the final D^2 tr(G_{T-1}) bound is plausible, but the displayed algebra should be corrected.
- [Section 4.1, Theorem 4.1] The symbol 'w*_{t+1}' is used for the minimizer of V(w), which is not the global optimum w*; renaming it (for example, widetilde{w}_{t+1}) would avoid confusion.
- [Section 5, Table 2] Table 2 reports 'last iterate' accuracy while the text says the last two iterates are averaged; please clarify which quantity is actually reported.
Circularity Check
No circularity: FedDuA's step size is computed from observed local updates, and the minimax and convergence theorems are independent derivations rather than restatements of the inputs.
full rationale
The paper's central step-size rule is derived, not fitted: η_t^g is set to h_t^{-1}(m_t), where m_t is defined directly from the norms of the clients' local updates and h_t is defined from the mirror-descent dual objective. Theorem 3.2 derives a lower bound on the optimal step size under the approximate projection condition, and FedDuA adopts that lower bound as its step size; this is an explicit design choice followed by a proof, not a parameter fitted to the quantity later called a prediction. Theorem 4.1 solves a genuine minimax problem: for the set H of optima satisfying A.P.C., it characterizes the unique minimizer of the worst-case Bregman distance difference and shows that this minimizer equals the FedDuA update for v_t = Δ̄_t. The Lagrangian argument identifies the condition θ_t − θ + λΔ̄_t = 0 for finiteness of the supremum, and then minimizes over λ; this is a real variational characterization, not a tautology. The convergence analysis in Theorem 4.3 uses standard L-smoothness and convexity assumptions and only uses the definition of η_t^g through the identity h_t(η_t^g) = m_t; that identity is algebraic, not an assumed conclusion. The A.P.C. condition is borrowed transparently from external prior work (Jhunjhunwala et al. 2023) as an explicit assumption, not smuggled in as an ansatz, and the authors do not rely on a self-citation chain to force the result. The momentum-variant minimax claim is not actually established by the Appendix C proof, which treats v_t = Δ̄_t rather than the FedDuAdam momentum vector; however, that is a proof gap or correctness concern, not an input-output circularity. No step in the paper reduces, by definition or by self-citation, to its own inputs.
Assumptions & free parameters
assumptions (6)
- domain assumption Approximate Projection Condition (Assumption 3.1): (1/M) sum_i ||w^t + Delta_i^t - w*||^2 <= ||w^t - w*||^2
- domain assumption Strong A.P.C.: <Delta_bar^t, w* - w^t> >= (1/M) sum_i ||Delta_i^t||^2
- domain assumption Local loss functions F_i are convex and L-smooth, and (1/M) sum_i ||grad F_i(w*)||^2 <= sigma_*^2
- ad hoc to paper Full-batch SGD and full client participation in the convergence analysis
- ad hoc to paper Anisotropic decay model [Delta_bar^t]_k = Theta(a_t k^{-beta}) with beta > 1 and a_t decreasing
- domain assumption H, the set of w* satisfying A.P.C., is nonempty
Cite this review
Pith. "Pith review of FedDuA: Doubly Adaptive Federated Learning." pith.science (2026). https://pith.science/paper/Q7B437BX
@misc{pith2026250511126,
author = {Pith},
title = {Pith review of: FedDuA: Doubly Adaptive Federated Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/Q7B437BX}},
note = {Machine review of arXiv:2505.11126}
}
read the original abstract
Federated learning is a distributed learning framework where clients collaboratively train a global model without sharing their raw data. FedAvg is a popular algorithm for federated learning, but it often suffers from slow convergence due to the heterogeneity of local datasets and anisotropy in the parameter space. In this work, we formalize the central server optimization procedure through the lens of mirror descent and propose a novel framework, called FedDuA, which adaptively selects the global learning rate based on both inter-client and coordinate-wise heterogeneity in the local updates. We prove that our proposed doubly adaptive step-size rule is minimax optimal and provide a convergence analysis for convex objectives. Although the proposed method does not require additional communication or computational cost on clients, extensive numerical experiments show that our proposed framework outperforms baselines in various settings and is robust to the choice of hyperparameters.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Here, we used the induction hypothesis for the first inequality and strong A.P.C
1 M MX i=1 ∆t i 2 +β 1mt−1 = 2mt. Here, we used the induction hypothesis for the first inequality and strong A.P.C. for the second inequality. Then, we obtain the result by induction. C Proof for Theorem 4.1 Givenw∈R d, the worst-case distance difference is defined as V(w) = sup w∗∈H V(w,w ∗) := sup w∗∈H Dψt(w∗|w)−D ψt(w∗|w t). From the definition of the ...
work page 2023
-
[3]
For ViT experiments, we use the architecture in omiita [2024]
We also provide the architecture of LSTM and CNN in Table 4 and Table 5, respectively. For ViT experiments, we use the architecture in omiita [2024]. Table 3: Datasets and models used in our experiments Dataset Task Model # of Classes License Synthetic dataset Regression Linear N/A N/A CIFAR-10 Image classification ResNet-18 10 MIT License CIFAR-100 Image...
work page 2024
-
[4]
S. P. Karimireddy, M. Jaggi, S. Kale, M. Mohri, S. J. Reddi, S. U. Stich, and A. T. Suresh. Mime: Mim- icking centralized stochastic algorithms in federated learning.arXiv preprint arXiv:2008.03606, 2020a. S. P. Karimireddy, S. Kale, M. Mohri, S. Reddi, S. Stich, and A. T. Suresh. Scaffold: Stochastic controlled averaging for federated learning. InInterna...
arXiv 2008
-
[5]
We see that FedDuA consistently outperforms other methods in terms of convergence speed and final accuracy. H.2 Comparison with FedProx 0 100 200 300 400 500 Communication Rounds 30 40 50 60 70Test Accuracy (%) CIFAR100 FedProx FedDuAdaGrad + FedProx FedAvg FedDuAdaGrad Figure 6: Comparison with FedProx. In this section, we provide a comparison with FedPr...
work page 2020
-
[6]
Other hyperparameters are kept the same across all methods. Following Jhunjhunwala et al. [2023], we use weight decay of 10−4, learning rate decay of0.998, and gradient clipping to stabilize the training for image classification tasks. Synthetic datasetFor the synthetic dataset, We tune ηl over{10−3,10−5/2,10−2,10−3/2,10−1}, andηg over{10−1,10−1/2,10 0,10...
work page 2023
-
[8]
URL https://openreview.net/ forum?id=aBO5SvgSt1. A. Tomihari and I. Sato. Understanding Why Adam Outperforms SGD: Gradient Heterogeneity in Transformers.arXiv preprint arXiv:2502.00213,
-
[15]
Image classification datasetsFor the image classification tasks, we tune ηl over {10−2,10−3/2,10−1,10−1/2,10 0}. The grid of ηg is{10−1,10−1/2,10 0,10 1/2,10 1} for Fe- dAvg(M) and SCAFFOLD, and{10−4,10−7/2,10−3,10−5/2,10−2} for FedOPT. The grid of ϵg is{10−3,10−5/2,10−2,10−3/2,10−1} for FedDuA, and{10−4,10−7/2,10−3,10−5/2,10−2} for FedExP(M). We fixϵ= 10...
work page 2000
- [2011]
Show all 16 references
-
[2015]
Koneˇcn`y, H
J. Koneˇcn`y, H. B. McMahan, F. X. Yu, P. Richtárik, A. T. Suresh, and D. Bacon. Federated learning: Strategies for improving communication efficiency.arXiv preprint arXiv:1610.05492,
-
[2016]
S. H. Lee, S. Sharma, M. Zaheer, and T. Li. Efficient Adaptive Federated Optimization.arXiv preprint arXiv:2410.18117,
-
[2019]
Caldas, S
S. Caldas, S. M. K. Duddu, P. Wu, T. Li, J. Koneˇcn`y, H. B. McMahan, V . Smith, and A. Talwalkar. Leaf: A benchmark for federated settings.arXiv preprint arXiv:1812.01097,
-
[2020]
Haddadpour and M
F. Haddadpour and M. Mahdavi. On the convergence of local descent methods in federated learning. arXiv preprint arXiv:1910.14425,
1910 arXiv
-
[2021]
C. Xie, O. Koyejo, I. Gupta, and H. Lin. Local adaalter: Communication-efficient stochastic gradient descent with adaptive learning rates.arXiv preprint arXiv:1911.09030,
1911 arXiv
-
[2022]
Nielsen, J.-D
F. Nielsen, J.-D. Boissonnat, and R. Nock. Bregman voronoi diagrams: Properties, algorithms and applications.arXiv preprint arXiv:0709.2196,
-
[2023]
ForR 1, we have R1 = 1 M MX i=1 ⟨wt−w∗,∆t i⟩ = ηl M MX i=1 τ−1X k=0 ⟨wt−w∗,∇Fi(wt i,k)⟩
for the last inequality. ForR 1, we have R1 = 1 M MX i=1 ⟨wt−w∗,∆t i⟩ = ηl M MX i=1 τ−1X k=0 ⟨wt−w∗,∇Fi(wt i,k)⟩. As shown in the proof of Theorem 1 in Jhunjhunwala et al. [2023], the right-hand side can be bounded as R1≥η lτ(F(wt)−F(w ∗))− ηlL 2M MX i=1 τ−1X k=0 wt i,k−wt 2 ....
2023
-
[2025]
J. Wang, Z. Xu, Z. Garrett, Z. Charles, L. Liu, and G. Joshi. Local adaptivity in federated learning: Convergence and consistency.arXiv preprint arXiv:2106.02305,
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.