REVIEW 4 major objections 6 minor 16 references
Distributed Sign Momentum with Local Steps for Training Transformers
T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Distributed sign momentum with local steps matches centralized rates and beats SlowMo.
desk verdict Useful empirical package with a plausible sign-momentum local-step method, but the main theorem's proof as printed doesn't establish the claimed rate; worth engaging after a corrected proof and stronger baselines. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Three parts carry the argument. The inner loop is any off-the-shelf base optimizer, whose local update directions $d^{(i)}_{t,k}$ are accumulated over $\tau$ steps; after an all-reduce, the scaled difference $(x_{t,0} - x_{t,\tau})/\gamma_t$ acts as a pseudo-gradient. The global step is Lion-style sign momentum: an auxiliary vector $u_{t+1}$ mixes the momentum buffer with the pseudo-gradient, the outer iterate moves by the sign of $u_{t+1}$ plus decoupled weight decay, and the momentum buffer $m_{t+1}$ is updated separately with coefficient $\beta_2$, allowing $\beta_2 > \beta_1$ for extra acceleration. The theory uses a randomized sign operator $S_r(v)$, defined so that $\mathbb{E}_S[S_r(v)] = v/B$ with $B = \tau R$, as a continuous surrogate for proofs with generic base optimizers, while the SGD theorem uses the actual sign and controls the momentum estimation error $\epsilon_{t+1} = m_{t+1} - \tau \nabla f(x_{t,0})$; the key identity is a descent lemma showing $\mathbb{E}\|\nabla f(x_{t,0})\|_1$ shrinks at the centralized sign-momentum rate, with local-step effects confined to lower-order terms.
What would settle it
Measure the running maximum $\max_{i,t,k}\|d^{(i)}_{t,k}\|_2$ during a GPT-2 pretraining run with AdamW as the base optimizer: if it grows without bound while the training loss is still decreasing, then Assumption 3 is violated and the stated $O(1/T^{1/4})$ guarantee is not what the experiment is exhibiting.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that taking the sign of a Lion-style momentum update on the accumulated local-step difference is a legitimate distributed optimizer: the outer iterate moves by $x_{t+1,0} = x_{t,0} - \eta\gamma_t(\mathrm{sign}(u_{t+1}) + \lambda x_{t,0})$, with $u_{t+1} = \beta_1 m_t + (1-\beta_1)(x_{t,0}-x_{t,\tau})/\gamma_t$. The momentum buffer itself evolves as $m_{t+1} = \beta_2 m_t + (1-\beta_2)(x_{t,0}-x_{t,\tau})/\gamma_t$, so the local steps generate a pseudo-gradient with no extra communication. The main theorem states that with SGD as the base optimizer and the actual sign operator, the average $\ell_1$ norm of the true gradient is bounded by terms whose leading order is $O(1/T^{1/4})$ in expectation for smooth nonconvex costs, under a uniform bound on local update directions. A randomized sign operator that is unbiased in expectation extends the analysis to generic base optimizers, and the GPT-2 Small, Medium, and Large experiments using AdamW, plus one Small experiment using Sophia, as base optimizers report lower validation losses than SlowMo at the same communication intervals, with less than half the performance drop of per-step AdamW in several configurations. The framework reduces to signed Lookahead with one worker and to signed SlowMo when $\beta_1 = \beta_2$.
Load-bearing premise
The load-bearing premise is Assumption 3: every local update direction at every worker and every iteration is uniformly bounded in Euclidean norm by a single constant $R$; if real gradients or normalized optimizer directions can grow without bound, the paper's convergence guarantees stop covering the algorithm as actually run.
Editorial extensions
If this is right
- Transformer pretraining can cut synchronization frequency by a factor of $\tau$ (12, 24, or 36 in the experiments) while, according to the reported runs, beating the strong local-step baseline SlowMo in validation loss.
- Local steps do not worsen the worst-case nonconvex rate: the SGD instance matches centralized sign momentum's $O(1/T^{1/4})$ bound in $\ell_1$ gradient norm.
- Because the global step is optimizer-agnostic, improvements like Sophia can be dropped into the local loop and still benefit from sign momentum on top of local steps.
- The reduction to signed Lookahead at $n=1$ suggests the same global update can serve as a general optimizer wrapper in single-machine settings, not only in distributed training.
Reading between the lines
- An implication the paper leaves implicit is that the global sign step could itself be compressed: once only the sign of $u_{t+1}$ matters, the communicated model differences might be reduced to 1-bit values without changing the outer update's functional form.
- The stated rates rely on Assumption 3, that every local update direction is uniformly bounded by $R$; for SGD this is a real restriction, and for AdamW or Sophia base optimizers, whose update directions are not globally bounded, the theorems do not directly cover the experimental regime.
- The ablation gap between signed SlowMo and Algorithm 1 suggests the $\beta_2 > \beta_1$ momentum asymmetry, not just the sign operation, is doing part of the work; a matched comparison at $\beta_1 = \beta_2$ versus $\beta_2 > \beta_1$ would isolate that contribution.
- The theory's linear-speedup term $\sqrt{d/(\tau n)}$ predicts the method is strongest when stochastic noise is large; testing with smaller per-worker batch sizes and larger $\tau$ would directly probe that prediction.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Algorithm 1, a distributed optimizer that performs several local steps with an arbitrary base optimizer and then applies a global Lion-style sign momentum update, using the accumulated local difference as a pseudo-gradient. The theoretical part gives a generic analysis for a randomized sign operator (Theorems 1 and 2), and claims an O(1/T^{1/4}) l1-gradient-norm rate for SGD local steps with the actual sign operator (Theorem 3). The experimental part compares Algorithm 1 with SlowMo for GPT-2 Small, Medium, and Large pretraining at 12x, 24x, and 36x communication reduction, and includes ablations with signed Lookahead, signed SlowMo, and global AdamW steps.
Significance. If Theorem 3 were correct, the paper would make a meaningful theoretical contribution: it would extend centralized sign-momentum convergence guarantees to the local-step distributed setting while preserving the known O(1/T^{1/4}) l1 rate. The empirical study is also potentially useful: the comparison with SlowMo on three GPT-2 sizes at several communication intervals, plus the ablation study isolating the role of the sign operation, is a real practical contribution. The manuscript is generally clearly written, the algorithm is simple to implement, the appendix contains substantial proof detail, and the authors provide a code link. However, as detailed below, the central theoretical rate is not established by the printed proof, and one row of the main empirical table contradicts the claimed consistency of the experiments.
major comments (4)
- [Appendix B.3.3, Eq. (43)] The bound on E3 uses ||x^{(i)}_{s,k} - x_{s,0}|| ≤ η k R, but the local update in Eq. (4) gives x^{(i)}_{s,k} - x_{s,0} = -γ Σ_{j<k} d^{(i)}_{s,j}, so under Assumption 3 this norm is at most γ k R, not η k R. Replacing η by γ in Eq. (43) changes the final term of Eq. (44) from (1/2)τ^2 η L R to (1/2)τ^2 γ L R. Substituting this corrected bound into Lemma 2 and taking η = 1/(L T^{3/4}) and 1-β = 1/√T, the final convergence inequality acquires a term √d τ γ L R that is independent of T for constant γ, whereas the theorem statement claims the last term is (√d τ R + γd/2)/T^{3/4}. Thus the printed proof does not establish the advertised O(1/T^{1/4}) rate, and in this proof no choice of constant γ can make all terms vanish.
- [Table 2] In the GPT-2 Large, 24x communication-reduction row, Algorithm 1 reports a final validation loss of 2.864, which is worse than SlowMo's 2.733, yet the table reports a 14.00% improvement and the text states that Algorithm 1 achieves 'consistent and robust performance improvements versus SlowMo under different communication intervals, τ = 12, 24, 36, across all model sizes'. As printed, this entry directly contradicts the empirical consistency claim; the number or the claim should be corrected.
- [Assumption 3 and §4 experiments] Assumption 3 requires every local update direction to satisfy ||d^{(i)}_{t,k}|| ≤ R uniformly. This may be a usable bounded-gradient condition for the SGD theory in Theorem 3, but the GPT-2 experiments use AdamW and Sophia as base optimizers. For these optimizers the update directions are not globally bounded in general: AdamW involves componentwise normalization plus weight decay, and Sophia adds a diagonal preconditioner. The convergence guarantees therefore do not cover the optimizer configurations for which the paper's main empirical claims are made. The authors should either prove a uniform bound for the optimizers actually used or explicitly state that the theory applies only to SGD local steps.
- [Theorem 3 / Remark 2] The theorem is called 'optimal' in the text, but no lower bound is proved; the claim that the rate matches the best-known centralized sign-momentum rate would be more precise and still sufficient to convey the contribution.
minor comments (6)
- [Section 1.2] The phrase 'its O(1/T^{-1/4}) convergence rate' should read O(T^{-1/4}), since the exponent is already negative in the big-O notation.
- [Tables 2 and 4] The header typo 'AadmW' should be 'AdamW'.
- [Assumption 3] The sentence contains the typo 'bounded at at all iteration'; it should read 'bounded at all iterations'.
- [Appendix, Algorithm 5] In the pseudocode for SlowMo, the base-optimizer update direction is indexed as d^{(i)}_{t,i}; this should be d^{(i)}_{t,k}.
- [Appendix B.3.3] The phrase 'the low of total expectations' should be 'the law of total expectations'.
- [Theorem 3 statement] The statement does not specify how the local learning rate γ is chosen; given the dependence of the corrected bound on γ, the parameter choice should be made explicit in the theorem.
Circularity Check
No circularity found: the convergence analysis is self-contained and the empirical claims are benchmarked against external baselines.
full rationale
The paper's derivations do not reduce to their inputs by construction. The convergence theorems are obtained from explicit regularity assumptions (L-smoothness, bounded variance, uniform bound on local update directions) and do not fit any parameter to make the rate appear. The randomized sign operator in Lemma 1 is deliberately constructed to satisfy E_S[S_r(v)] = v/B; this is an explicit linearization device used to analyze a relaxed algorithm, not a hidden reuse of the target result, and Theorem 3 analyzes the actual sign operator. The proof of Theorem 3 adapts the centralized sign-momentum analysis of Sun et al. (2023), which is an external, cited result, and the paper does not invoke any load-bearing self-citation or uniqueness theorem from the authors' own prior work. The empirical section compares Algorithm 1 against SlowMo, AdamW, and Sophia, all external baselines, with hyperparameters tuned independently; no quantity is fitted and then reported as a prediction. A reviewer's concern that Eq. (43) bounds the local drift with the global step size eta rather than the local learning rate gamma would be an internal correctness issue, not circularity: even if the bound is wrong, the theorem's conclusion is not identical to an assumption or to a fitted value. Accordingly, the derivation chain is self-contained and the circularity score is 0.
Assumptions & free parameters
free parameters (4)
- Global learning rate eta for Algorithm 1 in experiments =
not reported; tuned from {0.1, 0.5, 0.8, 1.0, 1.2, 1.5, 2.0}
- SlowMo momentum coefficient and global learning rate =
not reported
- Step-size schedule in Theorems 1-3 =
eta = 1/(L T^{3/4}), 1 - beta = 1/sqrt(T), gamma = (R/eta) sqrt(n tau / T)
- Peak learning rates for GPT-2 runs =
5e-4 (Small), 2e-4 (Medium), 2e-4 (Large)
assumptions (6)
- domain assumption Each local function f_i is differentiable and L-smooth (Assumption 1).
- domain assumption The average update direction d_{t,k} has bounded variance zeta^2 (Assumption 2).
- domain assumption Local update directions are uniformly bounded by R in l2 norm (Assumption 3).
- domain assumption Per-worker stochastic gradient variance is bounded by sigma^2 and heterogeneity by delta^2 (Theorem 2).
- domain assumption Stochastic gradient noise is independent across workers (used in E2 bound).
- standard math Known local SGD iterate deviation bound from Wang et al. 2019 and Yu et al. 2019.
Cite this review
Pith. "Pith review of Distributed Sign Momentum with Local Steps for Training Transformers." pith.science (2026). https://pith.science/paper/FW6K2W2X
@misc{pith2026241117866,
author = {Pith},
title = {Pith review of: Distributed Sign Momentum with Local Steps for Training Transformers},
year = {2026},
howpublished = {\url{https://pith.science/paper/FW6K2W2X}},
note = {Machine review of arXiv:2411.17866}
}
abstract
Pre-training Transformer models is resource-intensive, and recent studies have shown that sign momentum is an efficient technique for training large-scale deep learning models, particularly Transformers. However, its application in distributed training remains underexplored. This paper investigates a novel communication-efficient distributed sign momentum method with multiple local steps, to cope with the scenarios where communicating at every step is prohibitive. Our proposed method allows for a broad class of base optimizers for local steps, and uses sign momentum in the global step, where momentum is generated from differences accumulated during local steps. For generic base optimizers, by approximating the sign operator with a randomized version that acts as a continuous analog in expectation, we present a general convergence analysis, which specializes to an $O(1/\sqrt{T})$ rate for a particular instance. When local step is stochastic gradient descent, we show an optimal $O(1/T^{1/4})$ rate in terms of $\ell_1$ gradient norm for nonconvex smooth cost functions. We extensively evaluate our method on the pre-training of various sized GPT-2 models from scratch, and the empirical results show significant improvement compared to other distributed methods with multiple local steps.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
τ R η − 1 2 + 8τ β2 (1 − β)2 + 1 τ i . B.2 Proof of Theorem 2 Proof. Since d(i) t,k = ∇fi(x(i) t,k, ξ(i) t,k), we have Et,k[dt,k] = 1 n Pn i=1 ∇fi(x(i) t,k). Then, ∥∇f (xt,k) − Et,k[dt,k]∥2 = ∥ 1 n nX i=1 ∇fi(xt,k) − 1 n nX i=1 ∇fi(x(i) t,k)∥2 ≤ L2 n nX i=1 ∥xt,k − x(i) t,k∥2, (32) where in the last inequality we use Jensen’s inequality and L-smoothness o...
work page 2019
-
[3]
Marina: Faster non-convex distributed learning with compression
Eduard Gorbunov, Konstantin P Burlachenko, Zhize Li, and Peter Richt´ arik. Marina: Faster non-convex distributed learning with compression. In International Conference on Machine Learning , pages 3788–3798. PMLR, 2021a. Eduard Gorbunov, Filip Hanzely, and Peter Richt´ arik. Local sgd: Unified theory and new efficient methods. In International Conference ...
-
[7]
Communication efficient distributed training with distributed lion
Bo Liu, Lemeng Wu, Lizhang Chen, Kaizhao Liang, Jiaxu Zhu, Chen Liang, Raghuraman Krishnamoorthi, and Qiang Liu. Communication efficient distributed training with distributed lion. arXiv preprint arXiv:2404.00438, 2024a. Hong Liu, Zhiyuan Li, David Leo Wright Hall, Percy Liang, and Tengyu Ma. Sophia: A scalable stochastic second-order optimizer for langua...
-
[8]
Sgdr: Stochastic gradient descent with warm restarts
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983,
-
[12]
Slowmo: Improving communication- efficient distributed sgd with slow momentum
Jianyu Wang, Vinayak Tantia, Nicolas Ballas, and Michael Rabbat. Slowmo: Improving communication- efficient distributed sgd with slow momentum. arXiv preprint arXiv:1910.00643 ,
arXiv 1910
-
[13]
Component-wise vector multi- plication g2 t = gt ⊙gt, and √bvt means component- wise square root
with decoupled weight decay proposed by Loshchilov (2017). Component-wise vector multi- plication g2 t = gt ⊙gt, and √bvt means component- wise square root. Require: Initialization x0, momentum coeffi- cients β1, β2; weight decay λ; learning rate η; ϵ = 10−8 m0 ← 0, v0 ← 0 for t = 0 to T − 1 do Stochastic gradient: gt ← ∇f (xt, ξt) Momentum updates: mt+1 ...
work page 2017
-
[15]
For the iterates {xt,0} in Algorithm 1 with β1 = β2 = β, λ= 0, it satisfies that 1 T T −1X t=0 E ∥∇f (xt,0) ∥1 ≤ f (x0,0) − f∗ ηγT + 2 √ d τ T T −1X t=0 E ∥ϵt+1∥ + ηγdL 2 . (38) Proof. The proof of this Lemma is adapted from Sun et al. (2023). From Assumption 1 we have f (xt+1,0) ≤ f (xt,0) + ∇f (xt,0)⊤(xt+1,0 − xt,0) + L 2 ∥xt+1,0 − xt,0∥2 ≤ (37) f (xt,0...
work page 2023
-
[24]
The results show that Local AdamW is significantly slower than both SlowMo and our method, consistent with the findings of Sun et al. (2024). 20 40 60 80 100 Iteration Count (k Steps) 2.9 3.0 3.1 3.2 3.3 3.4 3.5Validation Losses GPT-2 Small 124M, interval = 12 AdamW Local AdamW SlowMo + AdamW, τ = 12 Algorithm 1 + AdamW, τ = 12 80 85 90 95 100 2.90 2.95 3...
work page 2024
Show all 16 references
-
[2012]
An image is worth 16x16 words: Transformers for image recognition at scale
Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929,
2010 arXiv
-
[2013]
Fedlion: Faster adaptive federated optimization with fewer communication
Zhiwei Tang and Tsung-Hui Chang. Fedlion: Faster adaptive federated optimization with fewer communication. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 13316–13320. IEEE,
2024
-
[2014]
Noise is not the main factor behind the gap between sgd and adam on transformers, but sign descent might be
Frederik Kunstner, Jacques Chen, Jonathan Wilder Lavington, and Mark Schmidt. Noise is not the main factor behind the gap between sgd and adam on transformers, but sign descent might be. arXiv preprint arXiv:2304.13960,
-
[2017]
How to scale distributed deep learning? arXiv preprint arXiv:1611.04581 ,
Peter H Jin, Qiaochu Yuan, Forrest Iandola, and Kurt Keutzer. How to scale distributed deep learning? arXiv preprint arXiv:1611.04581 ,
-
[2019]
Scalable training of deep learning machines by incremental block training with intra-block parallel optimization and blockwise model-update filtering
Kai Chen and Qiang Huo. Scalable training of deep learning machines by incremental block training with intra-block parallel optimization and blockwise model-update filtering. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 5880...
2016
-
[2020]
Adam: A method for stochastic optimization
Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 ,
-
[2021]
1-bit stochastic gradient descent and its application to data-parallel distributed training of speech dnns
Frank Seide, Hao Fu, Jasha Droppo, Gang Li, and Dong Yu. 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech dnns. In Interspeech, volume 2014, pages 1058–1062. Singapore,
2014
-
[2023]
Co2: Effi- cient distributed training with full communication-computation overlap
Weigao Sun, Zhen Qin, Weixuan Sun, Shidi Li, Dong Li, Xuyang Shen, Yu Qiao, and Yiran Zhong. Co2: Effi- cient distributed training with full communication-computation overlap. arXiv preprint arXiv:2401.16265 ,
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.