Pith. sign in

REVIEW 2 major objections 4 minor 61 references

HALoS: Hierarchical Asynchronous Local SGD over Slow Networks for Geo-Distributed Large Language Model Training

T0 review · 2 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A two-tier parameter-server scheme, HALoS trains geo-distributed LLMs up to 7.5x faster than DiLoCo and 2.1x faster than Async-Local-SGD while matching synchronous-SGD quality, backed by a non-convex convergence bound.

desk verdict A genuinely useful hierarchical async-local-SGD system with careful simulation, but the advertised convergence proof analyzes a different update rule than Algorithm 1, so the theoretical claim does not yet cover the reported method. read the letter →

arxiv 2506.04531 v1 pith:PNW35G4C submitted 2025-06-05 cs.LG

Geon-Woo Kim , Junbo Li , Shashidhar Gandham , Omar Baldonado , Adithya Gangidi , Pavan Balaji , Zhangyang Wang , Aditya Akella This is my paper
classification cs.LG MSC 68W1590C26
keywords hierarchicalasynchronoustraininglocalSGDgeo-distributedLLMparameterserverglobalmodelmergingnon-convexconvergenceanalysismomentumtrace-drivensimulation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

HALoS is a training method for large language models whose accelerators sit in different geographical regions connected by slow, low-bandwidth links. The paper's claim is that inserting a local parameter server inside each region — so workers talk only to a nearby server over fast links, that server accumulates their updates and periodically ships a single delta to a distant global server, and the returning global model is blended into the local model instead of replacing it — eliminates most of the idle waiting that slows both synchronous and flat asynchronous methods. If the claim holds, geo-distributed pretraining becomes practical at near-synchronous quality: experiments report up to 7.5x faster convergence to the same validation loss than DiLoCo, up to 2.1x faster than Async-Local-SGD, and a 68.6x time advantage over fully synchronous SGD, with matching or better benchmark accuracy. The paper also proves a non-convex convergence bound, Theorem 4.6, which it uses to explain why local momentum should stay high while global momentum must be moderate.

What carries the argument

The argument is carried by the two-tier update loop with server-side accumulation and $\alpha$-weighted global model merging. Workers take $H$ local SGD steps and send a gradient delta to their regional LPS; the LPS applies it with local momentum $\beta_l$, and every $K$ updates ships the accumulated difference $\Delta = \theta_t - \theta_{t_{\mathrm{last}}}$ to the GPS; the GPS applies it with global momentum $\beta_g$ and returns the new global model, which the LPS absorbs as $\theta_t \leftarrow (1-\alpha)\theta_t + \alpha\Theta_i$. The proof of Theorem 4.6 recasts the LPS step as a single combined update $\theta_{l,t+1} = (1-\alpha)\theta_{l,t} + \alpha\Theta_{lg(l,t)} - \eta_t[(1-\beta_l)\nabla_l(\theta_{l,t}) + \beta_l m_{l,t+1}]$, and the structure of the resulting bound — momentum terms of order $1/(1-\beta_l)$ locally but $\frac{1}{\beta_g^3(1-\beta_g)^3}$ globally — yields the paper's practical prescription: keep local momentum near 0.9, keep global momentum near 0.5, and drop global momentum entirely when regional data is strongly non-i.i.d.

What would settle it

A concrete check would be to replay Algorithm 1's exact event ordering (gradient updates whenever a worker's delta arrives, merge only when a global model returns) in the same simulator and compare the resulting model trajectories with the combined update rule $\theta_{l,t+1} = (1-\alpha)\theta_{l,t} + \alpha\Theta_{lg(l,t)} - \eta_t[(1-\beta_l)\nabla_l(\theta_{l,t}) + \beta_l m_{l,t+1}]$ that the proof analyzes; any material divergence means the theorem is not bounding the measured method. The decisive empirical falsifier is a live geo-distributed run in which HALoS fails to match the synchronous-SGD validation loss in the claimed time, or diverges when the worker-to-LPS ratio is uneven.

Watch

Extended reading notes

Core claim

On the paper's own terms, HALoS establishes that a two-level parameter-server hierarchy lets asynchronous local SGD hide both slow inter-region communication and straggling heterogeneous workers: each regional local server keeps applying momentum-based gradient updates while awaiting the slow global round trip, and when a fresh global model arrives it is merged in with weight $\alpha \approx 0.25$ rather than overwriting the local model, so no local progress is discarded. Empirically, the claim is that Pythia-70M/160M/410M models trained this way reach the same validation loss as fully synchronous SGD training — matching or exceeding it on eight downstream benchmarks — while cutting time-to-loss by up to 7.5x versus DiLoCo, 3.9x versus DiLoCo with dynamic updates, 2.1x versus Async-Local-SGD, and 68.6x versus synchronous SGD in the simulated geo-distributed environment. The theoretical claim is Theorem 4.6, a bound of the form $O(1/T)$ plus constant terms governed by heterogeneity $\sigma^2$ and asynchronous staleness $D_g^2 + D_l^2$, which the paper presents as the first convergence guarantee for hierarchical asynchronous distributed optimization with momentum.

Load-bearing premise

The convergence proof covers a simplified local-server update in which merging with the global model and applying the worker gradient happen together in one step, but the actual algorithm merges with a freshly received global model only occasionally, after several gradient updates have already been applied; if these two ways of updating are not the same, the mathematical guarantee does not cover the algorithm whose speedups the paper reports.

Editorial extensions

If this is right

  • Geo-distributed LLM pretraining can be made communication-efficient without a model-quality penalty: HALoS matches or exceeds synchronous-SGD accuracy on all eight reported downstream benchmarks while shortening time-to-loss.
  • The convergence bound gives a concrete rule for setting momentum in hierarchical asynchronous training (high local momentum near 0.9, moderate global momentum near 0.5) and warns that global momentum should be reduced or disabled when regions hold conflicting data.
  • Server-side update accumulation (K around 32) cuts the global server's workload and stabilizes cross-region gradient variance, so the approach scales to larger numbers of regions and workers.
  • The method carries over to other model families (Llama, Qwen) with hyperparameters tuned once on a small Pythia model, and composes with model parallelism by treating each worker as an accelerator group.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • All wall-clock numbers come from a trace-driven simulator that combines H100 profiling with published inter-region bandwidth measurements; a live multi-region deployment would be the decisive test of whether the 7.5x and 2.1x speedups survive real network variability.
  • The proof in Appendix E analyzes a combined merge-plus-gradient update, while Algorithm 1 merges only when a fresh global model arrives, possibly after several gradient updates; an equivalence argument or a proof for the event-driven rule is needed before Theorem 4.6 can be said to govern the measured algorithm.
  • The observed divergence under naive one-LPS-per-region grouping when worker counts differ (2 versus 6 per region) suggests HALoS's gains rest on balanced LPS workloads, making adaptive LPS formation a natural next step.
  • A testable extension falls out of the theory: the optimal global momentum should decrease as cross-region data heterogeneity increases, a prediction the Shakespeare experiment already points toward.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper proposes HALoS, a hierarchical asynchronous local-SGD framework for geo-distributed LLM training. HALoS places a local parameter server (LPS) in each region and a global parameter server (GPS) that receives accumulated updates from LPSs; workers perform H local SGD steps and send deltas to their LPS, which is allowed to keep updating while a stale global model is in flight and later merges the global model with the local model using a weight alpha. The paper contributes an algorithm, a non-convex convergence theorem (Theorem 4.6) with a proof in Appendix E, and a trace-driven simulation study reporting up to 7.5x faster time-to-loss than DiLoCo and up to 2.1x faster than Async-Local-SGD, with benchmark accuracy matching or exceeding synchronous SGD.

Significance. If Theorem 4.6 genuinely governed Algorithm 1, the paper would be a useful contribution to geo-distributed LLM training: it combines a concrete hierarchical design, a convergence analysis that explicitly separates local and global momentum effects, and an unusually careful simulation methodology using measured inter-region bandwidths, profiled H100 step times, and a published implementation. The empirical section is a strength: the trace-driven simulator is described in enough detail to be reproduced, the hyperparameter sweep is disclosed, and the baselines include DiLoCo with dynamic updates, which isolates the benefit of asynchrony. The main significance hinges on the theoretical claim, and that claim currently applies to an update rule that differs from the algorithm whose speedups are reported. With a corrected or properly scoped proof, the paper could be accepted; without it, the central claim of a rigorous convergence analysis is not established.

major comments (2)
  1. [Appendix E vs. Algorithm 1] Theorem 4.6 is proved for a LPS update rule that is not the rule in Algorithm 1. The recurrence for theta_{l,t+1} displayed in Appendix E immediately after Assumption E.5 applies the merge with the global model at every LPS time step, whereas Algorithm 1 (lines 9-20) applies a momentum-based ModelUpdate when a worker delta arrives and applies the convex combination theta_t <- (1-alpha)theta_t + alpha Theta_i only when a new global model arrives; with K > 1 there are gradient-only updates between merges. These two operations do not commute: in the proof's combined update the gradient step is scaled by (1-alpha) and the stale Theta pulls the model at every step, while Algorithm 1 takes pure gradient steps and then blends once. The proof also uses a gradient evaluated at the current LPS model, while the worker sends a delta produced after H local SGD steps, and the experiments (Appendix D) state that HALoS uses delayed Nesterov momentum with delays d_g and d_l, which do not appear in Algorithm 1 or in the proof. The GPS update in the proof is a convex combination of local gradients and momenta, not the difference delta = theta_t - theta_{t_last} received in Algorithm 1. Consequently, Theorem 4.6 does not, as written, govern the algorithm whose speedups are reported. This is load-bearing for the paper's claim of a rigorous convergence analysis, so the proof must be repaired or the theorem must be explicitly scoped to an idealized variant with an equivalence argument.
  2. [Section 4.2, Assumption 4.5 and Eq. (2)] The bounded-staleness constants D_g and D_l are assumed but never bounded in terms of the algorithm's parameters or the network model. The second constant term in Theorem 4.6 is O(L^2(D_g^2 + D_l^2)), yet the paper does not derive any dependence of D_g or D_l on H, K, alpha, worker speeds, or inter-region bandwidth/latency. As stated, the theorem cannot explain the central trade-off that larger K reduces communication but increases staleness, nor can it yield the claimed beta_g = 0.5 recommendation; those conclusions are supported only by the hyperparameter sweep in Figure 5. Please derive explicit bounds on D_g and D_l from the algorithm parameters (or state that the theorem is conditional on unknown staleness constants), and connect those bounds to the communication model used in the simulations.
minor comments (4)
  1. [Section 4.2, Theorem 4.6 statement] The theorem statement says 'Under certain assumptions' without naming them; it should explicitly reference Assumptions 4.1-4.5 so that the main theorem is self-contained.
  2. [Section 4.2, tightness claim] The sentence 'Overall, our derived bound is tight considering both perspectives' is not supported by a matching lower bound or a constructed example; please temper this claim or provide a formal tightness argument.
  3. [Appendix E, last displayed chain] In the final displayed chain of the proof, the term D_l^2 appears without the L^2 factor that multiplies D_g^2 and appears in the preceding lines; this appears to be a typo but should be fixed for consistency.
  4. [Section 5.1, benchmark comparison] The text says the HALoS model is trained on 223B tokens to match Pythia checkpoints trained on 207B tokens; please clarify whether the extra tokens are required by the method or are an artifact of the training schedule, since token counts are reported as a key efficiency metric.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the convergence proof is self-contained under standard assumptions, the beta_g recommendation follows from the derived bound rather than from a fitted value, and no load-bearing self-citation is present.

full rationale

The paper's derivation chain is self-contained. Theorem 4.6 is proved in Appendix E under explicit, standard assumptions (L-smoothness, bounded variance, bounded heterogeneity, bounded gradients, and bounded staleness), and the proof does not invoke any fitted value or empirical measurement as an input. The recommendation that beta_g be around 0.5 follows analytically from the functional form of the bound, O(1/(beta_g^3(1-beta_g)^3)), whose approximate minimizer is x=0.5; this is a mathematical consequence of the proof, not a parameter estimated from the experiments. The later hyperparameter sweep independently selects beta_g=0.5, but the theoretical statement does not depend on that sweep. There are no load-bearing self-citations: the cited prior results are external and used as comparisons or background, and the novelty claim rests on the paper's own derivation. The main concern noted by the reader is that the proof's LPS recurrence combines the gradient update and global-model merge at every step, whereas Algorithm 1 performs gradient updates on worker-delta arrivals and merges only when a new global model is received. That is a potential correctness gap between the analyzed model and the implemented algorithm, but it is not a circular reduction: the proof does not assume the conclusion, and the mismatch does not make the theorem's output equivalent to its input by construction. Therefore, on the circularity axis, the paper receives a score of 0.

Assumptions & free parameters 5 free parameters · 7 assumptions · 0 invented entities

The free parameters are all selected by hyperparameter sweep on Pythia-70M, not derived from theory. The theorem relies on five standard but unquantified assumptions and one unstated equivalence between Algorithm 1 and the Appendix E update rule. No invented physical or mathematical entities are introduced.

free parameters (5)
  • global momentum beta_g = 0.5
    Selected by hyperparameter sweep on Pythia-70M (Figure 5 bottom left); also claimed to minimize the theorem's 1/(beta_g^3(1-beta_g)^3) factor.
  • local momentum beta_l = 0.9
    Selected by hyperparameter sweep on Pythia-70M (Figure 5 bottom right).
  • merging weight alpha = 0.25
    Selected by hyperparameter sweep on Pythia-70M (Figure 5 top right).
  • local updates accumulation K = 32
    Selected by hyperparameter sweep on Pythia-70M (Figure 5 top left); adjusted to 8 in the heterogeneous grouping study.
  • number of local steps H = 32
    Selected by hyperparameter sweep on Pythia-70M and used for HALoS and baselines (Table 11).
assumptions (7)
  • domain assumption Training data is i.i.d. across workers (Section 3, footnote 2).
    Assumed for analytical clarity; the paper says the method does not rely on it in practice and tests a non-i.i.d. dataset.
  • domain assumption Local loss functions are L-smooth (Assumption 4.1).
    Standard in non-convex SGD analysis; not verified for transformer loss landscapes.
  • domain assumption Stochastic gradients are unbiased with bounded variance (Assumption 4.2).
    Standard SGD assumption at the worker level.
  • domain assumption Bounded heterogeneity between regions: E||grad_l(Theta) - grad F(Theta)||^2 <= sigma^2 (Assumption 4.3).
    Limits regional gradient disagreement; sigma^2 is not measured.
  • domain assumption Bounded gradients: ||grad F_i(x)|| <= G (Assumption 4.4).
    Used throughout the proof; justified only by appeal to gradient clipping.
  • domain assumption Bounded staleness: model difference between local training and global updates is bounded by D_g for GPS and D_l for LPS (Assumption 4.5).
    Not connected to H, K, alpha, or network delays; the theorem's error floor depends on these unquantified values.
  • ad hoc to paper Algorithm 1's ordered update-then-merge is equivalent to the combined LPS update rule analyzed in Appendix E.
    The proof's LPS equation has merge and gradient update in one step with gradient coefficient 1; Algorithm 1 merges only after receiving the global model, possibly after several gradient updates. No proof of equivalence is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HALoS: Hierarchical Asynchronous Local SGD over Slow Networks for Geo-Distributed Large Language Model Training." pith.science (2026). https://pith.science/paper/PNW35G4C

@misc{pith2026250604531,
  author       = {Pith},
  title        = {Pith review of: HALoS: Hierarchical Asynchronous Local SGD over Slow Networks for Geo-Distributed Large Language Model Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PNW35G4C}},
  note         = {Machine review of arXiv:2506.04531}
}
read the original abstract

Training large language models (LLMs) increasingly relies on geographically distributed accelerators, causing prohibitive communication costs across regions and uneven utilization of heterogeneous hardware. We propose HALoS, a hierarchical asynchronous optimization framework that tackles these issues by introducing local parameter servers (LPSs) within each region and a global parameter server (GPS) that merges updates across regions. This hierarchical design minimizes expensive inter-region communication, reduces straggler effects, and leverages fast intra-region links. We provide a rigorous convergence analysis for HALoS under non-convex objectives, including theoretical guarantees on the role of hierarchical momentum in asynchronous training. Empirically, HALoS attains up to 7.5x faster convergence than synchronous baselines in geo-distributed LLM training and improves upon existing asynchronous methods by up to 2.1x. Crucially, HALoS preserves the model quality of fully synchronous SGD-matching or exceeding accuracy on standard language modeling and downstream benchmarks-while substantially lowering total training time. These results demonstrate that hierarchical, server-side update accumulation and global model merging are powerful tools for scalable, efficient training of new-era LLMs in heterogeneous, geo-distributed environments.

Figures

Figures reproduced from arXiv: 2506.04531 by the authors.

Figure 1
Figure 1. Overview of HALoS. Local parameter servers leverage fast intra-region networks (red solid lines) to update their models with asynchronously computed gradients from workers. Accumu￾lated updates are sent to the global server via slow inter-region networks (blue dotted lines), and the latest global model is merged back into the local models. communication through asynchronous updates. In particu￾lar, local servers con… view at source ↗
Figure 2
Figure 2. (a) A geo-distributed training environment with 4 regions with 16 heterogenous workers where we adapt recent measurement of inter-region communication bandwidths from (Jaghouar et al., 2024). (b) Comparison of training time (in log scale) and token consumption for Pythia-70M model (Biderman et al., 2023) in (a) to reach the same validation loss under various training methods (“H” denotes the number of local updates … view at source ↗
Figure 3
Figure 3. Validation loss curves of the Pythia-160M model for different local SGD-based methods to reach the same validation loss: (left) loss vs. simulation wall-clock time and (right) loss vs. number of trained tokens. “DiLoCo + DynUpd,” which integrates dynamic local up￾dates into DiLoCo. Additionally, to ensure a fair comparison with Async-Local-SGD, HALoS incorporates both dynamic local updates and the delayed Nesterov o… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Test accuracies of the Pythia-160M model on the MMLU, Hellaswag, LAMBADA, and WinoGrande benchmarks, plotted against the number of trained tokens. The results compare syn￾chronous SGD and HALoS. The shaded regions represent the standard errors in benchmark evaluations.…
Figure 5
Figure 5. Figure 5: Hyperparameter sensitivity of HALoS: validation losses of Pythia-70M after training 12.9B tokens, varying (top left) up￾dates accumulation K, (top right) merging weight α, (bottom left) global server momentum βg, and (bottom right) local server mo￾mentum βl. The best l…
Figure 6
Figure 6. Figure 6: Simulation approach for evaluating geo-distributed training methods. The simulator accepts: (i) resource specifications (e.g., network bandwidths and the placements of the Global Parameter Server (GPS), Local Parameter Servers (LPSs), and workers along with their relat…
Figure 7
Figure 7. Figure 7: Validation loss curves for Pythia-70M, Pythia-160M, and Pythia-410M models trained using different methods from [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8 [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Validation loss curves for training the Pythia-70M model with different methods under (a) the default (1×) network bandwidths and (b) 2× network bandwidths. These results correspond to those presented in [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

61 extracted references · 31 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    On the choice of learning rate for local sgd

    Balles, L., Archambeau, C., et al. On the choice of learning rate for local sgd. Transactions on Machine Learning Research, 2024

    work page 2024

  4. [4]

    G., Bradley, H., O’Brien, K., Hallahan, E., Khan, M

    Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O’Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp.\ 2397--2430. PMLR, 2023

    2023

  5. [5]

    L., Gao, J., and Choi, Y

    Bisk, Y., Zellers, R., Bras, R. L., Gao, J., and Choi, Y. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020

    2020

  6. [6]

    Caldas, S., Duddu, S. M. K., Wu, P., Li, T., Kone c n \`y , J., McMahan, H. B., Smith, V., and Talwalkar, A. Leaf: A benchmark for federated settings. arXiv preprint arXiv:1812.01097, 2018

    arXiv 2018

  7. [7]

    Multi-level local sgd: Distributed sgd for heterogeneous hierarchical networks

    Castiglia, T., Das, A., and Patterson, S. Multi-level local sgd: Distributed sgd for heterogeneous hierarchical networks. In International Conference on Learning Representations, 2021

    work page 2021

  8. [8]

    Asynchronous online federated learning for edge devices with non-iid data

    Chen, Y., Ning, Y., Slawski, M., and Rangwala, H. Asynchronous online federated learning for edge devices with non-iid data. In 2020 IEEE International Conference on Big Data (Big Data), pp.\ 15--24. IEEE, 2020

    work page 2020

Show all 61 references
  1. [9]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  2. [10]

    Large scale distributed deep networks

    Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P., Yang, K., et al. Large scale distributed deep networks. Advances in neural information processing systems, 25, 2012

    2012

  3. [11]

    A., Chhaparia, R., Donchev, Y., Kuncoro, A., Ranzato, M., Szlam, A., and Shen, J

    Douillard, A., Feng, Q., Rusu, A. A., Chhaparia, R., Donchev, Y., Kuncoro, A., Ranzato, M., Szlam, A., and Shen, J. Diloco: Distributed low-communication training of language models. arXiv preprint arXiv:2311.08105, 2023

    2023 arXiv

  4. [12]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  5. [13]

    N., et al

    Gandhi, R., Tandon, K., Bhattacherjee, D., Padmanabhan, V. N., et al. Improving training time and gpu utilization in geo-distributed language model training. arXiv preprint arXiv:2411.14458, 2024

    2024

  6. [14]

    The pile: An 800gb dataset of diverse text for language modeling

    Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020

    2020 arXiv

  7. [15]

    M., B a doiu, V.-A., Agache, A., Dumitru, M.-V., Vasilescu, I., Mantu, R., and Raiciu, C

    Gherghescu, A. M., B a doiu, V.-A., Agache, A., Dumitru, M.-V., Vasilescu, I., Mantu, R., and Raiciu, C. I've got 99 problems but flops ain't one. In Proceedings of the 23rd ACM Workshop on Hot Topics in Networks, pp.\ 195--204, 2024

    2024

  8. [16]

    Measuring massive multitask language understanding

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  9. [17]

    M., and Hagemann, J

    Jaghouar, S., Ong, J. M., and Hagemann, J. Opendiloco: An open-source framework for globally distributed low-communication training. arXiv preprint arXiv:2407.07852, 2024

    2024 arXiv

  10. [18]

    MegaScale : Scaling large language model training to more than 10,000 gpus

    Jiang, Z., Lin, H., Zhong, Y., Huang, Q., Chen, Y., Zhang, Z., Peng, Y., Li, X., Xie, C., Nong, S., et al. MegaScale : Scaling large language model training to more than 10,000 gpus. In 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), pp.\ 745--760, 2024

    2024

  11. [19]

    Tighter theory for local sgd on identical and heterogeneous data

    Khaled, A., Mishchenko, K., and Richt \'a rik, P. Tighter theory for local sgd on identical and heterogeneous data. In International Conference on Artificial Intelligence and Statistics, pp.\ 4519--4529. PMLR, 2020

    2020

  12. [20]

    A unified theory of decentralized sgd with changing topology and local updates

    Koloskova, A., Loizou, N., Boreiri, S., Jaggi, M., and Stich, S. A unified theory of decentralized sgd with changing topology and local updates. In International Conference on Machine Learning, pp.\ 5381--5393. PMLR, 2020

    2020

  13. [21]

    Slow learners are fast

    Langford, J., Smola, A., and Zinkevich, M. Slow learners are fast. arXiv preprint arXiv:0911.0491, 2009

    2009 arXiv

  14. [22]

    Amp: Automatically finding model parallel strategies with heterogeneity awareness

    Li, D., Wang, H., Xing, E., and Zhang, H. Amp: Automatically finding model parallel strategies with heterogeneity awareness. Advances in Neural Information Processing Systems, 35: 0 6630--6639, 2022

    2022

  15. [23]

    Fednar: Federated optimization with normalized annealing regularization

    Li, J., Li, A., Tian, C., Ho, Q., Xing, E., and Wang, H. Fednar: Federated optimization with normalized annealing regularization. Advances in Neural Information Processing Systems, 36: 0 74753--74763, 2023

    2023

  16. [24]

    G., Park, J

    Li, M., Andersen, D. G., Park, J. W., Smola, A. J., Ahmed, A., Josifovski, V., Long, J., Shekita, E. J., and Su, B.-Y. Scaling distributed machine learning with the parameter server. In 11th USENIX Symposium on operating systems design and implementation (OSDI 14), pp.\ 583--598, 2014

    2014

  17. [25]

    K., Zaheer, M., Sanjabi, M., Talwalkar, A., and Smith, V

    Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., and Smith, V. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2: 0 429--450, 2020

    2020

  18. [26]

    Liang, X., Tang, J., and Quek, T. Q. Large-scale decentralized asynchronous federated edge learning with device heterogeneity. In ICC 2024-IEEE International Conference on Communications, pp.\ 4566--4571. IEEE, 2024

    2024

  19. [27]

    U., Patel, K

    Lin, T., Stich, S. U., Patel, K. K., and Jaggi, M. Don't use large mini-batches, use local sgd. arXiv preprint arXiv:1808.07217, 2018

    2018 arXiv

  20. [28]

    A., Shen, J., Szlam, A., and Ranzato, M

    Liu, B., Chhaparia, R., Douillard, A., Kale, S., Rusu, A. A., Shen, J., Szlam, A., and Ranzato, M. Asynchronous local-sgd training for language modeling. arXiv preprint arXiv:2401.09135, 2024

    2024 arXiv

  21. [29]

    An improved analysis of stochastic gradient descent with momentum

    Liu, Y., Gao, Y., and Yin, W. An improved analysis of stochastic gradient descent with momentum. Advances in Neural Information Processing Systems, 33: 0 18261--18271, 2020

    2020

  22. [30]

    Decoupled weight decay regularization

    Loshchilov, I. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  23. [31]

    McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp.\ 1273--1282. PMLR, 2017

    2017

  24. [32]

    Helix: Distributed serving of large language models via max-flow on heterogeneous gpus

    Mei, Y., Zhuang, Y., Miao, X., Yang, J., Jia, Z., and Vinayak, R. Helix: Distributed serving of large language models via max-flow on heterogeneous gpus. arXiv preprint arXiv:2406.01566, 2024

    2024 arXiv

  25. [33]

    Mixed precision training

    Micikevicius, P., Narang, S., Alben, J., Diamos, G., Elsen, E., Garcia, D., Ginsburg, B., Houston, M., Kuchaiev, O., Venkatesh, G., et al. Mixed precision training. arXiv preprint arXiv:1710.03740, 2017

    2017 arXiv

  26. [34]

    and Ulukus, S

    Mitra, P. and Ulukus, S. Timely asynchronous hierarchical federated learning: Age of convergence. In 2023 21st International Symposium on Modeling and Optimization in Mobile, Ad Hoc, and Wireless Networks (WiOpt), pp.\ 509--515. IEEE, 2023

    2023

  27. [35]

    Efficient large-scale language model training on gpu clusters using megatron-lm

    Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Patwary, M., Korthikanti, V., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International Confere...

    2021

  28. [36]

    Federated learning with buffered asynchronous aggregation

    Nguyen, J., Malik, K., Zhan, H., Yousefpour, A., Rabbat, M., Malek, M., and Huba, D. Federated learning with buffered asynchronous aggregation. In International Conference on Artificial Intelligence and Statistics, pp.\ 3581--3607. PMLR, 2022

    2022

  29. [37]

    Ortiz, J. J. G., Frankle, J., Rabbat, M., Morcos, A., and Ballas, N. Trade-offs of local sgd at scale: An empirical study. arXiv preprint arXiv:2110.08133, 2021

    2021 arXiv

  30. [38]

    Q., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fern \'a ndez, R

    Paperno, D., Kruszewski, G., Lazaridou, A., Pham, N. Q., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fern \'a ndez, R. The lambada dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computa...

    2016

  31. [39]

    Automatic mixed precision package - torch.amp

    PyTorch . Automatic mixed precision package - torch.amp. https://pytorch.org/docs/stable/amp.html, 2025

    2025

  32. [40]

    L., Bhagavatula, C., and Choi, Y

    Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64 0 (9): 0 99--106, 2021

    2021

  33. [41]

    Stich, S. U. Local sgd converges fast and communicates little. arXiv preprint arXiv:1805.09767, 2018

    2018 arXiv

  34. [42]

    Ml training with cloud gpu shortages: Is cross-region the answer? In Proceedings of the 4th Workshop on Machine Learning and Systems, pp.\ 107--116, 2024

    Strati, F., Elvinger, P., Kerimoglu, T., and Klimovic, A. Ml training with cloud gpu shortages: Is cross-region the answer? In Proceedings of the 4th Workshop on Machine Learning and Systems, pp.\ 107--116, 2024

    2024

  35. [43]

    Co2: Efficient distributed training with full communication-computation overlap

    Sun, W., Qin, Z., Sun, W., Li, S., Li, D., Shen, X., Qiao, Y., and Zhong, Y. Co2: Efficient distributed training with full communication-computation overlap. arXiv preprint arXiv:2401.16265, 2024

    2024 arXiv

  36. [44]

    H., and Zhang, J

    Sun, Y., Shao, J., Mao, Y., Wang, J. H., and Zhang, J. Semi-decentralized federated edge learning with data and device heterogeneity. IEEE Transactions on Network and Service Management, 20 0 (2): 0 1487--1501, 2023

    2023

  37. [45]

    On the importance of initialization and momentum in deep learning

    Sutskever, I., Martens, J., Dahl, G., and Hinton, G. On the importance of initialization and momentum in deep learning. In International conference on machine learning, pp.\ 1139--1147. PMLR, 2013

    2013

  38. [46]

    Fusionllm: A decentralized llm training system on geo-distributed gpus with adaptive compression

    Tang, Z., Kang, X., Yin, Y., Pan, X., Wang, Y., He, X., Wang, Q., Zeng, R., Zhao, K., Shi, S., et al. Fusionllm: A decentralized llm training system on geo-distributed gpus with adaptive compression. arXiv preprint arXiv:2410.12707, 2024

    arXiv

  39. [47]

    Optimization of collective communication operations in mpich

    Thakur, R., Rabenseifner, R., and Gropp, W. Optimization of collective communication operations in mpich. The International Journal of High Performance Computing Applications, 19 0 (1): 0 49--66, 2005

    2005

  40. [48]

    Valiant, L. G. A bridging model for parallel computation. Communications of the ACM, 33 0 (8): 0 103--111, 1990

    1990

  41. [49]

    and Joshi, G

    Wang, J. and Joshi, G. Cooperative sgd: A unified framework for the design and analysis of local-update sgd algorithms. Journal of Machine Learning Research, 22 0 (213): 0 1--50, 2021

    2021

  42. [50]

    Slowmo: Improving communication-efficient distributed sgd with slow momentum

    Wang, J., Tantia, V., Ballas, N., and Rabbat, M. Slowmo: Improving communication-efficient distributed sgd with slow momentum. arXiv preprint arXiv:1910.00643, 2019

    1910 arXiv

  43. [51]

    Wang, J., Liu, Q., Liang, H., Joshi, G., and Poor, H. V. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33: 0 7611--7623, 2020

    2020

  44. [52]

    and Wang, Y

    Wang, X. and Wang, Y. Asynchronous hierarchical federated learning. arXiv preprint arXiv:2206.00054, 2022

    2022 arXiv

  45. [53]

    F., and Gardner, M

    Welbl, J., Liu, N. F., and Gardner, M. Crowdsourcing multiple choice science questions. In Derczynski, L., Xu, W., Ritter, A., and Baldwin, T. (eds.), Proceedings of the 3rd Workshop on Noisy User-generated Text, pp.\ 94--106, Copenhagen, Denmark, September 2017. Association f...

    2017 doi

  46. [54]

    Asynchronous federated optimization

    Xie, C., Koyejo, S., and Gupta, I. Asynchronous federated optimization. arXiv preprint arXiv:1903.03934, 2019

    1903 arXiv

  47. [55]

    Hierarchical global asynchronous federated learning across multi-center

    Xie, W., Xiong, R., and Luo, J. Hierarchical global asynchronous federated learning across multi-center. In The 16th Asian Conference on Machine Learning (Conference Track), 2024

    2024

  48. [56]

    Qwen3 technical report

    Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025

    2025 arXiv

  49. [57]

    Parallel restarted sgd with faster convergence and less communication: Demystifying why model averaging works for deep learning

    Yu, H., Yang, S., and Zhu, S. Parallel restarted sgd with faster convergence and less communication: Demystifying why model averaging works for deep learning. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp.\ 5693--5700, 2019

    2019

  50. [58]

    S., Re, C., and Zhang, C

    Yuan, B., He, Y., Davis, J., Zhang, T., Dao, T., Chen, B., Liang, P. S., Re, C., and Zhang, C. Decentralized training of foundation models in heterogeneous environments. Advances in Neural Information Processing Systems, 35: 0 25464--25477, 2022

    2022

  51. [59]

    Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019

    Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019

    2019

  52. [60]

    E., and LeCun, Y

    Zhang, S., Choromanska, A. E., and LeCun, Y. Deep learning with elastic averaging sgd. Advances in neural information processing systems, 28, 2015

    2015

  53. [61]

    Y., and Decouchant, J

    Zuo, Y., Cox, B., Chen, L. Y., and Decouchant, J. Asynchronous multi-server federated learning for geo-distributed clients. arXiv preprint arXiv:2406.01439, 2024

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.