Pith. sign in

REVIEW 10 cited by

Don't Use Large Mini-Batches, Use Local SGD

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 1808.07217 v6 pith:X6C7HE6G submitted 2018-08-22 cs.LG stat.ML

Tao Lin , Sebastian U. Stich , Kumar Kshitij Patel , Martin Jaggi This is my paper · ORCID
classification cs.LGstat.ML
keywords efficiencyemphlargelocalmini-batchperformancescalabilitytraining
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Mini-batch stochastic gradient methods (SGD) are state of the art for distributed training of deep neural networks. Drastic increases in the mini-batch sizes have lead to key efficiency and scalability gains in recent years. However, progress faces a major roadblock, as models trained with large batches often do not generalize well, i.e. they do not show good accuracy on new data. As a remedy, we propose a \emph{post-local} SGD and show that it significantly improves the generalization performance compared to large-batch training on standard benchmarks while enjoying the same efficiency (time-to-accuracy) and scalability. We further provide an extensive study of the communication efficiency vs. performance trade-offs associated with a host of \emph{local SGD} variants.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. What's in a Smoothness Constant? Tighter Rates for Local SGD with Bounded Second-order Heterogeneity

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Local SGD provably improves over Mini-batch SGD under bounded second-order heterogeneity in the general convex setting, with nearly tight upper and lower bounds.

  2. Overcoming the Communication-Performance Tradeoff in LLM Pretraining

    cs.LG 2025-08 conditional novelty 7.0 of 10

    SparseLoCo combines error feedback with Top-k sparsification and 2-bit quantization to send 1-3% of the pseudo-gradient during LLM pre-training while matching or beating DiLoCo's dense updates.

  3. FedACT: Federated Adaptive Coordinate Trust Modulation for Robust Transformer Training under Data Heterogeneity

    cs.LG 2026-07 conditional novelty 6.5 of 10

    Global-aware coordinate trust modulation after corrected AdamW updates improves federated Transformer and LLM training under data heterogeneity over strong adaptive baselines.

  4. Energy Consumption in Parallel Neural Network Training

    cs.LG 2025-08 conditional novelty 6.0 of 10

    Energy use in data-parallel neural network training grows roughly linearly with GPU hours, but the energy cost per GPU hour varies by model, hardware, and the number of samples and gradient updates per GPU hour.

  5. HALoS: Hierarchical Asynchronous Local SGD over Slow Networks for Geo-Distributed Large Language Model Training

    cs.LG 2025-06 conditional novelty 6.0 of 10

    A hierarchical asynchronous local SGD method with regional parameter servers and global model merging is claimed to train small LLMs up to 7.5x faster than DiLoCo in simulated geo-distributed settings.

  6. Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch

    cs.CL 2025-01 conditional novelty 6.0 of 10

    Streaming DiLoCo trains billion-parameter LLMs at data-parallel quality while cutting the inter-datacenter bandwidth by about two orders of magnitude via partial, overlapped, and 4-bit-quantized synchronization.

  7. MuLoCo: Muon is a practical inner optimizer for DiLoCo

    cs.LG 2025-05 conditional novelty 5.0 of 10

    Using Muon instead of AdamW inside DiLoCo improves worker scaling and critical batch size for LLM pre-training across 150M to 15B parameters.

  8. Controlled Periodic Synchronization for Efficient Data-Parallel Training

    cs.DC 2026-07 conditional novelty 4.0 of 10

    Periodic gradient+parameter synchronization with SlowMo beats DDP by 2.44 pp (K=4) on a WAN while cutting average wall-clock time by 13.8%, but only under a fixed LR=0.1 protocol.

  9. FedEve: On Bridging the Client Drift and Period Drift for Cross-device Federated Learning

    cs.LG 2025-08 reject novelty 4.0 of 10

    FedEve uses a Kalman filter to combine server momentum (prediction) with client updates (observation) to offset period drift and client drift in cross-device federated learning.

  10. What Makes Local Updates Effective: The Role of Data Heterogeneity and Smoothness

    cs.LG 2025-06 conditional novelty 4.0 of 10

    Under bounded second-order heterogeneity, local updates are shown to achieve faster convergence than mini-batch SGD in several convex and non-convex regimes, with matching lower bounds.

Pith tools