REVIEW 7 cited by
Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
As we scale to more massive machine learning models, the frequent synchronization demands inherent in data-parallel approaches create significant slowdowns, posing a critical challenge to further scaling. Recent work develops an approach (DiLoCo) that relaxes synchronization demands without compromising model quality. However, these works do not carefully analyze how DiLoCo's behavior changes with model size. In this work, we study the scaling law behavior of DiLoCo when training LLMs under a fixed compute budget. We focus on how algorithmic factors, including number of model replicas, hyperparameters, and token budget affect training in ways that can be accurately predicted via scaling laws. We find that DiLoCo scales both predictably and robustly with model size. When well-tuned, DiLoCo scales better than data-parallel training with model size, and can outperform data-parallel training even at small model sizes. Our results showcase a more general set of benefits of DiLoCo than previously documented, including increased optimal batch sizes, improved downstream generalization with scale, and improved evaluation loss for a fixed token budget.
Forward citations
Cited by 7 Pith papers
-
What's in a Smoothness Constant? Tighter Rates for Local SGD with Bounded Second-order Heterogeneity
Local SGD provably improves over Mini-batch SGD under bounded second-order heterogeneity in the general convex setting, with nearly tight upper and lower bounds.
-
LoRDO: Distributed Low-Rank Optimization with Infrequent Communication
LoRDO combines global low-rank projections with full-rank quasi-hyperbolic momentum to let infrequent-synchronization distributed training match low-rank DDP at roughly 10x less communication.
-
Predictable Scale: Part II, Farseer: A Refined Scaling Law in Large Language Models
A newly fitted scaling law with model-size-dependent data exponents predicts LLM loss more accurately than Chinchilla, including at a held-out 25.1B model.
-
NoLoCo: No-all-reduce Low Communication Training Method for Large Models
NoLoCo trains large language models without any all-to-all synchronization by using pairwise weight averaging and random pipeline routing, matching or slightly beating DiLoCo in experiments.
-
DES-LOC: Desynced Low Communication Adaptive Optimizers for Training Foundation Models
DES-LOC synchronizes model parameters and Adam/ADOPT momentum states on separate schedules, matching Local Adam quality with about 2x less communication and 170x less than DDP in tests up to 1.7B parameters.
-
Distributed and Decentralised Training: Technical Governance Challenges in a Shifting AI Landscape
A policy analysis distinguishing distributed and decentralised AI training, arguing decentralised training may erode detectability and shutdownability while compute controls remain relevant.
-
MuLoCo: Muon is a practical inner optimizer for DiLoCo
Using Muon instead of AdamW inside DiLoCo improves worker scaling and critical batch size for LLM pre-training across 150M to 15B parameters.
Discussion (0). Continue with ORCID to comment.