REVIEW 19 cited by
DiLoCo: Distributed Low-Communication Training of Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large language models (LLM) have become a critical component in many applications of machine learning. However, standard approaches to training LLM require a large number of tightly interconnected accelerators, with devices exchanging gradients and other intermediate states at each optimization step. While it is difficult to build and maintain a single computing cluster hosting many accelerators, it might be easier to find several computing clusters each hosting a smaller number of devices. In this work, we propose a distributed optimization algorithm, Distributed Low-Communication (DiLoCo), that enables training of language models on islands of devices that are poorly connected. The approach is a variant of federated averaging, where the number of inner steps is large, the inner optimizer is AdamW, and the outer optimizer is Nesterov momentum. On the widely used C4 dataset, we show that DiLoCo on 8 workers performs as well as fully synchronous optimization while communicating 500 times less. DiLoCo exhibits great robustness to the data distribution of each worker. It is also robust to resources becoming unavailable over time, and vice versa, it can seamlessly leverage resources that become available during training.
Forward citations
Cited by 19 Pith papers
-
What's in a Smoothness Constant? Tighter Rates for Local SGD with Bounded Second-order Heterogeneity
Local SGD provably improves over Mini-batch SGD under bounded second-order heterogeneity in the general convex setting, with nearly tight upper and lower bounds.
-
Overcoming the Communication-Performance Tradeoff in LLM Pretraining
SparseLoCo combines error feedback with Top-k sparsification and 2-bit quantization to send 1-3% of the pseudo-gradient during LLM pre-training while matching or beating DiLoCo's dense updates.
-
Federated Lightweight Fine-Tuning
A federated fine-tuning method transmits only 1,280 latent floats per round and reaches near-FedAvg accuracy by exploiting the exact averaging identity of affine mapping networks.
-
Byzantine Accountability Without Consensus: Strong Eventual Consistency for Non-Associative, Stochastic, Robust Aggregation
Any pure function of a product of CRDTs inherits Strong Eventual Consistency, so multi-Krum and similar discontinuous robust selectors can be made coordinator-free and accountable via OR-Set contributions plus grow-on...
-
Not Every Sync Is Safe: Calibrated DiLoCo Scheduling for Shared AI Infrastructure
Matched-random merge placement is the right control for DiLoCo scheduling claims; only calibrated overlap plus EWMA forecasting beats it in real vLLM sidecar replay (6.54%→5.09% SLO violations).
-
LoRDO: Distributed Low-Rank Optimization with Infrequent Communication
LoRDO combines global low-rank projections with full-rank quasi-hyperbolic momentum to let infrequent-synchronization distributed training match low-rank DDP at roughly 10x less communication.
-
Learning Decentralized LLM Collaboration with Multi-Agent Actor Critic
Multi-agent actor-critic methods with a centralized critic improve decentralized LLM collaboration over Monte Carlo baselines in long-horizon and sparse-reward settings.
-
Compute Requirements for Algorithmic Innovation in Frontier AI Models
Estimated development compute for 36 LLM pretraining innovations shows half would remain possible under GPT-2-level or 8-H100 compute caps.
-
Model Parallelism With Subnetwork Data Parallelism
Training each GPU on a fixed overlapping subnetwork and averaging shared parameters cuts per-device memory by up to 60 percent without exchanging activations, matching DDP accuracy under FLOP-matched budgets.
-
DICE: Data Influence Cascade in Decentralized Learning
DICE defines and approximates multi-hop data influence in decentralized learning, showing that influence is shaped by data, topology, and loss curvature.
-
DiLoCoX: A Low-Communication Large-Scale Training Framework for Decentralized Cluster
DiLoCoX combines pipeline parallelism with dual optimizers, delayed communication overlap, and adaptive low-rank plus quantization compression to pre-train a 107B model over 1 Gbps links with a claimed 357x speedup ov...
-
NoLoCo: No-all-reduce Low Communication Training Method for Large Models
NoLoCo trains large language models without any all-to-all synchronization by using pairwise weight averaging and random pipeline routing, matching or slightly beating DiLoCo in experiments.
-
HALoS: Hierarchical Asynchronous Local SGD over Slow Networks for Geo-Distributed Large Language Model Training
A hierarchical asynchronous local SGD method with regional parameter servers and global model merging is claimed to train small LLMs up to 7.5x faster than DiLoCo in simulated geo-distributed settings.
-
Towards a future space-based, highly scalable AI infrastructure system design
Space-based AI compute is argued feasible via close-formation laser-linked satellites, radiation-survivable TPUs, and launch costs projected below $200/kg by the mid-2030s.
-
Distributed and Decentralised Training: Technical Governance Challenges in a Shifting AI Landscape
A policy analysis distinguishing distributed and decentralised AI training, arguing decentralised training may erode detectability and shutdownability while compute controls remain relevant.
-
PC-MoE: Memory-Efficient and Privacy-Preserving Collaborative Training for Mixture-of-Experts LLMs
PC-MoE shards the expert layers of an MoE LLM across parties and routes only sparse top-k activations between them, achieving near-centralized accuracy with about 70% memory savings and resistance to one partial-gradi...
-
Controlled Periodic Synchronization for Efficient Data-Parallel Training
Periodic gradient+parameter synchronization with SlowMo beats DDP by 2.44 pp (K=4) on a WAN while cutting average wall-clock time by 13.8%, but only under a fixed LR=0.1 protocol.
-
Paris: A Decentralized Trained Open-Weight Diffusion Model
A text-to-image diffusion model trained with eight fully isolated experts plus a routing network matches centralized quality at much lower data/compute cost, according to the authors' measurements.
-
AdLoCo: adaptive batching significantly improves communications efficiency and convergence for Large Language Models
AdLoCo claims faster distributed LLM training with fewer synchronization messages by adding adaptive batching, parallel model instances, and a gradient-accumulation fallback to DiLoCo.
Discussion (0). Sign in to comment.