REVIEW 6 cited by
Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Training of large language models (LLMs) is typically distributed across a large number of accelerators to reduce training time. Since internal states and parameter gradients need to be exchanged at each and every single gradient step, all devices need to be co-located using low-latency high-bandwidth communication links to support the required high volume of exchanged bits. Recently, distributed algorithms like DiLoCo have relaxed such co-location constraint: accelerators can be grouped into ``workers'', where synchronizations between workers only occur infrequently. This in turn means that workers can afford being connected by lower bandwidth communication links without affecting learning quality. However, in these methods, communication across workers still requires the same peak bandwidth as before, as the synchronizations require all parameters to be exchanged across all workers. In this paper, we improve DiLoCo in three ways. First, we synchronize only subsets of parameters in sequence, rather than all at once, which greatly reduces peak bandwidth. Second, we allow workers to continue training while synchronizing, which decreases wall clock time. Third, we quantize the data exchanged by workers, which further reduces bandwidth across workers. By properly combining these modifications, we show experimentally that we can distribute training of billion-scale parameters and reach similar quality as before, but reducing required bandwidth by two orders of magnitude.
Forward citations
Cited by 6 Pith papers
-
Not Every Sync Is Safe: Calibrated DiLoCo Scheduling for Shared AI Infrastructure
Matched-random merge placement is the right control for DiLoCo scheduling claims; only calibrated overlap plus EWMA forecasting beats it in real vLLM sidecar replay (6.54%→5.09% SLO violations).
-
LoRDO: Distributed Low-Rank Optimization with Infrequent Communication
LoRDO combines global low-rank projections with full-rank quasi-hyperbolic momentum to let infrequent-synchronization distributed training match low-rank DDP at roughly 10x less communication.
-
Prime Collective Communications Library -- Technical Report
PCCL is a fault-tolerant collective communication library that uses a master-coordinated micro-consensus protocol to support dynamic peer membership and bit-exact state synchronization for distributed ML over the internet.
-
Distributed and Decentralised Training: Technical Governance Challenges in a Shifting AI Landscape
A policy analysis distinguishing distributed and decentralised AI training, arguing decentralised training may erode detectability and shutdownability while compute controls remain relevant.
-
MuLoCo: Muon is a practical inner optimizer for DiLoCo
Using Muon instead of AdamW inside DiLoCo improves worker scaling and critical batch size for LLM pre-training across 150M to 15B parameters.
-
Toward a Global Regime for Compute Governance: Building the Pause Button
The paper argues that a global, enforceable compute pause is achievable through a layered framework of hardware controls, supply chain tracking, and regulation.
Discussion (0). Continue with ORCID to comment.