REVIEW 5 cited by
Zero Bubble Pipeline Parallelism
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Pipeline parallelism is one of the key components for large-scale distributed training, yet its efficiency suffers from pipeline bubbles which were deemed inevitable. In this work, we introduce a scheduling strategy that, to our knowledge, is the first to successfully achieve zero pipeline bubbles under synchronous training semantics. The key idea behind this improvement is to split the backward computation into two parts, one that computes gradient for the input and another that computes for the parameters. Based on this idea, we handcraft novel pipeline schedules that significantly outperform the baseline methods. We further develop an algorithm that automatically finds an optimal schedule based on specific model configuration and memory limit. Additionally, to truly achieve zero bubble, we introduce a novel technique to bypass synchronizations during the optimizer step. Experimental evaluations show that our method outperforms the 1F1B schedule up to 23% in throughput under a similar memory limit. This number can be further pushed to 31% when the memory constraint is relaxed. We believe our results mark a major step forward in harnessing the true potential of pipeline parallelism. We open sourced our implementation based on the popular Megatron-LM repository on https://github.com/sail-sg/zero-bubble-pipeline-parallelism.
Forward citations
Cited by 5 Pith papers
-
Opus: Photonic Rail-Optimized Fabric in ML Datacenters
Opus time-multiplexes a single photonic rail fabric across parallelism phases in ML training, achieving up to 23x network power reduction and 4x cost savings at under 6.7% training overhead in simulation.
-
Photonic Rails in ML Datacenters
A photonic rail design that reconfigures optical circuits between parallelism phases within a training job can emulate electrical rails with about 70% cost and 96% power savings and a few percent iteration-time overhe...
-
DeepCEE: Efficient Cross-Region Model Distributed Training System under Heterogeneous GPUs and Networks
DeepCEE groups heterogeneous GPUs by network and compute speed, schedules a compact zero-bubble pipeline across regions, and adapts micro-batch sizes to network fluctuations, reporting 1.3-2.8x higher training through...
-
The Hitchhiker's Guide to Agentic AI: From Foundations to Systems
A survey-style reference book mapping the full agentic-AI stack from transformer internals to production deployment, with no new research result.
-
DeepSeek: Paradigm Shifts and Technical Evolution in Large AI Models
A survey of DeepSeek's V3 and R1 models covering MLA, MoE, MTP, GRPO, and training engineering, with no new experimental results.
Discussion (0). Continue with ORCID to comment.