REVIEW 22 cited by
Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Large language models have led to state-of-the-art accuracies across a range of tasks. However, training these models efficiently is challenging for two reasons: a) GPU memory capacity is limited, making it impossible to fit large models on even a multi-GPU server, and b) the number of compute operations required to train these models can result in unrealistically long training times. Consequently, new methods of model parallelism such as tensor and pipeline parallelism have been proposed. Unfortunately, naive usage of these methods leads to fundamental scaling issues at thousands of GPUs, e.g., due to expensive cross-node communication or devices spending significant time waiting on other devices to make progress. In this paper, we show how different types of parallelism methods (tensor, pipeline, and data parallelism) can be composed to scale to thousands of GPUs and models with trillions of parameters. We survey techniques for pipeline parallelism and propose a novel interleaved pipeline parallelism schedule that can improve throughput by 10+% with memory footprint comparable to existing approaches. We quantitatively study the trade-offs between tensor, pipeline, and data parallelism, and provide intuition as to how to configure distributed training of a large model. Our approach allows us to perform training iterations on a model with 1 trillion parameters at 502 petaFLOP/s on 3072 GPUs with achieved per-GPU throughput of 52% of theoretical peak. Our code is open sourced at https://github.com/nvidia/megatron-lm.
Forward citations
Cited by 22 Pith papers
-
Trends in AI Supercomputers
Between 2019 and 2025, the performance of the largest AI supercomputers doubled every nine months, while hardware cost and power needs doubled every year.
-
MALT: Lightweight Curvature-Aware Muon via Diagonal Preconditioning
Two-sided diagonal preconditioning before Newton-Schulz orthogonalization, plus an adaptive scalar stepsize, improves Muon's GPT-2 pretraining loss at nearly unchanged cost.
-
A Training-Memory Regression in MLA Sequence Parallelism: Why Megatron-Core Forbids Absorption, and LAGA -- a Communication-Efficient Fix
LAGA replaces MLA training's per-head K/V all-to-all with a latent all-gather and local up-projection, matching explicit-form memory while cutting collective communication ~1.98x.
-
GIFT: Geometry-Informed Low-precision Gradient Communication for LLM Pretraining
Transforming gradients into K-FAC-based coordinates before FP8 quantization reduces communication error and improves downstream task preservation over Euclidean FP8, with a 7.6% end-to-end speedup on 64 GH200 GPUs.
-
Design-CP: Context Parallelism for Design of Protein Nanoparticles
Context-parallel inference for RFdiffusion 3 enables end-to-end all-atom design of large symmetric protein nanoparticles on multi-GPU hardware without retraining.
-
PHOENIX: Resilient LLM Training with Hot-Swapping via Zero-Overhead Checkpoint
PHOENIX recovers permanent node failures in LLM training via hot-swapping of spares using zero-overhead per-step in-memory optimizer-state replication, finishing recovery in under 40 s on up to 512 GPUs.
-
Speeding up Model Loading with fastsafetensors
A new safetensors loader copies groups of tensors directly to GPU memory, cutting LLM startup time by 4.8x to 7.5x compared with the default library.
-
Hardware-Efficient Attention for Fast Decoding
Grouped-Tied Attention and Grouped Latent Attention reduce KV-cache memory and speed up LLM decoding by up to 2x while matching the quality of GQA and MLA at up to 1.47B parameters.
-
Hetu v2: A General and Scalable Deep Learning System with Hierarchical and Heterogeneous Single Program Multiple Data Annotations
Hetu v2 introduces hierarchical and heterogeneous SPMD annotations plus graph specialization and dynamic graph switching, enabling efficient LLM training on heterogeneous, unstable, and mixed-length workloads.
-
Democratizing AI: Open-source Scalable LLM Training on GPU-based Supercomputers
AxoNN combines 3D parallel matrix multiplication with data parallelism to reach 1.423 exaflop/s on 6,144 H100 GPUs, and reports one-pass catastrophic memorization at the 70B scale that a masked-loss technique suppresses.
-
Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Pass
A single-pass transformer generalizes DUSt3R's pointmap regression from two views to all-to-all multi-view attention, reconstructing 1000+ images and estimating camera poses in one forward pass.
-
Mixture of Training: Recombining Small-Scale Scaffolded Pretraining Runs into a Larger Language Model
Independently trained layer blocks, trained inside a frozen aligner scaffold, can be recomposed into a 1.3B-parameter language model that reaches monolithic perplexity only when the scaffold is reused or the token bud...
-
LongStraw: Long-Context RL Beyond 2M Tokens under a Fixed GPU Budget
LongStraw caches a long prompt without autograd and replays response suffixes with gradients to fit 2M+ token GRPO-like training into fixed GPU budgets, but it never proves the resulting update equals the true RL gradient.
-
Efficient and Scalable Agentic AI with Heterogeneous Systems
A system design and preliminary simulation argue that pairing older H100 GPUs with Gaudi 3 accelerators can match or beat all-B200 clusters on total cost of ownership for AI agent workloads.
-
Automatically Planning Optimal Parallel Strategy for Large Language Models
A training-time simulator and pruned search space choose the fastest 3D parallel strategy, including micro-batch size, for LLM training on clusters.
-
Multi-matrix Factorization Attention
MFA and MFA-KR factorize the attention QK circuit with shared low-rank key and value projections, matching or beating MHA accuracy at a small fraction of the KV cache.
-
Revisiting Training Scale: An Empirical Study of Token Count, Power Consumption, and Parameter Efficiency
The paper's finding that training efficiency declines monotonically with token count is guaranteed by its efficiency metric, which divides by token count and power consumption.
-
Technical Report of TeleChat2, TeleChat2.5 and T1
The released T1-115B open-weight model outperforms OpenAI's o1-mini and GPT-4o on MATH500, AlignBench, and IFEval, despite using a standard dense transformer architecture.
-
Scaling Large Language Model Training on Frontier with Low-Bandwidth Partitioning
A topology-aware three-level partitioning of weights, gradients, and optimizer states speeds up ZeRO++ LLM training on Frontier by roughly 1.7x per GPU for 20B models with 0.94 scaling efficiency.
-
Memory-Efficient Activation Checkpointing with Sliding Window and Hirschberg's Algorithm for 0/1 Knapsack Solving in PyTorch
Sliding-window DP plus Hirschberg's algorithm reduces PyTorch's activation checkpointing knapsack solver memory from O(nW) to O(W) while preserving exact optimality.
-
Taming the Titans: A Survey of Efficient LLM Inference Serving
A survey that classifies LLM inference serving techniques into instance, cluster, emerging-scenario, and miscellaneous categories, with a companion GitHub index and four research directions.
-
The Hitchhiker's Guide to Agentic AI: From Foundations to Systems
A survey-style reference book mapping the full agentic-AI stack from transformer internals to production deployment, with no new research result.
Discussion (0). Continue with ORCID to comment.