REVIEW 5 cited by
Accelerating Distributed MoE Training and Inference with Lina
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Scaling model parameters improves model quality at the price of high computation overhead. Sparsely activated models, usually in the form of Mixture of Experts (MoE) architecture, have sub-linear scaling of computation cost with model size, thus providing opportunities to train and serve a larger model at lower cost than their dense counterparts. However, distributed MoE training and inference is inefficient, mainly due to the interleaved all-to-all communication during model computation. This paper makes two main contributions. First, we systematically analyze all-to-all overhead in distributed MoE and present the main causes for it to be the bottleneck in training and inference, respectively. Second, we design and build Lina to address the all-to-all bottleneck head-on. Lina opportunistically prioritizes all-to-all over the concurrent allreduce whenever feasible using tensor partitioning, so all-to-all and training step time is improved. Lina further exploits the inherent pattern of expert selection to dynamically schedule resources during inference, so that the transfer size and bandwidth of all-to-all across devices are balanced amid the highly skewed expert popularity in practice. Experiments on an A100 GPU testbed show that Lina reduces the training step time by up to 1.73x and reduces the 95%ile inference time by an average of 1.63x over the state-of-the-art systems.
Forward citations
Cited by 5 Pith papers
-
Hecate: Unlocking Efficient Sparse Model Training via Fully Sharded Sparse Data Parallelism
Fully Sharded Sparse Data Parallelism materializes expert parameters on the fly with SparseAllGather and SparseReduceScatter, avoiding the overhead of expert rearrangement in MoE training.
-
A Training-Memory Regression in MLA Sequence Parallelism: Why Megatron-Core Forbids Absorption, and LAGA -- a Communication-Efficient Fix
LAGA replaces MLA training's per-head K/V all-to-all with a latent all-gather and local up-projection, matching explicit-form memory while cutting collective communication ~1.98x.
-
ThAME: 3D Memory-Enabled Heterogeneous Accelerator for LLM Mixture of Experts
ThAME, a 3D FeFET-NAND + DRAM heterogeneous accelerator with an MOO-optimized hierarchical NoC, claims up to 15.7× per-token latency and 9.8× energy gains for MoE LLM inference in cycle-accurate simulation.
-
Communication-Aware Placement and Pruning for Efficient Mixture-of-Experts Inference
Communication-aware expert placement plus device-level pruning yields 1.23–1.86× MoE inference throughput and better accuracy at equal speedup than load-balance or sequential baselines.
-
Diagnosing Overhead in Dispatch Operations: Cross-architecture Observatory
Expert-parallel scaling leaves per-expert routing imbalance flat; mock-token benchmarks overestimate real-text imbalance and fake a batch-size trend; architectures split into data-resilient (MHA, Mamba-2) and persiste...
Discussion (0). Continue with ORCID to comment.