REVIEW 9 cited by
MegaBlocks: Efficient Sparse Training with Mixture-of-Experts
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We present MegaBlocks, a system for efficient Mixture-of-Experts (MoE) training on GPUs. Our system is motivated by the limitations of current frameworks, which restrict the dynamic routing in MoE layers to satisfy the constraints of existing software and hardware. These formulations force a tradeoff between model quality and hardware efficiency, as users must choose between dropping tokens from the computation or wasting computation and memory on padding. To address these limitations, we reformulate MoE computation in terms of block-sparse operations and develop new block-sparse GPU kernels that efficiently handle the dynamism present in MoEs. Our approach never drops tokens and maps efficiently to modern hardware, enabling end-to-end training speedups of up to 40% over MoEs trained with the state-of-the-art Tutel library and 2.4x over DNNs trained with the highly-optimized Megatron-LM framework.
Forward citations
Cited by 9 Pith papers
-
Route-Block Membership Selects Packed-AWQ Arithmetic: A Controlled Single-Fixture Mechanism Study
Moving a route across the block 40/41 boundary in a pinned MoE layer switches the kernel reduction schedule and transfers all four exact arithmetic surfaces; swapping routes inside a block preserves them.
-
Relax Within, Balance Across: Geometry-Guided Load Balancing for Vision-Language Mixture-of-Experts
The standard mixed image-text load-balancing loss lets modality errors cancel at one token mix; ReBA balances each modality separately and per image, reducing load across resolutions and tiles with comparable accuracy.
-
A Training-Memory Regression in MLA Sequence Parallelism: Why Megatron-Core Forbids Absorption, and LAGA -- a Communication-Efficient Fix
LAGA replaces MLA training's per-head K/V all-to-all with a latent all-gather and local up-projection, matching explicit-form memory while cutting collective communication ~1.98x.
-
Communication-Aware Placement and Pruning for Efficient Mixture-of-Experts Inference
Communication-aware expert placement plus device-level pruning yields 1.23–1.86× MoE inference throughput and better accuracy at equal speedup than load-balance or sequential baselines.
-
When Does Sparsity Mitigate the Curse of Depth in LLMs
Implicit and explicit sparsity reduce residual-stream variance and improve layer effectiveness metrics, enabling a depth-scaling recipe with about 4.6 points higher downstream accuracy.
-
Training Sparse Mixture Of Experts Text Embedding Models
Nomic Embed v2 applies sparse mixture-of-experts upcycling to a multilingual biencoder, reporting competitive BEIR and MIRACL scores with fewer active parameters than dense models of similar size.
-
Maximum Score Routing For Mixture-of-Experts
MaxScore casts MoE routing as min-cost max-flow with SoftTopk and claims better loss and eval scores at equal FLOPs; unverified because the full text is unreadable.
-
Apple Intelligence Foundation Language Models: Tech Report 2025
Apple's 3B on-device and larger server language models match or beat several similarly sized open models on MMLU, MMMLU, and MGSM, using new efficiency techniques like KV-cache sharing and 2-bit quantization.
-
Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insights
At 56B total parameters, fine-grained MoE with smaller, more numerous experts beats standard Switch and Mixtral-style MoE on validation loss and average downstream accuracy at matched FLOPs.
Discussion (0). Continue with ORCID to comment.