REVIEW 5 cited by
Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Scaling the capacity of language models has consistently proven to be a reliable approach for improving performance and unlocking new capabilities. Capacity can be primarily defined by two dimensions: the number of model parameters and the compute per example. While scaling typically involves increasing both, the precise interplay between these factors and their combined contribution to overall capacity remains not fully understood. We explore this relationship in the context of sparse Mixture-of-Experts (MoEs), which allow scaling the number of parameters without proportionally increasing the FLOPs per example. We investigate how varying the sparsity level, i.e., the fraction of inactive parameters, impacts model's performance during pretraining and downstream few-shot evaluation. We find that under different constraints (e.g., parameter size and total training compute), there is an optimal level of sparsity that improves both training efficiency and model performance. These results provide a better understanding of the impact of sparsity in scaling laws for MoEs and complement existing works in this area, offering insights for designing more efficient architectures.
Forward citations
Cited by 5 Pith papers
-
When Does Sparsity Mitigate the Curse of Depth in LLMs
Implicit and explicit sparsity reduce residual-stream variance and improve layer effectiveness metrics, enabling a depth-scaling recipe with about 4.6 points higher downstream accuracy.
-
Predictable Scale: Part II, Farseer: A Refined Scaling Law in Large Language Models
A newly fitted scaling law with model-size-dependent data exponents predicts LLM loss more accurately than Chinchilla, including at a held-out 25.1B model.
-
Towards Foundational Models for Dynamical System Reconstruction: Hierarchical Meta-Learning via Mixture of Experts
MixER uses K-means clustering and least squares to route each dynamical system to a specialist expert, improving reconstruction across loosely related ODE families in low-data regimes while underperforming on closely ...
-
Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient
A joint scaling law fitted to over 280 models shows that, under fixed memory or total-parameter budgets, MoE models can achieve lower loss than dense models when trained on more tokens.
-
Soup-of-Experts: Pretraining Specialist Models via Parameters Averaging
Soup-of-Experts pretrains a shared parameter bank and many expert vectors, plus a router, so a small specialist language model can be instantiated instantly from any domain-weight mixture without retraining.
Discussion (0). Continue with ORCID to comment.