REVIEW 7 cited by
Toward Inference-optimal Mixture-of-Expert Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Mixture-of-Expert (MoE) based large language models (LLMs), such as the recent Mixtral and DeepSeek-MoE, have shown great promise in scaling model size without suffering from the quadratic growth of training cost of dense transformers. Like dense models, training MoEs requires answering the same question: given a training budget, what is the optimal allocation on the model size and number of tokens? We study the scaling law of MoE-based LLMs regarding the relations between the model performance, model size, dataset size, and the expert degree. Echoing previous research studying MoE in different contexts, we observe the diminishing return of increasing the number of experts, but this seems to suggest we should scale the number of experts until saturation, as the training cost would remain constant, which is problematic during inference time. We propose to amend the scaling law of MoE by introducing inference efficiency as another metric besides the validation loss. We find that MoEs with a few (4/8) experts are the most serving efficient solution under the same performance, but costs 2.5-3.5x more in training. On the other hand, training a (16/32) expert MoE much smaller (70-85%) than the loss-optimal solution, but with a larger training dataset is a promising setup under a training budget.
Forward citations
Cited by 7 Pith papers
-
The power of fine-grained experts: Granularity boosts expressivity in Mixture of Experts
For MoE layers with equal active parameters, increasing the number of active experts from k' to k yields an exponential separation in L2 approximation error whenever the binomial coefficient binom(m,k) dominates binom(m',k').
-
Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient
A joint scaling law fitted to over 280 models shows that, under fixed memory or total-parameter budgets, MoE models can achieve lower loss than dense models when trained on more tokens.
-
Densing Law of LLMs
Maximum LLM capability per parameter, measured on five benchmarks, has grown exponentially, doubling about every three months.
-
Advancing Decoding Strategies: Enhancements in Locally Typical Sampling for LLMs
ASTS extends locally typical sampling with semantic scoring and dynamic thresholds, reporting improved perplexity, MAUVE, and diversity on story and summarization tasks.
-
Superposition in Transformers: A Novel Way of Building Mixture of Experts
Merging two GPT-2 variants with B-spline-blended hidden states plus autoencoders yields a single model that keeps both English and French perplexity closer to the best expert than linear interpolation.
-
Position: Enough of Scaling LLMs! Lets Focus on Downscaling
The paper advocates for downscaling LLMs, derives that training CO2 scales linearly with parameters and data, and proposes a condition under which ensembles of pruned small models beat the original at equal compute.
-
Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models
For MoE language models, under a fixed training compute budget, increasing sparsity while adding total parameters lowers pretraining loss, and the optimal sparsity grows with model size.
Discussion (0). Continue with ORCID to comment.