Pith. sign in

REVIEW 11 cited by

Scaling Laws for Fine-Grained Mixture of Experts

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.07871 v1 pith:7TN4LDUI submitted 2024-02-12 cs.LG cs.AIcs.CL

Jakub Krajewski , Jan Ludziejewski , Kamil Adamczewski , Maciej Pióro , Micha{l} Krutul , Szymon Antoniak , Kamil Ciebiera , Krystian Król
show 4 more authors
Tomasz Odrzygóźdź Piotr Sankowski Marek Cygan Sebastian Jaszczur
This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords expertsmodelssizebudgetcomputationallawsscalingtraining
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Mixture of Experts (MoE) models have emerged as a primary solution for reducing the computational cost of Large Language Models. In this work, we analyze their scaling properties, incorporating an expanded range of variables. Specifically, we introduce a new hyperparameter, granularity, whose adjustment enables precise control over the size of the experts. Building on this, we establish scaling laws for fine-grained MoE, taking into account the number of training tokens, model size, and granularity. Leveraging these laws, we derive the optimal training configuration for a given computational budget. Our findings not only show that MoE models consistently outperform dense Transformers but also highlight that the efficiency gap between dense and MoE models widens as we scale up the model size and training budget. Furthermore, we demonstrate that the common practice of setting the size of experts in MoE to mirror the feed-forward layer is not optimal at almost any computational budget.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Grouter: Decoupling Routing from Representation for Accelerated MoE Training

    cs.LG 2026-02 conditional novelty 6.0 of 10

    A frozen router distilled from a converged MoE teacher accelerates target MoE pretraining, reaching the same loss with about 4.3x less data and up to 33.5% higher throughput.

  2. BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsity

    cs.LG 2025-07 conditional novelty 6.0 of 10

    A ReLU-routed MoE with chunk-level sparsity training objectives and custom kernels combining activation sparsity with speculative decoding achieves over 70% 8-token chunk sparsity and up to 3.67x end-side speedup.

  3. A Theory of Inference Compute Scaling: Reasoning through Directed Stochastic Skill Search

    cs.LG 2025-06 conditional novelty 6.0 of 10

    A skill-graph random-walk model gives closed-form accuracy-versus-compute formulas for four reasoning strategies and connects them to training scaling.

  4. A Sovereign, Open-Source Foundation Model for German and English

    cs.CL 2026-07 conditional novelty 5.5 of 10

    Soofi S 30B-A3B, a hybrid Mamba-MoE model pretrained on ~27T tokens with deliberately up-weighted German, reports the highest English and German aggregate scores among fully open base models in its comparison while ma...

  5. Scale Weight Decay and Train Better

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Muon with weight decay scaled by η/η_max reaches the same MoE validation loss ~30% faster than constant-decay Muon while preserving asymptotic stationarity of the unregularized objective.

  6. SLAI T-Rex: Full-Parameter Post-training of the DeepSeek-V4 Family on Ascend SuperPOD

    cs.CL 2026-07 conditional novelty 5.0 of 10

    An Ascend-NPU training stack reaches 34.22% MFU on DeepSeek-V4-Pro, and a solver-verified CPT+SFT recipe raises OR benchmark averages to 71.81% (Flash) and 77.33% (Pro).

  7. Is MoE Routing a Huffman Code? Discovering the Frequency-Diversity Law in Chain-of-Thought

    cs.CL 2026-05 reject novelty 5.0 of 10

    MoE routers allocate expert diversity in proportion to operation rarity (the "Frequency-Diversity Law"), and subset-difference pruning can expose this pattern when load-balancing creates redundant experts.

  8. Maximum Score Routing For Mixture-of-Experts

    cs.LG 2025-08 unverdicted novelty 5.0 of 10

    MaxScore casts MoE routing as min-cost max-flow with SoftTopk and claims better loss and eval scores at equal FLOPs; unverified because the full text is unreadable.

  9. Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insights

    cs.LG 2025-06 conditional novelty 5.0 of 10

    At 56B total parameters, fine-grained MoE with smaller, more numerous experts beats standard Switch and Mixtral-style MoE on validation loss and average downstream accuracy at matched FLOPs.

  10. UltraMemV2: Memory Networks Scaling to 120B Parameters with Superior Long-Context Learning

    cs.LG 2025-08 conditional novelty 4.0 of 10

    A redesigned memory-layer architecture with five engineering improvements reaches performance parity with 8-expert MoE at similar compute, with lower memory access and stronger long-context memorization.

  11. $\mu$-MoE: Test-Time Pruning as Micro-Grained Mixture-of-Experts

    cs.LG 2025-05 conditional novelty 4.0 of 10

    Test-time Wanda pruning, reframed as a mixture of micro-experts, adapts the sparse weight mask to each prompt and improves perplexity and VQA accuracy over static pruning baselines.

Pith tools