pith. sign in

arxiv: 2502.05172 · v2 · pith:QK2Y7X4Snew · submitted 2025-02-07 · 💻 cs.LG · cs.AI· cs.CL

Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient

Jan Ludziejewski , Maciej Pi\'oro , Jakub Krajewski , Maciej Stefaniak , Micha{\l} Krutul , Jan Ma{\l}a\'snicki , Marek Cygan , Piotr Sankowski
show 3 more authors
Kamil Adamczewski Piotr Mi{\l}o\'s Sebastian Jaszczur
This is my paper
classification 💻 cs.LG cs.AIcs.CL
keywords modelsexpertslawsmemoryparametersscalingactivedense
0
0 comments X
read the original abstract

Mixture of Experts (MoE) architectures have significantly increased computational efficiency in both research and real-world applications of large-scale machine learning models. However, their scalability and efficiency under memory constraints remain relatively underexplored. In this work, we present joint scaling laws for dense and MoE models, incorporating key factors such as the number of active parameters, dataset size, and the number of experts. Our findings provide a principled framework for selecting the optimal MoE configuration under fixed memory and compute budgets. Surprisingly, we show that MoE models can be more memory-efficient than dense models, contradicting conventional wisdom. To derive and validate the theoretical predictions of our scaling laws, we conduct over 280 experiments with up to 2.7B active parameters and up to 5B total parameters. These results offer actionable insights for designing and deploying MoE models in practical large-scale training scenarios.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. MobileMoE: Scaling On-Device Mixture of Experts

    cs.LG 2026-05 unverdicted novelty 6.0

    MobileMoE introduces on-device MoE LLMs that match dense models with 2-4x fewer FLOPs and provide efficient smartphone inference.

  2. Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource

    cs.CL 2025-06 conditional novelty 6.0

    MoE models with activation rates in an optimal region outperform dense LLMs of identical total parameter count, training compute, and data budget, with the optimal region consistent across scales.

  3. DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Experts

    cs.AI 2026-05 unverdicted novelty 5.0

    DAG-MoE uses a lightweight module to learn DAG-based structural aggregation of selected experts, expanding combination space and enabling intra-layer multi-step reasoning compared to standard weighted-sum MoE.

  4. Dense vs Sparse Pretraining at Tiny Scale: Active-Parameter vs Total-Parameter Matching

    cs.CL 2026-05 accept novelty 5.0

    At tiny scale, MoE transformers lower validation loss versus dense models when active parameters match but raise it when total stored parameters match.

  5. Ling and Ring 2.6 Technical Report: Efficient and Instant Agentic Intelligence at Trillion-Parameter Scale

    cs.CL 2026-06 unverdicted novelty 4.0

    Technical report announcing Ling-2.6 and Ring-2.6 models with hybrid linear attention, evolutionary CoT, and KPop RL for efficient agentic intelligence at scale.