Pith. sign in

REVIEW 4 cited by

Efficient Large Scale Language Modeling with Mixtures of Experts

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2112.10684 v2 pith:PLYSZZHU submitted 2021-12-20 cs.CL cs.AIcs.LG

Mikel Artetxe , Shruti Bhosale , Naman Goyal , Todor Mihaylov , Myle Ott , Sam Shleifer , Xi Victoria Lin , Jingfei Du
show 16 more authors
Srinivasan Iyer Ramakanth Pasunuru Giri Anantharaman Xian Li Shuohui Chen Halil Akin Mandeep Baines Louis Martin Xing Zhou Punit Singh Koura Brian O'Horo Jeff Wang Luke Zettlemoyer Mona Diab Zornitsa Kozareva Ves Stoyanov
This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords modelsdenselanguageefficientmoesscalecomputeexperts
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

Mixture of Experts layers (MoEs) enable efficient scaling of language models through conditional computation. This paper presents a detailed empirical study of how autoregressive MoE language models scale in comparison with dense models in a wide range of settings: in- and out-of-domain language modeling, zero- and few-shot priming, and full-shot fine-tuning. With the exception of fine-tuning, we find MoEs to be substantially more compute efficient. At more modest training budgets, MoEs can match the performance of dense models using $\sim$4 times less compute. This gap narrows at scale, but our largest MoE model (1.1T parameters) consistently outperforms a compute-equivalent dense model (6.7B parameters). Overall, this performance gap varies greatly across tasks and domains, suggesting that MoE and dense models generalize differently in ways that are worthy of future study. We make our code and models publicly available for research use.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. On the Fitness Landscape in the $NK$ Model

    math.PR 2025-08 unverdicted novelty 7.0 of 10

    For the NK fitness landscape with K/N tending to alpha, exact limits for free energy and maximum fitness are identified, together with the geometry of near-fittest peaks.

  2. HAP: Hybrid Adaptive Parallelism for Efficient Mixture-of-Experts Inference

    cs.DC 2025-08 conditional novelty 6.0 of 10

    HAP uses ILP over module-specific hybrid parallel choices to speed up MoE inference, reporting up to 1.77x versus tensor parallelism on tested GPUs.

  3. A Survey of LLM $\times$ DATA

    cs.DB 2025-05 conditional novelty 5.0 of 10

    A comprehensive survey of the bidirectional links between LLMs and data management, organized as DATA4LLM and LLM4DATA with a new 'IaaS' data-quality framework.

  4. A Survey of End-to-End Modeling for Distributed DNN Training: Workloads, Simulators, and TCO

    cs.DC 2025-06 conditional novelty 2.0 of 10

    This survey classifies distributed DNN training simulators into analytical, profiling-based, and execution-driven categories, and compares them alongside TCO and carbon-emission models.

Pith tools