Pith. sign in

REVIEW 11 cited by

GLaM: Efficient Scaling of Language Models with Mixture-of-Experts

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2112.06905 v2 pith:X55NRAQW submitted 2021-12-13 cs.CL

Nan Du , Yanping Huang , Andrew M. Dai , Simon Tong , Dmitry Lepikhin , Yuanzhong Xu , Maxim Krikun , Yanqi Zhou
show 19 more authors
Adams Wei Yu Orhan Firat Barret Zoph Liam Fedus Maarten Bosma Zongwei Zhou Tao Wang Yu Emma Wang Kellie Webster Marie Pellat Kevin Robinson Kathleen Meier-Hellstern Toju Duke Lucas Dixon Kun Zhang Quoc V Le Yonghui Wu Zhifeng Chen Claire Cui
This is my paper
classification cs.CL
keywords languagemodelsglamgpt-3scalingdensemixture-of-expertsmodel
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Scaling language models with more data, compute and parameters has driven significant progress in natural language processing. For example, thanks to scaling, GPT-3 was able to achieve strong results on in-context learning tasks. However, training these large dense models requires significant amounts of computing resources. In this paper, we propose and develop a family of language models named GLaM (Generalist Language Model), which uses a sparsely activated mixture-of-experts architecture to scale the model capacity while also incurring substantially less training cost compared to dense variants. The largest GLaM has 1.2 trillion parameters, which is approximately 7x larger than GPT-3. It consumes only 1/3 of the energy used to train GPT-3 and requires half of the computation flops for inference, while still achieving better overall zero-shot and one-shot performance across 29 NLP tasks.

Discussion (0). Sign in to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 168 citations worldwide. Full citation record

  1. The Ray Tracing Sampler: Bayesian Sampling of Neural Networks for Everyone

    astro-ph.IM 2025-10 conditional novelty 7.0 of 10

    A new ray-tracing MCMC sampler keeps ray speed constant, making it far more robust to stochastic gradients and able to sample billion-parameter neural networks on one GPU.

  2. Breaking the MoE LLM Trilemma: Dynamic Expert Clustering with Structured Compression

    cs.CL 2025-09 conditional novelty 6.0 of 10

    Dynamic expert clustering plus shared-base, low-rank residual compression and two-stage routing cuts MoE parameter count by about 80% with a modest quality cost on GLUE and WikiText-103.

  3. Kinetics: Rethinking Test-Time Scaling Laws

    cs.LG 2025-06 conditional novelty 6.0 of 10

    A memory-aware test-time scaling law shows small models are overestimated and sparse attention is needed for efficient scaling.

  4. Safe and Certifiable AI Systems: Concepts, Challenges, and Lessons Learned

    cs.CY 2025-09 conditional novelty 5.0 of 10

    The paper presents the TÜV AUSTRIA Trusted AI audit catalog, a statistical framework based on the Stochastic Application Domain Definition, minimum performance requirements, and independent-sample testing for certifyi...

  5. Apple Intelligence Foundation Language Models: Tech Report 2025

    cs.LG 2025-07 conditional novelty 5.0 of 10

    Apple's 3B on-device and larger server language models match or beat several similarly sized open models on MMLU, MMMLU, and MGSM, using new efficiency techniques like KV-cache sharing and 2-bit quantization.

  6. AbbIE: Autoregressive Block-Based Iterative Encoder for Efficient Sequence Modeling

    cs.LG 2025-07 conditional novelty 5.0 of 10

    An iterative Transformer that reuses its middle layers with an extra residual connection matches a standard Transformer after one pass and improves perplexity and zero-shot accuracy when run for more passes at test time.

  7. Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insights

    cs.LG 2025-06 conditional novelty 5.0 of 10

    At 56B total parameters, fine-grained MoE with smaller, more numerous experts beats standard Switch and Mixtral-style MoE on validation loss and average downstream accuracy at matched FLOPs.

  8. Rethinking Data Mixture for Large Language Models: A Comprehensive Survey and New Perspectives

    cs.CL 2025-05 accept novelty 5.0 of 10

    A survey organizing LLM data mixture methods into offline and online families, with a fine-grained taxonomy based on optimization frameworks.

  9. Memory for Large Language Models

    cs.CL 2026-07 conditional novelty 4.0 of 10

    The paper organizes LLM memory architectures along three axes—implicit vs explicit representation, offline vs online update, and short vs long-term persistence—plus finer-grained update rules.

  10. A Vision Toward Energy-Efficient Domain-Specific Artificial Intelligence Models and Agents

    cs.AI 2025-10 unverdicted novelty 2.0 of 10

    A position paper proposing compact, domain-specific AI agents as the path to ≥1000× energy efficiency, without demonstrating the claim.

  11. The Carbon Cost of Conversation, Sustainability in the Age of Language Models

    cs.CY 2025-07 reject novelty 2.0 of 10

    A review-style essay that claims LLM training and use are environmentally expensive, but the quantitative evidence it presents is unreliable and internally inconsistent.

Pith tools