Pith. sign in

REVIEW 4 cited by

Mixture of Parrots: Experts improve memorization more than reasoning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.19034 v2 pith:4XHWHWUO submitted 2024-10-24 cs.LG

Samy Jelassi , Clara Mohri , David Brandfonbrener , Alex Gu , Nikhil Vyas , Nikhil Anand , David Alvarez-Melis , Yuanzhi Li
show 2 more authors
Sham M. Kakade Eran Malach
This is my paper · ORCID
classification cs.LG
keywords numberexpertsmoesreasoningtasksdenseparametersactive
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

The Mixture-of-Experts (MoE) architecture enables a significant increase in the total number of model parameters with minimal computational overhead. However, it is not clear what performance tradeoffs, if any, exist between MoEs and standard dense transformers. In this paper, we show that as we increase the number of experts (while fixing the number of active parameters), the memorization performance consistently increases while the reasoning capabilities saturate. We begin by analyzing the theoretical limitations of MoEs at reasoning. We prove that there exist graph problems that cannot be solved by any number of experts of a certain width; however, the same task can be easily solved by a dense model with a slightly larger width. On the other hand, we find that on memory-intensive tasks, MoEs can effectively leverage a small number of active parameters with a large number of experts to memorize the data. We empirically validate these findings on synthetic graph problems and memory-intensive closed book retrieval tasks. Lastly, we pre-train a series of MoEs and dense transformers and evaluate them on commonly used benchmarks in math and natural language. We find that increasing the number of experts helps solve knowledge-intensive tasks, but fails to yield the same benefits for reasoning tasks.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Theoretical limitations of multi-layer Transformer

    cs.LG 2024-12 conditional novelty 8.0 of 10

    An L-layer decoder-only Transformer requires polynomial model dimension to compute L-step sequential function composition, and this is proven without any unproven complexity conjecture.

  2. The power of fine-grained experts: Granularity boosts expressivity in Mixture of Experts

    cs.LG 2025-05 conditional novelty 7.0 of 10

    For MoE layers with equal active parameters, increasing the number of active experts from k' to k yields an exponential separation in L2 approximation error whenever the binomial coefficient binom(m,k) dominates binom(m',k').

  3. Understanding Factual Recall in Transformers via Associative Memories

    cs.LG 2024-12 accept novelty 6.0 of 10

    A one-layer transformer can store facts at near-optimal capacity by using attention value matrices or an MLP as associative memories, and training passes through a hallucination stage.

  4. Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models

    cs.LG 2025-01 conditional novelty 4.0 of 10

    For MoE language models, under a fixed training compute budget, increasing sparsity while adding total parameters lowers pretraining loss, and the optimal sparsity grows with model size.

Pith tools