Pith. sign in

REVIEW 6 cited by

A Review of Sparse Expert Models in Deep Learning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2209.01667 v1 pith:AWAZZCU2 submitted 2022-09-04 cs.LG cs.CL

William Fedus , Jeff Dean , Barret Zoph This is my paper · ORCID
classification cs.LGcs.CL
keywords modelsdeepexpertlearningsparsearchitectureconceptexample
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Sparse expert models are a thirty-year old concept re-emerging as a popular architecture in deep learning. This class of architecture encompasses Mixture-of-Experts, Switch Transformers, Routing Networks, BASE layers, and others, all with the unifying idea that each example is acted on by a subset of the parameters. By doing so, the degree of sparsity decouples the parameter count from the compute per example allowing for extremely large, but efficient models. The resulting models have demonstrated significant improvements across diverse domains such as natural language processing, computer vision, and speech recognition. We review the concept of sparse expert models, provide a basic description of the common algorithms, contextualize the advances in the deep learning era, and conclude by highlighting areas for future work.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Soup-of-Experts: Pretraining Specialist Models via Parameters Averaging

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Soup-of-Experts pretrains a shared parameter bank and many expert vectors, plus a router, so a small specialist language model can be instantiated instantly from any domain-weight mixture without retraining.

  2. EmoStyle: Affective Conditioning of Style-Specialist Experts for Emotional Image Generation

    cs.CV 2026-07 conditional novelty 5.0 of 10

    EmoStyle injects LLM-inferred valence-arousal and emotion labels into Z-Image via AdaLN-style residual modulation over style-bucket LoRA experts, plus VLM candidate ranking, and ranked first on AffectiveArt Track 1.

  3. Mixture of LoRA Experts with Multi-Modal and Multi-Granularity LLM Generative Error Correction for Accented Speech Recognition

    cs.SD 2025-07 conditional novelty 5.0 of 10

    A three-stage pipeline of mono-accent LoRA experts, hierarchical routing, and phoneme-plus-word LLM error correction cuts accented-English WER from 6.34% to 2.07% on a combined 9-accent test set.

  4. Heterogeneous Ranking in Industrial-Scale Recommender Systems: A Case Study

    cs.IR 2026-07 conditional novelty 4.0 of 10

    Heterogeneity-conditioned gating and expert modulation improved multi-task ranking in Google Discover, narrowing the articles-vs-videos ranking gap and lifting feed engagement in A/B tests.

  5. (GG) MoE vs. MLP on Tabular Data

    cs.LG 2025-02 conditional novelty 4.0 of 10

    A Gumbel-Softmax-gated mixture of experts with numerical embeddings matches MLP accuracy on 38 tabular datasets while using roughly 10x fewer parameters, but its edge over MLP is not statistically significant.

  6. Position: AI Scaling: From Up to Down and Out

    cs.LG 2025-02 conditional novelty 4.0 of 10

    AI scaling is reframed as three paradigms: Scaling Up, Scaling Down, and Scaling Out, with future gains predicted to come from down and out.

Pith tools