REVIEW 8 cited by
Mixture of A Million Experts
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The feedforward (FFW) layers in standard transformer architectures incur a linear increase in computational costs and activation memory as the hidden layer width grows. Sparse mixture-of-experts (MoE) architectures have emerged as a viable approach to address this issue by decoupling model size from computational cost. The recent discovery of the fine-grained MoE scaling law shows that higher granularity leads to better performance. However, existing MoE models are limited to a small number of experts due to computational and optimization challenges. This paper introduces PEER (parameter efficient expert retrieval), a novel layer design that utilizes the product key technique for sparse retrieval from a vast pool of tiny experts (over a million). Experiments on language modeling tasks demonstrate that PEER layers outperform dense FFWs and coarse-grained MoEs in terms of performance-compute trade-off. By enabling efficient utilization of a massive number of experts, PEER unlocks the potential for further scaling of transformer models while maintaining computational efficiency.
Forward citations
Cited by 8 Pith papers
-
Training, Reading, and Editing Legible Transformers
A variance-floor objective plus learned operator gates produce an end-to-end legible transformer whose crisp units are 50–184× more local to edit and can be reshaped from fan-out to fan-in circuits without quality loss.
-
OmniMoE: An Efficient MoE by Orchestrating Atomic Experts at Scale
OmniMoE's vector-level atomic experts + Cartesian-product router + expert-centric scheduling gives 50.9% zero-shot accuracy and a 10.9x inference speedup over PEER at matched 1.7B active parameters.
-
The SpiNNaker2 chip: a many-core platform for flexible and scalable brain-inspired computing
SpiNNaker2 delivers a measured many-core platform combining ARM cores, ML accelerators, and event routing that runs SNNs, DNNs, and hybrid event-based models on one scalable chip.
-
More Than Memory: Task-Conditioned Signed FFN Writes in Long-Context Retrieval
Scaling a model's own per-layer feed-forward writes shows signed, task-dependent retrieval roles, and the sign of the write–gradient dot product identifies suppressors from amplifiers.
-
MoLEx: Mixture of LoRA Experts in Speech Self-Supervised Models for Audio Deepfake Detection
MoLEx combines LoRA adapters with a top-K expert router inside a frozen WavLM model, achieving 5.56% EER on ASVSpoof 5 without augmentation.
-
PC-MoE: Memory-Efficient and Privacy-Preserving Collaborative Training for Mixture-of-Experts LLMs
PC-MoE shards the expert layers of an MoE LLM across parties and routes only sparse top-k activations between them, achieving near-centralized accuracy with about 70% memory savings and resistance to one partial-gradi...
-
UltraMemV2: Memory Networks Scaling to 120B Parameters with Superior Long-Context Learning
A redesigned memory-layer architecture with five engineering improvements reaches performance parity with 8-expert MoE at similar compute, with lower memory access and stronger long-context memorization.
-
Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors Approach
Llama-SMoP-DEDR, a sparse mixture of projectors with modality-specific experts and routers, lowers word error rate for LLM-based AVSR on LRS3, mainly with smaller LLMs.
Discussion (0). Continue with ORCID to comment.