REVIEW 7 cited by
Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-training
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Mixture-of-experts (MoE) models facilitate efficient scaling; however, training the router network introduces the challenge of optimizing a non-differentiable, discrete objective. Recently, a fully-differentiable MoE architecture, SMEAR, was proposed (Muqeeth et al., 2023), which softly merges experts in the parameter space; nevertheless, its effectiveness was only demonstrated in downstream fine-tuning on classification tasks. In this paper, we present Lory, the first approach that scales such architectures to autoregressive language model pre-training. Lory introduces two key techniques: (1) a causal segment routing strategy that achieves high efficiency for expert merging operations while preserving the autoregressive nature of language models; (2) a similarity-based data batching method that encourages expert specialization by grouping similar documents in training instances. We pre-train a series of Lory models on 150B tokens from scratch, with up to 32 experts and 30B (1.5B active) parameters. Experimental results show significant performance gains over parameter-matched dense models on both perplexity (+13.9%) and a variety of downstream tasks (+1.5%-11.1%). Despite segment-level routing, Lory models achieve competitive performance compared to state-of-the-art MoE models with token-level routing. We further demonstrate that the trained experts in Lory capture domain-level specialization without supervision. Our work highlights the potential of fully-differentiable MoE architectures for language model pre-training and advocates future research in this area.
Forward citations
Cited by 7 Pith papers
-
Full-bandwidth transformer
Fusing the previous top-layer hidden state into the next input token through a gated linear unit improves transformer pretraining data efficiency and decoding accuracy at 1B scale.
-
Grouter: Decoupling Routing from Representation for Accelerated MoE Training
A frozen router distilled from a converged MoE teacher accelerates target MoE pretraining, reaching the same loss with about 4.3x less data and up to 33.5% higher throughput.
-
BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsity
A ReLU-routed MoE with chunk-level sparsity training objectives and custom kernels combining activation sparsity with speculative decoding achieves over 70% 8-token chunk sparsity and up to 3.67x end-side speedup.
-
Guiding the Experts: Semantic Priors for Efficient and Focused MoE Routing
Guiding Soft MoE dispatch weights with foreground segmentation masks plus a zero-initialized LayerScale improves ImageNet-1K top-1 by 0.6% and ImageNet-100 by 1.4% over a reproduced baseline.
-
ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing
ReLU-based routing with adaptive L1 regularization outperforms TopK routing in MoE language models across model sizes, expert counts, and granularity.
-
The Evolution of Mixture-of-Experts Architectures in Large Language Models: Routing, Topology, Load Balancing, and Expert Parallelism
A survey proposes eight architectural milestones and four control planes for MoE LLMs, arguing the field is moving toward decoupling routing, compute budgets, and physical execution.
-
Mixture of Experts (MoE): A Big Data Perspective
A survey of MoE methods for big data that catalogs architectures, use cases, and open challenges without adding new results.
Discussion (0). Continue with ORCID to comment.