Pith. sign in

REVIEW 4 cited by

Upcycling Large Language Models into Mixture of Experts

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.07524 v2 pith:LUU4NGDJ submitted 2024-10-10 cs.CL cs.AIcs.LG

Ethan He , Abhinav Khattar , Ryan Prenger , Vijay Korthikanti , Zijie Yan , Tong Liu , Shiqing Fan , Ashwath Aithal
show 2 more authors
Mohammad Shoeybi Bryan Catanzaro
This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords upcyclingmodelsmodellanguageapproachtrainedachieveddense
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Upcycling pre-trained dense language models into sparse mixture-of-experts (MoE) models is an efficient approach to increase the model capacity of already trained models. However, optimal techniques for upcycling at scale remain unclear. In this work, we conduct an extensive study of upcycling methods and hyperparameters for billion-parameter scale language models. We propose a novel "virtual group" initialization scheme and weight scaling approach to enable upcycling into fine-grained MoE architectures. Through ablations, we find that upcycling outperforms continued dense model training. In addition, we show that softmax-then-topK expert routing improves over topK-then-softmax approach and higher granularity MoEs can help improve accuracy. Finally, we upcycled Nemotron-4 15B on 1T tokens and compared it to a continuously trained version of the same model on the same 1T tokens: the continuous trained model achieved 65.3% MMLU, whereas the upcycled model achieved 67.6%. Our results offer insights and best practices to effectively leverage upcycling for building MoE language models. Code is available.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Automatic Expert Discovery in LLM Upcycling via Sparse Interpolated Mixture-of-Experts

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Sparse Interpolated Mixture-of-Experts learns neuron-level masks and a per-prompt router to upcycle a dense LLM into a sparse expert model, improving instruction-tuning benchmarks with lower inference cost.

  2. MoLGE: Mixture of Language Group Experts for Efficient Scaling of Massively Multilingual Speech Recognition

    cs.CL 2026-07 conditional novelty 5.0 of 10

    Grouping 495 languages into roughly 16 clusters and routing speech to group-specific LoRA experts improves multilingual ASR error rates over dense and random baselines.

  3. Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insights

    cs.LG 2025-06 conditional novelty 5.0 of 10

    At 56B total parameters, fine-grained MoE with smaller, more numerous experts beats standard Switch and Mixtral-style MoE on validation loss and average downstream accuracy at matched FLOPs.

  4. ScatterPrism: convergence for generative simulation and inverse problems in particle and nuclear physics

    cs.LG 2026-04 unverdicted novelty 4.0 of 10

    Standard Conditional Flow Matching loss is a misleading early plateau; physics-informed metrics keep improving, so ScatterPrism and multi-metric diagnostics are needed for kinematic fidelity.

Pith tools