REVIEW 5 cited by
LocMoE: A Low-Overhead MoE for Large Language Model Training
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
The Mixtures-of-Experts (MoE) model is a widespread distributed and integrated learning method for large language models (LLM), which is favored due to its ability to sparsify and expand models efficiently. However, the performance of MoE is limited by load imbalance and high latency of All-to-All communication, along with relatively redundant computation owing to large expert capacity. Load imbalance may result from existing routing policies that consistently tend to select certain experts. The frequent inter-node communication in the All-to-All procedure also significantly prolongs the training time. To alleviate the above performance problems, we propose a novel routing strategy that combines load balance and locality by converting partial inter-node communication to that of intra-node. Notably, we elucidate that there is a minimum threshold for expert capacity, calculated through the maximal angular deviation between the gating weights of the experts and the assigned tokens. We port these modifications on the PanGu-Sigma model based on the MindSpore framework with multi-level routing and conduct experiments on Ascend clusters. The experiment results demonstrate that the proposed LocMoE reduces training time per epoch by 12.68% to 22.24% compared to classical routers, such as hash router and switch router, without impacting the model accuracy.
Forward citations
Cited by 5 Pith papers
-
Towards Paradigm-General Suicide Risk Detection via Speech LLM
A speech LLM with a mixture of DoRA experts (MoDE) improves cross-task suicide risk detection accuracy (0.656 vs 0.635 joint tuning) and calibration on 1,223 Chinese adolescents across ten speech tasks, though the uns...
-
Theory of Mixture-of-Experts for Mobile Edge Computing
For continual learning in mobile edge computing, a mixture-of-experts router with enough edge servers can bound overall generalization error to the minimum within-task-type gap, unlike standard offloading.
-
Software Engineering for Large Language Models: Research Status, Challenges and the Road Ahead
A literature review organizes LLM development into a six-phase software engineering lifecycle and identifies challenges and research directions for each phase.
-
MoE$^2$: Optimizing Collaborative Inference for Edge Large Language Models
MoE2 trains a gating network once on all edge LLMs, then selects a resource-feasible subset and a per-prompt top-k of experts to fuse token probabilities under latency and energy budgets.
-
A Survey on Inference Optimization Techniques for Mixture of Experts Models
A structured survey of MoE inference optimization that categorizes existing techniques into model, system, and hardware levels and summarizes reported speedups and memory savings.
Discussion (0). Continue with ORCID to comment.