REVIEW 5 cited by
EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large language models (LLMs) such as GPTs and Mixtral-8x7B have revolutionized machine intelligence due to their exceptional abilities in generic ML tasks. Transiting LLMs from datacenters to edge devices brings benefits like better privacy and availability, but is challenged by their massive parameter size and thus unbearable runtime costs. To this end, we present EdgeMoE, an on-device inference engine for mixture-of-expert (MoE) LLMs -- a popular form of sparse LLM that scales its parameter size with almost constant computing complexity. EdgeMoE achieves both memory- and compute-efficiency by partitioning the model into the storage hierarchy: non-expert weights are held in device memory; while expert weights are held on external storage and fetched to memory only when activated. This design is motivated by a key observation that expert weights are bulky but infrequently used due to sparse activation. To further reduce the expert I/O swapping overhead, EdgeMoE incorporates two novel techniques: (1) expert-wise bitwidth adaptation that reduces the expert sizes with tolerable accuracy loss; (2) expert preloading that predicts the activated experts ahead of time and preloads it with the compute-I/O pipeline. On popular MoE LLMs and edge devices, EdgeMoE showcase significant memory savings and speedup over competitive baselines. The code is available at https://github.com/UbiquitousLearning/mllm.
Forward citations
Cited by 5 Pith papers
-
DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inference
Fixed-footprint shared+top-1+draft-expert self-speculation with residual/router distillation, expansion-aware truncation, and prefetch raises end-device MoE decode throughput ~1.45× while keeping exact target outputs.
-
AirMoE: Statistic-Augmented Over-the-Air MoE for Collaborative Intelligence
AirMoE routes clients via compact prototype statistics and aggregates expert outputs over the air by transmitting them simultaneously, claiming communication savings and better segmentation accuracy.
-
Dissecting the Impact of Mobile DVFS Governors on LLM Inference Performance and Energy Efficiency
Independent Android DVFS governors waste energy and time during on-device LLM inference, and a coordinated governor (FUSE) recovers 7.0-36.8% of the lost performance.
-
Collaborative Inference and Learning between Edge SLMs and Cloud LLMs: A Survey of Algorithms, Execution, and Open Challenges
A survey that builds a taxonomy of edge-cloud LLM-SLM collaboration for inference and training, claiming to be the first to unify both phases.
-
MoE-GPS: Guidlines for Prediction Strategy for Dynamic Expert Duplication in MoE Load Balancing
MoE-GPS shows that predicting only the aggregate token-to-expert distribution, instead of exact token routing, can improve MoE inference latency by more than 23% in simulation, with guidelines for when each strategy wins.
Discussion (0). Sign in to comment.