REVIEW 7 cited by
Fast Inference of Mixture-of-Experts Language Models with Offloading
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
With the widespread adoption of Large Language Models (LLMs), many deep learning practitioners are looking for strategies of running these models more efficiently. One such strategy is to use sparse Mixture-of-Experts (MoE) - a type of model architectures where only a fraction of model layers are active for any given input. This property allows MoE-based language models to generate tokens faster than their dense counterparts, but it also increases model size due to having multiple experts. Unfortunately, this makes state-of-the-art MoE language models difficult to run without high-end GPUs. In this work, we study the problem of running large MoE language models on consumer hardware with limited accelerator memory. We build upon parameter offloading algorithms and propose a novel strategy that accelerates offloading by taking advantage of innate properties of MoE LLMs. Using this strategy, we build can run Mixtral-8x7B with mixed quantization on desktop hardware and free-tier Google Colab instances.
Forward citations
Cited by 7 Pith papers
-
Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study
On bandwidth-bound edge hardware, MoE inference cost tracks total parameters rather than active ones, so sparse activation fails to deliver the expected throughput or energy gains.
-
Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanism
MoEpic splits each MoE expert into a cached top segment and a prefetched bottom segment, raising cache hit rates and hiding loading latency to accelerate offloaded MoE inference.
-
Cache Management for Mixture-of-Experts LLMs -- extended version
The paper formalizes expert caching in MoE LLMs as layered paging, proves near-tight competitive lower bounds, and shows a layer-aware LRU variant beats standard LRU on real traces.
-
Broadcast Rate Limits in Wi-Fi: A Forgotten Bottleneck for Collaborative Edge LLM Inference
IEEE 802.11's low broadcast-rate cap, not the wireless medium itself, is the main bottleneck for one-to-many MoE embedding dispatch, and simulation finds optimal rates 32–64× above the cap.
-
A Scheduling Framework for Efficient MoE Inference on Edge GPU-NDP Systems
On simulated GPU-NDP-DIMM hardware, scheduling MoE experts with tensor parallelism, load balancing, and prefill-driven pre-fetching cuts end-to-end latency by 2.41x on average versus MoNDE.
-
MoE-Compression: How the Compression Error of Experts Affects the Inference Accuracy of MoE Model?
In a 26-layer MoE model, injecting Gaussian weight errors into middle-layer experts hurts math accuracy most, while deep-layer errors can sometimes improve instruction compliance.
-
On-the-Fly Adaptive Distillation of Transformer to Dual-State Linear Attention
On-the-fly distillation of Transformer layers to dual-state linear attention produces about 2.3x faster simulated LLM serving than Llama2-7B with roughly comparable benchmark accuracy.
Discussion (0). Sign in to comment.