REVIEW 8 cited by
MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
This paper presents MoE-Infinity, an efficient MoE inference system designed for personal machines with limited GPU memory capacity. The key idea for MoE-Infinity is that on personal machines, which are often single-user environments, MoE-based LLMs typically operate with a batch size of one. In this setting, MoE models exhibit a high degree of activation sparsity, meaning a small number of experts are frequently reused in generating tokens during the decode phase. Leveraging this idea, we design a sparsity-aware expert cache, which can trace the sparse activation of experts during inference and carefully select the trace that represents the sparsity pattern. By analyzing these selected traces, MoE-Infinity guides the replacement and prefetching of the expert cache, providing 3.1-16.7x per-token latency improvements over numerous state-of-the-art systems, including vLLM, Ollama, DeepSpeed and BrainStorm across various MoE models (DeepSeek and Mixtral) when handling different LLM tasks. MoE-Infinity's source code is publicly available at https://github.com/EfficientMoE/MoE-Infinity
Forward citations
Cited by 8 Pith papers
-
Beyond Global Routing Aggregation: Phase-Aware Expert Merging for MoE Vision-Language Models
Phase-aware expert merging based on routing role profiles preserves more MoE-VLM accuracy than global routing aggregation at matched compression ratios.
-
Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study
On bandwidth-bound edge hardware, MoE inference cost tracks total parameters rather than active ones, so sparse activation fails to deliver the expected throughput or energy gains.
-
HetRoute Heterogeneous and Cost-aware Collaborative Routing Framework for Distributed Edge MoE Inference
A routing framework for distributed edge MoE inference that co-optimizes expert placement, GPU/CPU residency, quantization precision, and per-token set-level routing, reporting large latency and throughput gains in si...
-
DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inference
Fixed-footprint shared+top-1+draft-expert self-speculation with residual/router distillation, expansion-aware truncation, and prefetch raises end-device MoE decode throughput ~1.45× while keeping exact target outputs.
-
Automated Tensor Scheduling for Hybrid CPU-GPU LLM Inference on Consumer Devices
Tensor-granularity hybrid CPU-GPU offloading with static placement plus load-aware dynamic transfer yields up to 1.94× prefill and 3.29× decode throughput for local LLMs on consumer devices.
-
Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanism
MoEpic splits each MoE expert into a cached top segment and a prefetched bottom segment, raising cache hit rates and hiding loading latency to accelerate offloaded MoE inference.
-
MoE-Compression: How the Compression Error of Experts Affects the Inference Accuracy of MoE Model?
In a 26-layer MoE model, injecting Gaussian weight errors into middle-layer experts hurts math accuracy most, while deep-layer errors can sometimes improve instruction compliance.
-
MoE-Beyond: Learning-Based Expert Activation Prediction on Edge Devices
An abstract-only MoE paper claiming 97.5% activation prediction accuracy and a 17% to 72% cache hit rate gain, whose full text is a different paper on functional equations, making the results unverifiable.
Discussion (0). Sign in to comment.