REVIEW 5 cited by
Who Says Elephants Can't Run: Bringing Large Scale MoE Models into Cloud Scale Production
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Mixture of Experts (MoE) models with conditional execution of sparsely activated layers have enabled training models with a much larger number of parameters. As a result, these models have achieved significantly better quality on various natural language processing tasks including machine translation. However, it remains challenging to deploy such models in real-life scenarios due to the large memory requirements and inefficient inference. In this work, we introduce a highly efficient inference framework with several optimization approaches to accelerate the computation of sparse models and cut down the memory consumption significantly. While we achieve up to 26x speed-up in terms of throughput, we also reduce the model size almost to one eighth of the original 32-bit float model by quantizing expert weights into 4-bit integers. As a result, we are able to deploy 136x larger models with 27% less cost and significantly better quality compared to the existing solutions. This enables a paradigm shift in deploying large scale multilingual MoE transformers models replacing the traditional practice of distilling teacher models into dozens of smaller models per language or task.
Forward citations
Cited by 5 Pith papers
-
Ecco: Improving Memory Bandwidth and Capacity for LLMs via Entropy-aware Cache Compression
Ecco compresses LLM weights and KV cache with shared k-means patterns and Huffman coding, and adds a parallel hardware decoder, to roughly quadruple memory capacity and speed up decode by up to ~3x.
-
MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design
MxMoE assigns mixed bitwidths at the linear-block level in MoE models using an ILP that balances measured quantization sensitivity and expert activation frequency, then auto-generates fused Group-GEMM kernels, improvi...
-
TileLang: A Composable Tiled Programming Model for AI Systems
TileLang is a tiled DSL and compiler that decouples scheduling from dataflow and reports performance at or near state-of-the-art vendor kernels on modern GPUs.
-
Flash Communication: Reducing Tensor Parallelization Bottleneck for Fast Large Language Model Inference
A quantized two-step all-reduce kernel reduces tensor-parallel communication overhead in LLM inference, achieving up to 3.18x faster all-reduce and 2.06x TTFT speedup on L40 GPUs.
-
A Survey on Inference Optimization Techniques for Mixture of Experts Models
A structured survey of MoE inference optimization that categorizes existing techniques into model, system, and hardware levels and summarizes reported speedups and memory savings.
Discussion (0). Continue with ORCID to comment.