Pith. sign in

REVIEW 4 cited by

SwapMoE: Serving Off-the-shelf MoE-based Large Language Models with Tunable Memory Budget

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2308.15030 v4 pith:CZBAALA2 submitted 2023-08-29 cs.AI

Rui Kong , Yuanchun Li , Qingtian Feng , Weijun Wang , Xiaozhou Ye , Ye Ouyang , Linghe Kong , Yunxin Liu This is my paper · ORCID
classification cs.AI
keywords expertsmemoryswapmoelargemodelslanguageservingaccuracy
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Mixture of experts (MoE) is a popular technique to improve capacity of Large Language Models (LLMs) with conditionally-activated parallel experts. However, serving MoE models on memory-constrained devices is challenging due to the large parameter size. Typical solutions such as memory swapping or expert pruning may lead to significantly higher latency or severe accuracy loss. In this paper, we introduce SwapMoE, a framework for efficient serving of MoE-based large language models with tunable memory budgets. The main idea of SwapMoE is to keep a small dynamic set of important experts, namely Virtual Experts, in the main memory for inference, while seamlessly maintaining how the Virtual Experts map to the actual experts. Experiments have shown that SwapMoE can reduce the memory footprint while maintaining reasonable accuracy. For example, on text summarization tasks with Switch Transformer, SwapMoE can reduce the memory consumption from 14.2 GiB to 4.7 GiB, together with 50\% latency reduction and a slight Rouge-2 score drop of 0.041.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. EcoServe: Designing Carbon-Aware AI Inference Systems

    cs.DC 2025-02 conditional novelty 6.0 of 10

    EcoServe combines four strategies (reuse, rightsize, reduce, recycle) in an ILP optimizer to cut modeled carbon emissions for LLM serving by up to 47% while keeping SLOs.

  2. A Replicate-and-Quantize Strategy for Plug-and-Play Load Balancing of Sparse Mixture-of-Experts LLMs

    cs.LG 2026-02 reject novelty 5.0 of 10

    Replicate-and-Quantize copies the busiest MoE expert as a quantized duplicate and compresses the least important expert, lowering a new Load-Imbalance Score by up to 1.4x while accuracy varies by roughly -1.2 to +3.0 points.

  3. MoE-Compression: How the Compression Error of Experts Affects the Inference Accuracy of MoE Model?

    cs.LG 2025-09 reject novelty 5.0 of 10

    In a 26-layer MoE model, injecting Gaussian weight errors into middle-layer experts hurts math accuracy most, while deep-layer errors can sometimes improve instruction compliance.

  4. Efficient Deployment of Vision-Language Models on Mobile Devices: A Case Study on OnePlus 13R

    cs.LG 2025-07 conditional novelty 5.0 of 10

    A side-by-side mobile benchmark shows VLM runtimes on a OnePlus 13R leave accelerators idle, push CPUs to thermal limits, and achieve order-of-magnitude power savings only when the GPU handles image and language kernels.

Pith tools