Pith. sign in

REVIEW 4 major objections 5 minor 36 references

PagedWeight quantizes MoE expert weights at runtime to free GPU memory for growing KV caches.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 21:05 UTC pith:LJIIUQDL

load-bearing objection PagedWeight has a genuinely new paging mechanism for MoE weights and a credible system, but the missing static-with-same-sensitivity comparison and the C4 calibration leak mean the dynamic advantage isn't yet established. the 4 major comments →

arxiv 2607.16184 v1 pith:LJIIUQDL submitted 2026-07-17 cs.LG

PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization

Yuchen Yang , Yifan Zhao , Anisha Dasgupta , Sasa Misailovic This is my paper
classification cs.LG
keywords Mixture-of-ExpertsLLM servingweight quantizationKV cachememory managementany-precisionmixed-precisionruntime planning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper proposes PagedWeight, a serving-time memory manager for mixture-of-experts (MoE) large language models that trades expert-weight precision for KV-cache space as contexts grow. The central claim is that dynamically quantizing individual expert linear-blocks—lowering bitwidths only when GPU memory pressure demands it—lands on a better quality-memory frontier than any fixed, uniform, or pre-runtime precision assignment. PagedWeight picks which weights to shrink using a hybrid damage estimate (offline Hessian sensitivity, online routing hotness, and prompt-conditioned residuals) and moves released bit-planes to CPU asynchronously, so inference is not interrupted. If correct, serving systems can run longer contexts and larger batches on the same GPU while keeping accuracy at the full-precision level. The paper reports FP16-equivalent accuracy with up to 72% GPU memory savings and 1.94× throughput gains.

Core claim

PagedWeight's discovery is that expert weights in MoE serving should be managed as a dynamic, pageable resource alongside the KV cache. Using a bit-plane weight format with lookup tables, each expert linear-block becomes a 'weight page' whose bitwidth can be lowered or restored. A runtime planner converts KV-cache pressure into a byte-reduction target, then greedily selects per-block bitwidth reductions by predicted quality damage per released byte. The damage estimate combines a Hessian sensitivity prior, routing-mass buckets protecting frequently used experts, and a prompt-residual correction computed from input norms. Offloading and reloading pages are asynchronous and committed only at s

What carries the argument

The central object is the 'weight page': one bit-plane (and its lookup table) of an expert linear-block, identified by layer, expert, and block type, with a committed bitwidth q_i and desired bitwidth d_i in a page table. The decision identity is the predicted damage per action, d̂ = max{ε, μ_β g_i η_i}, where g_i is an offline Hessian sensitivity score, μ_β is a routing-bucket multiplier, and η_i is an exponentiated prompt residual; the planner selects actions with the lowest damage per released byte until the KV-pressure byte target is met. This damage-per-byte ranking is what maps memory pressure into quality-aware page movements without interrupting inference.

Load-bearing premise

The planner's predicted quality damage faithfully ranks true harm to the model output; if offline sensitivity scores and prompt-residual heads are miscalibrated for an unseen prompt, the wrong expert weights get quantized and the accuracy–memory tradeoff collapses even though memory and throughput behave as designed.

What would settle it

On held-out prompts with unusual routing patterns, compare the planner's predicted damage ranking against the actual perplexity or accuracy change from applying each reduction individually. A Spearman correlation near zero between predicted and actual damage would falsify the core claim; a concrete test is to log the planner's top-100 chosen actions, revert them one by one, and measure the per-action quality change.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • MoE serving can hold full-precision accuracy at far lower GPU memory (up to 72% savings), enabling longer contexts or larger batches on the same hardware.
  • Runtime, pageable weight quantization dominates static uniform and static mixed-precision policies across language modeling, reasoning, and long-context tasks.
  • Throughput stays within a few percent (≤4.1%) of a uniform low-bit baseline while memory footprint adapts to KV-cache growth.
  • Granular per-linear-block control makes the method effective across MoE architectures with very different expert counts and top-k choices.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same paging abstraction could extend to other compressible weight formats (e.g., attention projections or shared experts), broadening the dynamic memory-tradeoff idea beyond routed experts.
  • The damage predictor could be made self-correcting by using online task-quality feedback instead of proxy perplexity, potentially improving robustness on out-of-distribution prompts.
  • For workloads with highly skewed routing, the routing-statistics component should yield even larger quality gains than reported, because hot-expert protection becomes more informative.
  • A testable extension: adapt the page-granularity and bitwidth floor per token-generation phase, since early tokens may tolerate more quantization than later ones.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. PagedWeight proposes a runtime system for MoE LLM serving that treats Any-Precision bit-plane/LUT state of each expert linear-block as a swappable weight page. A planner combines offline Hessian sensitivity (Eq. 4), online routing-mass buckets, and prompt-residual corrections (Eqs. 5-8) to select low-damage bitwidth reductions when the KV cache demands memory. Page offload/reload is asynchronous and a fused mixed-precision MoE kernel executes variable bitwidths. The paper reports quality-memory, long-context, throughput, and ablation experiments on Qwen1.5-MoE-A2.7B, Mixtral-8x7B, and Gemma-4-26B-A4B, claiming FP16-equivalent accuracy at up to 72.0% GPU memory savings and 1.94x throughput improvement.

Significance. The system contribution is concrete and timely: the weight-page abstraction, asynchronous movement pipeline, and fused kernel are engineering contributions that go beyond static quantization. The planner's combination of Hessian sensitivity, routing statistics, and prompt residual is interesting and is evaluated on three architecturally distinct MoE models with a range of baselines. However, the evaluation as presented does not yet establish the central 'dynamic' claim or the exact headline operating point, and the C4 calibration overlap reduces the weight of the perplexity evidence. With the missing static-dynamic comparison and a de-circularized evaluation, the paper would be a solid systems contribution.

major comments (4)
  1. [§4, §5.1 (Fig. 5, Table 4)] Planner calibration leaks into evaluation: the policy-construction paragraph in §4 states that the sensitivity table, prompt-residual heads, routing buckets, bucket multipliers, depth caps, and residual weights are built on the C4 calibration set, and then C4 perplexity is reported as an evaluation metric in Fig. 5 and used in every ablation row in Table 4. Since C4 is the same distribution used to fit the planner's quality model, the C4 rows are partly circular and cannot by themselves support the quality-preservation claim. Please either hold out a calibration split and evaluate on the held-out portion, or designate C4 as a calibration diagnostic and base the quality claim on Wikitext2, GSM8K, MATH-500, and LongBench.
  2. [§5.1/§5.4 (Fig. 5, Table 4)] The central claim is that runtime page movement improves the quality-memory tradeoff, but no baseline implements a static PagedWeight policy with the same planner components (Eq. 4 sensitivity, routing-bucket multipliers, prompt residual). The 'w/o page movement' ablation in Table 4 is a static plan, yet it is reported at a single operating point, has no memory figure, and (by construction) omits the prompt residual; it cannot be placed on the quality-memory plane of Fig. 5. Add a full static-mixed-precision PagedWeight frontier (planner run offline at each memory budget, then fixed during generation) and compare it with dynamic paging. If the curves coincide, the dynamic mechanism is not load-bearing; if they differ, the paper needs to say where and why.
  3. [§5.2, Table 2] The LongBench results are non-monotonic in memory: average score 17.0 at 9.86GB, 15.7 at 11.82GB, 16.5 at 12.79GB, 16.8 at 14.83GB, and Passage Retrieval drops from 17.5% at 9.86GB to 14.5% at 11.82GB. Since additional memory should not degrade quality under the paper's premise, these numbers suggest high variance or uncontrolled differences between configurations. No error bars, seeds, or repeated runs are reported. RQ2's conclusion that PagedWeight preserves long-context quality at the 10GB budget therefore needs statistical support or a mechanistic explanation.
  4. [Abstract, Tables 2-3] The headline 'FP16-equivalent accuracy with up to 72.0% GPU memory savings and 1.94x throughput improvement' is obtained by combining different rows. The 72.0% savings corresponds to the 9.86GB row of Table 2 (vs. 35.25GB FP16), while the 1.94x throughput is the 7.63GB row of Table 3 (130.1 vs. 67.1 TPS); no quality is reported for the 7.63GB row, and no throughput is reported for the 9.86GB row. Please state the operating point at which both conditions hold, or revise the claim so that the memory-savings and throughput figures are traceable to the same configuration.
minor comments (5)
  1. [§2.1, Fig. 2] The y-axis label 'Usage' is not the routing mass defined in Eq. 1; define the plotted quantity so the reader can connect the figure to the formal definition.
  2. [§3.2, Eq. (7)] Define what happens when D=0 but pressure is triggered; the relationship between Tblk and the offload threshold is not explicit.
  3. [§5.1 baseline section] MxMoE is only shown in its 3.25-bit config; clarify why other configurations were not used, since the comparison is meant to span the memory-quality frontier.
  4. [Abstract] The '39.3% quality improvement' is not tied to a specific baseline, task, or memory budget; please make the reference point explicit.
  5. [General] No code release or artifact link is mentioned; for a systems paper, releasing the implementation would materially help reproducibility.

Circularity Check

1 steps flagged

C4-based calibration leaks into the C4 perplexity evaluation arm; independent benchmarks keep the central claim from collapsing.

specific steps
  1. fitted input called prediction [Section 4 'Policy Construction' / Section 5.1 'Quality-Memory Tradeoff' (Figure 5)]
    "Policy Construction. The quantization and runtime hyperparameters are built offline on the C4 calibration set. We first form a prompt-independent sensitivity table for each routed expert linear-block and supported bitwidth transition. Then we train the prompt residual from routing weighted input norm features, and use a small calibration search to choose the routing buckets, bucket multipliers, depth caps, and residual weight used by the online planner."

    The planner's predicted damage for each bitwidth reduction (Eq. 8: d̂ = max{ε, μ_β g_i η_i}) is composed entirely of objects fitted on C4 — the calibrated sensitivity table (Eq. 4), the trained prompt-residual heads (Eqs. 5–6), and the calibration-searched bucket multipliers and depth caps. The same C4 set is then used as one of the two perplexity benchmarks in Figure 5 and in the Table 4 ablations. Therefore the C4 arm reports an in-sample fit as if it were an out-of-sample prediction: the planner was tuned to agree with C4 damage estimates, so its C4 perplexity advantage is partly forced by construction. The other benchmarks (Wikitext2, GSM8K, MATH-500, LongBench) are not used in calibration, so the central quality-memory claim still has independent support.

full rationale

The only identifiable circularity is the C4 calibration/evaluation overlap: the planner's sensitivity scores, prompt-residual heads, bucket multipliers, and depth caps are all 'built offline on the C4 calibration set,' yet C4 perplexity is reported as evidence in Section 5.1 and Table 4. This makes the C4 arm partly self-confirming. I do not find the other suspected circularities: no load-bearing self-citation (ARQ [32] appears only in related work; the APL, MxMoE, DP-LLM baselines are external), no uniqueness theorem, and no ansatz smuggled through citation. The independent Wikitext2/GSM8K/MATH-500/LongBench results mean the central dynamic-quantization quality-memory claim does not reduce to the fitted inputs. The missing static-same-sensitivity quality-memory comparison noted by the skeptic is an evaluation gap that weakens the dynamic-adaptation attribution, but it is a correctness/experimental-design concern, not a circular derivation, and therefore does not change the score.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

No new physical entities are introduced; 'weight page', 'page table', and 'prompt-residual head' are software abstractions or fitted ML components covered under free_parameters and axioms.

free parameters (6)
  • Routing-bucket damage multipliers μ_β = not reported (selected by calibration search)
    Chosen in a small calibration search on C4 (Sec. 4) to protect hot experts; directly scales predicted damage in Eq. 8.
  • Prompt-residual strength α, confidence c, clipping bounds = not reported
    Eq. 6; hyperparameters decided in calibration (Sec. 4).
  • Prompt-residual linear regression weights w and bias a = trained on C4
    Eq. 5; predicts per-prompt sensitivity residual, trained offline on the C4 calibration set.
  • Bitwidth floors and depth caps per routing bucket = not reported
    Bitwidth floor stages in Algorithm 1 and depth caps; selected in the calibration search (Sec. 4).
  • Damage floor ε and free-block threshold T_blk = not reported
    ε floors Eq. 8; T_blk sets the pressure trigger in Eq. 7; no numeric values are given.
  • Offline global sensitivity table s_i^b = Hessian-derived scores, not listed
    Calibrated on C4 (Sec. 4) and used as the prior in Eq. 4.
axioms (5)
  • domain assumption Hessian-weighted sensitivity scores are a valid proxy for end-to-end quality loss from quantizing each linear-block.
    Used to build offline global damage in Eq. 4; no evidence that this proxy correlates with task accuracy beyond PPL.
  • domain assumption Expert routing mass measured on calibration/runtime traffic reflects the importance of each expert for current requests.
    Fig. 2 shows imbalance, but assumes stable ranking; bucket multipliers rely on this.
  • domain assumption Any-Precision LLM bit-plane/LUT representation supports switching bitwidths at runtime with no additional accuracy penalty beyond the chosen bitwidth.
    Borrowed from APL [24]; PagedWeight builds its page state on this representation.
  • domain assumption Asynchronous page offload/reload can be fully hidden behind inference compute.
    Supports the 'at most 4.1% throughput loss' claim; depends on transfer bandwidth vs compute ratio, not measured in isolation.
  • ad hoc to paper C4 is a suitable calibration set for planner hyperparameters; evaluation on C4 measures held-out quality.
    Section 4 says policy construction is on C4; Section 5.1/5.4 report C4 perplexity, creating an overlap between tuning and eval.

pith-pipeline@v1.3.0-alltime-deepseek · 12526 in / 13603 out tokens · 116189 ms · 2026-08-01T21:05:09.195940+00:00 · methodology

0 comments
read the original abstract

Mixture-of-Experts (MoE) is a popular class of large language models (LLMs), offering high efficiency and accuracy. However, in KV-cache-intensive serving scenarios, MoEs often exhibit a tension between the GPU memory requirements of the model weights and the growing KV cache. We propose PagedWeight, a novel management method for MoE LLM serving that dynamically quantizes MoE model's weights at runtime and balances expert-weight precision with the KV cache sizes. PagedWeight exposes and effectively navigates the complex tradeoff between the model's task accuracy, memory consumption, and throughput/latency. Across several memory-sensitive MoE serving scenarios, PagedWeight improves the quality-memory tradeoff over several existing quantization baselines. PagedWeight achieves FP16-equivalent accuracy with up to 72.0% GPU memory savings and 1.94$\times$ throughput improvement, and improves quality over quantization methods by up to 39.3% at a similar memory budget with at most 4.1% throughput loss.

Figures

Figures reproduced from arXiv: 2607.16184 by Anisha Dasgupta, Sasa Misailovic, Yifan Zhao, Yuchen Yang.

Figure 1
Figure 1. Figure 1: Static quantization fixes MoE weights; PagedWeight offloads weight pages to free KV [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Expert activation patterns on Qwen1.5-MoE-A2.7B. The y-axis shows the normalized [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: PagedWeight system overview. Dashed arrows show control flow, and solid arrows show weight data movement. APL stores weights in a shared bit-plane format, where different subsets of bit-planes correspond to different effective bitwidths. Each quantized tensor is also associated with a lookup table (LUT), which stores the bitwidth-specific centroid values used by the AP representation. 3 PagedWeight System … view at source ↗
Figure 4
Figure 4. Figure 4: Asynchronous page-movement pipeline. PagedWeight planner keeps preparing precision￾transition plans during runtime. At the offload threshold, the latest plan lowers the committed bitwidth and releases weight pages from GPU memory. At the reload threshold, the weight pages are restored before the higher bitwidth is committed. b → b ′ : ρb b→b ′ i = (w b→b ′ u ) ⊤ϕi + a b→b ′ u . (5) Here, ϕi is a three-dime… view at source ↗
Figure 5
Figure 5. Figure 5: Quality-memory tradeoff across three MoE models and four evaluation tasks. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

36 extracted references · 1 canonical work pages · 1 internal anchor

  1. [1]

    Dynamic expert quantization for scalable mixture-of-experts inference, 2025

    Kexin Chu, Dawei Xiang, Zixu Shen, Yiwei Yang, Zecheng Liu, and Wei Zhang. Dynamic expert quantization for scalable mixture-of-experts inference, 2025. URL https://arxiv. org/abs/2511.15015

    arXiv 2025

  2. [2]

    SpQR: A sparse-quantized representation for near-lossless LLM weight compression, 2023

    Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. SpQR: A sparse-quantized representation for near-lossless LLM weight compression, 2023. URL https://arxiv.org/ abs/2306.03078

    Pith/arXiv arXiv 2023

  3. [3]

    MxMoE: Mixed-precision quantization for MoE with accuracy and performance co-design

    Haojie Duanmu, Xiuhong Li, Zhihang Yuan, Size Zheng, Jiangfei Duan, Xingcheng Zhang, and Dahua Lin. MxMoE: Mixed-precision quantization for MoE with accuracy and performance co-design. InProceedings of the 42nd International Conference on Machine Learning, volume 267 ofProceedings of Machine Learning Research, pages 14793–14806. PMLR, 2025. URL https://pr...

    2025

  4. [4]

    TTKV: Temporal-tiered KV cache for long-context LLM inference, 2026

    Gradwell Dzikanyanga, Weihao Yang, Hao Huang, Donglei Wu, Shihao Wang, Wen Xia, and Sanjeeb K C. TTKV: Temporal-tiered KV cache for long-context LLM inference, 2026. URL https://arxiv.org/abs/2604.19769. 10

    Pith/arXiv arXiv 2026

  5. [5]

    Klotski: Efficient mixture-of-expert inference via expert-aware multi-batch pipeline, 2025

    Zhiyuan Fang, Yuegui Huang, Zicong Hong, Yufeng Lyu, Wuhui Chen, Yue Yu, Fan Yu, and Zibin Zheng. Klotski: Efficient mixture-of-expert inference via expert-aware multi-batch pipeline, 2025. URLhttps://arxiv.org/abs/2502.06888

    Pith/arXiv arXiv 2025

  6. [6]

    Switch transformers: scaling to trillion parameter models with simple and efficient sparsity.J

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: scaling to trillion parameter models with simple and efficient sparsity.J. Mach. Learn. Res., 23(1), January 2022. ISSN 1532-4435

    2022

  7. [7]

    GPTQ: Accurate post-training quantization for generative pre-trained transformers, 2023

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post-training quantization for generative pre-trained transformers, 2023. URL https://arxiv.org/abs/ 2210.17323. ICLR 2023

    Pith/arXiv arXiv 2023

  8. [8]

    EAQuant: Enhancing post-training quantization for MoE models via expert-aware optimization,

    Zhongqian Fu, Tianyi Zhao, Ning Ding, Xianzhi Yu, Xiaosong Li, Yehui Tang, and Yunhe Wang. EAQuant: Enhancing post-training quantization for MoE models via expert-aware optimization,

  9. [9]

    Gemma-4-26B-A4B

    Google DeepMind. Gemma-4-26B-A4B. https://huggingface.co/google/ gemma-4-26B-A4B, 2026. Hugging Face model card

    2026

  10. [10]

    Tsang, and Yew Soon Ong

    Xin He, Shunkang Zhang, Kaijie Tang, Shaohuai Shi, Yuxin Wang, Zihao Zeng, Zhenheng Tang, Xiaowen Chu, Haiyan Yin, Ivor W. Tsang, and Yew Soon Ong. ExpertFlow: Efficient mixture-of-experts inference via predictive expert caching and token scheduling, 2026. URL https://arxiv.org/abs/2410.17954. DAC 2026

    arXiv 2026

  11. [11]

    MiLo: Efficient quantized MoE inference with mixture of low-rank compensators, 2025

    Beichen Huang, Yueming Yuan, Zelei Shao, and Minjia Zhang. MiLo: Efficient quantized MoE inference with mixture of low-rank compensators, 2025. URL https://arxiv.org/abs/ 2504.02658

    Pith/arXiv arXiv 2025

  12. [12]

    DyMoE: Dynamic expert orchestration with mixed-precision quantization for efficient MoE inference on edge, 2026

    Yuegui Huang, Zhiyuan Fang, Weiqi Luo, Ruoyu Wu, Wuhui Chen, and Zibin Zheng. DyMoE: Dynamic expert orchestration with mixed-precision quantization for efficient MoE inference on edge, 2026. URLhttps://arxiv.org/abs/2603.19172

    arXiv 2026

  13. [13]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven...

    Pith/arXiv arXiv 2024

  14. [14]

    KVComp: A high-performance, LLM-aware, lossy compression framework for KV cache, 2025

    Bo Jiang, Taolue Yang, Youyuan Liu, Chengming Zhang, Xubin He, and Sian Jin. KVComp: A high-performance, LLM-aware, lossy compression framework for KV cache, 2025. URL https://arxiv.org/abs/2509.00579

    Pith/arXiv arXiv 2025

  15. [15]

    Mixture of quantized experts (MoQE): Complementary effect of low-bit quantization and robustness, 2023

    Young Jin Kim, Raffy Fahim, and Hany Hassan Awadalla. Mixture of quantized experts (MoQE): Complementary effect of low-bit quantization and robustness, 2023. URL https: //arxiv.org/abs/2310.02410

    Pith/arXiv arXiv 2023

  16. [16]

    Lee, and Yeonhong Park

    Sangwoo Kwon, Seong Hoon Seo, Jae W. Lee, and Yeonhong Park. DP-LLM: Runtime model adaptation with dynamic layer-wise precision assignment, 2025. URL https://arxiv.org/ abs/2508.06041. NeurIPS 2025

    arXiv 2025

  17. [17]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. InProceedings of the 29th ACM Symposium on Operating Systems Principles, SOSP ’23, pages 611–626, 2023. doi: 10.1145/3600006.3613165. URLhttps://doi...

    arXiv 2023

  18. [18]

    GShard: Scaling giant models with condi- tional computation and automatic sharding

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling giant models with condi- tional computation and automatic sharding. InInternational Conference on Learning Represen- tations, 2021. URLhttps://openreview.net/forum?id=qrwe7XHTmYb. 11

    2021

  19. [19]

    QuantMoE-Bench: Ex- amining post-training quantization for mixture-of-experts, 2024

    Pingzhi Li, Xiaolong Jin, Zhen Tan, Yu Cheng, and Tianlong Chen. QuantMoE-Bench: Ex- amining post-training quantization for mixture-of-experts, 2024. URL https://arxiv.org/ abs/2406.08155

    Pith/arXiv arXiv 2024

  20. [20]

    KVTuner: Sensitivity-aware layer-wise mixed- precision KV cache quantization for efficient and nearly lossless LLM inference, 2025

    Xing Li, Zeyu Xing, Yiming Li, Linping Qu, Hui-Ling Zhen, Wulong Liu, Yiwu Yao, Sinno Jialin Pan, and Mingxuan Yuan. KVTuner: Sensitivity-aware layer-wise mixed- precision KV cache quantization for efficient and nearly lossless LLM inference, 2025. URL https://arxiv.org/abs/2502.04420. ICML 2025

    arXiv 2025

  21. [21]

    AWQ: Activation-aware weight quantization for on-device LLM compression and acceleration

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. AWQ: Activation-aware weight quantization for on-device LLM compression and acceleration. InProceedings of Machine Learning and Systems, volume 6, 2024. URL https://proceedings.mlsys.org/paper_files/paper/ 2024/hash/42a452cb...

    2024

  22. [22]

    HybridGen: Efficient LLM generative inference via CPU-GPU hybrid computing, 2026

    Mao Lin, Xi Wang, Guilherme Cox, Dong Li, and Hyeran Jeon. HybridGen: Efficient LLM generative inference via CPU-GPU hybrid computing, 2026. URL https://arxiv.org/abs/ 2604.18529

    Pith/arXiv arXiv 2026

  23. [23]

    He, Hanser Jiang, Zion Wang, Alan Zhao, and Patrick P

    Qingxiu Liu, Cyril Y . He, Hanser Jiang, Zion Wang, Alan Zhao, and Patrick P. C. Lee. FluxMoE: Decoupling expert residency for high-performance MoE serving, 2026. URL https://arxiv. org/abs/2604.02715

    Pith/arXiv arXiv 2026

  24. [24]

    Yeonhong Park, Jake Hyun, Sanglyul Cho, Bonggeun Sim, and Jae W. Lee. Any-precision LLM: Low-cost deployment of multiple, different-sized LLMs. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors,Proceedings of the 41st International Conference on Machine Learning, volume 23...

    2024

  25. [25]

    Qwen1.5-MoE-A2.7B

    Qwen Team. Qwen1.5-MoE-A2.7B. https://huggingface.co/Qwen/Qwen1.5-MoE-A2. 7B, 2024. Hugging Face model card

    2024

  26. [26]

    Fu, Zhiqiang Xie, Beidi Chen, Clark Barrett, Joseph E

    Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Daniel Y . Fu, Zhiqiang Xie, Beidi Chen, Clark Barrett, Joseph E. Gonzalez, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. FlexGen: High-throughput generative inference of large language models with a single GPU, 2023. URLhttps://arxiv.org/abs/2303.06865

    Pith/arXiv arXiv 2023

  27. [27]

    MoQAE: Mixed-precision quantization for long-context LLM inference via mixture of quantization-aware experts

    Wei Tao, Haocheng Lu, Xiaoyang Qu, Bin Zhang, Kai Lu, Jiguang Wan, and Jianzong Wang. MoQAE: Mixed-precision quantization for long-context LLM inference via mixture of quantization-aware experts. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors,Proceedings of the 63rd Annual Meeting of the Association for Computation...

    2025

  28. [28]

    Aqa: Network-accelerated memory offloading for LLMs in scale-up GPU domains

    Abhishek Vijaya Kumar, Gianni Antichi, and Rachee Singh. Aqa: Network-accelerated memory offloading for LLMs in scale-up GPU domains. InProceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ASPLOS ’25, New York, NY , USA, 2025. Association for Computing Machinery. doi: 10.1...

    arXiv 2025

  29. [29]

    MoE-SpeQ: Speculative quantized decoding with proactive expert prefetching and offloading for mixture-of-experts, 2025

    Wenfeng Wang, Jiacheng Liu, Xiaofeng Hou, Xinfeng Xia, Peng Tang, Mingxuan Zhang, Chao Li, and Minyi Guo. MoE-SpeQ: Speculative quantized decoding with proactive expert prefetching and offloading for mixture-of-experts, 2025. URL https://arxiv.org/abs/ 2511.14102

    arXiv 2025

  30. [30]

    SmoothQuant: Accurate and efficient post-training quantization for large language mod- els

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. SmoothQuant: Accurate and efficient post-training quantization for large language mod- els. InProceedings of the 40th International Conference on Machine Learning, volume 202 ofProceedings of Machine Learning Research, pages 38087–38099. PMLR, 2023. URL https://proceedings.mlr.p...

    2023

  31. [31]

    LayerKV: Optimizing large language model serving with layer-wise KV cache management, 2024

    Yi Xiong, Hao Wu, Changxu Shao, Ziqing Wang, Rui Zhang, Yuhong Guo, Junping Zhao, Ke Zhang, and Zhenxuan Pan. LayerKV: Optimizing large language model serving with layer-wise KV cache management, 2024. URLhttps://arxiv.org/abs/2410.00428

    Pith/arXiv arXiv 2024

  32. [32]

    ARQ: A Mixed-Precision Quantization Framework for Accurate and Certifiably Robust DNNs

    Yuchen Yang, Yifan Zhao, Shubham Ugare, Gagandeep Singh, and Sasa Misailovic. ARQ: A mixed-precision quantization framework for accurate and certifiably robust DNNs. In Proceedings of the 35th ACM SIGSOFT International Symposium on Software Testing and Analysis, 2026. doi: 10.48550/arXiv.2410.24214. URL https://arxiv.org/abs/2410. 24214

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2410.24214 2026

  33. [33]

    MoE-Lens: Towards the hardware limit of high- throughput MoE LLM serving under resource constraints, 2025

    Yichao Yuan, Lin Ma, and Nishil Talati. MoE-Lens: Towards the hardware limit of high- throughput MoE LLM serving under resource constraints, 2025. URL https://arxiv.org/ abs/2504.09345

    Pith/arXiv arXiv 2025

  34. [34]

    Yanqi Zhang, Yuwei Hu, Runyuan Zhao, John C. S. Lui, and Haibo Chen. DiffKV: Differentiated memory management for large language models with parallel KV compaction, 2025. URL https://arxiv.org/abs/2412.03131. SOSP 2025

    Pith/arXiv arXiv 2025

  35. [35]

    Ruidong Zhu, Ziheng Jiang, Chao Jin, Peng Wu, Cesar A. Stuardo, Dongyang Wang, Xinlei Zhang, Huaping Zhou, Haoran Wei, Yang Cheng, Jianzhe Xiao, Xinyi Zhang, Lingjun Liu, Haibin Lin, Li-Wen Chang, Jianxi Ye, Xiao Yu, Xuanzhe Liu, Xin Jin, and Xin Liu. MegaScale- Infer: Serving mixture-of-experts at scale with disaggregated expert parallelism, 2025. URL ht...

    Pith/arXiv arXiv 2025

  36. [2025]

    URLhttps://arxiv.org/abs/2506.13329

    arXiv