Pith. sign in

REVIEW 3 major objections 5 minor 41 references

Who Should Own the Expert Cache? Kernel-Managed Tiering for Trillion-Parameter MoE Inference

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The page cache, with the kernel's default LRU eviction, is the right owner of the expert tier for trillion-parameter MoE inference.

desk verdict Careful, honest expert-cache study whose central 1.09x mechanism claim leans on an unverified zero-copy read path from an anonymous companion paper; worth refereeing, but the premise needs a public artifact or protocol. read the letter →

arxiv 2608.12103 v1 pith:LZ3SSDTR submitted 2026-08-12 cs.OS

Yuan Si , Yufeng Lin , Daming Li , Jialu Zhang This is my paper · ORCID
classification cs.OS
keywords Mixture-of-ExpertsinferenceexpertcachingpagecacheLRUevictionmemorytieringGPUoffloadrouterlookaheadreclaimamplification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that when a mixture-of-experts model's expert pool dwarfs DRAM, the operating system's page cache is the right place to cache experts, and the kernel's default recency eviction is the right policy. The evidence comes from replaying router traces from models with 128 to 896 experts per layer against a 1.45 TB pool on a coherent-link node. At an enforced equal memory budget, untuned kernel LRU serves 75.3% of demand versus 74.6% for a same-domain oracle frequency table, and it keeps 70-71% hit rate when the oracle's tuned table collapses off-domain. End to end, enabling kernel-managed caching speeds up decode by 1.09-1.10x with token-identical outputs. If right, serving systems can stop building expert-granular pinned caches and instead spend model knowledge on admission and advice.

What carries the argument

The central object is the page cache acting as the expert tier, with the full expert pool mmapped and read through normal file-backed I/O; the kernel's default LRU reclaim carries eviction at page granularity. The identity that carries the argument is the constant-free demand model: simulated LRU misses times expert size compared against measured block-layer bytes, which yields a per-capacity amplification diagnostic. A second mechanical result is the ownership calculus: on a coherent link, HBM promotion breaks even at k* = 1.06 reuses, derived from measured bandwidth constants of 346 GB/s overlapped copy, 327 GB/s file-backed read, and 2938 GB/s HBM. Model knowledge enters only as admission control and as router-lookahead advice delivered through fadvise hints, never as a reimplementation of eviction.

What would settle it

Re-measure the equal-memory 256 GB comparison with the zero-copy read path removed: if the page cache's per-iteration time rises from 1.13 s toward or beyond the pinned arena's 1.04 s, the kernel tier's advantage is an artifact of the read path rather than of eviction policy.

Watch

Extended reading notes

Core claim

The paper's central claim is that for MoE models whose expert pools exceed DRAM, the kernel page cache is the right owner of the expert tier: at an enforced equal-memory budget, untuned kernel LRU serves 75.3% of demand versus 74.6% for a same-domain oracle frequency table at 256 GB, and the oracle's only advantage is a 1.09x mechanism edge from kernel lookup and reclaim work, not from choosing better bytes. Off-domain, the oracle's hit rate collapses to 21-34% while LRU holds 70-71%. The authors state the design principle directly: let the kernel own eviction, and spend model-specific knowledge on admission and advice. The end-to-end price of admitting the cache is 1.09-1.10x decode speedup with token-identical outputs, achieved simply by letting expert reads enter the page cache instead of setting the unbuffered flag.

Load-bearing premise

The whole comparison presupposes that file-backed expert pages resident in the page cache can be read by the GPU at the same bandwidth as framework-owned pinned allocations; the paper cites a companion submission for this contract, and if that contract fails on real systems, the 1.09x edge over the pinned arena is not a fair mechanism comparison.

Editorial extensions

If this is right

  • Serving systems for models with giant expert pools can simply mmap the pool and let kernel recency own eviction; this costs at most 1.09x against an oracle no deployment has, and it survives domain drift.
  • DRAM capacity becomes a practical sizing knob: iteration time and block-level device traffic are smooth, reproducible functions of cache size, so a floor on latency translates directly into a DRAM requirement.
  • Router lookahead is worth using only as kernel readahead advice, not as synchronous prefetch; at 64.7% recall it buys 0.3%, and the Belady bound leaves about a quarter of misses as the remaining addressable headroom.
  • Cache admission never changes correctness: all nine balanced end-to-end pairs produce token-identical outputs, so the 1.09-1.10x speedup comes without any quality trade.
  • Balloon-based capacity studies should cross-check at least one point against a cgroup limit or physical memory removal; with MGLRU and a mostly-mlocked host they can overstate pressured-end device traffic by about 2x.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension of the advice result is recall-gated advice: because per-layer recall ranges from 1% to 84%, withholding fadvise hints below a threshold could push the measured 0.3% gain toward the Belady-addressable quarter of misses.
  • The kernel tier's system-wide sharing suggests a multi-tenant benefit the paper does not quantify: several model replicas or different models sharing one node could all hit the same warm expert pages, effectively pooling their working sets across processes.
  • On machines without a coherent file-backed read path, the 1.09x residual may flip direction; the ownership verdict should be re-tested per platform before treating 'kernel owns eviction' as a universal rule.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper argues that for trillion-parameter mixture-of-experts inference, the kernel page cache, rather than a user-space frequency-pinned arena, should own the expert tier. Using native replay of router traces from three MoE models against a full 1.45 TB expert pool on GH200 nodes, the authors report four main results: cache capacity is a smooth, reproducible sizing knob; the deep-pressure knee is a reclaim artifact requiring both MGLRU and balloon-style mostly-mlocked memory; under an enforced equal-memory wall, untuned kernel LRU matches a same-domain oracle frequency table on-domain (75.3% vs. 74.6% hit rate at 256 GB) while remaining robust off-domain; and router lookahead advice buys little. End to end, kernel-managed caching gives 1.09-1.10x decode speedup with token-identical outputs. The paper concludes that the kernel should own eviction and model-specific knowledge should be spent on admission and advice.

Significance. If the main results hold, the paper is a significant corrective to the current design consensus for MoE serving caches. It combines several strong methodological features: three independent capacity-enforcement mechanisms, per-run accounting proofs for accepted measurement cells, an oracle deliberately biased in favor of the incumbent, token-identical output verification, and a disclosed two-round audit history. The parameter-free derivations in Section 8 and the falsifiable tie-band prediction are also valuable. The main significance risk is that the central mechanism comparison depends on an unverified zero-copy file-backed read path delegated to an anonymous companion submission, so the core claim is currently conditional on an external result.

major comments (3)
  1. [Section 2 ('The read path') and Section 6 (equal-memory comparison)] The central comparison between the page-cache tier and the oracle-pinned arena presupposes that file-backed pages resident in the page cache are readable by the GPU over the coherent link at hostAlloc-class bandwidth with zero copies. This premise is delegated entirely to anonymous companion submission [4], which is concurrently under review and ships no artifact. Section 8 asserts local re-measurement of the relevant bandwidth constants, but gives no measurement protocol, and Section 9's end-to-end A/B runs through the engine's own pread reader, which does not isolate the GPU zero-copy path. If file-backed reads carry a hidden copy or bandwidth penalty, then Figure 4's 1.09x mechanism edge is not a fair mechanism comparison and the conclusion that the kernel tier is 'never the wrong owner' is not established. The paper must either include a full, self-contained description of the zero-copy read-path verification, provide a public artifact for it, or explicitly scope all mechanism conclusions to the unverified path established by [4].
  2. [Section 8 (Rules 2 and 3)] The ownership calculus rests on measured bandwidth constants (327 GB/s file-backed, 330 GB/s hostAlloc, 346 GB/s overlapped streaming, 23 GB/s unregistered malloc) that are asserted without any experimental setup. The breakeven k*=1.06 and the 5.6% adopt-vs-stream tie-band are load-bearing for the placement policy, yet a reader cannot reproduce or even assess these numbers without knowing how the file-backed reads were issued, how page residency was ensured, whether GPU reads were synchronous or overlapped, and how the coherent-link ceiling was measured. This is a reproducibility gap in a central derivation and should be closed by adding the measurement methodology or a reference to a publicly available artifact.
  3. [Section 6 and Conclusion] The conclusion that the kernel tier is 'never the wrong configuration' is stated without adequately bounding the regime. Section 8 itself notes that on PCIe machines without a coherent read path the adopt row disappears and user-space HBM caches 'genuinely earn their complexity,' which means the ownership claim is specific to coherent-link hosts. The off-domain hit-rate and reclaim findings may be mechanism-independent, but the measured 1.09x edge is not. The paper should explicitly state that the 'never wrong' conclusion applies only when the zero-copy file-backed read path of [4] is available, so that the claim's scope matches the evidence.
minor comments (5)
  1. [Table 1] The row 'LRU vs. static (B>=32)' appears as 'tie LRU wins LRU wins' across three model columns; this should be formatted as three separate entries or separated by clear delimiters.
  2. [Section 5 and Section 10] Section 5 refers to a 'round-6 campaign,' but the methods ledger in Section 10 describes only round-1 and round-2 audits; the round numbering should be introduced or reconciled.
  3. [Section 2 (canonical trace parser)] The parser is said to reconstruct 'exactly 299 accepted tokens per trace against the engine's requested 300'; the missing token is never explained and should be clarified.
  4. [General] The paper contains no data or code availability statement. Given the audit-heavy methodology and the external dependency on [4], a public release of the replay harness, traces, and measurement scripts would materially aid replication.
  5. [Section 8 (Rule 1)] The heading 'Admission to DRAM is free' is stronger than the rule's own content, which acknowledges that admission can evict a hotter working set; consider a more precise heading such as 'Admission is free on the miss path.'

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the oracle frequency table is a disclosed upper bound, the k* breakeven is derived from measured bandwidth constants, and the companion [4] dependency is an external premise with local re-measurement.

full rationale

The claimed derivations are not equivalent to their inputs. The oracle frequency table in Sections 6 and 10 is explicitly computed from the replayed trace and labeled an upper bound no deployed system reaches; it functions as a benchmark, not as a disguised prediction. The Section 8 ownership calculus derives k*=1.06 and the 5.6% tie-band directly from measured bandwidth constants (1/346, 1/327, and 1/2938) through the stated closed-form formula; no parameter is fitted to the target outcome. The capacity curves, hit rates, and amplification diagnostics are measured at the block layer or computed from simulated LRU over the replayed trace, and the paper repeatedly discloses audit corrections rather than retrofitting numbers. The one load-bearing external input is companion [4]'s zero-copy read contract, but Sections 2 and 8 both state that the bandwidth equivalence is re-measured on the paper's own node, so the central equal-memory comparison does not reduce to [4]'s conclusion. An unverified companion artifact would be a correctness or artifact risk, not a circular reduction; no equation in the paper is shown to be its own input by construction.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central measurements rest on domain assumptions about trace representativeness and the zero-copy execution contract. The paper introduces no free parameters fitted to its headline claims: bandwidth constants in Section 8 are directly measured machine properties, and the oracle frequency table is derived from the replayed trace but explicitly labeled an upper bound. The fair-window split is the one hand-chosen evaluation threshold affecting hit-rate comparisons. No new entities are postulated.

free parameters (1)
  • Fair-window warmup fraction (60% warm, 40% test) = 0.6/0.4
    Hit-rate comparisons in Sections 3 and 6 score only the last 40% of iterations after 60% warmup; the split is hand-chosen and sensitivity is not reported, so the LRU-versus-oracle gap could shift with the window.
assumptions (3)
  • domain assumption Zero-copy execution contract from companion submission [4]: file-backed pages reach accelerator-class bandwidth with zero copies.
    Invoked in Section 2 'The read path' and Section 8; the page-cache-versus-pinned conclusion assumes residency policy, not copy mechanics, decides performance. The companion paper is anonymous and not publicly verifiable.
  • domain assumption Recorded traces (242-290 engine iterations per domain, 299 accepted tokens) are representative of production routing behavior.
    Section 10 states traces are short and four prompt families stand in for domains; the 4,000-step asymptote comes from the 128-expert model, not the trillion-parameter model.
  • domain assumption Greedy decoding makes the replayed I/O stream exactly the engine-issued stream.
    Section 10 'Correctness invariance'; if sampling or nondeterminism were present, replay would not equal the engine's I/O.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Who Should Own the Expert Cache? Kernel-Managed Tiering for Trillion-Parameter MoE Inference." pith.science (2026). https://pith.science/paper/LZ3SSDTR

@misc{pith2026260812103,
  author       = {Pith},
  title        = {Pith review of: Who Should Own the Expert Cache? Kernel-Managed Tiering for Trillion-Parameter MoE Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LZ3SSDTR}},
  note         = {Machine review of arXiv:2608.12103}
}
abstract

Mixture-of-experts models whose expert pools dwarf DRAM force every serving system to contain a cache, yet existing systems typically implement this cache in user space using expert-granular, frequency-ranked, explicitly pinned tiers. We characterize the alternative already provided by the OS: the page cache as the expert tier. We use router traces from three MoE models spanning 128 to 896 experts per layer, including a production trillion-parameter model with a 1.45 TB expert pool, and replay them natively against the full pool on GH200 nodes with capacity enforced by three independent mechanisms. We find four main results. First, iteration time and device traffic are smooth, reproducible functions of cache capacity, making DRAM a practical sizing knob for trillion-parameter serving. Second, the deep-pressure knee is a reclaim artifact requiring both MGLRU and balloon-style mostly-mlocked memory; cgroup limits and physical-memory configurations show no such amplification, indicating that balloon-based studies can overstate pressured-end device traffic by approximately 2$\times$. Third, under an enforced equal-memory wall, untuned kernel LRU serves essentially as much demand as a same-domain oracle frequency table (75.3% versus 74.6% at 256 GB), while the oracle's mechanism advantage is only 1.09$\times$ and disappears off-domain, where LRU remains at 70--71% hit rate. Fourth, router lookahead with 64.7% measured recall provides only a 0.3% benefit as kernel readahead advice and no benefit as synchronous prefetch. End to end, enabling kernel-managed caching improves decode performance by 1.09--1.10$\times$ with token-identical outputs across nine balanced pairs. The resulting design principle is simple: in this regime, let the kernel own eviction, while model-specific knowledge is best spent on admission and advice.

Figures

Figures reproduced from arXiv: 2608.12103 by the authors.

Figure 1
Figure 1. Fair-window hit rates at the two ends of the expert￾count range (4-domain mean±range); 𝐸=256 sits between them ( [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Capacity response: iteration time (left, three ran￾domized runs shown individually) and block-layer device traffic (right, against an inverse-capacity reference—a rail, not a fit). 𝐶 (GB) med (s) spread GB/it 𝑘/𝐶 dev miss (GB) amp 448 0.99 0.5% 5.3 +0.0% 5.3 0.99× 256 1.37 2.1% 9.8 +5.0% 8.1 1.22× 128 1.95 4.0% 17.4 -6.7% 11.6 1.50× 64 2.76 1.4% 33.6 -9.7% 15.1 2.22× [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. The reclaim finding. Left: the original MGLRU A/B at 𝐶=64GB under balloon pressure on machine A (rail = nominal miss demand). Middle: the same capacity on machine B under three mechanisms—only the high-locked-fraction balloon amplifies. Right: reclaim-neutral cells agree across mechanisms and machines. 256 128 C (GB) 0.0 0.5 1.0 1.5 2.0 s per iteration page cache + FADV_RANDOM 256 128 C (GB) 0 5 10 15 device GB / it… view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: The oracle table off-domain: prose-trained pinning versus table-free LRU at equal budget, fair windows. Policy: the kernel already keeps the right bytes. The same-denominator comparison the old draft never ran: what fraction of total demand does each mode serve without…
Figure 6
Figure 6. Figure 6: One bounded prefetch worker: no prefetch, oracle next-pass hints, and the real PILOT prediction plan, each delivered as advice or as blocking reads. Axis starts at zero. maintain per-workload tables and accept the off-domain cliff; §9 argues the better spend is admissi…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 31 canonical work pages

  1. [4]

    Anonymous. 2026. The Ingestion Tax: Adopting File-Backed Weights in Tensor Frameworks. Companion paper, concurrently under sub- mission

    work page 2026

  2. [1]

    Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar

    Keivan Alizadeh, Iman Mirzadeh, Dmitry Belenko, Karen Khatamifard, Minsik Cho, Carlo C. Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar. 2024. LLM in a Flash: Efficient Large Language Model Inference with Limited Memory. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL)

    work page 2024

  3. [2]

    Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, and Yuxiong He. 2022. DeepSpeed-Inference: En- abling Efficient Inference of Transformer Models at Unprecedented Scale. InProceedings of the International Conference for High Perfor- mance Computing, Network...

    work page 2022

  4. [3]

    Spyros Angelopoulos, Loris Marchal, Adrien Obrecht, and Bertrand Simon. 2025. Cache Management for Mixture-of-Experts LLMs. arXiv:2509.02408(2025)

    work page Pith review arXiv 2025

  5. [5]

    Antonios Antoniadis, Christian Coester, Marek Eliáš, Adam Polak, and Bertrand Simon. 2020. Online Metric Algorithms with Untrusted Predictions. InProceedings of the 37th International Conference on Machine Learning (ICML)

    work page 2020

  6. [6]

    Laszlo A. Belady. 1966. A Study of Replacement Algorithms for a Virtual-Storage Computer.IBM Systems Journal5, 2 (1966)

    work page 1966

  7. [7]

    Andrew Crotty, Viktor Leis, and Andrew Pavlo. 2022. Are You Sure You Want to Use MMAP in Your Database Management System?. In Proceedings of the 12th Conference on Innovative Data Systems Research (CIDR)

    work page 2022

  8. [8]

    DeepSeek-AI. 2024. DeepSeek-V3 Technical Report.arXiv:2412.19437 (2024)

    arXiv 2024

Show all 41 references
  1. [9]

    Peter J. Denning. 1968. The Working Set Model for Program Behavior. Commun. ACM11, 5 (1968)

    1968

  2. [10]

    Dai, Simon Tong, Dmitry Lep- ikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al

    Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lep- ikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. 2022. GLaM: Efficient Scaling of Language Mod- els with Mixture-of-Experts. InInternational Conference on Machine Learning (ICML)

    2022

  3. [11]

    Zhixu Du, Shiyu Li, Yuhao Wu, Xiangyu Jiang, Jingwei Sun, Qilin Zheng, Yongkai Wu, Ang Li, Hai (Helen) Li, and Yiran Chen. 2024. SiDA: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models. InProceedings of Machine Learning and Systems (MLSys)

    2024

  4. [12]

    Artyom Eliseev and Denis Mazur. 2023. Fast Inference of Mixture-of- Experts Language Models with Offloading.arXiv:2312.17238(2023)

    2023 arXiv

  5. [13]

    William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch Trans- formers: Scaling to Trillion Parameter Models with Simple and Effi- cient Sparsity.Journal of Machine Learning Research23, 120 (2022)

    2022

  6. [14]

    Georgi Gerganov and contributors. 2023. llama.cpp: LLM Inference in C/C++.https://github.com/ggml-org/llama.cpp

    2023

  7. [15]

    Jim Gray and Franco Putzolu. 1987. The 5 Minute Rule for Trading Memory for Disc Accesses and the 10 Byte Rule for Trading Memory for CPU Time.ACM SIGMOD Record16, 3 (1987)

    1987

  8. [16]

    Ranggi Hwang, Jianyu Wei, Shijie Cao, Changho Hwang, Xiaohu Tang, Ting Cao, and Mao Yang. 2024. Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference. InPro- ceedings of the 51st International Symposium on Computer Architecture (ISCA)

    2024

  9. [17]

    Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, et al

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, et al. 2024. Mixtral of Experts.arXiv:2401.04088(2024)

    2024 arXiv

  10. [18]

    Keisuke Kamahori, Yile Gu, Kan Zhu, and Baris Kasikci. 2024. Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models.arXiv:2402.07033(2024)

    2024 arXiv

  11. [19]

    Kimi Team. 2025. Kimi K2: Open Agentic Intelligence.arXiv:2507.20534 (2025)

    2025 arXiv

  12. [20]

    Jinwei Kong, Runqi Meng, Fanyi Wang, Wentao Qiu, Haotian Hu, Yongjian Zhou, and Zhenhua Ge. 2026. SpecPrefetch: Parameter- Efficient Expert Prefetching for Sparse MoE Foundation Models. arXiv:2607.24787(2026)

    2026 arXiv

  13. [21]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica

  14. [22]

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2021. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. InInternational Conference on Learning Representations (ICLR)

    2021

  15. [23]

    Thodoris Lykouris and Sergei Vassilvitskii. 2018. Competitive Caching with Machine Learned Advice. InProceedings of the 35th International Conference on Machine Learning (ICML)

    2018

  16. [24]

    Hasan Al Maruf, Hao Wang, Abhishek Dhanotia, Johannes Weiner, Niket Agarwal, Pallab Bhattacharya, Chris Petersen, Mosharaf Chowd- hury, Shobhit Kanaujia, and Prakash Chauhan. 2023. TPP: Transparent Page Placement for CXL-Enabled Tiered-Memory. InProceedings of the 28th Interna...

    2023

  17. [25]

    Nimrod Megiddo and Dharmendra S. Modha. 2003. ARC: A Self- Tuning, Low Overhead Replacement Cache. InProceedings of the 2nd USENIX Conference on File and Storage Technologies (FAST)

    2003

  18. [26]

    Moonshot AI. 2026. Kimi-K3 Open-Weights Release. Hugging Face model repository

    2026

  19. [27]

    NVIDIA Corporation. 2023. NVIDIA GH200 Grace Hopper Superchip Architecture Whitepaper

    2023

  20. [28]

    Qwen Team. 2025. Qwen3 Technical Report.arXiv:2505.09388(2025)

    2025 arXiv

  21. [29]

    Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yux- iong He. 2022. DeepSpeed-MoE: Advancing Mixture-of-Experts Infer- ence and Training to Power Next-Generation AI Scale. InInternational Conference on Machi...

    2022

  22. [30]

    Dhruv Rohatgi. 2020. Near-Optimal Bounds for Online Caching with Machine Learned Advice. InProceedings of the 31st ACM-SIAM Sym- posium on Discrete Algorithms (SODA)

    2020

  23. [31]

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. In International Conference on Learning Representations (ICLR)

    2017

  24. [32]

    Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. 2023. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU. InInternational Conference on Machine Learning (ICML)

    2023

  25. [33]

    Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen. 2024. PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU. In Proceedings of the 30th Symposium on Operating Systems Principles (SOSP)

    2024

  26. [34]

    Waldspurger

    Carl A. Waldspurger. 2002. Memory Resource Management in VMware ESX Server. InProceedings of the 5th USENIX Symposium on Operating Systems Design and Implementation (OSDI)

    2002

  27. [35]

    Johannes Weiner, Niket Agarwal, Dan Schatzberg, Leon Yang, Hao Wang, Blaise Sanouillet, Bikash Sharma, Tejun Heo, Mayank Jain, Chunqiang Tang, and Dimitrios Skarlatos. 2022. TMO: Transparent Memory Offloading in Datacenters. InProceedings of the 27th Interna- tional Conference...

    2022

  28. [36]

    Leyang Xue, Yao Fu, Zhan Lu, Luo Mai, and Mahesh Marina. 2024. MoE- Infinity: Offloading-Efficient MoE Model Serving. InarXiv:2401.14361

    2024 arXiv

  29. [37]

    Rongjie Yi, Liwei Guo, Shiyun Wei, Ao Zhou, Shangguang Wang, and Mengwei Xu. 2023. EdgeMoE: Fast On-Device Inference of MoE-Based Large Language Models.arXiv:2308.14352(2023)

    2023 arXiv

  30. [38]

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A Distributed Serving System for Transformer-Based Generative Models. InProceedings of the 16th USENIX Symposium on Operating Systems Design and Implementa- tion (OSDI)

    2022

  31. [39]

    Hanfei Yu, Xingqi Cui, Hong Zhang, Hao Wang, and Hao Wang. 2026. Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading. InProceedings of the 21st European Conference on Computer Systems (EuroSys)

    2026

  32. [40]

    Yu Zhao. 2023. Multi-Gen LRU: the Next Generation of the Linux Page Reclaim Mechanism. Linux kernel documentation, admin-guide/mm/multigen_lru. 13

    2023

  33. [2023]

    InProceedings of the 29th Symposium on Operating Systems Principles (SOSP)

    Efficient Memory Management for Large Language Model Serving with PagedAttention. InProceedings of the 29th Symposium on Operating Systems Principles (SOSP)

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.