Pith. sign in

REVIEW 4 major objections 5 minor 81 references

HyMCache claims that a CXL-hybrid memory tier—SSD capacity plus a small DRAM staging window—can serve reused KV-cache blocks at near-DRAM latency when the serving stack prefetches them in the order multi-turn prompts will consume them.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 15:51 UTC pith:5FXFPOKT

load-bearing objection Real prototype, plausible staging-prefetch idea, but the DRAM accounting and the missing sustained-refill-bandwidth measurement keep me from taking the headline ratios at face value. the 4 major comments →

arxiv 2607.18141 v2 pith:5FXFPOKT submitted 2026-07-20 cs.DC

HyMCache: A KV Cache Framework for Multi-Turn LLM Serving with CXL-Hybrid Memory

Hakbeom Jang , Inho Song , Sam H. Noh , Jongryool Kim This is my paper
classification cs.DC
keywords KV cacheCXL-hybrid memoryLLM servingmulti-turn inferenceprefetchingtiered memorySSD-backed memoryPD-disaggregated serving
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that multi-turn LLM serving has a memory bottleneck: reusing long conversation context makes the KV cache a read-dominated, predictable, append-only workload. It claims that a CXL-hybrid memory device, normally managed as a transparent LRU cache, fails on this workload because prefix reads are sequential, one-hit-wonder scans. HyMCache instead manages the device's small DRAM as an explicitly controlled staging window: the serving stack reveals the ordered list of KV blocks a request will read, the device stages them from SSD ahead of time, and the worker fetches them from DRAM via RDMA. On a real prototype, this delivers 3.0x the serving performance of local DRAM caching under the same DRAM budget, and about 70% of a 1 TB distributed-DRAM cache while using 16x less DRAM. The reason to care is that TB-scale shared context becomes affordable without dropping to storage latency.

Core claim

HyMCache's central claim is that the predictable, read-heavy, append-only access pattern of multi-turn KV caches lets a memory device replace its internal cache policy with a software-controlled prefetch pipeline. The device exposes an issue–wait–release API: the serving runtime passes the exact order in which KV objects will be consumed; the device stages each object from SSD into its small internal DRAM before the worker's RDMA read arrives; after consumption, the staging slot is released. Because reads are served from DRAM and writes are buffered asynchronously in large batches, the SSD-backed tier behaves like remote DRAM for reused prefixes. The paper shows this on a prototype: peak rea

What carries the argument

The load-bearing object is the 'KV-object prefetch window' with a bounded issue–consume–release protocol (the prefetch-object/wait/release API). It converts SSD-to-DRAM refill from a cache-miss event into a pipelined background activity, so internal DRAM acts as a latency-hiding staging buffer sized to the window, not to the total KV footprint. Read-prioritized write isolation is the second mechanism: new KV writes are admitted only when a small write-side buffer has space, and are flushed to SSD in large batches when read pressure is low, keeping the SSD path clean for prefetches.

Load-bearing premise

The load-bearing premise is that the serving runtime knows, before execution, the exact ordered list of KV objects a request will consume; if the predicted order is wrong, the staging window misses and latency falls to SSD levels.

What would settle it

Run the same system on an agentic trace in which the next tool call or turn selection depends on prior model output, so future reads are not known in advance. If the device-DRAM prefetch hit rate drops below roughly 50% and per-turn TTFT converges toward the no-prefetch or recomputation baseline, the central claim fails. A simpler microbenchmark randomizing read order within a turn would produce the same test.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Multi-turn and agentic workloads with high prefix reuse can keep most redundant prefill work off the critical path even after the prefix working set exceeds local DRAM capacity.
  • The CXL-HM tier preserves contiguous prefix chains across turns, so hit rates stay near 90% while local caches' LRU eviction breaks the chain and forces recomputation.
  • Because the full SSD-backed space is RDMA-addressable, a worker can fetch a KV block in one round trip without a second metadata lookup or software staging on the storage node.
  • Scaling the tier means adding SSD-backed devices or larger SSDs rather than DRAM, so the marginal cost per gigabyte of shared context stays near flash cost.
  • The same DRAM budget can serve a much larger shared context pool, directly addressing the TB-scale context requirement of agentic workloads.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Boundary: if agentic turns branch on model or tool outputs, the ordered-prefix assumption breaks; a natural test is measuring prefetch hit rate on tool-use traces where future read order is not known in advance.
  • Transfer: the hint-based staging contract could apply to other predictable, read-once server workloads (e.g., embedding caches or retrieval-augmented context), where the consumer knows access order before issuing reads.
  • Extension: the paper's roughly 128 MB per-request prefetch window and fixed refill pipeline depth leave a tunable knob; increasing staging depth should push performance closer to the distributed-DRAM tier, suggesting an explicit cost-performance tradeoff.
  • Implication for cluster design: a 30% TTFT penalty for 16x less DRAM means capacity-constrained deployments may prefer this design, and the gap likely shrinks as footprints grow beyond 1 TB.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes HyMCache, a KV-cache framework that uses CXL-hybrid memory (CXL-HM) as a remote Tier-3 backend for multi-turn LLM serving. CXL-HM combines a small device-DRAM staging area with SSD-backed capacity exposed through CXL.mem; HyMCache adds master, lookup, connector, and KV-manager modules to Dynamo/vLLM that prefetch known prefix KV objects into device DRAM and defer newly written KV blocks through an isolated path. The authors evaluate on an FPGA-based CXL-HM prototype against LMCache, Mooncake, and NVMe-oF baselines, reporting 3.0x TTFT improvement over local LMCache in single-node serving, 1.45x in PD-disaggregated serving, and about 70% of a 1 TB distributed-DRAM Mooncake's performance with 16x less DRAM.

Significance. If the central claims hold, the paper contributes a concrete system point between expensive remote DRAM and slow remote SSD: it demonstrates a real FPGA-based CXL-HM prototype, integrates it with vLLM/Dynamo, and evaluates it under both single-node and PD-disaggregated configurations. The use of real hardware, multiple baselines, and a cost discussion are strengths. However, the headline comparisons rest on inconsistent DRAM accounting, and the paper does not directly measure the sustained SSD-to-DRAM refill bandwidth that is the load-bearing mechanism of the design. The prefetch-window parameters are also tuned on the same prototype and workloads used in the headline evaluation, so the reported gains may be optimistic without a sensitivity study.

major comments (4)
  1. [Abstract, §5, §6.1] The 'same DRAM budget' claim is not supported by the stated setup. In PD-disaggregated serving, LMCache is configured with 64 GB of local DRAM on each of the four prefill workers (256 GB total, §5), whereas HyMCache's remote CXL-HM prototype contains 64 GB of device DRAM in total. Thus the 1.45x TTFT improvement is obtained with roughly 4x less DRAM, not the same budget. The single-node comparison may be apples-to-apples (64 GB local vs 64 GB device), but the abstract and introduction state both modes. In addition, the '16x less DRAM than Mooncake' claim counts only the 1 TB remote DRAM tier and omits the 20 GB/node transfer-buffer DRAM that Fig. 10 shows Mooncake needs to reach its headline performance. The DRAM accounting should be made consistent, or the claims reworded.
  2. [§3.3, §6.2.3] The central mechanism is that the bounded internal-DRAM staging window remains populated by SSD-to-DRAM refill. The paper states this condition ('as long as the refill pipeline keeps this window populated') but never measures sustained refill bandwidth. Fig. 13 reports only relative average/peak bandwidth improvements and a 14% E2E gain; no absolute GB/s over a full turn-sized prefix is provided. With two Gen5 x4 SSDs backing a 200 Gbps RDMA path, a sustained-bandwidth deficit could drain the 64 GB staging window and cause on-demand SSD reads, reverting to the collapse regime shown in Fig. 5. Add a microbenchmark reporting sustained prefetch bandwidth for a turn-sized working set, with and without concurrent foreground reads and write flushes, and the resulting window-drain behavior.
  3. [§6.1.2] The NVMe-oF baseline is evaluated only through an XFS mount, which achieves 6.5 GB/s, while the paper itself reports that the raw NVMe-oF block-device path reaches about 23 GB/s. The conclusion that HyMCache avoids the 'slow remote storage' regime is therefore not cleanly isolated from filesystem-stack overhead. A raw-block or otherwise I/O-bypass KV-storage baseline could plausibly perform far closer to HyMCache. Please add such a baseline, or at least quantify how much of HyMCache's advantage remains after eliminating the XFS overhead.
  4. [§4.2, §6.2.1, §6.2.2] Key prefetch parameters — the 128 MB maximum prefetch window per request, the 32-block lookup batch, and the 5 GB worker staging buffer — appear to be fixed based on the same prototype and workloads used for the headline evaluation, with no sensitivity analysis. Because the staging-window depth directly determines how much SSD latency can be hidden, a reader cannot assess how sensitive the TTFT gains are to these choices. Report a sweep over prefetch depth/window/batch size on at least one workload, or justify the values from independent measurements.
minor comments (5)
  1. [Fig. 8] Fig. 8(b) label block 'Turn 1 to 77.3 11.1 16.1 17.2 27.4' is ambiguous; label bars directly with system names and values.
  2. [§3.2] The commercial CMM-H device used for Fig. 5 is not identified (vendor/model/firmware), and the microbenchmark configuration is only in the caption. This makes the motivating 'LRU collapse' result hard to reproduce.
  3. [§6.1, Fig. 8, Fig. 11] Headline TTFT results are medians without dispersion (error bars/CDFs) for the key comparisons. A CDF or per-run variance would help judge whether the 1.45x and 3.0x differences are stable.
  4. [Table 1] The cost-footnote says the CXL-HM estimate is for 'four FPGA-based prototypes,' but the PD experiment uses one prototype; clarify the capacity/cost basis so the cost ratios are reproducible.
  5. [§2.3, §5] The motivating agentic workloads from [65] are not evaluated; the experiments use AIPerf synthetic traces and LMSYS chat. A direct agentic-trace evaluation, or an explicit argument that these workloads satisfy the same prefix-order property, would strengthen the generality claim.

Circularity Check

0 steps flagged

No significant circularity: HyMCache's headline claims are empirical comparisons against independent baselines; self-citations are background only.

full rationale

HyMCache's claimed contribution is an empirical system result, not a derivation-to-fit loop. The workload observations in Sec. 3.1 are measured (remote-DRAM setting, LMSYS) or cited to external traces [65]; the design directives in Sec. 3.2-3.3 follow as engineering choices, not as formal derivations. The prefetch staging window in Sec. 4.2 is tuned on the prototype ('Based on our prototype evaluation, allocating up to approximately 128 MB of prefetched KV data per request provides sustained performance'), but this is a configuration choice, and its benefit is then directly measured against prefetch-off in Fig. 13; the headline 3.0x/1.45x results compare against LMCache and Mooncake, which are external systems with fixed configurations. The self-citations to ITME [24] and TraCT [72] appear only in background and related-work descriptions of CXL-HM/CXL pooling; they do not supply the load-bearing evidence for the paper's performance or cost claims, which rest on the paper's own prototype measurements and Table 1 cost estimates. The conditional in Sec. 3.3 ('As long as the refill pipeline keeps this window populated') is an explicitly stated precondition for the design's benefit; whether it holds under agentic branching is a workload-generalization question, not circularity. No equation or fitted parameter is renamed as a prediction, so no circular step can be exhibited.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The central design depends on tuned prefetch parameters (128 MB per request, 32-block lookup batches, 5 GB staging buffers) and on workload-predictability assumptions. No new physical entities are introduced.

free parameters (3)
  • prefetch_window_per_request = ~128 MB (8 x 16 MB or 4 x 32 MB blocks)
    Section 4.2: 'Based on our prototype evaluation, allocating up to approximately 128 MB of prefetched KV data per request provides sustained performance.' This is tuned to the prototype, not derived from a workload model.
  • lookup_batch_size = 32 blocks per request
    Section 4.1: 'up to 32 blocks per request by default.' Chosen without analysis and affects prefetch timing and metadata batching.
  • staging_buffer_size = 5 GB per worker
    Section 4.1: 'We use a 5 GB staging buffer by default, matching common practice in existing systems.' This is a convention, not a derived value.
axioms (4)
  • domain assumption KV reuse is read-dominant, sequential, and append-only
    Section 3.1 Observations 1-4; supported by traces but assumed for the design. If workloads have random or write-heavy KV access, prefetching and write isolation lose their benefit.
  • domain assumption Request prefix order is known before execution
    Section 3.3: 'Multi-turn prefix-cache lookup reveals the ordered list of KV objects that an inference request will consume.' The entire prefetch mechanism depends on this predictability.
  • domain assumption Commercial CMM-H collapse is caused by LRU refills and dirty evictions
    Section 3.2: 'which we verify internally'—the attribution is asserted but the verification data is not released.
  • domain assumption FPGA CXL-HM prototype is representative of production CXL-HM
    Section 3.3 and Section 5 use an FPGA-based prototype with 64 GB DRAM and 2 TB SSD; results may not transfer directly to commercial CXL-HM devices with different controllers and policies.

pith-pipeline@v1.3.0-alltime-deepseek · 23135 in / 15594 out tokens · 147671 ms · 2026-08-01T15:51:34.480074+00:00 · methodology

0 comments
read the original abstract

Long-context, multi-turn, and agentic LLM workloads increasingly reuse previously processed context, making KV-cache reuse essential for reducing redundant computation. However, this reuse shifts the bottleneck to the memory tier that stores and serves reusable KV states at cluster scale. GPU HBM and host DRAM are too costly to scale to TB-scale shared context capacity, motivating remote tiers built from lower-cost, higher-capacity media. This paper presents HyMCache, a KV-cache framework that integrates CXL-hybrid memory (CXL-HM) for multi-turn LLM serving. CXL-HM combines a small amount of in-device DRAM with large SSD-backed capacity behind a CXL interface. By exploiting the read-dominant, predictable, and append-only nature of multi-turn KV-cache access, HyMCache rethinks DRAM management within CXL-HM to efficiently support TB-scale SSD-backed KV reuse. It uses request-level prefix prefetching and opportunistic write buffering to stage latency-critical reads in device DRAM, enabling DRAM-scale KV-cache efficiency at SSD-level cost. We evaluate HyMCache on a real CXL-HM prototype under both single-aggregator and PD-disaggregated serving configurations. Under the same DRAM budget, HyMCache outperforms local LMCache by 3.0x in single-node serving and 1.45x in PD-disaggregated serving. Compared with 1 TB distributed-DRAM Mooncake, HyMCache incurs about 30% lower performance but uses 16x less DRAM.

Figures

Figures reproduced from arXiv: 2607.18141 by Hakbeom Jang, Inho Song, Jongryool Kim, Sam H. Noh.

Figure 1
Figure 1. Figure 1: CXL-HM across deployment models DRAM and storage. Overall, KV-cache tiering exposes a trade-off among latency, capacity, cost, energy, and data movement. 2.2 Available CXL Solutions and Trade-offs Compute Express Link (CXL) has emerged as a memory intercon￾nect that extends server memory capacity beyond local DIMMs while preserving a memory-like access model. The CXL specifica￾tion has evolved from single-… view at source ↗
Figure 2
Figure 2. Figure 2: CXL-HM opportunity for multi-turn LLM serving. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Comparison of RDMA-based remote access paths. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Impact of model-dependent KV block size on CMM [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Overview of CXL-HM prototype and its interaction [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: HyMCache overview and walkthrough example. [PITH_FULL_IMAGE:figures/full_fig_p007_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: TTFT in a 4P–1D–1S setup using Qwen2.5-32B. Recomputation performs no KV reuse, GPU prefix caching stores [PITH_FULL_IMAGE:figures/full_fig_p009_8.png] view at source ↗
Figure 10
Figure 10. Figure 10: Qwen2.5-32B serving performance with the Dis [PITH_FULL_IMAGE:figures/full_fig_p010_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: End-to-end single-node serving performance comparing HyMCache against baselines. [PITH_FULL_IMAGE:figures/full_fig_p011_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Comparison with Redis-backed LMCache. buffer also scales with KV-loading bandwidth; 100–400 Gbps links or multiple NICs need more staging space to keep KV reads in flight. In the following set of experiments, we isolate this transfer￾buffer effect and examine how much local host DRAM is needed to sustain memory-based remote KV reuse by varying the per-node transfer-buffer budget for both HyMCache and Dist… view at source ↗
Figure 13
Figure 13. Figure 13: Performance impact of HyMCache prefetching [PITH_FULL_IMAGE:figures/full_fig_p012_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

81 extracted references · 7 linked inside Pith

  1. [1]

    Aguilera, Emmanuel Amaro, Nadav Amit, Erika Hunhoff, Anil Yelam, and Greg Zellweger

    Marcos K. Aguilera, Emmanuel Amaro, Nadav Amit, Erika Hunhoff, Anil Yelam, and Greg Zellweger. 2023. Memory Disaggregation: Why Now and What Are the Challenges. ACM SIGOPS Operating Systems Review 57, 1 (2023), 38–46

    2023

  2. [2]

    ai-dynamo. 2025. NVIDIA Dynamo AIPerf. https://github.com/ai-dynamo/aiperf

    2025

  3. [3]

    AI-Dynamo Project. 2025. NVIDIA Inference Xfer Library (NIXL). https://github. com/ai-dynamo/nixl

    2025

  4. [4]

    Hasan Al Maruf, Haoyang Zhong, Heming Wang, Moray Dhanotia, Sanjay Shakkottai, Klara Nahrstedt, and Indranil Gupta. 2023. TPP: Transparent Page Placement for CXL-Enabled Tiered Memory. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS 23)

    2023

  5. [5]

    Alibaba Cloud. 2024. Revisiting Distributed Memory in the CXL Era. https://www. alibabacloud.com/blog/revisiting-distributed-memory-in-the-cxl-era_600809

    2024

  6. [6]

    Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar

    Keivan Alizadeh, Iman Mirzadeh, Dmitry Belenko, Karen Khatamifard, Minsik Cho, Carlo C. Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar. 2023. LLM in a Flash: Efficient Large Language Model Inference with Limited Memory. arXiv preprint arXiv:2312.11514 (2023)

    Pith/arXiv arXiv 2023

  7. [7]

    Amazon / NEMIX RAM. 2026. NEMIX RAM 256GB DDR5 RDIMM Server Memory. https://www.amazon.com/NEMIX-RAM-Registered-Compatible- M321RBJA0M22-CLP/dp/B0F9B49KZ5

    2026

  8. [8]

    Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang, Ammar Ah- mad Awan, Cheng Li, Du Li, Elton Zheng, Jeff Rasley, Shaden Smith, Olatunji Ruwase, and Yuxiong He. 2022. DeepSpeed-Inference: Enabling Efficient In- ference of Transformer Models at Unprecedented Scale. In Proceedings of the International Conference for High Performance Computing, Networ...

    2022

  9. [9]

    Anthropic. 2025. Computer Use Tool. https://docs.anthropic.com/en/docs/build- with-claude/computer-use

    2025

  10. [10]

    Deepseek api docs. 2026. DeepSeek API Context Caching on Disk. DeepSeek API Documentation. Accessed: 2026-06-11

    2026

  11. [11]

    Mustafa Rafique, and Sudharshan Vazhkudai

    Moiz Arif, Kevin Assogba, M. Mustafa Rafique, and Sudharshan Vazhkudai. 2022. Exploiting CXL-based Memory for Distributed Deep Learning. InProceedings of the 51st International Conference on Parallel Processing (ICPP 22). ACM, 19:1– 19:11

    2022

  12. [12]

    Broadcom. 2020. Broadcom Stingray SmartNIC Accelerates Baidu Cloud Services. https://www.broadcom.com/company/news/product-releases/53106

    2020

  13. [13]

    Chelsio Communications. 2026. Chelsio Unified Wire Adapters. https://www. chelsio.com/nic/unified-wire-adapters/

    2026

  14. [14]

    Yihua Cheng, Yuhan Liu, Jiayi Yao, Yuwei An, Xiaokun Chen, Shaoting Feng, Yuyang Huang, Samuel Shen, Kuntai Du, and Junchen Jiang. 2025. LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference. arXiv preprint arXiv:2510.09665 (2025)

    arXiv 2025

  15. [15]

    CXL Consortium. 2023. CXL Consortium Member Spotlight: UniFab- riX. https://computeexpresslink.org/blog/cxl-consortium-member-spotlight- unifabrix-1078/

    2023

  16. [16]

    Pouya Esmaili-Dokht, Francesco Sgherzi, Valéria Soldera Girelli, Isaac Boix- aderas, Mariana Carmin, Alireza Monemi, Adrià Armejach, Estanislao Mercadal, German Llort, Petar Radojkovic, Miquel Moretó, Judit Giménez, Xavier Mar- torell, Eduard Ayguadé, Jesús Labarta, Emanuele Confalonieri, Rishabh Dubey, and Jason Adlard. 2024. A Mess of Memory System Benc...

    2024

  17. [17]

    Yehonatan Fridman, Suprasad Mutalik Desai, Navneet Singh, Thomas Willhalm, and Gal Oren. 2023. CXL Memory as Persistent Memory for Disaggregated HPC: A Practical Approach. In Proceedings of the SC ’23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis. ACM, 983–994

    2023

  18. [18]

    Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, and Pengfei Zuo. 2024. Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention. In arXiv preprint arXiv:2403.19708v3

    Pith/arXiv arXiv 2024

  19. [19]

    Goumas, Zeshan Chishti, and Nandita Vijaykumar

    Christina Giannoula, Kailong Huang, Jonathan Tang, Nectarios Koziris, Geor- gios I. Goumas, Zeshan Chishti, and Nandita Vijaykumar. 2023. DaeMon: Archi- tectural Support for Efficient Data Movement in Fully Disaggregated Systems. Proceedings of the ACM on Measurement and Analysis of Computing Systems (POMACS 23) 7, 1 (2023), 16:1–16:36. https://doi.org/10...

    doi:10.1145/3579445 2023

  20. [20]

    Google AI for Developers. 2025. Long Context. https://ai.google.dev/gemini- api/docs/long-context

    2025

  21. [21]

    Donghyun Gouk, Miryeong Kwon, Hanyeoreum Bae, Sangwon Lee, and My- oungsoo Jung. 2023. Memory Pooling With CXL. IEEE Micro 43, 2 (2023), 48–57. https://doi.org/10.1109/MM.2023.3237491

    arXiv 2023

  22. [22]

    Zerui Guo, Hua Zhang, Chenxingyu Zhao, Yuebin Bai, Michael Swift, and Ming Liu. 2023. LEED: A Low-Power, Fast Persistent Key-Value Store on SmartNIC JBOFs. In Proceedings of the ACM SIGCOMM 2023 Conference (SIGCOMM 23). 1012–1027

    2023

  23. [23]

    Hyperscalers. 2026. Samsung PM1743 15.36TB PCIe Gen5 Enterprise SSD. https://www.hyperscalers.com/Samsung-PM1743-NVMe-PCIe-Gen5-15. 36TB-MZWLO15THBLA-00A07

    2026

  24. [24]

    Hakbeom Jang, Younghoon Min, Sunwoong Kim, Taeyoung Ahn, Hanyee Kim, Youngpyo Joo, Hoshik Kim, and Jongryool Kim. 2026. ITME: Infer- ence Tiered Memory Expansion with Disaggregated CXL-Hybrid Memories. arXiv:2606.12556 [cs.DC] https://arxiv.org/abs/2606.12556

    Pith/arXiv arXiv 2026

  25. [25]

    Juhyun Jang, Donghyun Gouk, Miryeong Kwon, Sangwon Han, Myoungsoo Kim, and Myoungsoo Jung. 2023. CXL-ANNS: Software-Hardware Collaborative Memory Disaggregation and Computation. In Proceedings of the 2023 USENIX Annual Technical Conference (ATC 23)

    2023

  26. [26]

    2007.numa(3): NUMA policy library

    Michael Kerrisk. 2007.numa(3): NUMA policy library. Linux man-pages project

    2007

  27. [27]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. arXiv preprint arXiv:2309.06180 (2023)

    Pith/arXiv arXiv 2023

  28. [28]

    Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. 2024. InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, Santa Clara, CA, 155–172. https://www.usenix.org/conference/osdi24/presentation/lee

    2024

  29. [29]

    Liqid. 2026. Liqid Composable Memory Solutions. https://www.liqid.com/ products/composable-memory-solutions

    2026

  30. [30]

    LMCache. 2026. LMCache Redis Backend Documentation. https://docs.lmcache. ai/kv_cache/redis.html

    2026

  31. [31]

    Marvell Technology. 2026. Marvell OCTEON Data Processing Units. https: //www.marvell.com/products/data-processing-units.html

    2026

  32. [32]

    Marvell Technology. 2026. NVMe over Fabrics Technology Brief. https://www.marvell.com/content/dam/marvell/en/public-collateral/fibre- channel/marvell-nvme-over-fabrics-technology-brief.pdf

    2026

  33. [33]

    Jaehong Min, Ming Liu, Tapan Chugh, Chenxingyu Zhao, Andrew Wei, In Hwan Doh, and Arvind Krishnamurthy. 2021. Gimbal: Enabling Multi-tenant Storage Disaggregation on SmartNIC JBOFs. InProceedings of the 2021 ACM SIGCOMM 2021 Conference (SIGCOMM 21). 106–122

    2021

  34. [34]

    NVIDIA. 2023. Mastering LLM Techniques: Inference Optimization. https:// developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/

    2023

  35. [35]

    NVIDIA. 2025. Architecture of AIPerf. https://docs.nvidia.com/aiperf/architecture-internals/architecture-of-ai-perf

    2025

  36. [36]

    NVIDIA. 2025. NVIDIA DOCA NVMe-oF Documentation. https: //docs.nvidia.com/doca/archive/2-5-4/NVME-oF%2B-%2BNVM%2BExpress% 2Bover%2BFabrics/index.html

    2025

  37. [37]

    NVIDIA. 2025. NVIDIA Dynamo: A Low-Latency Distributed Inference Framework for Scaling Reasoning AI Models. https://developer.nvidia.com/blog/ introducing-nvidia-dynamo-a-low-latency-distributed-inference-framework- for-scaling-reasoning-ai-models/

    2025

  38. [38]

    NVIDIA. 2025. Trace Replay with Mooncake Traces. https://docs.nvidia.com/aiperf/benchmark-modes/trace-replay-with- mooncake-traces

    2025

  39. [39]

    NVIDIA. 2026. Introducing NVIDIA BlueField-4-Powered CMX Context Memory Storage Platform for the Next Frontier of AI. NVIDIA Technical Blog (2026). https://developer.nvidia.com/blog/introducing-nvidia-bluefield-4-powered- inference-context-memory-storage-platform-for-the-next-frontier-of-ai/

    2026

  40. [40]

    NVIDIA. 2026. NVIDIA BlueField Networking Platform. https://www.nvidia. com/en-us/networking/products/data-processing-unit/

    2026

  41. [41]

    NVM Express. 2017. Accelerating NVMe™over Fabrics with Hardware Of- floads at 100Gb/s and Beyond. https://nvmexpress.org/wp-content/uploads/ Accelerating-NVMe-over-Fabrics-with-Hardware-Offloads.pdf

    2017

  42. [42]

    OpenAI. 2025. Introducing ChatGPT Agent: Bridging Research and Action. https://openai.com/index/introducing-chatgpt-agent/

    2025

  43. [43]

    Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2025. Mooncake: Trading More Storage for Less Computation — A KVCache-Centric Architecture for Serving LLM Chatbots. In Proceedings of the USENIX Conference on File and Storage Technologies (FAST 25)

    2025

  44. [44]

    Samsung Electronics. 2024. CXL Memory Module Box: CMM-B. https://semiconductor.samsung.com/news-events/tech-blog/cxl-memory- module-box-cmm-b/

    2024

  45. [45]

    Samsung Electronics. 2026. Samsung CMM-D: CXL Memory Module-DRAM. https://semiconductor.samsung.com/cxl-memory/cmm-d/

    2026

  46. [46]

    Samsung Electronics. 2026. Samsung PM1743 Enterprise SSD. https: //semiconductor.samsung.com/ssd/enterprise-ssd/pm1743/

    2026

  47. [47]

    ServerPartDeals. 2026. Dell/Micron 7450 PRO 15.36TB PCIe Gen4 TLC SSD. https://serverpartdeals.com/products/dell-micron-7450-pro-mtfdkcc15t3tfr- 15-36tb-1-dwpd-pcie-gen-4-0-x4-8gb-s-3d-tlc-u-3-nvme-2-5-ssd

    2026

  48. [48]

    ServerSupply. 2026. Dell AC830719 128GB DDR5-5600 ECC RDIMM Server Memory. https://www.serversupply.com/MEMORY/PC5-44800/128GB/DELL/ AC830719_386128.htm. 13

    2026

  49. [49]

    ServeTheHome. 2022. Compute Express Link CXL Latency: How Much Is Added? https://www.servethehome.com/compute-express-link-cxl-latency- how-much-is-added-at-hc34/

    2022

  50. [50]

    Graham Lopez, Matthew B

    Pavel Shamis, Manjunath Gorentla Venkata, M. Graham Lopez, Matthew B. Baker, Oscar Hernandez, Yossi Itigin, Mike Dubman, Gilad Shainer, Richard L. Graham, Liran Liss, et al . 2015. UCX: An Open Source Framework for HPC Network APIs and Beyond. In 2015 IEEE 23rd Annual Symposium on High-Performance Interconnects. IEEE, 40–43

    2015

  51. [51]

    Jiacheng Shen, Pengfei Zuo, Xuchuan Luo, Yuxuan Su, Jiazhen Gu, Hao Feng, Yangzhou Zhou, and Michael R. Lyu. 2023. Ditto: An elastic and adaptive memory-disaggregated caching system. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP 23). 675–691

    2023

  52. [52]

    Fu, Zhiqiang Xie, Beidi Chen, Clark Barrett, Joseph E

    Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Daniel Y. Fu, Zhiqiang Xie, Beidi Chen, Clark Barrett, Joseph E. Gonzalez, and Ion Stoica

  53. [53]

    SK hynix. 2024. SK hynix Presents AI Memory Solutions at CXL DevCon

    2024

  54. [54]

    SK hynix. 2025. SK hynix Completes Customer Validation of CXL-based DDR5 Memory Module. https://news.skhynix.com/sk-hynix-completes-customer- validation-of-cxl-based-ddr5/

    2025

  55. [55]

    SK hynix. 2026. SK hynix PS1000 Series PCIe NVMe Gen5 Enterprise SSD. https://product.skhynix.com/products/ssd/essd/ps1000.go

    2026

  56. [56]

    Solidigm. 2023. Solidigm D5-P5430 3.84TB PCIe Gen4 QLC Enterprise SSD. ServerSupply product listing. Accessed: 2026-06-11; listed price example: $2,350

    2023

  57. [57]

    Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen. 2023. PowerInfer: Fast Large Language Model Serving with a Consumer-Grade GPU. arXiv preprint arXiv:2312.12456 (2023)

    Pith/arXiv arXiv 2023

  58. [58]

    Gongjin Sun. 2025. Revisiting Memory Hierarchies with CMM-H: Us- ing Device-side Caching to Integrate DRAM and SSD for a Hybrid CXL Memory. https://www.hotstorage.org/2025/slides/Revisiting%20Memory% 20Hierarchies%20with%20CMM-H.pdf

    2025

  59. [59]

    Xun Sun, Mingxing Zhang, Yingdi Shan, Kang Chen, Jinlei Jiang, and Yongwei Wu. 2025. Scalio: Scaling up DPU-based JBOF Key-value Store with NVMe-oF Target Offload. In 19th USENIX Symposium on Operating Systems Design and Implementation (OSDI 25). USENIX Association

    2025

  60. [60]

    Super Micro Computer, Inc. 2024. Petascale JBOF All-Flash Array for AI Data Pipeline Acceleration. https://www.supermicro.org.cn/en/products/jbof

    2024

  61. [61]

    Supermicro. 2026. Supermicro Certified 15.36TB KIOXIA CD8P-R PCIe Gen5 TLC SSD. https://store.supermicro.com/us_en/15-36tb-nvme-hds-t3n-kcd8dpje15t3. html

    2026

  62. [62]

    Terasic. 2023. The Agilex 7 FPGA I-Series Development Kit. https: //www.terasic.com.tw/cgi-bin/page/archive.pl?CategoryNo=142&Language= English&No=1279

    2023

  63. [63]

    The Next Platform. 2022. Just How Bad Is CXL Memory Latency? https://www.nextplatform.com/store/2022/12/05/just-how-bad-is-cxl- memory-latency/1653612

    arXiv 2022

  64. [64]

    Xi Wang, Jie Liu, Jianbo Wu, Shuangyan Yang, Jie Ren, Bhanu Shankar, and Dong Li. 2024. Exploring and Evaluating Real-World CXL: Use Cases and System Adoption. arXiv preprint arXiv:2405.14209 (2024)

    arXiv 2024

  65. [65]

    Yongtong Wu, Shaoyuan Chen, Yinmin Zhong, Rilin Huang, Yixuan Tan, Wentao Zhang, Liyue Zhang, Shangyan Zhou, Yuxuan Liu, Shunfeng Zhou, Mingxing Zhang, Xin Jin, and Panpan Huang. 2026. DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference. https://arxiv.org/abs/2602. 21548

    2026

  66. [66]

    XCENA. 2026. XCENA MX1 CXL Computational Memory. https://xcena.com/ computational_memory

    2026

  67. [67]

    XConn Technologies. 2026. XC50256 CXL 2.0 Switch Chip. https://www.marvell. com/products/cxl.html

    2026

  68. [68]

    Xsight Labs. 2026. Xsight Labs E-Series DPU. https://xsightlabs.com/dpu

    2026

  69. [69]

    Jingsen Xu, Yu Qiu, Yuan Chen, Yang Wang, Wei Lin, Yi Lin, Shuai Zhao, Yan Liu, Yu Wang, and Wenguang Chen. 2024. Performance Characterization of SmartNIC NVMe-over-Fabrics Target Offloading. In Proceedings of the 17th ACM International Systems and Storage Conference (SYSTOR 24). 14–24

    2024

  70. [70]

    Juncheng Yang, Yazhuo Zhang, Ziyue Qiu, Yao Yue, and Rashmi Vinayak. 2023. FIFO queues are all you need for cache eviction. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP 23)

    2023

  71. [71]

    Xinjun Yang, Qingda Hu, Junru Li, Feifei Li, Yuqi Zhou, Yicong Zhu, Qiuru Lin, Jian Dai, Yang Kong, Jiayu Zhang, Guoqiang Xu, and Qiang Liu. 2025. Beluga: A CXL-Based Memory Architecture for Scalable and Efficient LLM KVCache Management. https://arxiv.org/abs/2511.20172

    arXiv 2025

  72. [72]

    Noh, and Jongryool Kim

    Dongha Yoon, Younghoon Min, Hoshik Kim, Sam H. Noh, and Jongryool Kim

  73. [73]

    Lingfan Yu, Jinkun Lin, and Jinyang Li. 2023. Stateful Large Language Model Serving with Pensieve. arXiv preprint arXiv:2312.05516 (2023)

    Pith/arXiv arXiv 2023

  74. [74]

    Ling Zhan, Kai Lu, Yiqin Xiong, Jiguang Wan, and Zixuan Yang. 2024. Trick- leKV: A High-Performance Key-Value Store on Disaggregated Storage with Low Network Traffic. IEEE Access (2024)

    2024

  75. [75]

    Ming Zhang, Yu Hua, Pengfei Zuo, and Limin Liu. 2022. FORD: Fast one-sided RDMA-based distributed transactions for disaggregated persistent memory. In Proceedings of the 20th USENIX Conference on File and Storage Technologies (FAST 22). 51–68

    2022

  76. [76]

    Xing, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhang- hao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric P. Xing, Joseph E. Gonzalez, Ion Stoica, and Hao Zhang. 2024. LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset. In Proceedings of the International Conference on Learning Representations (ICLR 24)

    2024

  77. [77]

    Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Huang, Jeff Sun, Chao H. Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, et al . 2023. SGLang: Efficient Execution of Structured Language Model Programs. https: //arxiv.org/abs/2312.07104

    Pith/arXiv arXiv 2023

  78. [78]

    Yuchen Zhou, Jianping Zeng, and Changhee Jung. 2024. LightWSP: Whole- System Persistence on the Cheap. In 2024 57th IEEE/ACM International Symposium on Microarchitecture (MICRO 24). IEEE, 215–230. 14

    2024

  79. [2023]

    In Proceedings of the 40th International Conference on Machine Learning (ICML 23)

    FlexGen: High-Throughput Generative Inference of Large Language Mod- els with a Single GPU. In Proceedings of the 40th International Conference on Machine Learning (ICML 23)

  80. [2024]

    https://news.skhynix.com/sk-hynix-presents-ai-memory-solutions-at- cxl-devcon-2024/

    2024

Showing first 80 references.