Pith. sign in

REVIEW 4 major objections 6 minor 6 cited by

Strata: Hierarchical Context Caching for Long Context Language Model Serving

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Strata claims long-context LLM serving is loading-bound, not compute-bound, and fixes it by moving cache pages with GPU threads and scheduling around the load.

desk verdict Strata makes a real systems contribution to long-context KV-cache serving, but the headline speedup numbers overstate what the evaluation actually shows; referee it, and insist on code. read the letter →

arxiv 2508.18572 v1 pith:FX4GAA55 submitted 2025-08-26 cs.DC

Zhiqiang Xie , Ziyi Xu , Mark Zhao , Yuwei An , Vikram Sharma Mailthody , Scott Mahlke , Michael Garland , Christos Kozyrakis This is my paper · ORCID
classification cs.DC
keywords hierarchicalKVcachelong-contextLLMservingGPU-assistedI/Ocache-awareschedulingTTFTPagedAttentionmemoryhierarchycontextcaching
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Long-context LLM serving relies on reusing previously computed key-value (KV) caches, but when those caches live in CPU memory or disk, loading them back to the GPU becomes the bottleneck: paged layouts break transfers into tiny fragments that use only about 22% of PCIe bandwidth, and schedulers that ignore loading time leave the GPU idle. Strata claims that two changes fix this: a GPU-assisted I/O kernel that copies fragmented pages at near full bandwidth while using only one or two GPU blocks, and a cache-aware scheduler that defers requests that would cause duplicate computation, balances batches so loading is hidden by prefill compute, and fills unavoidable stalls with decoding work. Evaluated on long-context benchmarks with models from 8B to 70B, the paper reports up to 5x lower time-to-first-token and up to 5x higher throughput than existing hierarchical-caching systems, without short-context regression. If true, Strata makes cheap, large-scale context reuse practical on current hardware.

What carries the argument

Three mechanisms carry the argument. First, GPU-assisted I/O: a CUDA kernel with thousands of threads, each moving a small chunk through register files while bypassing cache, is launched with a small number of large blocks so the hardware scheduler confines it to one or two SMs; this turns small fragmented page copies into near-bandwidth transfers and allows on-the-fly layout transformation between a page-first host layout and a layer-first GPU layout. Second, the HiRadixTree, an extension of SGLang's RadixTree prefix tree that acts as a page table and records transient in-queue and in-flight nodes so the scheduler can detect delay hits. Third, a cache-aware scheduler with three policies: deferral on delay hits, balanced batch formation using a load-to-compute ratio threshold (default 100), and bubble filling that overlaps loading stalls with decoding batches.

What would settle it

Run Strata's I/O kernel concurrently with prefill and decode on a non-H200 GPU and measure sustained host-to-GPU and GPU-to-host bandwidth along with the prefill and decode slowdown; if sustaining roughly 50 GB/s requires more than the one-to-two-block quota, or if prefill or decode degradation exceeds the claimed 5% or 10%, the end-to-end throughput gains over baselines should erode proportionally.

Watch

Extended reading notes

Core claim

Strata's central claim is that long-context serving slowdowns come mostly from how cached context is moved, not from how it is computed or stored. The paper argues that small pages, which are needed for high cache hit rates, make standard DMA copies inefficient, and that layer-wise overlap assumptions break when cached tokens dwarf new prefill tokens. Strata's GPU-assisted I/O kernel replaces many small cudaMemcpyAsync calls with a CUDA kernel whose threads copy 128-byte chunks directly between pinned CPU memory and GPU memory; by launching only one or two large thread blocks, the kernel stays confined to a few streaming multiprocessors and reaches about 50 GB/s while keeping prefill degradation under 5% and decode degradation under 10%. The cache-aware scheduler then treats CPU-GPU bandwidth as a first-class resource: it defers requests that would hit a cache that is still being computed, forms batches whose load-to-compute ratio stays below a profiled threshold, and inserts decoding batches into remaining loading stalls. Together these mechanisms deliver the reported 1.9x to 5x improvements over SGLang-HiCache, vLLM+LMCache, and TensorRT-HiCache on LooGLE, and keep performance comparable on short-context ShareGPT workloads.

Load-bearing premise

The load-bearing premise is that one or two GPU compute blocks can move data between CPU and GPU memory at roughly 50 GB/s while slowing prefill by under 5% and decode by under 10%, and that this holds on other GPUs and under real concurrent workloads; if it does not, the I/O-driven gains shrink to a small scheduling improvement.

Editorial extensions

If this is right

  • Hierarchical caching becomes practical for prefill-dominated workloads without inflating page size and sacrificing cache hit rate.
  • The page-size tuning burden disappears: Strata claims consistently high I/O efficiency at page sizes from 1 token upward, where SGLang-HiCache peaks at 93% of Strata's throughput at its best page size.
  • On high-bandwidth platforms like Grace-Hopper, software that uses GPU-assisted I/O and bandwidth-aware scheduling can approach an oracle with infinite CPU-GPU bandwidth; the paper shows Strata-GH roughly matches Strata-Oracle.
  • Short-context workloads are unaffected, because the extra I/O and scheduling layers only engage when hierarchical cache traffic is significant.
  • The scheduling principles generalize: treating interconnect bandwidth as a first-class resource prevents loading-bound behavior that layer-wise overlap alone cannot hide.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the reported CPU/disk tiers, the same GPU-assisted transfer kernel should apply to disaggregated KV caches reached over network fabrics, where small fragmented transfers are equally punishing; this is an extrapolation, since Strata evaluates only local CPU DRAM and disk.
  • Strata's default deferral threshold of 100 token matches is presented as a tuned constant; a natural test is to sweep it under different request arrival rates and cache distances to see whether a fixed threshold remains optimal.
  • Because Strata removes the page-size penalty, one could push cache hit rates further by using the smallest legal page size and measure whether the hit-rate gain outweighs any remaining transfer overhead; the paper does not report this extreme.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. Strata is a hierarchical KV-cache caching framework for long-context LLM serving, built on SGLang. It contributes (i) GPU-assisted I/O, a small CUDA kernel that copies fragmented KV-cache pages between CPU and GPU memory while confining itself to one or two SMs, with on-the-fly layout transformation between the layer-first GPU layout and the page-first host layout; and (ii) cache-aware scheduling, comprising delay-hit deferral via transient radix-tree nodes, balanced batch formation that caps the load/compute ratio, and bubble-filling that overlaps loading stalls with decoding. The paper evaluates Strata against SGLang-HiCache, vLLM+LMCache, and TensorRT-LLM-HiCache on H200 and GH200 with Llama-8B, Llama-70B, and Qwen2.5-14B across LooGLE, NarrativeQA, ReviewMT, and ShareGPT. It reports up to 5x throughput gains at equal TTFT on Llama-70B, with a roughly equal split between the I/O and scheduling components in the Qwen-14B breakdown, and no short-context degradation on ShareGPT.

Significance. The potential significance is high if the results transfer: the paper identifies and quantifies two real bottlenecks (fragmented small-page transfers and loading-bound scheduling) and offers a concrete, deployable fix. The evaluation is unusually thorough for this area: it includes a component-level breakdown (Figure 9), a workload-pattern study (Figure 11), a page-size robustness study (Figure 10), a disk-layout microbenchmark, and a GH200 comparison. The decomposition of the end-to-end gain into I/O and scheduling contributions is a particular strength. The main caveats are that the headline quantitative claim is stated as a TTFT reduction although the reported metric is throughput at equal TTFT, the I/O interference result rests on a single platform and a single co-run shape, and the scheduler depends on several default thresholds for which no sensitivity analysis is provided.

major comments (4)
  1. [Abstract; §5.2.1] The abstract's 'up to 5x lower TTFT' and '3.75x speedup over TensorRT-LLM' are not supported by the evaluation as described. Section 5.2.1 reports throughput gains at the same TTFT (e.g., 'up to 3.2x, 2.6x, and 1.9x higher throughput at the same TTFT' for Llama-8B), and for Llama-70B the 'gains reach 5x, 5x, and 3.75x' appear to be throughput gains too. No experiment in the paper measures TTFT at matched throughput. Please either restate the headline as 'up to 5x higher throughput at equal TTFT' or add the missing TTFT-versus-throughput comparison that justifies the current abstract wording.
  2. [§4.2, Figure 5] The central I/O efficiency claim rests on a microbenchmark performed on a single H200 with one fixed co-run shape (two 4k-token prefill requests and one 16-request decode batch). The claim in §4.2 that the end-to-end evaluation confirms an 'overall performance impact under 5%' is not supported by any end-to-end interference measurement: Figure 8 and the breakdowns report aggregate throughput and TTFT, not the marginal cost of the I/O kernel on co-running prefill and decode. Please report interference at several prefill batch sizes, decode batch sizes, and with tensor-parallel 70B traffic, and either remove the 'under 5%' statement or back it with an explicit measurement.
  3. [§5.1, §3.1, Figure 2] The evaluation sets page size to 1 for Strata and SGLang but to 32 for SGLang-HiCache, vLLM-LMCache, and TensorRT-HiCache. Since §3.1 and Figure 2 show that page size materially changes cache hit rate and TTFT, the headline comparison mixes the effect of Strata's ability to use small pages with the effect of its I/O and scheduling mechanisms. Figure 10 addresses this only for one model and workload. Please add a controlled comparison (e.g., SGLang-HiCache with page size 1, and Strata with page size 32) for the main end-to-end benchmarks, or explicitly quantify the page-size component for each model.
  4. [§4.3, §4.2] The scheduler's two main thresholds are engineering constants: the load/compute ratio bound of 100 and the delay-hit deferral threshold of 100 tokens, plus the I/O block quotas. No sensitivity analysis is presented, so it is unclear how robust the reported 1.7x-5x gains are to these choices or whether they were tuned on the evaluation workloads. Please report throughput and TTFT for a range of threshold values (e.g., load/compute ratio 50-200 and deferral threshold 50-200) on at least one long-context workload, and state how the defaults were selected.
minor comments (6)
  1. [§5.3.4] The text says 'Figure 6 presents a micro-benchmark demonstrating the benefit,' but Figure 6 is the layout diagram; the relevant bar chart is Figure 12. Please correct the cross-reference.
  2. [§5.2, Figure 8] No error bars or confidence intervals are reported for the throughput-latency curves, and the number of repetitions is not stated. Given the heavy-tailed nature of TTFT, please report P90 TTFT and/or multiple runs for the main comparisons.
  3. [§5.4] The 'Strata-Oracle' simulation is not defined. Please specify what it simulates (e.g., zero transfer time or infinite interconnect bandwidth) and how the TTFT values were obtained.
  4. [§4.2] The statement that 'the granularity required for efficient GPU-assisted I/O is only 128 bytes' is asserted without a citation. Please provide a reference or a measurement.
  5. [§4.2] ROCm compatibility is claimed but never evaluated. If the claim is retained, support it with at least a brief microbenchmark or state explicitly that AMD support is untested.
  6. [§1] The production-deployment claim ('deployed in production environments at a leading AI company') is unverifiable. Consider removing it or adding a public reference.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular reduction: Strata's claims are benchmark measurements against external baselines; self-citations are background and tuning constants are reported as profiled settings.

full rationale

Walking the claimed derivation chain, I find no step in which a predicted quantity is equivalent to an input by construction. The central claims—up to 3.2x/2.6x/1.9x throughput at equal TTFT for Llama-8B and up to 5x/5x/3.75x for Llama-70B relative to SGLang-HiCache, vLLM-LMCache, and TensorRT-HiCache—are direct measurements against external baselines (Section 5.2.1, Figure 8). The two key design components are evaluated independently in Section 5.3.1: Strata-Schedule-Only and Strata-IO each contribute measurable, separable gains, so the combined result is not a single fitted parameter. The scheduler thresholds (load/compute ratio 100 and deferral threshold 100) are explicitly presented as hardware- and model-dependent profiled defaults ('This threshold is hardware- and model-dependent and thus can be profiled separately; in practice, Strata uses a default ratio of 100, corresponding to the point where stalls begin to appear showed in Figure 1'), not as predictions derived from the end-to-end result. The two-block GPU-assisted I/O quota is selected from the microbenchmark in Figure 5 and reported as an engineering choice, and its interference claim is separately tested end-to-end; even if the quota were wrong on another platform, that is a portability risk, not circularity. The paper cites prior work by its own authors [26, 30, 37] for the general idea of GPU-assisted I/O and for the observation that GPU schedulers manage contention poorly, but these citations are background and inspiration: the paper's own Figure 5 measures the contention behavior, and no uniqueness theorem or load-bearing derivation is imported from those citations. The delay-hit concept is explicitly credited to networking prior work [5], and 'bundle hit' is a descriptive term for batching same-context requests, not a renamed derivation. Overall, the paper is self-contained against external benchmarks and contains no reduction of a conclusion to its own inputs.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claims are empirical and rest on hardware behavior assumptions plus a few tuned scheduling constants, not on a mathematical derivation with hidden inputs.

free parameters (4)
  • loading_bound ratio threshold = 100 (default)
    Algorithm 1 uses this ratio of aggregated load tokens to compute tokens to classify a batch as loading-bound. The paper states it is hardware- and model-dependent and profiled separately; default 100 corresponds to where stalls begin in Figure 1.
  • delay-hit deferral threshold = 100 active token matches
    Requests are deferred only when transient-node token matches exceed this threshold. Chosen in practice, not derived, and it controls scheduling behavior.
  • GPU-assisted I/O block quota = 2 SMs for host-to-GPU loads; 1 for GPU-to-host backup
    Selected from H200 microbenchmarks in Figure 5 as a trade-off between bandwidth and interference. Hardware-specific and not derived from first principles.
  • I/O kernel block size = 1024 threads per block
    Chosen in microbenchmarks; not justified against alternatives, and the interference profile depends on this choice.
assumptions (5)
  • standard math Little's Law: C = lambda * L, giving X = C*S/L for stable throughput.
    Used in Section 3.1 to argue that transfer size S is the practical lever for bandwidth utilization. Assumes a stable-state queueing model for I/O.
  • domain assumption Achieving 75-80% of PCIe 5.0 bandwidth requires megabyte-scale transfer sizes (1-2 MB).
    Empirical claim cited in Section 3.1; motivates why small KV cache pages underutilize bandwidth. This is a hardware behavior assumption, not proven in the paper.
  • domain assumption A few large CUDA blocks can confine the I/O kernel to a small number of SMs and avoid cache pollution.
    Core to GPU-assisted I/O in Section 4.2. Supported by H200 microbenchmarks but assumed to generalize across GPUs and concurrent workloads.
  • domain assumption Decode batches saturate HBM bandwidth while cache loads use PCIe bandwidth, so the two can overlap with little contention.
    Used in Section 4.3.3 for bubble filling. If a platform shares a single interconnect or the GPU has different bandwidth topology, this overlap may not hold.
  • domain assumption Long-context production workloads maintain high cache hit rates (around 95%).
    Evaluation datasets are constructed with repeated access to the same documents. The system's benefit depends on such reuse; low-hit-rate workloads would reduce the value of caching.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Strata: Hierarchical Context Caching for Long Context Language Model Serving." pith.science (2026). https://pith.science/paper/FX4GAA55

@misc{pith2026250818572,
  author       = {Pith},
  title        = {Pith review of: Strata: Hierarchical Context Caching for Long Context Language Model Serving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FX4GAA55}},
  note         = {Machine review of arXiv:2508.18572}
}
read the original abstract

Large Language Models (LLMs) with expanding context windows face significant performance hurdles. While caching key-value (KV) states is critical for avoiding redundant computation, the storage footprint of long-context caches quickly exceeds GPU memory capacity, forcing production systems to adopt hierarchical caching across memory hierarchies. However, transferring large cached contexts back to the GPU introduces severe performance bottlenecks: fragmented I/O from paged layouts prevents full bandwidth utilization, and existing schedulers fail to account for cache-loading delays, leaving systems loading-bound rather than compute-bound. We present Strata, a hierarchical context caching framework designed for efficient long context LLM serving. Strata introduces GPU-assisted I/O to combat KV cache fragmentation, decoupling GPU and CPU memory layouts and employs cache-aware request scheduling to balance compute with I/O latency and overlapping unavoidable stalls with complementary tasks. Built on SGLang and deployed in production, Strata achieves up to 5x lower Time-To-First-Token (TTFT) compared to vLLM + LMCache and 3.75x speedup over NVIDIA TensorRT-LLM on long-context benchmarks, without degrading short-context performance.

Figures

Figures reproduced from arXiv: 2508.18572 by the authors.

Figure 1
Figure 1. Benchmark profile for Qwen2.5-14B on the LooGLE dataset. The x-axis shows the Load / Compute Ra￾tio (tokens loaded from CPU memory relative to new input tokens) per prefill batch. The right axis displays the I/O stall percentage, representing the amount of prefill execution time attributed to I/O stall. See §5.3 for full benchmark details. CPU memory to GPU HBM) increases substantially. How￾ever, current systems ach… view at source ↗
Figure 2
Figure 2. Large page sizes decrease cache hit rate and in￾crease TTFT, benchmarked on H200 for Mistral-24B using the ShareGPT dataset. 16 32 64 384 Theoretical Bandwidth (GB/s) 0 20 40 60 80 100 120 Latency (ms) PCIe 3.0 PCIe 4.0 PCIe 5.0 GH200 Measured Latency (ms) Theoretical Latency (ms) Sustained Utilization 0.0 0.2 0.4 0.6 0.8 1.0 Bandwidth Utilization (%) [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Latency and bandwidth utilization of loading KV caches of 8192 tokens (using page size 32) of Llama-3.1-8B from CPU to GPU on different platforms. by providers such as OpenAI [35] and Google [14]. To ex￾tend capacity, caches are stored in slower tiers such as CPU memory [13, 19, 44], distributed memory pools [16, 24, 36], or even disk [11, 18, 25]. Recent systems, e.g., CachedAt￾tention [11], overlap cache loading w… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: System Architecture of Strata 4 Strata’s Design and Implementation 4.1 Overview Motivated by challenges discussed in §3, we built Strata, a sys￾tem with two key components. The Strata Cache Controller (§4.2) manages the data plane elements throughout the mem￾ory hierar…
Figure 6
Figure 6. Figure 6: Layer-first v.s. Page-first layouts degrading the performance of both the I/O operations and concurrent computational kernels. We observe that efficient data transfer does not need to mo￾nopolize the entire GPU. Strata employs a strategy of launch￾ing a small number of…
Figure 7
Figure 7. Figure 7: Scheduling Policies of Strata, where orange blocks indicate prefill batches experiencing cache miss, green indi￾cates cache hit on device, purple indicates cache hit on host memory, blue indicates data transfer, and the one decoding batch is colored in gray. the layer-…
Figure 8
Figure 8. Figure 8: End-to-end benchmark performance comparison on H200. workloads. To understand the performance of Strata on short￾context workloads, the final row of [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Breakdown of I/O and scheduling of Strata. mechanism from §4.2, Strata-Schedule-Only, which applies the scheduling policy from §4.3, and Strata-IO-LPM, which integrates a longest prefix match (LPM) policy [45]. The results show that both the Strata-scheduling and Strat…
Figure 11
Figure 11. Figure 11: Breakdown of Strata’ optimization attributions on different workload patterns. Llama-3.1-8B Qwen-2.5-14B Llama-3.1-70B 0 1 2 3 Loading Latency (s) 1.687 1.739 2.102 0.420 0.638 1.202 Original Page-first [PITH_FULL_IMAGE:figures/full_fig_p010_11.png]
Figure 12
Figure 12. Figure 12: Latency of loading KV caches of 8192 tokens (using page size 32) of different models from a local disk to CPU memory with different memory layouts. of Strata-IO’s performance, primarily due to a 2.4% lower cache hit rate. 5.3.3 Can Strata adapt to varying cache distan…
Figure 13
Figure 13. Figure 13: A zooming in comparison between benchmarks on PCIe-5.0 and Grace-Hopper platform. 0 50 100 150 Sustained Bandwidth (GB/s) 10.80 19.43 40.30 150.50 SGLang-HiCache-PCIe SGLang-HiCache-GH Strata-IO-PCIe Strata-IO-GH [PITH_FULL_IMAGE:figures/full_fig_p011_13.png]
Figure 14
Figure 14. Figure 14: Sustained Bandwidth comparison. 5.3.4 Does the decoupled memory layout benefit disk caching? GPU-assisted I/O enables the use of a page-first layout in CPU memory without requiring changes to the GPU memory layout, as discussed in §4.2.1 [PITH_FULL_IMAGE:figures/full…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. HiSparse: Scaling Sparse-Attention Decoding with Hierarchical KV Cache Management

    cs.DC 2026-08 conditional novelty 7.0 of 10

    A hierarchical KV cache for sparse-attention LLM serving bounds each request's GPU memory by a small LRU cache and fetches misses from host memory, raising long-context decoding throughput up to 4.7x with unchanged outputs.

  2. Heterogeneous LLM Serving with General-Purpose Processing-Near-Memory for Retrieval-Based Sparse Attention

    cs.AR 2026-08 conditional novelty 7.0 of 10

    Heterogeneous serving that moves the KV cache and retrieval-based sparse attention to general-purpose processing-near-memory devices improves simulated decode throughput per TDP by 2.09-6.13x over a GPU-only baseline.

  3. vToken: Token-Level Virtualization for Reclaimable KV Caches

    cs.AI 2026-08 conditional novelty 6.0 of 10

    A translation table plus asynchronous live-token repacking lets token-granular KV eviction reclaim underutilized physical blocks in PagedAttention-style serving runtimes, improving memory efficiency and throughput in ...

  4. HBF Sucks! A Full-Stack Characterization of High-Bandwidth Flash for KV-Centric LLM Serving

    cs.AR 2026-08 conditional novelty 6.0 of 10

    Replacing an SSD KV-offload tier with High-Bandwidth Flash in an SSD-style LLM serving stack raises average end-to-end latency 2 to 5.5 times and cuts SLO goodput, because transient KV is write-heavy and off the criti...

  5. Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Serving

    cs.DC 2026-03 conditional novelty 6.0 of 10

    Pattern-aware speculative tool execution cuts agent end-to-end latency by roughly half and observed tool latency by about 1.8× by overlapping predicted tools with LLM generation.

  6. ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System

    cs.OS 2026-02 conditional novelty 6.0 of 10

    A program-aware scheduler and tool-lifecycle manager for LLM agent workflows raises serving throughput by 1.5–3.6x and RL rollout throughput by 1.8–3.9x over vLLM/Continuum baselines.

Reference graph

Works this paper leans on

47 extracted references · 31 canonical work pages · cited by 6 Pith papers

  1. [1]

    [n. d.]. Artificial Analysis Model Leaderboards. https:// artificialanalysis.ai/leaderboards/models

    work page

  2. [2]

    Advanced Micro Devices, Inc. 2025. ROCm™ Software 6.4.3 Documen- tation. https://rocm.docs.amd.com/en/latest/. Accessed: 2025-08-20

    work page 2025

  3. [3]

    Anthropic. 2024. Claude 3.5 Sonnet.https://www.anthropic.com/news/ claude-3-5-sonnet

    work page 2024

  4. [4]

    Anthropic. 2024. Prompt Caching. https://docs.anthropic.com/en/ docs/build-with-claude/prompt-caching . Accessed: 2025-08-18

    work page 2024

  5. [5]

    Nirav Atre, Justine Sherry, Weina Wang, and Daniel S. Berger. 2020. Caching with Delayed Hits. In Proceedings of the Annual Conference of the ACM Special Interest Group on Data Communication on the Applica- tions, Technologies, Architectures, and Protocols for Computer Communi- cation (Virtual Event, USA)(SIGCOMM ’20). Association for Computing Machinery,...

    arXiv 2020

  6. [6]

    Qizhe Cai, Shubham Chaudhary, Midhul Vuppalapati, Jaehyun Hwang, and Rachit Agarwal. 2021. Understanding host network stack over- heads. In Proceedings of the 2021 ACM SIGCOMM 2021 Conference (Virtual Event, USA) (SIGCOMM ’21). Association for Computing Ma- chinery, New York, NY, USA, 65–77. doi:10.1145/3452296.3472888

    arXiv 2021

  7. [7]

    LMDeploy Contributors. 2023. LMDeploy: A Toolkit for Compressing, Deploying, and Serving LLM. https://github.com/InternLM/lmdeploy

    work page 2023

  8. [8]

    DeepSeek. [n. d.]. Prompt caching. https://api-docs.deepseek.com/ guides/kv_cache

    work page

Show all 47 references
  1. [9]

    DeepSeek-AI. 2025. DeepSeek-V3 Technical Report. arXiv:2412.19437 [cs.CL] https://arxiv.org/abs/2412.19437

    2025 arXiv

  2. [10]

    Assaf Eisenman, Asaf Cidon, Evgenya Pergament, Or Haimovich, Ryan Stutsman, Mohammad Alizadeh, and Sachin Katti. 2019. Flashield: a hybrid key-value cache that controls flash write amplification. In 16th USENIX Symposium on Networked Systems Design and Implementation (NSDI 19). 65–78

    2019

  3. [11]

    Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, and Pengfei Zuo. 2024. Cost- efficient large language model serving for multi-turn conversations with CachedAttention. In Proceedings of the 2024 USENIX Conference on Usenix An...

    2024

  4. [12]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. 2023. Retrieval- augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997 2, 1 (2023)

    2023 arXiv

  5. [13]

    In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khan- delwal, and Lin Zhong. 2024. Prompt Cache: Modular Attention Reuse for Low-Latency Inference. In Proceedings of Machine Learning and Systems, P. Gibbons, G. Pekhimenko, and C. De Sa (Eds.), Vol. 6. 325–338. https:...

    2024

  6. [14]

    Google. [n. d.]. Prompt caching. https://cloud.google.com/vertex- ai/generative-ai/docs/context-cache/context-cache-overview

  7. [15]

    Google DeepMind. 2023. Gemini: Google DeepMind’s Most Capa- ble and General AI Models. https://deepmind.google/technologies/ gemini/

    2023

  8. [16]

    Cunchen Hu, Heyang Huang, Junhao Hu, Jiang Xu, Xusheng Chen, Tao Xie, Chenxi Wang, Sa Wang, Yungang Bao, Ninghui Sun, and Yizhou Shan. 2024. MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool. arXiv:2406.17565 [cs.DC] https://arxiv.org/abs/2406.17565

    2024 arXiv

  9. [17]

    2023.{ARK}:{GPU-driven} code execution for distributed deep learning

    Changho Hwang, KyoungSoo Park, Ran Shu, Xinyuan Qu, Peng Cheng, and Yongqiang Xiong. 2023.{ARK}:{GPU-driven} code execution for distributed deep learning. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23) . 87–101

    2023

  10. [18]

    Jinwoo Jeong and Jeongseob Ahn. 2025. Accelerating LLM Serving for Multi-turn Dialogues with Efficient Resource Management. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (Rotterdam, Ne...

    2025

  11. [19]

    Chao Jin, Zili Zhang, Xuanlin Jiang, Fangyue Liu, Xin Liu, Xuanzhe Liu, and Xin Jin. 2024. RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation. arXiv:2404.12457 [cs.DC] https: //arxiv.org/abs/2404.12457

    2024 arXiv

  12. [20]

    Shuowei Jin, Xueshen Liu, Qingzhao Zhang, and Z Morley Mao

  13. [21]

    Tomáš Kočisk `y, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018. The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguistics 6 (2018), 317–328

    2018

  14. [22]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Sto- ica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Princ...

    2023

  15. [23]

    Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. 2023. LooGLE: Can Long-Context Language Models Understand Long Con- texts? arXiv preprint arXiv:2311.04939 (2023)

    2023 arXiv

  16. [24]

    Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Anantha- narayanan, et al. 2024. Cachegen: Kv cache compression and streaming for fast large language model serving. In Proceedings of the ACM SIG- COMM 2024 Co...

    2024

  17. [25]

    LMCache. 2025. LMCache. https://lmcache.ai. Accessed: 2025-05-14

    2025

  18. [26]

    Lingxiao Ma, Zhiqiang Xie, Zhi Yang, Jilong Xue, Youshan Miao, Wei Cui, Wenxiang Hu, Fan Yang, Lintao Zhang, and Lidong Zhou. 2020. Rammer: Enabling Holistic Deep Learning Compiler Optimizations with rTasks. In 14th USENIX Symposium on Operating Systems Design and Implementati...

    2020

  19. [27]

    Berger, Nathan Beck- mann, and Gregory R

    Sara McAllister, Benjamin Berg, Julian Tutuncu-Macias, Juncheng Yang, Sathya Gunasekar, Jimmy Lu, Daniel S. Berger, Nathan Beck- mann, and Gregory R. Ganger. 2021. Kangaroo: Caching Billions of Tiny Objects on Flash. In Proceedings of the ACM SIGOPS 28th Symposium on Operating...

    2021

  20. [28]

    meta ai. 2024. Introducing Llama 3.1: Our most capable models to date. https://ai.meta.com/blog/meta-llama-3-1/

    2024

  21. [29]

    meta ai. 2025. The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation. https://ai.meta.com/blog/llama- 4-multimodal-intelligence/

    2025

  22. [30]

    Seung Won Min, Vikram Sharma Mailthody, Zaid Qureshi, Jinjun Xiong, Eiman Ebrahimi, and Wen-mei Hwu. 2020. EMOGI: efficient memory-access for out-of-memory graph-traversal in GPUs. Proc. VLDB Endow. 14, 2 (Oct. 2020), 114–127. doi:10.14778/3425879.3425883

    2020

  23. [31]

    NVIDIA. 2024. 5x Faster Time to First Token with NVIDIA TensorRT- LLM KV Cache Early Reuse. https://developer.nvidia.com/blog/5x- faster-time-to-first-token-with-nvidia-tensorrt-llm-kv-cache-early- reuse/. Accessed: 2025-05-01

    2024

  24. [32]

    NVIDIA. 2025. NVIDIA GH200 Grace Hopper Superchip. https://www. nvidia.com/en-us/data-center/grace-hopper-superchip/ . Accessed: 2025-05-14

    2025

  25. [33]

    NVIDIA. 2025. NVIDIA TensorRT-LLM. https://docs.nvidia.com/ tensorrt-llm/index.html. Accessed: 2025-05-14

    2025

  26. [34]

    NVIDIA Corporation. 2025. Parallel Thread Execution ISA . https: //docs.nvidia.com/cuda/parallel-thread-execution/ Accessed: Aug. 21, 2025

    2025

  27. [35]

    OpenAI. [n. d.]. Prompt caching. https://platform.openai.com/docs/ guides/prompt-caching

  28. [36]

    Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2025. Moon- cake: Trading More Storage for Less Computation — A KVCache- centric Architecture for Serving LLM Chatbot. In 23rd USENIX Confer- ence on File and Stora...

    2025

  29. [37]

    Zaid Qureshi, Vikram Sharma Mailthody, Isaac Gelado, Seungwon Min, Amna Masood, Jeongmin Park, Jinjun Xiong, C. J. Newburn, Dmitri Vainbrand, I-Hsin Chung, Michael Garland, William Dally, and Wen-mei Hwu. 2023. GPU-Initiated On-Demand High-Throughput Storage Access in the BaM ...

    2023

  30. [38]

    Cheng Tan, Dongxin Lyu, Siyuan Li, Zhangyang Gao, Jingxuan Wei, Siqi Ma, Zicheng Liu, and Stan Z Li. 2024. Peer review as a multi-turn and long-context dialogue with role-based interactions. arXiv preprint arXiv:2406.05688 (2024)

    2024 arXiv

  31. [39]

    Qwen Team. 2025. Qwen2.5-1M: Deploy Your Own Qwen with Context Length up to 1M Tokens. https://qwenlm.github.io/blog/qwen2.5-1m/

    2025

  32. [40]

    The SGLang Team. 2024. SGLang v0.4: Zero-Overhead Batch Scheduler, Cache-Aware Load Balancer, Faster Structured Outputs. https://lmsys. org/blog/2024-12-04-sglang-v0-4/

    2024

  33. [41]

    vLLM Contributors. 2025. vLLM Configuration API Reference. https: //docs.vllm.ai/en/latest/api/vllm/vllm.config.html. Accessed: 2025-05- 01

    2025

  34. [43]

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A Distributed Serving System for Transformer-Based Generative Models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). USENIX Asso- ciation, Carlsbad, C...

    2022

  35. [44]

    Lingfan Yu, Jinkun Lin, and Jinyang Li. 2025. Stateful Large Lan- guage Model Serving with Pensieve. In Proceedings of the Twentieth European Conference on Computer Systems (Rotterdam, Netherlands) (EuroSys ’25). Association for Computing Machinery, New York, NY, USA, 144–158....

    2025

  36. [45]

    Gonzalez, Clark Barrett, and Ying Sheng

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Sto- ica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. 2024. SGLang: Efficient Execution of Structured Language Model Pro- grams. In Advances in Neural In...

    2024

  37. [46]

    2024.{DistServe}: Disaggregating prefill and decoding for goodput-optimized large language model serving

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xu- anzhe Liu, Xin Jin, and Hao Zhang. 2024.{DistServe}: Disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation ...

    2024

  38. [47]

    2025.{NanoFlow}: Towards Optimal Large Language Model Serving Throughput

    Kan Zhu, Yufei Gao, Yilong Zhao, Liangyu Zhao, Gefei Zuo, Yile Gu, Dedong Xie, Zihao Ye, Keisuke Kamahori, Chien-Yu Lin, et al . 2025.{NanoFlow}: Towards Optimal Large Language Model Serving Throughput. In 19th USENIX Symposium on Operating Systems Design and Implementation (O...

    2025

  39. [2024]

    Compute or load kv cache? why not both? arXiv preprint arXiv:2410.03065 (2024)

    2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.