Pith. sign in

REVIEW 4 major objections 4 minor 39 references

vToken: Token-Level Virtualization for Reclaimable KV Caches

T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read vToken adds a token-level virtualization layer that decouples KV token liveness from physical block placement, reclaiming up to 72.3% of retained KV blocks and doubling feasible concurrency without changing attention kernels.

desk verdict The indirection-layer design is genuinely new and the paired evaluation is clean, but the headline numbers compare against a strawman baseline while the cited state of the art goes unmeasured. read the letter →

arxiv 2608.13263 v1 pith:YICLPSJF submitted 2026-08-13 cs.AI cs.DCcs.OS

Yuanhang Gao , Xiangrui Yang , Yuanfeng Chen , Hongjia Chen , Qianru Lv , Wenfei Wu , Dongsheng Li This is my paper · ORCID
classification cs.AIcs.DCcs.OS
keywords KVcachemanagementtoken-levelevictionblockfragmentationmemoryvirtualizationLLMinferencePagedAttentionlazycompactionCUDAGraph
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

vToken is a runtime layer that fixes a granularity mismatch: token-level cache eviction decides which tokens are dead, but block-based serving runtimes can only free whole blocks, so blocks that still hold any live token stay allocated even if most of their slots are holes. The paper proposes a per-sequence token table that maps logical token identities to physical slots and a reclamation backend that asynchronously repacks live tokens out of underutilized blocks, turning logical liveness into reusable physical memory. On paired tests where both variants make identical eviction decisions, vToken reduces retained KV blocks per request by 27.2%–72.3%, raises SLA-constrained throughput by up to 1.37x, and extends the maximum feasible concurrency by up to 2x under a fixed KV budget. If true, this means existing block-based serving systems can get the memory benefits of token-level eviction without changing attention kernels or giving up CUDA Graph execution.

What carries the argument

The token table is the central object: a per-sequence array that maps each logical token ID to a physical location (block ID, offset) and a liveness bit. It exposes three operations—mark a token evicted, register newly generated tokens, and apply a list of moves after relocation copies finish—so eviction policies never touch blocks and the runtime never reasons about token importance. Around it sits a lazy-compaction reclamation backend that monitors per-block live-token counts, admits relocation plans only when projected block reduction is positive and destination headroom exists, and copies KV entries asynchronously after the current forward pass, using a CUDA event to make relocated data visible to the next attention kernel only when needed. The combination turns token liveness into reclaimable physical capacity without modifying attention kernels.

What would settle it

Run an implemented page-aligned or bounded-budget reclamation system under the same eviction policies, models, and KV budgets; if retained blocks per request drop as much without a token-table indirection layer, the claimed capability gap does not exist.

Watch

Extended reading notes

Core claim

The paper's central claim is that token-level KV eviction delivers its memory savings only when a runtime boundary decouples logical token liveness from physical block placement; without that boundary, the savings stay trapped as intra-block fragmentation. vToken realizes the boundary with a logical address space per request: policies call an evict-token operation that marks a token dead in a token table while the KV entry remains physically in place, and a physical reclamation backend later compacts live tokens from low-utilization blocks into destination blocks, updates the table, and returns emptied blocks to the allocator. Relocation copies run after the current decoding step on a separate stream, with a CUDA-event dependency guarding the next attention launch, which preserves the existing slot-mapping mechanism and CUDA Graph replay. The paper argues this is a missing abstraction layer, not merely an engine-specific optimization, because the same hooks port to any PagedAttention-style runtime.

Load-bearing premise

The load-bearing premise is that both tested variants differ only in the reclamation layer, so the measured block savings come from reclamation rather than from some other change; if a competing system already recovers most partially live blocks, the headline gains would be smaller.

Editorial extensions

If this is right

  • A block-based serving system can host token-level eviction policies without redesigning its allocator or attention kernels, so H2O-, StreamingLLM-, and Scissorhands-style policies become drop-in modules.
  • Under memory pressure, fewer retained blocks per request means more concurrent requests fit in the same KV block pool, directly raising the feasible concurrency frontier.
  • Because reclamation is deferred and batched, eviction cost is paid only when fragmentation is actionable, keeping the steady-state decoding path nearly free.
  • The benefit is workload- and policy-dependent: it is largest when live tokens are scattered (Random, Scissorhands) and smaller when retention is structured (H2O), so the layer matters most where policies already reduce logical KV demand aggressively.
  • vToken is a pressure-activated extension: the native full-KV path remains the right choice when memory is not the bottleneck.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If measured against a competing runtime that already relocates live tokens out of partial blocks, the reported 27.2%–72.3% reduction may shrink; the paper's contribution is the missing boundary, not the relocation trick itself.
  • The logical-token view suggests the same layer could manage KV tiering or offload: liveness is already separated from placement, so deciding which physical storage tier a token occupies becomes a placement policy.
  • Composing the layer with representation-changing compression (quantization, mixed precision, depth compression) seems natural because the token table is independent of the KV tensor's physical shape, though the paper evaluates only uniform full-precision KV.
  • A cheap test of transferability: enable the layer with a very low eviction ratio and dense blocks; near-zero block reduction would confirm the gains scale with fragmentation rather than with indirection overhead.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes vToken, a token-level virtualization layer for block-managed KV caches in LLM serving. It introduces a per-request token table that decouples logical token liveness from physical block placement, and a reclamation backend that asynchronously repacks live tokens from partially live blocks and returns freed blocks to the allocator. The design is implemented on vLLM and evaluated with H2O, Random, and Scissorhands policies on Mistral-7B, Llama-3.1-8B, and Qwen2.5-14B. The headline claims are that vToken reduces retained KV blocks per request by 27.2%-72.3%, improves SLA-constrained throughput by up to 1.37x, extends maximum feasible concurrency by up to 2x, preserves PagedAttention kernels and CUDA Graph compatibility, and reduces per-policy integration footprint from 500+ to under 50 lines.

Significance. If validated, vToken addresses a real and underappreciated gap: token-level eviction policies are semantically finer than the block-level reclamation interface of PagedAttention-style runtimes, so token-level decisions leave physical capacity trapped in partially live blocks. The paired methodology cleanly isolates the effect of the reclamation backend from the eviction policy, the capacity-frontier result is replicated on a third model (Qwen2.5-14B), and the CUDA Graph compatibility argument is concrete. However, the quantitative advantage over existing reclamation-capable systems is not demonstrated, and the paper contains several internal inconsistencies in the headline numbers. The value of the abstraction is plausible, but the evidence as presented does not support all of the claims made in the abstract and conclusion.

major comments (4)
  1. [§7 / §5.1] The evaluation compares vToken only against Native vLLM and Naive-Evict, both of which retain partially live blocks because they perform no physical reclamation. Section 7 names three systems that already perform block-level compaction or page-aligned reclamation—PagedEviction, DiffKV, and Zipage—but none is implemented or measured. Since Naive-Evict deliberately disables any reclamation, the reported 27.2%-72.3% block reduction and up to 2x concurrency extension show the gap between no reclamation and any reclamation, not a capability advantage over systems that already recover partially live blocks. This is load-bearing because the abstract and conclusion present these numbers as the primary evidence of practical value. The authors should either implement and measure at least one state-of-the-art reclamation baseline, or substantially weaken the claim to an isolation of the reclamation mechanism rather than a comparison against existing systems.
  2. [Abstract vs §1 vs §5.3] The headline numbers are internally inconsistent. The abstract and conclusion state retained blocks are reduced by 27.2%-72.3%, but §1 states 16%-85%. The abstract states SLA-constrained throughput improvement of up to 1.37x, while §5.3 reports Scissorhands throughput gains of 33.3%-103.7%, which is up to roughly 2.04x, and §5.4 reports concurrency extension of up to 2x. These ranges cannot all be correct without explicit qualifiers defining the experimental setup for each number. The authors need to reconcile the ranges and make clear which policies, models, and workloads each number corresponds to.
  3. [§5.3, Fig. 8] Most headline quantitative claims are presented without uncertainty quantification. Figures 7 and 8 and the throughput/latency percentages in §5.3 are point estimates with no error bars or run counts; only the capacity-frontier experiment in §5.4 states that throughput is averaged over three runs. Given that SLA-constrained throughput and p95 latency are noisy metrics, the claims of 'up to 1.37x' throughput and 'up to 2x' concurrency need either confidence intervals or repeated-run data to be properly evaluated.
  4. [§5.6] The fragmentation threshold θ_F=0.25 is chosen by empirical exploration on the evaluation workloads and then used in all default runs. The sensitivity analysis in Fig. 12 shows throughput is relatively stable across the threshold range, which mitigates the concern, but retained KV capacity does move with the threshold. The paper should state explicitly whether the headline block-reduction and concurrency results persist when θ_F is re-selected per workload, or report the headline metrics across the full θ_F range.
minor comments (4)
  1. [§2.1] The arithmetic '2×40×5120×2 B≈0.8 MB' gives 819,200 bytes, which is approximately 0.78 MiB; the units should be stated consistently.
  2. [§5.7, Table 1] The degree-1 row reports ThroughputΔ=-2.9% while the text says the degree-1 case 'shows only the small overhead'; the sign convention for Δ (negative meaning throughput decrease) should be clarified in the caption or text.
  3. [General] The paper provides no artifact, repository link, or instructions for reproducing the experiments; a code-release statement would strengthen the reproducibility of the claims.
  4. [Figure 1] The preliminary experiment in Fig. 1 reports low-utilization block fractions without error bars or run counts; a sentence describing replication would help the reader assess the 40%-60% waste claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's central results are paired empirical measurements, and the one tuned threshold has a demonstrated modest effect.

full rationale

vToken's load-bearing claims are system measurements, not deductions whose conclusions are assumed in their inputs. The main comparison fixes identical token-level eviction decisions across Naive-Evict and vToken, differing only in whether token-table indirection and physical reclamation are enabled. The direction of the block-reduction result is expected from the experimental construction, but the reported magnitudes (27.2%-72.3% block reduction, up to 1.37x throughput, up to 2x concurrency) are measured outcomes rather than consequences forced by an equation or by a fitted parameter. The only runtime parameter tuned on the evaluation workloads is the fragmentation threshold theta_F=0.25, chosen by the empirical exploration in §5.6 and used in the default runs; however, the paper's own sensitivity sweep shows that 'vToken does not require fine-grained trigger tuning,' so this does not make the headline results fitted predictions in any load-bearing sense. No self-citation chain or imported uniqueness theorem is used to justify the central design choice, and the related-work discussion treats PagedEviction, DiffKV, and Zipage as alternatives without claiming they are invalid. The absence of implemented state-of-the-art baselines is a substantive evidence-quality concern, but it is not circularity: the paper's claims are not defined into existence by its inputs.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The paper is an engineering contribution, so most background consists of domain assumptions about existing systems rather than mathematical axioms. The central claim rests on the PagedAttention/vLLM substrate, CUDA stream/event semantics, and the premise that token-level eviction decisions do not interact with relocation. The only fitted numerical parameter is the fragmentation threshold theta_F=0.25, chosen on the evaluation workloads. No new physical or mathematical entities are postulated; the token table and reclamation backend are software components, not theoretical entities.

free parameters (1)
  • fragmentation threshold theta_F = 0.25
    Set through empirical exploration on the same workloads in §5.6 and used as the default in all main comparisons. Sensitivity analysis shows the threshold has a modest effect, but it is still a fitted knob rather than a derived constant.
assumptions (4)
  • domain assumption The PagedAttention/vLLM block substrate provides the only physical KV allocation interface, and attention kernels consume mutable slot mappings without modification.
    vToken's compatibility claims rely on the block semantics and slot-mapping interface described in §2.1 and §3.1.
  • domain assumption CUDA stream/event semantics guarantee that relocated KV data are visible to a dependent attention kernel after a stream-level wait_event.
    Invariant I3 in §3.5 is asserted as structural, and the pre-attention synchronization hook in §3.4 depends on this without an empirical race test under contention.
  • domain assumption Token-level eviction decisions and their effect on output quality are independent of the physical placement and repacking of tokens.
    Section 5.8 delegates any task-quality change to the eviction policy and validates only that vToken matches Naive-Evict, not unmodified full-retention serving.
  • domain assumption The paired Naive-Evict baseline differs from vToken only in token-table indirection and the physical reclamation backend.
    Section 5.1 defines the paired protocol, but vToken also changes slot mapping and scheduler hooks; if those alter scheduling order, the comparison would conflate effects.

how reviews work

0 comments
Cite this review

Pith. "Pith review of vToken: Token-Level Virtualization for Reclaimable KV Caches." pith.science (2026). https://pith.science/paper/YICLPSJF

@misc{pith2026260813263,
  author       = {Pith},
  title        = {Pith review of: vToken: Token-Level Virtualization for Reclaimable KV Caches},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YICLPSJF}},
  note         = {Machine review of arXiv:2608.13263}
}
abstract

Large language model serving faces a critical memory bottleneck: the KV cache grows with sequence length and batch size. PagedAttention uses fixed-size memory blocks to reduce allocator-level fragmentation, but recent KV eviction algorithms operate at a token granularity finer than block-level management. This mismatch causes intra-block fragmentation, leaving a large fraction of allocated KV memory unreclaimable. We present vToken, a lightweight token-level virtualization layer that decouples logical token liveness from physical block placement. vToken maintains a stable logical token view through token-table indirection and realizes physical reclamation by repacking live tokens asynchronously. The design preserves PagedAttention kernels and CUDA Graph compatibility. We implement vToken in vLLM and evaluate it with H2O, Random, and Scissorhands across models. Compared with a paired Naive-Evict baseline, vToken reduces retained KV blocks per request by 27.2\%--72.3\% and improves SLA-constrained throughput by up to 1.37$\times$. Under a constrained active-KV budget, it extends the maximum feasible concurrency by up to 2$\times$, while reducing the per-policy integration footprint from 500+ lines to under 50.

Figures

Figures reproduced from arXiv: 2608.13263 by the authors.

Figure 1
Figure 1. Low-utilization KV blocks under token-level evic￾tion. Bars show allocated blocks with ≤50% utilization, split into 0–25% and 25–50% bins; such partially live blocks re￾main unreclaimable in a block-granular runtime. Results [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Token table as virtualization metadata for the logical token address space (§3.2); a physical reclamation backend batches reclamation in the background to satisfy C3 and to drive the asynchronous copies of C2 (§3.3); and reclamation-aware scheduler hooks insert the slot-mapping refresh and CUDA-event dependency that close C2’s safety requirement (§3.4). 3.2 Token Table and Logical Address Space The token table is th… view at source ↗
Figure 4
Figure 4. Physical reclamation workflow The canonical table lives on the CPU for simplicity and compatibility, while a GPU-resident lookup cache acceler￾ates slot translation in the steady-state decoding path. After eviction or relocation, the cached translation array is re￾freshed before constructing the next attention slot mapping. When a sequence grows append-only, the cache is updated by appending the new entries; a full … view at source ↗
Figures from the paper (8 more)
Figure 5
Figure 5. Figure 5: Stage-aware asynchronous copy. vToken avoids putting KV copy on the critical path and reduces HBM con￾tention by launching copy after forward; a CUDA event guards the next attention step only when needed. is HBM-bandwidth bound; (b) FFN forward, which is compute bound;…
Figure 6
Figure 6. Figure 6: Policy-adapter procedure and integration footprint in vToken. 5 Evaluation Our evaluation answers six questions: whether token-level eviction leaves physical capacity trapped in partially live blocks (§5.2), whether vToken improves the paired eviction frontier under id…
Figure 7
Figure 7. Figure 7: Memory efficiency under token-level eviction. vToken improves effective KV cache usage by reclaiming partially live blocks that remain allocated in Naive-Evict. 200 225 250 Throughput (tokens/s) H2O, ShareGPT 120 140 160 H2O, LongBench 150 175 200 225 Random, ShareGPT …
Figure 8
Figure 8. Figure 8: SLA-constrained throughput frontier on Mistral-7B under token-level eviction. The SLA threshold is 1.05× the p95 latency of Naive-Evict at 𝐶ref = 8, and the star marks 𝐶sel, the maximum-throughput feasible point. vLLM is the unmodified full-retention serving baseline w…
Figure 9
Figure 9. Figure 9: Active-KV capacity frontier at gpu_mem_util=0.35. vToken extends the verified feasible region from 𝐶 = 5 to 𝐶 = 8; × marks infeasible points. 1 2 3 4 5 6 7 8 12 Active concurrency C 0 50 100 150 200 KV block demand (%) 100% KV block budget C=5 C=8 Native vLLM Naive-Evi…
Figure 10
Figure 10. Figure 10: KV-block demand normalized by usable budget. At gpu_mem_util=0.35 (left) and 0.50 (right), vToken ex￾tends the verified boundary from 𝐶 = 5 to 𝐶 = 8 and from 𝐶 = 11 to 𝐶 = 22. to larger batches. Thus, the SLA-frontier gain comes from reclaiming capacity that already e…
Figure 11
Figure 11. Figure 11: Physical reclamation overhead and overlap. [PITH_FULL_IMAGE:figures/full_fig_p010_11.png]
Figure 12
Figure 12. Figure 12: Sensitivity analysis of block size, eviction ratio, and fragmentation threshold under H2O. overlaps copy with decode, but copy/decode contention can appear under heavier operating pressure. 5.6 Sensitivity Analysis We use one-factor-at-a-time experiments to evaluate h…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 15 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al . 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774(2023)

    arXiv 2023

  2. [2]

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. 2023. GQA: Training General- ized Multi-Query Transformer Models from Multi-Head Checkpoints. arXiv:2305.13245 [cs.CL]https://arxiv.org/abs/2305.13245

    arXiv 2023

  3. [3]

    Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, et al. 2022. Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale. InSC22: International Conference for High Performance Computing, Networking, Storage and Analysis....

    work page 2022

  4. [4]

    Anthropic. 2026. Claude Code.https://github.com/anthropics/claude- code. GitHub repository, accessed April 22, 2026

    work page 2026

  5. [5]

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding. arXiv:2308.14508 [cs.CL] https://arxiv.org/abs/2308.14508

    arXiv 2024

  6. [6]

    Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Yucheng Li, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Junjie Hu, et al. 2024. Pyramidkv: Dynamic kv cache compression based on pyramidal information fun- neling.arXiv preprint arXiv:2406.02069(2024)

    arXiv 2024

  7. [7]

    Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot im- pressing gpt-4 with 90%* chatgpt quality.See https://vicuna. lmsys. org (accessed 14 April 2023)2, 3 (2023), 6

    work page 2023

  8. [8]

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas An- gelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al . 2024. Chatbot arena: An open platform for evaluating llms by human preference.arXiv preprint arXiv:2403.04132(2024)

    arXiv 2024

Show all 39 references
  1. [9]

    Krishna Teja Chitty-Venkata, Jie Ye, Siddhisanket Raskar, Anthony Kougkas, Xian Sun, Murali Emani, Venkatram Vishwanath, and Bog- dan Nicolae. 2026. PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inference. InFindings of the Associatio...

    2026 doi

  2. [10]

    Dom Eccleston. 2023. ShareGPT: Share your ChatGPT conversations with one click.https://github.com/domeccleston/sharegpt. GitHub repository

    2023

  3. [11]

    Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. 2024. Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs. InThe Twelfth International Conference on Learning Representations (ICLR).https://arxiv.org/abs/2310.01801

    2024 arXiv

  4. [12]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. 2025. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning.Nature645, 8081 (2025), 633–638

    2025

  5. [13]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yifan Wu, YK Li, et al. 2024. DeepSeek- Coder: when the large language model meets programming–the rise of code intelligence.arXiv preprint arXiv:2401.14196(2024)

    2024 arXiv

  6. [14]

    Yefei He, Luoming Zhang, Weijia Wu, Jing Liu, Hong Zhou, and Bohan Zhuang. 2024. Zipcache: Accurate and efficient kv cache quantization with salient token identification.Advances in Neural Information Processing Systems37 (2024), 68287–68307

    2024

  7. [15]

    Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Ma- honey, Yakun S Shao, Kurt Keutzer, and Amir Gholami. 2024. Kvquant: Towards 10 million context length llm inference with kv cache quan- tization.Advances in Neural Information Processing Systems37 (2024), 1270–1303

    2024

  8. [16]

    Tom Kilburn, David BG Edwards, Michael J Lanigan, and Frank H Sumner. 2009. One-level storage system.IRE Transactions on Electronic Computers2 (2009), 223–235

    2009

  9. [17]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica

  10. [18]

    Mengqi Liao, Lu Wang, Chaoyun Zhang, Bo Qiao, Si Qin, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, and Huaiyu Wan. 2026. Zipage: Maintain High Request Concurrency for LLM Reasoning through 13 Gao et al. Compressed PagedAttention. arXiv:2603.08743 [cs.DC]https://arxiv. org/abs...

    2026

  11. [19]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al . 2024. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437(2024)

    2024 arXiv

  12. [20]

    Akide Liu, Jing Liu, Zizheng Pan, Yefei He, Gholamreza Haffari, and Bohan Zhuang. 2024. Minicache: Kv cache compression in depth dimension for large language models.Advances in Neural Information Processing Systems37 (2024), 139997–140031

    2024

  13. [21]

    Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Anantha- narayanan, et al. 2024. Cachegen: Kv cache compression and stream- ing for fast large language model serving. InProceedings of the ACM SIGCOMM 2024 Con...

    2024

  14. [22]

    Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. 2023. Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time.Advances in Neural Information Pr...

    2023

  15. [23]

    Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. 2024. Kivi: A tuning- free asymmetric 2bit quantization for kv cache.arXiv preprint arXiv:2402.02750(2024)

    2024 arXiv

  16. [24]

    ModelTC. 2026. LightLLM.https://github.com/ModelTC/lightllm. GitHub repository, accessed April 22, 2026

    2026

  17. [25]

    NVIDIA. 2026. TensorRT-LLM.https://github.com/NVIDIA/TensorRT- LLM. GitHub repository, accessed April 22, 2026

    2026

  18. [26]

    OpenAI. 2026. Codex: Lightweight Coding Agent.https://github.com/ openai/codex. GitHub repository, accessed April 22, 2026

    2026

  19. [27]

    OpenClaw Contributors. 2026. OpenClaw: Personal AI Assistant Gateway.https://github.com/openclaw/openclaw. GitHub repository, accessed April 22, 2026

    2026

  20. [28]

    Ramya Prabhu, Ajay Nayak, Jayashree Mohan, Ramachandran Ramjee, and Ashish Panwar. 2025. vattention: Dynamic memory management for serving llms without pagedattention. InProceedings of the 30th ACM International Conference on Architectural Support for Program- ming Languages a...

    2025

  21. [29]

    Noam Shazeer. 2019. Fast Transformer Decoding: One Write-Head is All You Need. arXiv:1911.02150 [cs.NE]https://arxiv.org/abs/1911. 02150

    2019 arXiv

  22. [30]

    Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. 2023. Flexgen: High-throughput generative inference of large language models with a single gpu. InInternational Conference on Machine Learning....

    2023

  23. [31]

    Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. 2024. Quest: Query-aware sparsity for efficient long- context llm inference.arXiv preprint arXiv:2406.10774(2024)

    2024 arXiv

  24. [32]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2023. Attention Is All You Need. arXiv:1706.03762 [cs.CL]https://arxiv.org/ abs/1706.03762

    2023 arXiv

  25. [33]

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2023. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453(2023)

    2023 arXiv

  26. [34]

    Zhiqiang Xie, Ziyi Xu, Mark Zhao, Yuwei An, Vikram Sharma Mailthody, Scott Mahlke, Michael Garland, and Christos Kozyrakis

  27. [35]

    Yanqi Zhang, Yuwei Hu, Runyuan Zhao, John C. S. Lui, and Haibo Chen. 2025. DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compaction. InProceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles(Lotte Hotel World, Seoul, R...

    2025

  28. [36]

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. 2023. H2o: Heavy-hitter oracle for efficient generative inference of large language models.Advances in Neural Information Processing ...

    2023

  29. [37]

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody H Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. 2024. Sglang: Efficient execution of structured lan- guage model programs.Advances in neural information processing systems37 (2...

    2024

  30. [2023]

    InProceedings of the 29th symposium on operating systems principles

    Efficient memory management for large language model serving with pagedattention. InProceedings of the 29th symposium on operating systems principles. 611–626

  31. [2025]

    arXiv:2508.18572 [cs.DC]https://arxiv.org/abs/ 2508.18572

    Strata: Hierarchical Context Caching for Long Context Lan- guage Model Serving. arXiv:2508.18572 [cs.DC]https://arxiv.org/abs/ 2508.18572

    arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.