Pith. sign in

REVIEW 4 major objections 6 minor 44 references

The key-value entries a long-context LLM needs for its next token can be predicted from a speculative token, letting the system prefetch them before attention runs.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 15:01 UTC pith:XT4LTWHM

load-bearing objection Genuine systems contribution but overclaimed novelty and a measurement gap in the central predictability claim. the 4 major comments →

arxiv 2607.26475 v1 pith:XT4LTWHM submitted 2026-07-29 cs.DC

DualDecoder: Accelerate Long Context LLM Inference by Predictive Prefetch

Zuning Liang , Zhiyi Yao , Qi Chen , Yuedong Xu , Hao Dai , Zhiqiang Ding , Tongkai Yang , Jinlong Hou
show 1 more author
Yuan Cheng
This is my paper
classification cs.DC
keywords long-context LLM inferenceKV cache offloadingsparse attentionspeculative decodingprefetchingGPU memoryserving throughputdual-token pipeline
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper sets out to establish that the small set of cached key-value (KV) entries a long-context LLM will attend to in the next decoding step can be predicted one step in advance, from a speculative token generated alongside the real output token. If that predictability holds, the expensive host-to-device transfer of those entries can be issued early and hidden under GPU computation, so a sparse KV cache can stay mostly in host memory without the GPU-resident selection or reconstruction state that current systems need. The paper's DualDecoder implements this with a dual-token decoding pipeline that co-computes the speculative token, a layer-aware transfer schedule, and a two-layer ping-pong buffer, and reports up to 2.62x decoding throughput over state-of-the-art long-context serving systems at preserved latency and quality. A sympathetic reader would care because the claimed mechanism attacks the memory wall that currently caps request batch size in long-context serving, converting scarce GPU memory into larger batches rather than auxiliary index state.

Core claim

DualDecoder's central claim is that in dynamic sparse KV cache serving, the retrieval index for step S_{i+1}—the set of historical key-value entries attention will need—can be computed from a speculated token T'_{i+1} plus the static GPU-resident landmarks, before the true token T_{i+1} is known. Because the speculative token is generated in the same pass as the real token, the system can co-compute it with little extra cost, then issue host-to-device fetches for the predicted entries early. The paper shows retrieval-index prediction accuracy averaging 88% even when token prediction itself is imperfect, and argues that with a prioritized urgent stream for the 12% misses and selective low-ran

What carries the argument

The load-bearing mechanism is the dual-token decoding pipeline: within one decoding step, the model emits both the real output token and a speculative token for the position immediately after, using a causal mask so the real token is unaffected by the speculative one. Because the next step's KV retrieval index is a function of the token being decoded and fixed landmarks, the speculative token converts an unavailable future value into a computable prediction. Around it sit a layer-aware transfer schedule that starts prefetching for layer i+2 after layer i's attention finishes, an urgent-stream mechanism that fetches missed entries before attention consumes them, and a two-layer ping-pong spar

Load-bearing premise

The load-bearing premise is that a speculative token generated in the same decoding pass, without seeing the true next token, selects nearly the same sparse KV retrieval index as the true next token would; the paper's reported accuracy comes from initializing the first speculative token with the ground-truth second token, and the production choice of a model-generated first token is never directly measured.

What would settle it

Measure retrieval-index overlap under DualDecoder's actual production initialization—pre-decoding step produces the first speculative token from the model itself—over many long-context decoding traces. If average overlap lands near the 56% seen with random initialization instead of the 82-88% reported with ground-truth initialization, most prefetches would miss, urgent fetches would dominate, and the throughput benefit would largely disappear.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If prediction accuracy stays in the high range reported, host-to-device KV transfer moves off the critical path, so sparse KV offloading can preserve latency without GPU-resident selection or reconstruction state.
  • Removing auxiliary residency raises the maximum request batch size, which directly raises average token throughput under long-context workloads.
  • The two-layer buffer keeps prefetched data bounded independently of model depth, so memory savings do not evaporate as models get deeper.
  • Quality is preserved because missed entries are fetched urgently before attention, so the attention result equals the ground-truth sparse selection.
  • Throughput gains grow with context length, where full-KV baselines hit OOM and sparse systems otherwise stall on PCIe transfers.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the paper's production initialization—using the model-generated first speculative token rather than the ground-truth one—would settle how much of the reported accuracy is achievable outside the paper's favorable initialization; the paper's random-initialization baseline suggests accuracy could drop toward 56%.
  • The same predictive-prefetch shape could be applied to disaggregated or remote KV pools, where the transfer latency is even higher than PCIe, making prediction lead time more valuable.
  • Because prediction accuracy is measured per-layer, a version could skip prefetching entirely for layers with low predictability and instead rely on reconstruction, tuning the prefetch/reconstruct split online rather than by offline profiling.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents DualDecoder, a serving system for long-context LLM inference with dynamic sparse KV caches. It observes that the KV retrieval indices for the next decoding step can be predicted from a speculative token co-generated with the current output token, enabling prefetching of sparse KV entries from host memory and overlapping transfer with computation. This is intended to remove most GPU-resident auxiliary state (low-rank K, quantized KV) used by prior systems such as ShadowKV and SpeCache. The system contributes a dual-token decoding pipeline, a layer-aware transfer scheduler with prioritized miss handling and adaptive key reconstruction, and a two-layer ping-pong GPU buffer. On a testbed with Hopper GPUs and PCIe 5.0, DualDecoder reports up to 2.62× decoding throughput over ShadowKV, lower memory footprint, and preserved quality on RULER NIAH tasks.

Significance. If the predictability claim holds broadly, the paper identifies a useful system-level opportunity: replacing residency-heavy KV retrieval assistance with predictive prefetching. The implementation is nontrivial and the end-to-end results are encouraging, spanning multiple model sizes, context lengths, and baselines. The paper also gives a clear decomposition of the auxiliary-state memory problem, which is often overlooked. However, the evidence for the central predictability claim is currently limited to needle-retrieval workloads, and the compute overhead of dual-token decoding is not isolated against single-token decoding. The memory 'elimination' claim is overstated because landmarks and an adaptive low-rank cache remain. I note that the stress-test concern about the first speculative token does not land: §5.2's pre-decoding step generates T2 from T1, and because T2 is exactly the token the model will process next, the production initialization coincides with the 'ground-truth T2' case in Figure 7(b), not with the random-initialization case. With the required additional experiments and wording corrections, the contribution would be solid.

major comments (4)
  1. [§6.3, Fig. 16] KV prediction accuracy is reported only for NIAH-1 and NIAH-2, which are needle-in-haystack retrieval tasks with highly concentrated attention. The central claim in the abstract is general ('the critical KV entries required for decoding the next token can be accurately predicted'). Please report prediction hit rate per RULER task family (QA, variable tracking, etc.) and, if possible, on a non-retrieval benchmark. If accuracy drops on non-retrieval tasks, the scope of the claim and the expected speedup must be revised.
  2. [§5.2, Fig. 17] The dual-token compute overhead is measured only against a two-pass baseline (two separate decoding passes). The relevant baseline for understanding the system's overhead is single-token decoding, which is what ShadowKV and SpeCache execute. Please add per-layer/per-step latency for single-token decoding and quantify the added compute of co-batched dual-token execution. Without this, the claim of 'negligible computational overhead' is not established by the modular study.
  3. [§3.2/§5.3] The paper claims to 'eliminate the GPU memory overhead of auxiliary states' (abstract, §4.2), but DualDecoder still keeps GPU-resident K landmarks for retrieval-index prediction (§4.1, Fig. 8) and a low-rank key cache for adaptively reconstructed layers (§5.3). These are auxiliary states under the paper's own definition of GPU-resident metadata used for KV selection/reconstruction. Please quantify the remaining auxiliary residency and state precisely which ShadowKV/SpeCache states are removed.
  4. [Contributions/§7] The contribution statement says this is 'the first work to characterize and exploit KV retrieval predictability.' This is contradicted by the paper's own related-work description of SpeCache [13], which 'employs a draft model and exploits the similarity between the sparse KV indices selected by speculative tokens and those required by the actual output tokens, enabling proactive KV prefetching.' Please restate the novelty as eliminating the draft-model/auxiliary-state overhead, rather than being first to exploit predictability.
minor comments (6)
  1. [§3.1] Typo in section title: 'Throughtput' should be 'Throughput'.
  2. [§6.2] 'exposively' should be 'explosively'; 'ShadowKVshadowkv' has inconsistent capitalization.
  3. [Appendix A] Appendix A refers to 'StreamDecoder' instead of 'DualDecoder'.
  4. [§6.1] The RULER citation appears as 'RULER []' — fill in the reference.
  5. [§6.3, Table 1] 'strictly lossless quality' is imprecise: sparse attention is approximate, and the table shows QA-2 = 0.58/0.43, not exact equality with full attention. Rephrase to 'comparable or better than ShadowKV'.
  6. [§5.2] The notation T'_i+1 and the description of the causal-mask transformation are hard to follow; a small pseudocode or diagram would help.

Circularity Check

0 steps flagged

No significant circularity; the KV-predictability claim is empirical and benchmarked, not derived from its own inputs.

full rationale

DualDecoder's central claim is that a speculative token produced in step S_i can predict the sparse-KV retrieval index of step S_{i+1}. This is tested empirically by measuring overlap between predicted and ground-truth indices (Figure 16: 'the accuracy stays above 82.6%'), not by defining the prediction to equal the target. The speculative token T'_{i+1} is model-generated, not the ground-truth T_{i+1}, so the reported accuracy is not true by construction. The sensitivity analysis in Figure 7(b) shows that initialization matters and that random initialization degrades accuracy, but an unmeasured deployment regime is a correctness limitation, not a circular reduction. The adaptive key-reconstruction layer set is chosen via offline profiling (§5.3), but that is engineering configuration tuned to measured prediction accuracy, not a fitted parameter renamed as a prediction. The implementation builds on the external ShadowKV system and is compared against external baselines; no load-bearing step depends on a self-citation chain or on an author-imported uniqueness theorem. Even though SpeCache and SpeContext explore similar speculative-prefetch ideas, the paper acknowledges them and adds different mechanisms (dual-token co-execution, layer-scoped ping-pong buffering, prioritized miss handling); whether the contribution is sufficiently novel is a prior-art question, not a circularity. No equation or construction in the paper makes the output equivalent to the input by definition.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

DualDecoder introduces no new physical entities; the speculative token is inherited from speculative-decoding literature. The main load-bearing choices are system constants and scheduling decisions, not fitted parameters in a derivation.

free parameters (3)
  • Prefetch lead distance = 2 layers (layer i+2)
    Chosen to balance hiding PCIe transfer latency against buffer footprint; no sensitivity study is shown (§5.3).
  • Adaptive key-reconstruction layer set = Low-prediction-accuracy layers selected by offline profiling
    The system enables low-rank K reconstruction only for layers where profiling shows low retrieval-index prediction accuracy; this is a workload-tuned choice (§5.3).
  • Sparse context length / sparse KV budget = 1024 in NIAH accuracy experiments
    Inherited from ShadowKV-style chunked sparse selection; affects hit rate and transfer volume, and no sensitivity analysis is provided (Table 1).
axioms (4)
  • domain assumption Retrieval index for token T_{i+1} is computable from token T_i and fixed GPU-resident landmarks.
    Adopted from ShadowKV/QUEST-style dynamic sparse KV cache design (§4.1).
  • ad hoc to paper A speculative token co-computed with the output token carries enough contextual information to predict the next step's retrieval indices.
    This is the paper's core empirical hypothesis; it is validated only partially and, for the first token, only with ground-truth initialization (§4.1, §5.2).
  • domain assumption CPU-to-GPU KV transfer can be overlapped with layer computation using PCIe 5.0 bandwidth.
    The layer-aware schedule assumes that prefetching two layers ahead is enough to hide transfer latency (§5.3).
  • domain assumption CUDA streams and events between compute and communication streams prevent data hazards.
    Used to justify the ping-pong buffer and prioritized transfer streams (§5.4).

pith-pipeline@v1.3.0-daily-deepseek · 23895 in / 12869 out tokens · 148987 ms · 2026-08-01T15:01:10.305183+00:00 · methodology

0 comments
read the original abstract

Long-context inference is becoming a fundamental capability for modern LLM serving, especially driven by emerging agentic applications. Yet it faces a severe memory wall that the KV cache scales proportionally with increasing context length and request concurrency. Existing sparse KV cache methods offload most KV entries to host memory and retrieve only the critical KV entries needed by each decoding step. However, they commonly introduce substantial auxiliary states in GPU memory for KV retrieval management. Our measurements show that these often-overlooked auxiliary states introduce significant memory overhead and become a new bottleneck under high-concurrency workloads. In this paper, we present DualDecoder, a lightweight serving system for long-context LLM inference that enables efficient sparse KV cache retrieval from host memory. Our key insight is that the critical KV entries required for decoding the next token can be accurately predicted from the preceding speculated token. This predictability enables KV retrieval to be proactively prefetched and overlapped with decoding computation, effectively eliminating the GPU memory overhead of auxiliary states. To achieve this prefetching efficiently, DualDecoder leverages a novel dual-token decoding pipeline that accurately identifies critical KV entries with negligible computational overhead, and designs a layer-aware transfer schedule to overlap KV prefetching with model computation and a layer-scoped memory manager to reduce the GPU runtime buffer. Experimental results show that DualDecoder improves decoding throughput by up to 2.62$\times$ over state-of-the-art systems while preserving decoding latency and model quality.

Figures

Figures reproduced from arXiv: 2607.26475 by Hao Dai, Jinlong Hou, Qi Chen, Tongkai Yang, Yuan Cheng, Yuedong Xu, Zhiqiang Ding, Zhiyi Yao, Zuning Liang.

Figure 1
Figure 1. Figure 1: LLM decoding step with dynamic sparse KV cache. many noncontiguous KV entries from host memory before or during the host-to-device transfer, which further increases retrieval latency. To reduce this overhead, recent systems introduce GPU-resident auxiliary state for KV selection, re￾construction, or communication scheduling. ShadowKV [28] keeps a low-rank representation of the key cache on the GPU and reco… view at source ↗
Figure 2
Figure 2. Figure 2: Maximum request batch size of decoding using full KV cache, ShadowKV and ideal scaling [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: End-to-end decoding throughput. system, but it provides a useful upper bound on the serving concurrency that the GPU could support if KV cache mem￾ory were not the bottleneck. As shown in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 6
Figure 6. Figure 6: KV retrieval prediction via speculative token. Retrieval index prediction. To overcome this dependency, we use a lightweight token prediction mechanism to es￾timate the retrieval index of the next decoding step. Re￾cent studies [34] provide approaches like speculative de￾coding to precisely predict the decoding tokens. However, they either introduce heavy computation and memory over￾head by running a draft… view at source ↗
Figure 7
Figure 7. Figure 7: Accuracy of KV prediction. of decoding. Thus, we can directly apply it during decoding and prefetch the predicted KV cache of the next step, which we call the speculative KV cache. This prefetching offers a direct opportunity to move sparse KV retrieval out of the critical path of decoding. Since they are not consumed by the current decoding step, their host-to-device transfer can be issued early and overl… view at source ↗
Figure 8
Figure 8. Figure 8: Overview of DualDecoder KV entries from host memory to GPU HBM before the next attention computation consumes them. The transfer sched￾ule is designed to hide CPU-to-GPU latency while avoiding excessive early buffering. Meanwhile, DualDecoder handles prediction misses by giving missing KV entries higher trans￾fer priority than background prefetches and by selectively reconstructing key entries when fetchin… view at source ↗
Figure 10
Figure 10. Figure 10: Streams that ensure prioritized transmission. stall decoding. DualDecoder therefore separates urgent cor￾rection from background prefetching. As shown in [PITH_FULL_IMAGE:figures/full_fig_p008_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: DualDecoder two-layer ping-pong buffer. the memory benefit of KV offloading. To address this prob￾lem, DualDecoder introduces a layer-scoped KV memory manager that maintains only two active sparse KV buffers on the GPU and coordinates their reuse across transformer layers with CUDA stream events. Two-layer ping-pong buffering. DualDecoder uses a two￾layer ping pong buffer to decouple the sparse KV working… view at source ↗
Figure 12
Figure 12. Figure 12: End-to-end serving throughput. (RCT), while maintaining latency comparable to existing sys￾tems. We further evaluate the KV cache prediction accuracy to validate our key insight on KV predictability. Meanwhile, we measure GPU memory usage and decoding latency to quantify the effectiveness of DualDecoder in reducing mem￾ory footprint and to assess the contribution of its individual design components. 6.2 E… view at source ↗
Figure 13
Figure 13. Figure 13: P99 request completion time across different arrival rates for the 32B model serving [PITH_FULL_IMAGE:figures/full_fig_p011_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Memory footprint over the same batch sizes (12, 6 and 4 for 8B, 14B and 32B models respectively) [PITH_FULL_IMAGE:figures/full_fig_p011_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Memory footprint across different context lengths [PITH_FULL_IMAGE:figures/full_fig_p011_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: KV retrieval prediction accuracy across models [PITH_FULL_IMAGE:figures/full_fig_p011_16.png] view at source ↗
Figure 19
Figure 19. Figure 19: KV retrieval latency using DualDecoder GPU gathering. 2048. The significant improvement stems from that DualDe￾coder performs asynchronous, contiguous block prefetching over PCIe and leverages the massive parallelism of GPU ker￾nels to execute a monolithic memory reorder only after the ground-truth tokens are confirmed. 7 Related Work Dynamic sparse KV cache techniques. A substantial body of work has expl… view at source ↗
Figure 18
Figure 18. Figure 18: Per-step latency with DualDecoder priority sched￾uling for missed KV entries. speculative tokens into a single batch dimension, DualDe￾coder drastically reduces both the computational latencies of attention and linear layers. For example, computing two tokens via our Co-batched attention takes only 0.035 ms, saving 49% of the time compared to the 0.069 ms required by consecutively launching two separate p… view at source ↗
Figure 20
Figure 20. Figure 20: TTFT and TPOT comparison across different systems. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_20.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

44 extracted references · 10 linked inside Pith

  1. [1]

    Gulavani, Alexey Tumanov, and Ramachandran Ramjee

    Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Alexey Tumanov, and Ramachandran Ramjee. 2024. Taming Throughput-Latency Tradeoff in LLM In- ference with Sarathi-Serve. In18th USENIX Symposium on Operat- ing Systems Design and Implementation, OSDI 2024, Santa Clara, CA, USA, July 10-12, 2024, Ada Gavrilovska a...

    2024

  2. [2]

    Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2025. LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)...

    2025

  3. [3]

    DeepSeek-AI. 2025. DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models.CoRRabs/2512.02556 (2025).https://doi.org/ 10.48550/ARXIV.2512.02556arXiv:2512.02556

    Pith/arXiv arXiv doi:10.48550/arxiv.2512.02556 2025

  4. [4]

    Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. 2024. LongRoPE: Ex- tending LLM Context Window Beyond 2 Million Tokens. InForty-first International Conference on Machine Learning, ICML 2024, Vienna, Aus- tria, July 21-27, 2024 (Proceedings of Machine Learning Research), Ruslan Salakhutdinov, Zico...

    2024

  5. [5]

    Aly, Beidi Chen, and Carole-Jean Wu

    Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agar- wal, Ahmed Roman, Ahmed A. Aly, Beidi Chen, and Carole-Jean Wu

  6. [6]

    Yu Fu, Zefan Cai, Abedelkadir Asi, Wayne Xiong, Yue Dong, and Wen Xiao. 2025. Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning. InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net.https://openreview. net/forum?id=FJFVmeXusW

    2025

  7. [7]

    Shiwei Gao, Youmin Chen, and Jiwu Shu. 2025. Fast State Restora- tion in LLM Serving with HCache. InProceedings of the Twenti- eth European Conference on Computer Systems, EuroSys 2025, Rotter- dam, The Netherlands, 30 March 2025 - 3 April 2025. ACM, 128–143. https://doi.org/10.1145/3689031.3696072

    arXiv 2025

  8. [8]

    Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Synnaeve. 2024. Better & Faster Large Language Models via Multi-token Prediction. InForty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 (Proceed- ings of Machine Learning Research), Ruslan Salakhutdinov, Zico Kolter, Kathe...

    2024

  9. [9]

    Jiaao He and Jidong Zhai. 2024. FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelines.CoRR abs/2403.11421 (2024).https://doi.org/10.48550/ARXIV.2403.11421 arXiv:2403.11421

    Pith/arXiv arXiv doi:10.48550/arxiv.2403.11421 2024

  10. [10]

    Yongjun He, Yao Lu, and Gustavo Alonso. 2024. Deferred Continuous Batching in Resource-Efficient Large Language Model Serving. In Proceedings of the 4th Workshop on Machine Learning and Systems, EuroMLSys 2024, Athens, Greece, 22 April 2024. ACM, 98–106.https: //doi.org/10.1145/3642970.3655835

    arXiv 2024

  11. [11]

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024. RULER: What’s the Real Context Size of Your Long-Context Language Models? (2024). arXiv:cs.CL/2404.06654https://arxiv.org/abs/2404.06654

    Pith/arXiv arXiv 2024

  12. [12]

    Cunchen Hu, Heyang Huang, Junhao Hu, Jiang Xu, Xusheng Chen, Tao Xie, Chenxi Wang, Sa Wang, Yungang Bao, Ninghui Sun, and Yizhou Shan. 2024. MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool.CoRRabs/2406.17565 (2024). https://doi.org/10.48550/ARXIV.2406.17565arXiv:2406.17565

    Pith/arXiv arXiv doi:10.48550/arxiv.2406.17565 2024

  13. [13]

    Shibo Jie, Yehui Tang, Kai Han, Zhi-Hong Deng, and Jing Han. 2025. SpeCache: Speculative Key-Value Caching for Efficient Generation of LLMs. InInternational Conference on Machine Learning (ICML). https://icml.cc/virtual/2025/poster/45386

    2025

  14. [14]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica

  15. [15]

    Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. 2024. InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management. In18th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2024, Santa Clara, CA, USA, July 10-12, 2024, Ada Gavrilovska and Douglas B. Terry (Eds.). USENIX Association, 155–17...

    2024

  16. [16]

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast Inference from Transformers via Speculative Decoding.International Conference on Machine Learning (ICML)(2023)

    2023

  17. [17]

    Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024. SnapKV: LLM Knows What You are Looking for 13 Before Generation. InAdvances in Neural Information Process- ing Systems 37: Annual Conference on Neural Information Process- ing Systems 2024, NeurIPS 2024, Vancouver, BC, Can...

    2024

  18. [18]

    Zhiyu Li, Shichao Song, Hanyu Wang, Simin Niu, Ding Chen, Jiawei Yang, Chenyang Xi, Huayi Lai, Jihao Zhao, Yezhaohui Wang, Jun- peng Ren, Zehao Lin, Jiahao Huo, Tianyi Chen, Kai Chen, Kehang Li, Zhiqiang Yin, Qingchen Yu, Bo Tang, Hongkang Yang, Zhi-Qin John Xu, and Feiyu Xiong. 2025. MemOS: An Operating System for Memory- Augmented Generation (MAG) in La...

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.22101 2025

  19. [19]

    Di Liu, Meng Chen, Baotong Lu, Huiqiang Jiang, Zhenhua Han, Qianxi Zhang, Qi Chen, Chengruidong Zhang, Bailu Ding, Kai Zhang, Chen Chen, Fan Yang, Yuqing Yang, and Lili Qiu. 2024. RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval.CoRR abs/2409.10516 (2024).https://doi.org/10.48550/ARXIV.2409.10516 arXiv:2409.10516

    Pith/arXiv arXiv doi:10.48550/arxiv.2409.10516 2024

  20. [20]

    Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Anantha- narayanan, Michael Maire, Henry Hoffmann, Ari Holtzman, and Junchen Jiang. 2024. CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving. InProceedings of the ACM SIG- COMM 2024 Conference, ACM SIGCOMM 20...

    arXiv 2024

  21. [21]

    Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen (Henry) Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. 2024. KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache. InForty- first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 (Proceedings of Machine Learning Research), Ruslan Salakhutdinov, ...

    2024

  22. [22]

    Meta AI Llama Team. 2024. The Llama 3 Herd of Models.arXiv e- prints, Article arXiv:2407.21783 (July 2024), arXiv:2407.21783 pages. https://doi.org/10.48550/arXiv.2407.21783arXiv:cs.AI/2407.21783

    Pith/arXiv arXiv doi:10.48550/arxiv.2407.21783 2024

  23. [23]

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li,...

    Pith/arXiv arXiv 2025

  24. [24]

    Luka Ribar, Ivan Chelombiev, Luke Hudlass-Galley, Charlie Blake, Carlo Luschi, and Douglas Orr. 2024. SparQ Attention: Bandwidth- Efficient LLM Inference. InForty-first International Conference on Ma- chine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 (Proceed- ings of Machine Learning Research), Ruslan Salakhutdinov, Zico Kolter, Katherine A. H...

    2024

  25. [25]

    Tal Schuster et al. 2023. Accelerating Large Language Model Decoding with Speculative Sampling.arXiv preprint arXiv:2302.01318(2023)

    Pith/arXiv arXiv 2023

  26. [26]

    Ning Shang, Li Lyna Zhang, Siyuan Wang, Gaokai Zhang, Gilsinia Lopez, Fan Yang, Weizhu Chen, and Mao Yang. 2025. LongRoPE2: Near- Lossless LLM Context Window Scaling. InForty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025 (Proceedings of Machine Learning Research), Aarti Singh, Maryam Fazel, Daniel ...

    2025

  27. [27]

    Noam Shazeer. 2019. Fast Transformer Decoding: One Write-Head is All You Need.CoRRabs/1911.02150 (2019). arXiv:1911.02150http: //arxiv.org/abs/1911.02150

    Pith/arXiv arXiv 2019

  28. [28]

    Hanshi Sun, Li-Wen Chang, Wenlei Bao, Size Zheng, Ningxin Zheng, Xin Liu, Harry Dong, Yuejie Chi, and Beidi Chen. 2025. ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference. InInternational Conference on Machine Learning (ICML).https://icml. cc/virtual/2025/poster/44053

    2025

  29. [29]

    Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. 2024. QUEST: Query-Aware Sparsity for Efficient Long-Context LLM Inference. InForty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net.https://openreview.net/forum?id=KzACYw0MTV

    2024

  30. [30]

    Bingyang Wu, Shengyu Liu, Yinmin Zhong, Peng Sun, Xuanzhe Liu, and Xin Jin. 2024. LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism. InProceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles, SOSP 2024, Austin, TX, USA, November 4-6, 2024, Emmett Witchel, Christopher J. Rossbach, Andr...

    arXiv 2024

  31. [31]

    Yongtong Wu, Shaoyuan Chen, Yinmin Zhong, Rilin Huang, Yixuan Tan, Wentao Zhang, Liyue Zhang, Shangyan Zhou, Yuxuan Liu, Shun- feng Zhou, Mingxing Zhang, Xin Jin, and Panpan Huang. 2026. Du- alPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference.CoRRabs/2602.21548 (2026).https://doi.org/10.48550/ARXIV. 2602.21548arXiv:2602.21548

    doi:10.48550/arxiv.2602.21548 2026

  32. [32]

    Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. 2025. DuoAttention: Ef- ficient Long-Context LLM Inference with Retrieval and Streaming Heads. InThe Thirteenth International Conference on Learning Repre- sentations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. https://openreview.net/forum...

    2025

  33. [33]

    Guangxuan Xiao, Yaohui Tian, Beidi Chen, et al. 2024. StreamingLLM: Efficient Streaming Inference of Large Language Models with Atten- tion Sinks. InInternational Conference on Learning Representations (ICLR).https://iclr.cc/media/iclr-2024/Slides/18794.pdfSlides

    2024

  34. [34]

    2026.SpeContext:Enabling Efficient Long-context Reasoning with Speculative Context Sparsity in LLMs

    Jiaming Xu, Jiayi Pan, Hanzhen Wang, Yongkang Zhou, Jiancai Ye, Yu Wang, and Guohao Dai. 2026.SpeContext:Enabling Efficient Long-context Reasoning with Speculative Context Sparsity in LLMs. InProceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ASPLOS 2026, Pittsburgh, PA,...

    arXiv 2026

  35. [35]

    Jiaming Xu, Jiayi Pan, Yongkang Zhou, Siming Chen, Jinhao Li, Yaoxiu Lian, Junyi Wu, and Guohao Dai. 2025. SpecEE: Accelerating Large Language Model Inference with Speculative Early Exiting. InPro- ceedings of the 52nd Annual International Symposium on Computer Architecture, ISCA 2025, Tokyo, Japan, June 21-25, 2025. ACM, 467–481. https://doi.org/10.1145/...

    arXiv 2025

  36. [36]

    Minghao Yan, Saurabh Agarwal, and Shivaram Venkataraman. 2025. Decoding Speculative Decoding. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, NAACL 2025 - Volume 1: Long Papers, Albuquerque, New Mexico, USA, April 29 - May 4, 2025, Luis Chiruzzo, A...

    doi:10.18653/v1/2025 2025

  37. [37]

    Dongjie Yang, Xiaodong Han, Yan Gao, Yao Hu, Shilin Zhang, and Hai Zhao. 2024. PyramidInfer: Pyramid KV Cache Compression for 14 High-throughput LLM Inference. InFindings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024 (Findings of ACL), Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Ed...

    doi:10.18653/v1/2024.findings- 2024

  38. [38]

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A Distributed Serving System for Transformer-Based Generative Models. In16th USENIX Symposium on Operating Systems Design and Implementation (OSDI). 521–538. https://dblp.org/rec/conf/osdi/YuJKKC22

    2022

  39. [39]

    Yanqi Zhang, Yuwei Hu, Runyuan Zhao, John C. S. Lui, and Haibo Chen. 2025. DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compaction. InProceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles, SOSP 2025, Lotte Hotel World, Seoul, Republic of Korea, October 13-16, 2025, Youjip Won, Youngjin Kwon, Di...

    arXiv 2025

  40. [40]

    Barrett, Zhangyang Wang, and Beidi Chen

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark W. Barrett, Zhangyang Wang, and Beidi Chen. 2023. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models. InAdvances in Neural Information Processing Systems (NeurIPS).https: //dl.acm.org/doi/10.5555...

    arXiv 2023

  41. [41]

    Jun Zhao, Can Zu, Xu Hao, Yi Lu, Wei He, Yiwen Ding, Tao Gui, Qi Zhang, and Xuanjing Huang. 2024. LONGAGENT: Achieving Question Answering for 128k-Token-Long Documents through Multi- Agent Collaboration. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, Yaser Al-Onai...

    doi:10.18653/v1/2024.emnlp-mai 2024

  42. [42]

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xu- anzhe Liu, Xin Jin, and Hao Zhang. 2024. DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving. In18th USENIX symposium on operating systems design and implementation, OSDI 2024, santa clara, CA, USA, july 10-12, 2024, Ada Gavrilovska and Douglas B....

    2024

  43. [2023]

    InProceedings of the 29th ACM Symposium on Operating Systems Principles (SOSP)

    Efficient Memory Management for Large Language Model Serv- ing with PagedAttention. InProceedings of the 29th ACM Symposium on Operating Systems Principles (SOSP). 611–626.https://doi.org/10. 1145/3600006.3613165

    arXiv

  44. [2024]

    LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding. InProceedings of the 62nd Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, 12622–12642.https://do...

    doi:10.18653/v1/2024.acl-long.681 2024