Pith. sign in

REVIEW 3 major objections 7 minor 88 references

Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generation

T0 review · 3 major / 7 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A RAG chunk's cached attention state stays reusable at any prompt position if you recompute only its most context-entangled tokens — about 30% of them, preserving 90% of answer quality.

desk verdict Chunk-level KV reuse with selective recomputation is a genuinely useful idea for RAG serving, but the quality guarantee has a causal-direction problem: tokens to recompute are chosen from the old context, so new-prefix damage can escape repair. read the letter →

arxiv 2502.15734 v1 pith:7ALGNZKT submitted 2025-02-05 cs.DC cs.AIcs.CLcs.LGcs.OS

Shubham Agarwal , Sai Sundaresan , Subrata Mitra , Debabrata Mahapatra , Archit Gupta , Rounak Sharma , Nirmal Joshua Kapu , Tong Yu
show 1 more author
Shiv Saini
This is my paper · ORCID
classification cs.DCcs.AIcs.CLcs.LGcs.OS
keywords retrieval-augmentedgenerationKV-cachereusechunk-cachemanagementprefilloptimizationprefixcachingselectivetokenrecomputationLLMservingsystemsattention-basedcontextualization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Every time a retrieval-augmented generation (RAG) system answers a question, the language model re-executes attention prefill over retrieved text chunks it has often already seen, wasting GPU hours on repeated computation. Cache-Craft tries to prove this is largely avoidable: the per-token key-value states (KV-caches) of a chunk can be stored once, then reused even when the chunk lands in a new prompt at a new position, as long as a small, precisely chosen set of tokens is recomputed to repair the stale context. On two real production RAG workloads the paper reports a 51% reduction in redundant computation over state-of-the-art prefix caching, a 75% reduction over full recomputation, and, under continuous batching, a 1.6× throughput gain with roughly 2× lower end-to-end latency at about 30% token recomputation. If correct, Cache-Craft converts the dominant cost of RAG serving — the quadratic prefill — from a per-request expense into a mostly amortized one that grows only with the chunks that change.

What carries the argument

The chunk-cache: the per-layer key and value vectors of a single retrieved chunk, computed during an earlier prefill and stored along with the prefix that contextualized it. The reusability scores: CCI (Cache Context Impact), the sigmoid of the layer-averaged ratio of normalized inter-chunk to intra-chunk attention, which measures how much a chunk's representation depends on outside tokens; $\beta'$, the prefix-overlap score penalized by a normalized Kendall-Tau order distance; and the cache-fix overhead $\mathrm{CFO} = \alpha \cdot \mathrm{CCI} \cdot (1-\beta')$, which sets how many tokens of the chunk get recomputed. The fix: recompute the top-$\lceil \mathrm{CFO} \cdot |C| \rceil$ tokens ranked by inter-attention from the old prefix, then stop early for chunks the user question stops attending to, using a change-point detector on cumulative question-to-chunk attention. Supporting machinery: rotary position embeddings are stripped from stored keys and re-applied at the new position, a custom attention mask replaces the triangular causal mask, and chunk-caches are tiered across GPU, host memory, and SSD with layer-wise preloading so that load time hides behind prefill compute.

What would settle it

Take a model with a different positional-encoding scheme (relative rather than rotary positions), cache chunks under one retrieval distribution, then serve a second distribution where chunks are densely cross-referential, and check whether the same 30% recomputation fraction keeps ROUGE-F1 above 90% of full recomputation. A more direct check: for a set of reused chunks, compare the tokens Cache-Craft selects by inter-attention against the tokens whose key-value outputs actually deviate most from a full forward pass with the prefix swapped; if the two sets barely overlap, or if output deviation at fixed $\mathrm{CFO}$ is uncorrelated with $\mathrm{CCI}\cdot(1-\beta')$, the heuristic is the wrong proxy.

Watch

Extended reading notes

Core claim

The paper's central finding is that whether a chunk's cached key-value states can be transplanted into a new prompt is visible in the attention weights themselves. A chunk dominated by intra-chunk attention — its tokens attending mostly to each other — carries almost no baggage from its old prefix and can be reused directly; a chunk that was heavily contextualized by its old prefix can still be salvaged by recomputing only the tokens with the highest inter-attention scores. Cache-Craft codifies this in three scores: CCI, the sigmoid of the layer-averaged inter-to-intra attention ratio; $\beta$, the share of old-prefix inter-attention that survives into the new prompt; and $\gamma$, a normalized Kendall-Tau penalty for prefix reordering; these combine into a cache-fix overhead $\mathrm{CFO} = \alpha \cdot \mathrm{CCI} \cdot (1-\beta')$, where $\beta' = \beta(1-\gamma)$, giving the fraction of each chunk's tokens to recompute, with $\alpha$ tuned on a validation set to meet a target F1. With roughly 30% of tokens recomputed, the paper reports answer quality at about 90% of the full-recomputation ROUGE-F1 while avoiding 75% of prefill computation, and a user study finds cached answers acceptable 79% of the time versus 84% for exact recomputation on SQuAD.

Load-bearing premise

The load-bearing premise is that attention-weight statistics measured when a chunk was cached — the ratio of inter-chunk to intra-chunk attention and the prefix overlap with the new prompt — reliably predict both which caches can be reused and exactly which tokens must be recomputed to preserve answer quality.

Editorial extensions

If this is right

  • RAG prefill stops scaling with knowledge-base reuse: each repeated chunk becomes a cache hit, so the quadratic part of attention is paid only once per chunk instead of once per request, and the per-request cost is dominated by the chunks that change.
  • Operators get one quality knob: $\alpha$ in $\mathrm{CFO} = \alpha\cdot\mathrm{CCI}\cdot(1-\beta')$ is calibrated on a validation set to a target F1, and at the calibrated 30% recomputation the reported trade is about 90% of base ROUGE-F1 for a 1.6× throughput gain and a 2× latency cut.
  • Exact-prefix caching, the current state of the art, is nearly useless in RAG: in the studied production workloads only 8% of requests and 18% of prefill tokens share a full prefix, so chunk-level reuse captures reuse that prefix matching structurally cannot.
  • The gains widen with sequence length, batch size, and model depth, since the avoided work is the $O(n^2)$ part of attention: TTFT speedups over full recomputation reached 1.6× for LLaMA-3-8B and 2.3× for LLaMA-3-70B at batch size 4.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The CCI/CFO heuristic is training-free but probably not transfer-free: it is demonstrated on LLaMA-3 with rotary position embeddings, and $\alpha$ is retuned per workload, so the 30%-recomputation-for-90%-quality rule should be re-validated on other model families and on retrieval distributions with densely cross-referential chunks before it is treated as a general constant.
  • Because the fix recomputes only tokens inside the chunk, the ceiling on savings is set by how self-contained the knowledge base's chunks are; corpora whose chunks repeatedly reference each other will push more tokens into the top-N recomputation set and shrink the reported 75% reduction.
  • A sharper test of the proxy: compare the tokens Cache-Craft selects (highest inter-attention from the old prefix) against the tokens whose KV values actually change most when the prefix is swapped; if the overlap is low, the CFO formula is tracking a correlated signal rather than the mechanism that breaks cached attention.
  • The quality guarantee is stated on ROUGE-F1, a surface overlap metric; reporting exact-match or human-rated accuracy on multi-step reasoning tasks at the same recomputation budget would test whether the 'maintaining quality' claim holds where answer correctness, not phrasing, is what matters.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper presents Cache-Craft, a chunk-level KV-cache management system for retrieval-augmented generation. Instead of relying on exact prefix matches, it stores per-chunk KV caches from previous requests, decides which caches can be reused by computing attention-derived scores (CCI, β′, CFO), and recomputes the KV values of a small selected fraction of tokens to repair contextualization errors. The implementation is integrated into vLLM and evaluated on two production RAG workloads and several public datasets with LLaMA-3-8B and LLaMA-3-70B. The central claims are a 51% reduction in redundant computation over prefix caching, 1.6× throughput and about 2× latency improvements under continuous batching, and preservation of about 90% of the base ROUGE-F1 score when recomputing 30% of tokens.

Significance. If the central empirical claims hold, Cache-Craft would be a valuable contribution to RAG serving: it directly targets the prefill bottleneck, which the paper convincingly shows dominates RAG workloads, and it does so with a real vLLM implementation rather than a simulation. The paper has several genuine strengths: evaluation on real production workloads, two model sizes, six public datasets, a 250-participant user study, ablations of the design components, and a concrete hierarchical cache/preloading design. The risk is concentrated in the quality-preservation heuristic: the token-selection signal is derived from the old cached context, and the quality metric is agreement with full recomputation rather than with ground-truth answers. These issues are fixable with targeted experiments, but they are load-bearing for the headline quality/compute trade-off.

major comments (3)
  1. [§3.2.1, Eq. (14)] The token set T(C_i) in Eq. (14) is selected using inter-attention scores from the old prefix that existed when the cache was created. For a new request whose prefix contains chunks that were not present in that old prefix, the tokens whose hidden states are most distorted by the new chunks need not lie in T(C_i); the scalar β′ (Eq. 8) only measures overall prefix overlap and ordering and cannot locate token-level damage. The headline claim that 30% recomputation retains about 90% ROUGE-F1 is therefore not guaranteed for requests containing unseen prefix chunks, and the high chunk overlap in the evaluated workloads (e.g., Fig. 6a) may mask this failure mode. Please add a controlled experiment that varies the fraction of retrieved prefix chunks absent from the cached context and reports both the actual high-distortion token set (e.g., measured by hidden-state or attention deviation under the new prefix) and ROUGE-F1 at a fixed recomputation budget.
  2. [§5.2, Fig. 20] The ROUGE-F1 scores are computed against the output of Full-Recomp, not against ground-truth answers; Full-Recomp is treated as having ROUGE=1 by construction. A score of 0.9 therefore measures agreement with full recomputation, not correctness on the underlying task. Since the user study in Table 2 shows that Full-Recomp itself receives only 76.9% and 83.7% human Yes ratings on 2wikiMQA and SQuAD, the statement that quality is 'maintained' is overstated. Please report ground-truth ROUGE or human acceptance for all datasets, or explicitly reframe the claim as fidelity to exact recomputation.
  3. [§5.1.2 and §3.2, Eq. (13)] The paper does not state whether the F1 values in Fig. 20 come from a held-out evaluation set or from the same validation data used to select α in Eq. (13). Because α is calibrated to meet F1_desired on a validation set, reporting the 90%-quality result on the same data would be partly circular. In addition, with 200 questions per dataset and no confidence intervals, the precise 'about 90%' claim is not statistically supported. Please specify the calibration/evaluation split and report variability (e.g., bootstrap CIs) for the main quality results.
minor comments (7)
  1. [§5.4] The phrase 'we copious scope by setting that 60% of the prefill tokens will have a prefix match' appears to contain a typo; it should probably read 'we give copious scope' or 'we generously set'.
  2. [§6] The text 'using LLaMA-80B with 30% recomputation' appears to be a typo for 'LLaMA-3-8B'.
  3. [§2.3] The reference to 'Fig 5a shows the retrieval hit rates' should point to Fig. 6a; the same cross-reference error appears in §6 when discussing high cache hit rates.
  4. [Eq. (7)] The denominator T = m(m−1)/2 is undefined for m < 2; please state the convention used for m = 0 or m = 1.
  5. [Fig. 22] The axis labels 'QPMM' should be 'QPM' (queries per minute).
  6. [Fig. 23a] The caption states '2.3X speedup' but the panel labels list multiple speedup values (2.3×, 2.5×, 2.8×, 1.9×, 1.3×); please reconcile the caption with the figure.
  7. [§5.1.4] The Set-Cache baseline is described only briefly; please specify exactly how RPE is modified and how the longest prefix match is computed after reordering.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the quality/compute trade-off is an empirically measured operating point, not a consequence of the definitions.

full rationale

The central derivation is a heuristic (CCI/CFO, Eqs. 9-12, and top-N token selection, Eq. 14) whose validity is established by measurement: ROUGE-F1, Jaccard, and user-study scores are computed against full-recomputation outputs on external datasets and a production workload, not implied by the heuristic itself. The quality claim '30% tokens are recomputed which maintains 90% of the base ROUGE F1' is a reported operating point; alpha is calibrated via Eq. 13 to meet F1_desired, but the actual F1 values are measured and the recompute fraction is the argmin, so the trade-off is not forced by construction. A residual caveat is that the paper does not state an explicit validation/test split for alpha, so the 90% figure may be in-sample; this is a robustness/rigor concern, not circularity. The paper's related-work self-citations (e.g., Ref. [4]) are inspirational only and do not supply any load-bearing premise, uniqueness theorem, or ansatz. The skeptic concern that Eq. 14 selects tokens using old-context inter-attention only is a correctness/transfer-risk argument about failure modes when new prefixes contain unseen chunks, not a reduction of the output to the input. Overall, no step of the claimed derivation is equivalent to its inputs by definition.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

No new physical entities are posited; all introduced components (chunk-cache, metadata store, focused-chunk selection) are software artifacts and heuristics. The main load-bearing elements are the ad hoc scoring functions and the assumption that causal attention effects can be approximated by attention-weight proxies.

free parameters (2)
  • alpha (Cache Fix Overhead scaling) = 1 (default; tuned on validation via Eq. 13)
    Scales CCI*(1-beta') to set the recomputation fraction; alpha=1 used in evaluations, values 0.5-3 explored in ablation.
  • layer confidence window w = Not specified
    Window length in Algorithm 1 for declaring focused chunks stable; no default or sensitivity analysis reported.
assumptions (5)
  • domain assumption Attention weights are a valid proxy for the extent to which a chunk's KV representation is contextualized by outside tokens.
    Eqs. 3-4 and the CCI definition in Section 3.1 rely on inter/intra attention sums as the measure of contextualization.
  • ad hoc to paper Selectively recomputing the top-N tokens with highest inter-attention scores restores output quality.
    Section 3.2.1, Eq. 14; justified only by empirical observations in Figs. 14-15, no formal guarantee.
  • ad hoc to paper CFO = alpha * CCI * (1 - beta') predicts the fraction of tokens needing recomputation.
    Eq. 12 and Fig. 12-13; no first-principles derivation, calibrated via the alpha hyperparameter.
  • domain assumption Retrieved RAG chunks are mostly self-contained, with low inter-chunk attention.
    Section 6 states intra-attention is 2.18x inter-attention on average for large chunks in Sys-X; this motivates the whole reuse strategy.
  • domain assumption Queue-wait time can mask cache loading latency from CPU/SSD.
    Section 3.5 gives average queue wait 0.32s vs SSD load 0.59s; this is workload-specific and may not hold under low load.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generation." pith.science (2026). https://pith.science/paper/7ALGNZKT

@misc{pith2026250215734,
  author       = {Pith},
  title        = {Pith review of: Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7ALGNZKT}},
  note         = {Machine review of arXiv:2502.15734}
}
read the original abstract

Retrieval-Augmented Generation (RAG) is often used with Large Language Models (LLMs) to infuse domain knowledge or user-specific information. In RAG, given a user query, a retriever extracts chunks of relevant text from a knowledge base. These chunks are sent to an LLM as part of the input prompt. Typically, any given chunk is repeatedly retrieved across user questions. However, currently, for every question, attention-layers in LLMs fully compute the key values (KVs) repeatedly for the input chunks, as state-of-the-art methods cannot reuse KV-caches when chunks appear at arbitrary locations with arbitrary contexts. Naive reuse leads to output quality degradation. This leads to potentially redundant computations on expensive GPUs and increases latency. In this work, we propose Cache-Craft, a system for managing and reusing precomputed KVs corresponding to the text chunks (we call chunk-caches) in RAG-based systems. We present how to identify chunk-caches that are reusable, how to efficiently perform a small fraction of recomputation to fix the cache to maintain output quality, and how to efficiently store and evict chunk-caches in the hardware for maximizing reuse while masking any overheads. With real production workloads as well as synthetic datasets, we show that Cache-Craft reduces redundant computation by 51% over SOTA prefix-caching and 75% over full recomputation. Additionally, with continuous batching on a real production workload, we get a 1.6X speed up in throughput and a 2X reduction in end-to-end response latency over prefix-caching while maintaining quality, for both the LLaMA-3-8B and LLaMA-3-70B models.

Figures

Figures reproduced from arXiv: 2502.15734 by the authors.

Figure 1
Figure 1. Distribution of number tokens in prefill (left) and decode (right phases [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. Chunk-cache hit rate pdf for both Sys-X and RAG datasets. systems with large knowledge bases, precomputed KV-caches may not remain in GPU memory, as space is required for storing a) the LLM parameters and b) the growing KV-cache during the decode phase. The latency of loading precomputed KV-caches into GPU memory must not negate the savings from bypassing recomputa￾tion, requiring efficient system design and impleme… view at source ↗
Figure 5
Figure 5. Token distribution of differ￾ent prompt components (Mother prompt, RAG chunks, Examples, Query, etc.) across RAG use cases. 0.05 Ratio of Chunks SYS SYS- -XX Musique normalized 1-chunk index (a) Individual chunks 1 2 3 4 5 Prefix Length 0 10 20 30 40 50 60 70 Reuse Density SYS-X 2WikiMQA Musique (b) Reuse Density Ratio of Chunks 0.58 SYS-X Musique normalized 5-chunk index (c) Observed 5-tuples [PITH_FULL_IMAGE:figu… view at source ↗
Figures from the paper (9 more)
Figure 9
Figure 9. Figure 9: Inter-attention (C1, C2) and intra-attention (C2, C2) distributions for [PITH_FULL_IMAGE:figures/full_fig_p004_9.png]
Figure 11
Figure 11. Figure 11: Chunk-cache reuse scenarios. Inter and intra-attention for the blue [PITH_FULL_IMAGE:figures/full_fig_p005_11.png]
Figure 13
Figure 13. Figure 13: 𝐶𝐹𝑂 as a function of 𝐶𝐶𝐼 and 1 − 𝛽 ′ to find 𝛼 in Eq. 12 [PITH_FULL_IMAGE:figures/full_fig_p006_13.png]
Figure 19
Figure 19. Figure 19: Layer-wise preloading of chunk-caches into GPU memory to eliminate [PITH_FULL_IMAGE:figures/full_fig_p008_19.png]
Figure 20
Figure 20. Figure 20: Rouge F1 of answer generated using Llama-3 8B and 70B on multi-hop QA, single-hop QA, text summarization, and on production Sys-X. Short Ans True/False Jac. Sim MuSiQue SQuAD DROP Our Full Cache Full Recomp Prefill H20 Random Recomp Prefix Cache Set Cache 2wikiMQA Acc…
Figure 21
Figure 21. Figure 21: Jaccard Similarity and Accuracy of short answers and True/False generated using Llama-3 8B on multi-hop and single-hop datasets. 5.2 Generation Quality with KV Chunk Reuse 5.2.1 Evaluation of Recomputation Strategy We evaluate the recom￾putation strategy of Cache-Craf…
Figure 22
Figure 22. Figure 22: Throughput and overall system response latency speedup under varying computational loads for Cache-Craft deployed with vLLM using ORCA. the text tokens of the chunks and the question to LLaMA-3-8B for a Full-Recomp to get the answers, (2) use Full-Cache that simply re…
Figure 24
Figure 24. Figure 24: Evaluation of Cache-Craft on Sys-X trace on LLaMA-3-8B [PITH_FULL_IMAGE:figures/full_fig_p012_24.png]
Figure 26
Figure 26. Figure 26: Design elements on Rouge F1 vs. recompute tokens for 2wikiMQA with LLaMA-3-8B and 30% recomp. The dotted line con￾nects different 𝛼. 𝛼=1 w/o (𝛽 and focus) 2500 5000 7500 10000 Prefill Length (Tokens) 0.65 0.70 0.75 0.80 0.85 0.90 Rouge F1 Score More Chunks Longer Chun…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

88 extracted references · 30 canonical work pages

  1. [1]

    [n. d.]. Amazon EC2 P4d Instances – AWS. https://aws.amazon.com/ec2/instance- types/p4/. (Accessed on 10/18/2024)

    2024

  2. [2]

    Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, Prashant Nair, Ilya Solovey- chik, and Purushotham Kamath. 2024. Keyformer: Kv cache reduction through key tokens selection for efficient generative inference. Proceedings of Machine Learning and Systems 6 (2024), 114–127

    2024

  3. [3]

    Shubham Agarwal, Gromit Yeuk-Yin Chan, Shaddy Garg, Tong Yu, and Subrata Mitra. 2023. Fast Natural Language Based Data Exploration with Samples. In Companion of the 2023 International Conference on Management of Data . 155–158

    2023

  4. [4]

    Shubham Agarwal, Subrata Mitra, Sarthak Chakraborty, Srikrishna Karanam, Koyel Mukherjee, and Shiv Kumar Saini. 2024. Approximate Caching for Effi- ciently Serving{Text-to-Image} Diffusion Models. In 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24) . 1173–1189

    2024

  5. [5]

    Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. 2024. Taming {Throughput-Latency} Tradeoff in{LLM} Inference with{Sarathi-Serve}. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). 117–134

    2024

  6. [6]

    Ghazi Shazan Ahmad, Shubham Agarwal, Subrata Mitra, Ryan Rossi, Manav Doshi, Vibhor Porwal, and Syam Manoj Kumar Paila. 2024. ScaleViz: Scaling Visualization Recommendation Models on Large Data. In Pacific-Asia Conference on Knowledge Discovery and Data Mining . Springer, 93–104

    2024

  7. [7]

    Friedman, Thomas Williams, Ramesh K

    Sohaib Ahmad, Hui Guan, Brian D. Friedman, Thomas Williams, Ramesh K. Sitara- man, and Thomas Woo. 2024. Proteus: A High-Throughput Inference-Serving Sys- tem with Accuracy Scaling. In Proceedings of the 29th ACM International Confer- ence on Architectural Support for Programming Languages and Operating Systems, Volume 1 (La Jolla, CA, USA)(ASPLOS ’24). A...

    arXiv 2024

  8. [8]

    Samaneh Aminikhanghahi, Tinghui Wang, and Diane J Cook. 2018. Real-time change point detection with application to smart home time series data. IEEE Transactions on Knowledge and Data Engineering 31, 5 (2018), 1010–1023

    2018

Show all 88 references
  1. [9]

    AI Anthropic. 2024. The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card 1 (2024)

    2024

  2. [10]

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al . 2023. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508 (2023)

    2023 arXiv

  3. [11]

    Anthony Chen, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. Eval- uating question answering evaluation. In Proceedings of the 2nd workshop on machine reading for question answering . 119–124

    2019

  4. [12]

    Jack Choquette, Wishwesh Gandhi, Olivier Giroux, Nick Stam, and Ronny Krashinsky. 2021. NVIDIA A100 Tensor Core GPU: Performance and Innovation. IEEE Micro 41, 2 (2021), 29–35. https://doi.org/10.1109/MM.2021.3061394

    2021

  5. [13]

    Vincent A Cicirello. 2019. Kendall tau sequence distance: Extending Kendall tau from ranks to sequences. arXiv preprint arXiv:1905.02752 (2019)

    2019 arXiv

  6. [14]

    Gonzalez, Ion Stoica, and Alexey Tumanov

    Daniel Crankshaw, Gur-Eyal Sela, Corey Zumar, Xiangxi Mo, Joseph E. Gonzalez, Ion Stoica, and Alexey Tumanov. 2020. InferLine: ML Prediction Pipeline Provi- sioning and Management for Tight Latency Objectives. arXiv:1812.01776 [cs.DC]

    2020 arXiv

  7. [15]

    Daniel Crankshaw, Xin Wang, Guilio Zhou, Michael J Franklin, Joseph E Gonzalez, and Ion Stoica. 2017. Clipper: A{Low-Latency} online prediction serving system. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17). 613–627

    2017

  8. [16]

    Florin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice, Cesare Campagnano, Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri. 2024. The power of noise: Redefining retrieval for rag systems. In Proceedings of the 47th International ACM SIGIR Conference o...

    2024

  9. [17]

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. Flashat- tention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems 35 (2022), 16344–16359

    2022

  10. [18]

    Jeffrey Dean and Sanjay Ghemawat. 2008. MapReduce: simplified data processing on large clusters. Commun. ACM 51, 1 (2008), 107–113

    2008

  11. [19]

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems 35 (2022), 30318–30332

    2022

  12. [20]

    Harry Dong, Xinyu Yang, Zhenyu Zhang, Zhangyang Wang, Yuejie Chi, and Beidi Chen. 2024. Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference. arXiv preprint arXiv:2402.09398 (2024)

    2024 arXiv

  13. [21]

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. A survey on in-context learning. arXiv preprint arXiv:2301.00234 (2022)

    2022 arXiv

  14. [22]

    Shichen Dong, Wen Cheng, Jiayu Qin, and Wei Wang. 2024. QAQ: Quality Adaptive Quantization for LLM KV Cache.arXiv preprint arXiv:2403.04643 (2024)

    2024 arXiv

  15. [23]

    Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. arXiv preprint arXiv:1903.00161 (2019)

    2019 arXiv

  16. [24]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

    2024 arXiv

  17. [25]

    Karima Echihabi. 2020. High-dimensional vector similarity search: from time series to deep network embeddings. In Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data . 2829–2832

    2020

  18. [26]

    Karima Echihabi, Kostas Zoumpatianos, and Themis Palpanas. 2021. New trends in high-d vector similarity search: al-driven, progressive, and distributed. Pro- ceedings of the VLDB Endowment 14, 12 (2021), 3198–3201

    2021

  19. [27]

    Elias Frantar and Dan Alistarh. 2023. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning . PMLR, 10323–10337

    2023

  20. [28]

    Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, and Pengfei Zuo. 2024. AttentionStore: Cost- effective Attention Reuse across Multi-turn Conversations in Large Language Model Serving. arXiv preprint arXiv:2403.19708 (2024)

    2024 arXiv

  21. [29]

    Minos N Garofalakis and Phillip B Gibbons. 2001. Approximate Query Processing: Taming the TeraBytes.. In VLDB, Vol. 10. 645927–672356

    2001

  22. [30]

    In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong. 2024. Prompt cache: Modular attention reuse for low-latency inference. Proceedings of Machine Learning and Systems 6 (2024), 325–338

    2024

  23. [31]

    Arpan Gujarati, Reza Karimi, Safya Alzayat, Wei Hao, Antoine Kaufmann, Ymir Vigfusson, and Jonathan Mace. 2020. Serving{DNNs} like clockwork: Perfor- mance predictability from the bottom up. In14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20) . 443–462

    2020

  24. [32]

    Cong Guo, Jiaming Tang, Weiming Hu, Jingwen Leng, Chen Zhang, Fan Yang, Yunxin Liu, Minyi Guo, and Yuhao Zhu. 2023. Olive: Accelerating large language models via hardware-friendly outlier-victim pair quantization. In Proceedings of the 50th Annual International Symposium on Co...

    2023

  25. [33]

    Tae Jun Ham, Yejin Lee, Seong Hoon Seo, Soosung Kim, Hyunji Choi, Sung Jun Jung, and Jae W Lee. 2021. ELSA: Hardware-software co-design for efficient, lightweight self-attention mechanism in neural networks. In 2021 ACM/IEEE 48th Annual International Symposium on Computer Arch...

    2021

  26. [34]

    Mingcong Han, Hanze Zhang, Rong Chen, and Haibo Chen. 2022. Microsecond- scale preemption for concurrent{GPU-accelerated}{ DNN} inferences. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22) . 539–558

    2022

  27. [35]

    Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reason- ing Steps. In Proceedings of the 28th International Conference on Computational Linguistics, Donia Scott, Nuria Bel, and Chengqing...

    2020 doi

  28. [36]

    Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. 2024. Kvquant: Towards 10 million context length llm inference with kv cache quantization. arXiv preprint arXiv:2401.18079 (2024)

    2024 arXiv

  29. [37]

    GI Ivchenko and SA Honov. 1998. On the jaccard similarity test. Journal of Mathematical Sciences 88 (1998), 789–794

    1998

  30. [38]

    Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023. Llmlingua: Compressing prompts for accelerated inference of large language models. arXiv preprint arXiv:2310.05736 (2023)

    2023 arXiv

  31. [39]

    Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023. Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression. arXiv preprint arXiv:2310.06839 (2023)

    2023 arXiv

  32. [40]

    Chao Jin, Zili Zhang, Xuanlin Jiang, Fangyue Liu, Xin Liu, Xuanzhe Liu, and Xin Jin. 2024. RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation. arXiv preprint arXiv:2404.12457 (2024)

    2024 arXiv

  33. [41]

    Woosuk Kwon, Sehoon Kim, Michael W Mahoney, Joseph Hassoun, Kurt Keutzer, and Amir Gholami. 2022. A fast post-training pruning framework for transform- ers. Advances in Neural Information Processing Systems 35 (2022), 24101–24116

    2022

  34. [42]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Princip...

    2023

  35. [43]

    2024.{InfiniGen}: Efficient generative inference of large language models with dynamic {KV} cache management

    Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. 2024.{InfiniGen}: Efficient generative inference of large language models with dynamic {KV} cache management. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). 155–172

    2024

  36. [44]

    Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, Daniel Haziza, Luca Wehrstedt, Jeremy Reizenstein, and Grigory Sizov

  37. [45]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. 14 Cache-Craft: Managing Chunk-Caches for E...

    2020

  38. [46]

    Muyang Li, Tianle Cai, Jiaxin Cao, Qinsheng Zhang, Han Cai, Junjie Bai, Yangqing Jia, Kai Li, and Song Han. 2024. Distrifusion: Distributed parallel inference for high-resolution diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni...

    2024

  39. [47]

    Shuo Li, Sangdon Park, Insup Lee, and Osbert Bastani. 2024. TRAQ: Trust- worthy Retrieval Augmented Question Answering via Conformal Prediction. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language...

    2024 doi

  40. [48]

    Tatiana Likhomanenko, Qiantong Xu, Gabriel Synnaeve, Ronan Collobert, and Alex Rogozhnikov. 2021. Cape: Encoding relative positions with continuous augmented positional embeddings. Advances in Neural Information Processing Systems 34 (2021), 16079–16092

    2021

  41. [49]

    Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. 74–81

    2004

  42. [50]

    Chin-Yew Lin and FJ Och. 2004. Looking for a few good metrics: ROUGE and its evaluation. In Ntcir workshop

    2004

  43. [51]

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics 12 (2024), 157–173

    2024

  44. [52]

    Shu Liu, Asim Biswal, Audrey Cheng, Xiangxi Mo, Shiyi Cao, Joseph E Gonza- lez, Ion Stoica, and Matei Zaharia. 2024. Optimizing llm queries in relational workloads. arXiv preprint arXiv:2403.05821 (2024)

    2024 arXiv

  45. [53]

    Yuhan Liu, Hanchen Li, Kuntai Du, Jiayi Yao, Yihua Cheng, Yuyang Huang, Shan Lu, Michael Maire, Henry Hoffmann, Ari Holtzman, et al. 2023. Cachegen: Fast context loading for language model applications. arXiv preprint arXiv:2310.07240 (2023)

    2023 arXiv

  46. [54]

    Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. 2024. Scissorhands: Ex- ploiting the persistence of importance hypothesis for llm kv cache compression at test time. Advances in Neural Information...

    2024

  47. [55]

    Chen-Yi Lu, Shubham Agarwal, Md Mehrab Tanjim, Kanak Mahadik, Anup Rao, Subrata Mitra, Shiv Kumar Saini, Saurabh Bagchi, and Somali Chaterji. 2024. RECON: Training-Free Acceleration for Text-to-Image Synthesis with Retrieval of Concept Prompt Trajectories. In European Conferen...

    2024

  48. [56]

    Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2024. Deepcache: Accelerating diffusion models for free. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 15762–15772

    2024

  49. [57]

    Jesse Mu, Xiang Li, and Noah Goodman. 2024. Learning to compress prompts with gist tokens. Advances in Neural Information Processing Systems 36 (2024)

    2024

  50. [58]

    Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. 2016. Ab- stractive text summarization using sequence-to-sequence rnns and beyond.arXiv preprint arXiv:1602.06023 (2016)

    2016 arXiv

  51. [59]

    Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745 (2018)

    2018 arXiv

  52. [60]

    Georg Ofenbeck, Ruedi Steinmann, Victoria Caparros, Daniele G Spampinato, and Markus Püschel. 2014. Applying the roofline model. In 2014 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) . IEEE, 76–85

    2014

  53. [61]

    Yongjoo Park, Barzan Mozafari, Joseph Sorenson, and Junhao Wang. 2018. Ver- dictdb: Universalizing approximate query processing. In Proceedings of the 2018 International Conference on Management of Data . 1461–1476

    2018

  54. [62]

    Zheng Qu, Liu Liu, Fengbin Tu, Zhaodong Chen, Yufei Ding, and Yuan Xie. 2022. Dota: detect and omit weak attentions for scalable transformer acceleration. In Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating S...

    2022

  55. [63]

    P Rajpurkar. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250 (2016)

    2016 arXiv

  56. [64]

    Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arX...

    2024 arXiv

  57. [65]

    Kai Ren, Qing Zheng, Joy Arulraj, and Garth Gibson. 2017. SlimDB: A space- efficient key-value storage engine for semi-sorted data. Proceedings of the VLDB Endowment 10, 13 (2017), 2037–2048

    2017

  58. [66]

    Yadwadkar, and Christos Kozyrakis

    Francisco Romero, Qian Li, Neeraja J. Yadwadkar, and Christos Kozyrakis

  59. [67]

    Haichen Shen, Lequn Chen, Yuchen Jin, Liangyu Zhao, Bingyu Kong, Matthai Philipose, Arvind Krishnamurthy, and Ravi Sundaram. 2019. Nexus: A GPU cluster engine for accelerating DNN-based video analysis. In Proceedings of the 27th ACM Symposium on Operating Systems Principles . 322–337

    2019

  60. [68]

    Yining Shi, Zhi Yang, Jilong Xue, Lingxiao Ma, Yuqing Xia, Ziming Miao, Yuxiao Guo, Fan Yang, and Lidong Zhou. 2023. Welder: Scheduling deep learning memory access via tile-graph. In 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23) . 701–718

    2023

  61. [69]

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing 568 (2024), 127063

    2024

  62. [70]

    Philippe Tillet. 2021. Triton: Open-source GPU programming for neural networks. https://openai.com/index/triton/

    2021

  63. [71]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)

    2023 arXiv

  64. [72]

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal

  65. [73]

    A Vaswani. 2017. Attention is all you need. Advances in Neural Information Processing Systems (2017)

    2017

  66. [74]

    Kefei Wang and Feng Chen. 2023. Catalyst: Optimizing Cache Management for Large In-memory Key-value Systems. Proceedings of the VLDB Endowment 16, 13 (2023), 4339–4352

    2023

  67. [75]

    Transactions of the Association for Computational Linguistics 10 (05 2022), 539–554

    ♪ MuSiQue: Multihop Questions via Single-hop Question Composition. Transactions of the Association for Computational Linguistics 10 (05 2022), 539–554. https://doi.org/10.1162/tacl_a_00475 arXiv:https://direct.mit.edu/tacl/article- pdf/doi/10.1162/tacl_a_00475/2020694/tacl_a_00475.pdf

    2022 doi

  68. [76]

    Zhiruo Wang, Jun Araki, Zhengbao Jiang, Md Rizwan Parvez, and Graham Neubig

  69. [77]

    2018.{VideoChef}: Efficient Approximation for Streaming Video Processing Pipelines

    Ran Xu, Jinkyu Koo, Rakesh Kumar, Peter Bai, Subrata Mitra, Sasa Misailovic, and Saurabh Bagchi. 2018.{VideoChef}: Efficient Approximation for Streaming Video Processing Pipelines. In 2018 USENIX Annual Technical Conference (USENIX ATC 18). 43–56

    2018

  70. [78]

    Kefei Wang, Jian Liu, and Feng Chen. 2020. Put an elephant into a fridge: opti- mizing cache efficiency for in-memory key-value stores. Proceedings of the VLDB Endowment 13, 9 (2020)

    2020

  71. [79]

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung- Gon Chun. 2022. Orca: A distributed serving system for{Transformer-Based} generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). 521–538

    2022

  72. [80]

    Haoyu Zhang, Ganesh Ananthanarayanan, Peter Bodik, Matthai Philipose, Paramvir Bahl, and Michael J Freedman. 2017. Live video analytics at scale with approximation and{Delay-Tolerance}. In 14th USENIX Symposium on Net- worked Systems Design and Implementation (NSDI 17) . 377–392

    2017

  73. [81]

    Hong Zhang, Yupeng Tang, Anurag Khandelwal, and Ion Stoica. 2023. {SHEPHERD}: Serving{DNNs} in the wild. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23) . 787–808

    2023

  74. [82]

    Yu Yan, Jiusheng Chen, Weizhen Qi, Nikhil Bhendawade, Yeyun Gong, Nan Duan, and Ruofei Zhang. 2021. El-attention: Memory efficient lossless attention for generation. In International Conference on Machine Learning. PMLR, 11648–11658

    2021

  75. [83]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhang- hao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric P Xing, et al. 2023. Lmsys- chat-1m: A large-scale real-world llm conversation dataset. arXiv preprint arXiv:2309.11998 (2023)

    2023 arXiv

  76. [84]

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Jeff Huang, Chuyue Sun, Cody_Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. 2023. Efficiently Programming Large Language Models using SGLang. (2023). 15

    2023

  77. [86]

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. 2024. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing...

    2024

  78. [2020]

    arXiv:1905.13348 [cs.DC] https://arxiv.org/abs/1905.13348

    INFaaS: A Model-less and Managed Inference Serving System. arXiv:1905.13348 [cs.DC] https://arxiv.org/abs/1905.13348

    1905 arXiv

  79. [2022]

    https: //github.com/facebookresearch/xformers

    xFormers: A modular and hackable Transformer modelling library. https: //github.com/facebookresearch/xformers

  80. [2023]

    Learning to filter context for retrieval-augmented generation.arXiv preprint arXiv:2311.08377 (2023)

    2023 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.