Rethinking Caching for LLM Serving Systems: Beyond Traditional Heuristics

Chanwoo Moon; Heejin Kim; Jaeheon Lee; Jungwoo Kim; Minsang Kim; Sungjin Lee; Taeho Hwang; Woosuk Chung; Yeseong Kim

REVIEW 4 major objections 5 minor 82 references

Rethinking Caching for LLM Serving Systems: Beyond Traditional Heuristics

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SISO claims that caching one answer per cluster of similar queries, evicting by long-term semantic popularity, and loosening the similarity threshold under overload lifts LLM cache hit ratios up to 1.71x over GPTCache.

desk verdict SISO is a solid, well-evaluated semantic-caching design for single-turn LLM workloads, but the headline hit-ratio and accuracy claims need tighter validation before I'd trust them. read the letter →

arxiv 2508.18736 v1 pith:7ADWNATD submitted 2025-08-26 cs.DB cs.LG

Jungwoo Kim , Minsang Kim , Jaeheon Lee , Chanwoo Moon , Heejin Kim , Taeho Hwang , Woosuk Chung , Yeseong Kim

show 1 more author

Sungjin Lee

This is my paper

classification cs.DBcs.LG

keywords semanticcachingLLMservingcentroid-basedcachereplacementlocalitydynamicthresholdSLOattainmentembeddingsimilarity

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SISO is a semantic caching layer for LLM serving that argues traditional caching instincts — store each query separately, evict by recency or frequency, always serve a hit — are wrong for LLM workloads. Instead it stores one centroid vector per cluster of semantically similar questions, keeps the centroids that represent the most future queries, and dynamically relaxes the similarity threshold when the serving system is overloaded so more requests are answered from cache instead of hitting the model. The payoff claimed is concrete: up to 1.71x higher cache hit ratios than GPTCache, better SLO attainment under heavy and bursty load, and roughly 1.85x less memory to reach the same hit ratio. The authors are explicit that the benefit is scoped to single-turn, meaning-based queries (information and advice seeking), and that the underlying similar-input-yields-similar-output assumption breaks on multi-turn queries and on coding and debugging tasks.

What carries the argument

The central object is the centroid of a query cluster: a single embedding vector — carrying a stored answer, a cluster_size, and an access_count — that stands in for many semantically similar queries. Around it two further mechanisms do their work: replacement is driven by cluster_size interpreted as semantic locality, which the paper measures as stable over weeks (96.1% of centroids change rank by no more than 10%), so re-clustering is triggered only when about 10% new queries accumulate; and SLO-aware serving uses a threshold-to-hit-ratio (T2H) table plus an M/D/1 queueing estimate of expected waiting time to pick the loosest similarity threshold that still meets the target SLO. The load-b

What would settle it

A controlled replay would settle it: take a corpus of multi-turn or code-editing queries where near-identical prompts legally have different answers (for example, fix-this-error prompts with different stack traces), run SISO with the threshold tuned to meet a given SLO, and measure how often the cached centroid answer is wrong, say with F1 below a fixed bar against a reference completion. If the wrong-answer rate exceeds what a deployment tolerates at the threshold that delivers the advertised hit-ratio gain, the core observation fails; the paper's own category split in §6, where coding and de

Watch

Extended reading notes

Core claim

SISO's central claim is that semantic caching should cache answers to questions, not answers to query strings. The paper establishes Observation #1 (high cosine similarity between queries strongly indicates duplicate meaning, with median similarities of 0.82 for duplicate pairs versus 0.62 for non-duplicates on QQP, MRPC, and MQP) and Observation #4 (input similarity correlates with output similarity), then builds a system in which historical query logs are clustered into centroids, each carrying one generated answer, a cluster_size measuring semantic locality, and an access counter. Cache replacement is triggered rarely, on a long window, swapping in centroids with large cluster_size; acces

Load-bearing premise

The whole system leans on the premise that semantically similar questions get semantically similar answers; where that premise fails — multi-turn context, coding, debugging — the cached answer can be wrong, so the hit-ratio gain turns directly into an accuracy loss.

Editorial extensions

If this is right

Semantic caching becomes substantially more memory-efficient: at equal hit ratios on Quora and Reddit, GPTCache needs roughly 1.85x more cache memory, and centroid caching reaches a given hit ratio with far smaller capacity.
SLO attainment survives heavy and bursty load: where vLLM and GPTCache fall below the target SLO as requests-per-second or the coefficient of variation rises, SISO keeps attainment high by converting load into cache hits instead of computation.
The approach is a drop-in complement to the serving stack: it sits alongside vLLM with no architecture-level changes, unlike scheduling, kernel-level, and KV-cache memory optimizations.
Quality is traded only when necessary: under light load the threshold rises so output quality matches uncached serving, while under heavy load the average win rate against vLLM stays near 42%.
The gains concentrate in single-turn, meaning-based queries (information seeking, advice seeking, editing, reasoning), which the paper argues dominate real workloads: 99% of API calls and 67% of chatbot interactions are single-turn.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If centroid caching works as claimed, the same collapse of repeated work into one representative applies beyond LLM caches — RAG retrievals, code-assist completions, and database result caches could store cluster representatives instead of individual entries; the measured stability of semantic locality (96% of centroids shift rank by under 10%) is what would make periodic re-clustering cheap enoug
A testable extension: apply the dynamic threshold per query category, since the paper's own category analysis shows the input-output similarity correlation is much stronger for advice/information seeking than for coding/brainstorming; routing only high-locality categories through the semantic cache could keep the SLO gains while removing the concentrated accuracy loss on code tasks.
The 6.9% average accuracy drop hides a concentrated error distribution: on tasks where the core assumption fails, mistakes cluster, so production deployments would need a guard such as detecting multi-turn or context-heavy queries and bypassing the cache — an extension of the repeated-query bypass SISO already implements.
An economic consequence left implicit: lowering the threshold under load turns cached answers into an admission-control mechanism, letting a provider absorb spikes without GPU over-provisioning; with the paper's estimate that clustering roughly a year of queries costs under $300, the break-even point is reachable at moderate query volumes.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a real systems paper. What's new in the LLM-serving context is caching cluster centroids instead of individual query vectors, plus semantic-locality-aware replacement and a load-driven similarity threshold. The evaluation covers five datasets, two model sizes, and the authors honestly report where it fails — multi-turn queries, coding and debugging. It deserves a serious referee.

What I like: the design is clean and the motivating observations are tested. Centroid caching is a genuine departure from GPTCache's per-vector LRU, and the memory-efficiency argument (1.27–1.14x hit-ratio gains at equal cache size, or ~1.85x less memory for equal hits) is plausible from Fig. 3. The dynamic threshold controller built on an M/D/1 queue plus a T2H table is practical and implemented, not just simulated. The stability analysis of semantic locality gives a principled reason to replace centroids slowly rather than with LRU/LFU. And the limitation section is refreshing: they explicitly say similar inputs do not always yield similar outputs, and they verify the consequence with category-level SLO results.

Where I get nervous: the headline hit-ratio gains are only as good as the hit definition. At theta_ret = 0.60, which their controller uses under load, the median cosine similarity for non-duplicate pairs is 0.62. So many of those extra hits are not semantically equivalent queries; they are just close in embedding space. The paper's accuracy checks (Alpaca-eval win rate, F1) are averaged over thresholds and request rates, so they don't tell us how many low-threshold hits are actually useful. The reported win-rate drop (~8 points at high RPS) signals real quality loss. That doesn't kill the paper, but it means "1.71x higher hit ratio" is not yet evidence of "1.71x more useful answers." Also, the abstract says "up to 1.71x" while the conclusion says "1.71x on average"; those are different claims and the paper should pin one down.

No code, no error bars. For a systems paper whose evidence is measurements, that is a real reproducibility problem. The design thresholds are tuned on the same embedding statistics used for accuracy evaluation, so the circularity is mild but worth acknowledging.

Bottom line: if you work on LLM serving or semantic caching, this is a useful contribution and the core design argument holds up. But treat the headline numbers as conditional until code and quality-gated hit analysis are available. I'd send it to peer review and ask for a major revision: release code, add error bars, and break down hit quality by threshold.

Referee Report

4 major / 5 minor

Summary. This paper proposes SISO, a semantic caching layer for LLM serving systems. SISO clusters historical queries into centroids and caches one representative output per centroid, replaces centroids based on semantic locality rather than recency/frequency, and dynamically adjusts the retrieval similarity threshold theta_t to balance response quality and SLO attainment under varying load. The evaluation compares SISO with vLLM and GPTCache on MSMARCO, NQ, Quora, Reddit, and ShareGPT using LLaMa-3.1-8B and 70B, reporting SLO attainment, E2E latency, cache hit ratios, and response quality measured by Alpaca-eval win rates and F1 against GPT-4. The paper claims up to 1.71x higher hit ratios, stronger SLO attainment, and only a marginal accuracy drop, while acknowledging that the approach is limited for multi-turn and coding/debugging queries.

Significance. Semantic caching is a promising orthogonal optimization for LLM serving, and SISO makes a useful conceptual step by caching centroids instead of individual query vectors and by making the similarity threshold workload-aware. The authors honestly report poor performance on ShareGPT and complex query categories, include two model sizes, and provide an ablation that isolates dynamic threshold adjustment. However, the central quantitative claims are not yet established. The hit-ratio gains are measured at thresholds where, by the paper's own similarity statistics, many non-duplicate query pairs would be counted as hits, and the quality metrics are aggregated separately from the hit-ratio experiments. If the authors can directly validate that low-threshold hits return acceptable outputs, the contribution is solid; with the current evidence the headline hit-ratio and SLO-improvement figures are not trustworthy.

major comments (4)

[Sec. 3.3, Fig. 7; Sec. 5.2.1, Fig. 18] The central hit-ratio claim is not validated as semantically useful. In Sec. 3.1/Fig. 2, non-duplicate query pairs have a median cosine similarity of 0.62, while Sec. 3.3/Fig. 7 explicitly considers theta_t = 0.60 and the controller in Sec. 4.3 lowers theta_t under heavy load. At that threshold a large fraction of non-duplicate pairs are within the match radius, so a measured 'hit' does not imply that the cached answer addresses the query. The headline 1.71x/1.54x hit ratios (abstract, Sec. 3.1) therefore conflate true and false hits. The quality checks in Sec. 5.2.7 (Figs. 14-15) are aggregate over all RPS and over all hits and misses, so they do not gate the hits counted in Figs. 9-13. The Sec. 6 limitation about multi-turn and coding tasks does not address this issue, since the false-hit problem also affects single-turn information-seeking queries at low theta_t. Please report hit rat
[Sec. 3.1, Fig. 3(a); Sec. 4.1] The choice of theta_c = 0.86 and the 'Impact on accuracy' analysis are circular. The threshold is selected from the duplicate/non-duplicate cosine-similarity distributions in Fig. 2, and the accuracy of centroid caching is then measured by the cosine similarity between the cached answer and the original answer (Fig. 3(a)). This uses the same type of metric as the design decision, so it cannot independently establish that cached centroids produce correct or acceptable outputs. The paper should validate with an independent quality metric, such as exact/semantic answer equivalence or human/LLM-judged correctness, and include a sensitivity analysis over theta_c and theta_t.
[Sec. 4.3, Eq. (2)] The M/D/1 model in Eq. (2) does not match the actual service-time distribution. With semantic caching, each request's service time is bimodal: near-zero for a cache hit and a full LLM generation for a miss. The derivation E[T] = S(1 - h(theta_t)) uses only the mean service time, but the M/D/1 waiting-time formula assumes deterministic service times. Feeding a mean into a deterministic-formula model can materially misestimate the waiting time, especially as h(theta_t) changes. Since dynamic thresholding is one of the three core contributions, the controller should either use an M/G/1 Pollaczek-Khinchine formula with measured service-time variance, or the paper should justify the deterministic assumption with empirical service-time distributions and a comparison of predicted vs. actual latency across RPS.
[Sec. 5.1, Figs. 9-13] The evaluation reports single-run point estimates without error bars, confidence intervals, or significance tests, and no code is released. The workloads use randomized Poisson arrivals and dataset splits, so the claim that SISO provides 'consistently' stronger SLO attainment (e.g., Figs. 9-10) needs variance across multiple seeds or runs. The absence of code also prevents checking details that the paper states but does not fully specify, such as the HNSW hierarchy placement, T2H table construction, and the re-clustering trigger. Adding reproducibility material or at least multi-run statistics is necessary to support the quantitative strength of the central claims.

minor comments (5)

[Figures throughout] Several figures have garbled axis labels and legend text after rendering (e.g., Fig. 14 y-axis appears as 't in rate', Fig. 9 legends appear as 'MaMa-3.1-8', etc.). Please regenerate the figures so all labels are legible and unambiguous.
[Sec. 3.1 and Sec. 4.1] The symbols theta_c and theta_t are introduced with different levels of formality. Please define both explicitly at first use and use them consistently in all equations, figures, and table captions.
[References] Reference [41] appears to duplicate reference [40] (AlpaServe) with a different proceedings name; please verify and merge or correct. Other reference URLs should be checked for consistency and accessibility.
[Sec. 5.2.7 and Introduction] The introduction states a 'marginal accuracy drop of 6.9%' but the results section does not derive this number directly. Please define the calculation (e.g., relative F1 drop, win-rate drop) and point to the specific figure or table that supports it.
[Appendix B] The statement that SISO deliberately lowers hit ratio under light load to improve output quality should be stated in the main text near Fig. 18, since readers may otherwise interpret the hit-ratio drop as an inconsistency with the headline hit-ratio improvement.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SISO's headline hit-ratio and SLO gains are empirical results from external comparisons; design thresholds and the T2H table are fitted controls, not predictions.

full rationale

I walked the paper's derivation chain. The abstract's 1.71x hit-ratio and SLO-attainment claims are measured results from §5, comparing SISO against vLLM, GPTCache, and SISO-NoDTA; they are not derived from the assumptions. §3's observations are empirical characterizations used to justify design choices: theta_c is chosen from duplicate/non-duplicate cosine-similarity distributions (Fig. 2), and the T2H table (§4.3) is an online estimator used by the controller, not a fitted quantity later reported as a prediction. Response quality is separately assessed with Alpaca-eval win rates and F1 scores against GPT-4 (§5.2.7), so the headline hit ratios are not self-graded by the same cosine metric used for clustering. The §6 limitation, that similar-input-similar-output fails for multi-turn and coding/debugging queries, is an acknowledged assumption boundary rather than a circular step. No load-bearing self-citations or imported uniqueness theorems appear; the system components are evaluated against external baselines. Finding: no significant circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

No new physical entities are introduced. The free parameters are design thresholds and policies chosen from the data analysis; the axioms are the semantic similarity premise, the embedding-space clustering premise, the queueing model, and the transferability of the clustering setup. These axioms are stated in the paper rather than proven from first principles.

free parameters (5)

clustering threshold theta_c = 0.86
Chosen from Observation #1 as the median duplicate cosine similarity across QQP, MRPC, MQP; controls cluster granularity and centroid count, directly affecting hit ratios.
retrieval threshold theta_t = 0.86 default, dynamically adjusted 0.60-0.98
Controls whether a query is a cache hit; fixed at 0.86 for GPTCache and SISO-NoDTA, dynamically tuned using the T2H table for SISO.
cache capacity = 6% of dataset vectors
Set by the authors referring to ChatGPT query log estimates; capacity directly determines the comparative hit-ratio results in §5.2.5.
re-clustering trigger = 10% new queries
Ad hoc policy to amortize clustering overhead; affects how quickly semantic locality updates, described in §4.1.
cluster_size decay factor = 1.1
In Algorithm 1, cluster_size is divided by 1.1 in each filtering step; chosen by hand to gradually age centroids.

assumptions (4)

domain assumption Similar inputs yield similar outputs (Observation #4)
Stated in §3.3 and §6; underpins the accuracy of returning cached outputs. The paper itself notes it fails for multi-turn, coding, and debugging queries.
domain assumption Embedding space clusters reflect query semantics
Used in §3.1-3.2 to justify centroid-based caching and semantic locality; depends on the chosen embedding model paraphrase-albert-small-v2.
domain assumption M/D/1 queue model with Poisson arrivals, deterministic service, FIFO
Used in §4.3 Eq. (1)-(2) to estimate waiting time W and adjust theta_t. Arrivals are simulated as Poisson in evaluation, so the model matches the test setup, but real workloads may not be Poisson.
domain assumption Community detection with cosine threshold produces good centroids
Clustering algorithm chosen in §4.1 based on a QQP experiment; the threshold theta_c = 0.86 is assumed to transfer across datasets.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Rethinking Caching for LLM Serving Systems: Beyond Traditional Heuristics." pith.science (2026). https://pith.science/paper/7ADWNATD

@misc{pith2026250818736,
  author       = {Pith},
  title        = {Pith review of: Rethinking Caching for LLM Serving Systems: Beyond Traditional Heuristics},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7ADWNATD}},
  note         = {Machine review of arXiv:2508.18736}
}

abstract

Serving Large Language Models (LLMs) at scale requires meeting strict Service Level Objectives (SLOs) under severe computational and memory constraints. Nevertheless, traditional caching strategies fall short: exact-matching and prefix caches neglect query semantics, while state-of-the-art semantic caches remain confined to traditional intuitions, offering little conceptual departure. Building on this, we present SISO, a semantic caching system that redefines efficiency for LLM serving. SISO introduces centroid-based caching to maximize coverage with minimal memory, locality-aware replacement to preserve high-value entries, and dynamic thresholding to balance accuracy and latency under varying workloads. Across diverse datasets, SISO delivers up to 1.71$\times$ higher hit ratios and consistently stronger SLO attainment compared to state-of-the-art systems.

Figures

Figures reproduced from arXiv: 2508.18736 by the authors.

**Figure 1.** Overall organization of SISO 3 Design Principles of SISO To overcome the limitations of caching for the LLM serving system, we propose SISO, a novel semantic caching framework. SISO is designed to leverage the semantic nature of LLM queries while maintaining the practical benefits of traditional caching, such as simple deployment and no architectural changes to underlying serving systems. The proposed SISO manage… view at source ↗

**Figure 4.** Hit ratios by policies Quora Reddit 0 50 100 Percenta e [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗

**Figure 6.** Heatmap of input/output cosine similarities 0.6 0.8 1 resold 0 0.5 1 it ratio (a) Quora 0.6 0.8 1 resold (b) Reddit [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗

Figures from the paper (7 more)

**Figure 8.** Figure 8: Implementation overview of SISO Impact of on hit ratio. Observation #4 indicates that the quality of outputs can be controlled by adjusting . Setting high (e.g., 0.98) ensures that only the centroids highly close to input queries are selected, thereby producing output…

**Figure 9.** Figure 9: In MSMARCO and NQ, SISO-NoDTA exhibits higher SLO attainment rates than GPTCache, highlighting the effectiveness of the centroid-based caching over LRU. However, at approximately 20 RPS, its SLO attainment declines due to the rising query volume. In contrast, SISO ma…

**Figure 10.** Figure 10: Impact of CV on SLO attainment 10 20 30 0 10 20 MGMHRCI 10 20 30 NQ 10 20 30 JaK LLaMa-3.1-8M GNareOPP 10 20 30 Quora 10 20 30 Reddit 1 2 RPG 0 30 60 1 2 RPG 1 2 RPG JQK LLaMa-3.1-70M 1 2 RPG 1 2 RPG R 2R latency S Ts [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]

**Figure 12.** Figure 12: Comparison of various cache replacement policies 0 1 Cacce capacity 0 0.1 0.2 dit ratio eaf MgMhRCi 0 1 Cacce capacity 0 0.2 0.4 ejf NQ 0 1 Cacce capacity 0 0.2 0.4 ecf gcarekPl mnmo pPqCacre [PITH_FULL_IMAGE:figures/full_fig_p011_12.png]

**Figure 13.** Figure 13: compares the cache hit ratios of SISO and GPTCache depending on cache size with fixed at 0.86. SISO stores only centroid vectors in the cache, but if free space remains, SISO caches individual vectors and manages them using LRU. Since the results for Quora and Reddit …

**Figure 14.** Figure 14: Win rate of SISO against vLLM 0 10 20 30 RP} 0 0.2 0.4 ~1 score M}MRC 0 10 20 30 RP} NQ 0 10 20 30 RP} }areP 0 10 20 30 RP} Quora 0 10 20 30 RP} Reddit M PCace }}-NoD }} [PITH_FULL_IMAGE:figures/full_fig_p012_14.png]

**Figure 17.** Figure 17: The topics in the shade of blue (Advice seeking, Information Seeking, Editing and Reasoning) are simple queries, and the ones in the shade of red (Planning, Data analysis, Creative writing, Coding&debugging and Brainstorming) are complex queries. These also correspon…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

82 extracted references · 67 canonical work pages

[1]

Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. 2024. Taming Throughput-Latency Tradeoﬀ in LLM Infer- ence with Sarathi-Serve. In Proceedings of the Symposium on Operat- ing Systems Design and Implementation

2024
[2]

Friedman, Thomas Williams, Ramesh K

Sohaib Ahmad, Hui Guan, Brian D. Friedman, Thomas Williams, Ramesh K. Sitaraman, and Thomas Woo. 2024. Proteus: A High- Throughput Inference-Serving System with Accuracy Scaling. In Pro- ceedings of the Conference on Architectural Support for Pro gramming Languages and Operating Systems

2024
[3]

Amazon Web Services, Inc. 2025. Amazon EC2 P5 instances. h/t_tps://aws.amazon.com/ec2/instance-types/p5/. Accessed 2025-08- 19

work page 2025
[4]

Mihael Ankerst, Markus M Breunig, Hans-Peter Kriegel, and Jörg Sander. 1999. OPTICS: Ordering Points to Identify the Clustering Structure. ACM Special Interest Group on Management of Data Record (1999)

work page 1999
[5]

Diochnos, and György Turán

Vanda Balogh, Gábor Berend, Dimitrios I. Diochnos, and György Turán. 2020. Understanding the Semantic Content of Sparse Word Embeddings Using a Commonsense Knowledge Base. In Proceedings of the Association for the Advancement of Artiﬁcial Intelli gence

work page 2020
[6]

Fu Bang. 2023. GPTCache: An Open-Source Semantic Cache for LL M Applications Enabling Faster Answers and Cost Savings. In Proceed- ings of the Workshop for Natural Language Processing Open Source Soft- ware

work page 2023
[7]

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeﬀrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin,...

work page 2020
[8]

Ricardo JGB Campello, Davoud Moulavi, and Jörg Sander. 2013. Density-based Clustering Based on Hierarchical Density Estimates. In Proceedings of the Conference on Knowledge Discovery and Da ta Mining

work page 2013

Show all 82 references

[9]

CBOT.ai. 2025. CBOT-LLM. h/t_tps://www.cbot.ai/cbot-llm/

2025
[10]

Harrison Chase. 2022. LangChain. h/t_tps://github.com/langchain-ai/langchain

2022
[11]

Chungmin Melvin Chen and Nicholas Roussopoulos. 1994. The Im - plementation and Performance Evaluation of the ADMS Query Op- timizer: Integrating Query Result Caching and Matching. In Proceed- ings of the International Conference on Extending DatabaseTechnology: Advances in Da...

1994
[12]

Han-Yi Chou and Sayan Ghosh. 2023. Batched Graph Community Detection on GPUs. In Proceedings of the International Conference on Parallel Architectures and Compilation Techniques

2023
[13]

Fu, Stefano Ermon, Atri Rudra, and Christoph er Ré

Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christoph er Ré. 2022. FlashAttention: Fast and Memory-eﬃcient Exact Attention with IO-awareness. In Proceedings of the International Conference on Neural Information Processing Systems

2022
[14]

Franklin, Björn Þór Jónsson, Divesh Srivast ava, and Michael Tan

Shaul Dar, Michael J. Franklin, Björn Þór Jónsson, Divesh Srivast ava, and Michael Tan. 1996. Semantic Data Caching and Replacement. In Proceedings of the International Conference on Very Large D ata Bases

1996
[15]

Redis Developers. 2025. Redis. h/t_tp://redis.io/

2025
[16]

Bill Dolan and Chris Brockett. 2005. Automatically Construc ting a Corpus of Sentential Paraphrases. In Proceedings of the International Workshop on Paraphrasing

2005
[17]

Fabio Duarte. 2025. Number of ChatGPT Users. h/t_tps://explodingtopics.com/blog/chatgpt-users

2025
[18]

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Ka- dian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The Llama 3 Herd of Models. arXiv:2407.21783

2024 arXiv
[19]

Hashimoto

Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. 2024. Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators. arXiv:2404.04475

2024 arXiv
[20]

Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu . 1996. A Density-based Algorithm for Discovering Clusters in Large Spatial Databases with Noise. In Proceedings of the Conference on Knowledge Discovery and Data Mining

1996
[21]

Hugging Face. 2025. What is Zero-Shot Classiﬁcation? h/t_tps://huggingface.co/tasks/zero-shot-classification

2025
[22]

Waris Gill, Mohamed Elidrisi, Pallavi Kalapatapu, Ammar Ahm ed, Ali Anwar, and Muhammad Ali Gulzar. 2024. MeanCache: User- Centric Semantic Cache for Large Language Model Based Web Ser- vices. arXiv:2403.02694

2024 arXiv
[23]

Google LLC. 2025. M2 Machine Series — Memory- optimized Machine Family for Compute Engine. h/t_tps://cloud.google.com/compute/docs/memory-optimized-machines

2025
[24]

Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive Re p- resentation Learning on Large Graphs. In Proceedings of the Advances in Neural Information Processing Systems

2017
[25]

Guseul Heo, Sangyeop Lee, Jaehong Cho, Hyunmin Choi, Sanghyeo n Lee, Hyungkyu Ham, Gwangsun Kim, Divya Mahajan, and Jongse Park. 2024. NeuPIMs: NPU-PIM Heterogeneous Acceleration for Batched LLM Inferencing. In Proceedings of the ACM International Conference on Architectural S...

2024
[26]

Connor Holmes, Masahiro Tanaka, Michael Wyatt, Ammar Ahmad Awan, Jeﬀ Rasley, Samyam Rajbhandari, Reza Yazdani Aminabadi, Heyang Qin, Arash Bakhtiari, Lev Kurilenko, and Yuxiong He. 2024. DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferenc...

2024 arXiv
[27]

Ke Hong, Guohao Dai, Jiaming Xu, Qiuli Mao, Xiuhong Li, Jun Liu, Kangdi Chen, Yuhan Dong, and Yu Wang. 2024. FlashDecod- ing++: Faster Large Language Model Inference with Asynchroniza- tion, Flat GEMM Optimization, and Heuristics. In Proceedings of Ma- chine Learning and Systems

2024
[28]

Ke Hong, Xiuhong Li, Lufang Chen, Qiuli Mao, Guohao Dai, Xuefei Ning, Shengen Yan, Yun Liang, and Yu Wang. 2025. SOLA: Optimizing SLO Attainment for Large Language Model Serving with State-Aware Scheduling. In Proceedings of Machine Learning and Systems

2025
[29]

Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami

Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W . Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. 2025. KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization. In Proceedings of the International Conference on Neural Informati...

2025
[30]

Cunchen Hu, Heyang Huang, Junhao Hu, Jiang Xu, Xusheng Chen, Tao Xie, Chenxi Wang, Sa Wang, Yungang Bao, Ninghui Sun, et al
[31]

Jinqi Huang, Yi Xiong, Xuebing Yu, Wenjie Huang, Entong Li, Li Zeng, and Xin Chen. 2025. SLO-Aware Scheduling for Large Lan- guage Model Inferences. arXiv:2504.14966

2025 arXiv
[32]

Shankar Iyer, Nikhil Dandekar, and Kornel Csernai
[33]

Jinwoo Jeong and Jeongseob Ahn. 2025. Accelerating LLM Serving for Multi-turn Dialogues with Eﬃcient Resource Management. InPro- ceedings of the ACM International Conference on Architectural Support for Programming Languages and Operating Systems . 13 Jungwoo Kim, Minsang Kim,...

2025
[34]

Keller and J

A.M. Keller and J. Basu. 1994. A Predicate-based Caching Sch eme for Client-server Database Architectures. In Proceedings of International Conference on Parallel and Distributed Information System s

1994
[35]

Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redﬁeld, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polo- sukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Sla...

2019
[36]

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica
[37]

Sanghyeon Lee, Hongbeen Kim, Soojin Hwang, Guseul Heo, Minwoo Noh, and Jaehyuk Huh. 2025. Eﬃcient LLM Inference with Activation Checkpointing and Hybrid Caching. arXiv:2501.01792

2025
[38]

Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. 2024. InﬁniGen: Eﬃcient Generative Inference of Large Language Models with Dynamic KV Cache Management. In Proceedings of the USENIX Conference on Operating Systems Design and Implementation

2024
[39]

Jiaxing Li, Chi Xu, Feng Wang, Isaac M von Riedemann, Cong Zhang, and Jiangchuan Liu. 2024. SCALM: Towards Semantic Caching for Automated Chat Services with Large Language Models. arXiv:2406.00025

2024 arXiv
[40]

Gonza- lez, and Ion Stoica

Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E. Gonza- lez, and Ion Stoica. 2023. AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving. In Proceedings of the Symposium ...

2023
[41]

Gonza- lez, and Ion Stoica

Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E. Gonza- lez, and Ion Stoica. 2023. AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving. In Proceedings of the USENIX Sym...

2023
[42]

James MacQueen et al. 1967. Some Methods for Classiﬁcatio n and Analysis of Multivariate Observations. In Proceedings of the Sympo- sium on Mathematical Statistics and Probability

1967
[43]

Malkov and D

Yu A. Malkov and D. A. Yashunin. 2020. Eﬃcient and Robust Approx- imate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence (2020)

2020
[44]

Clara H McCreery, Namit Katariya, Anitha Kannan, Manish Chablani, and Xavier Amatriain. 2020. Eﬀective Transfer Learning for Identify- ing Similar Questions: Matching User Questions to COVID-19 FAQs. In Proceedings of the Conference on Knowledge Discovery & Data Min- ing

2020
[45]

Xupeng Miao, Chunan Shi, Jiangfei Duan, Xiaoli Xi, Dahua Lin, Bin Cui, and Zhihao Jia. 2024. SpotServe: Serving Generative Large Lan- guage Models on Preemptible Instances. In Proceedings of the ACM In- ternational Conference on Architectural Support for Programming Lan- guage...

2024
[46]

Microsoft. 2025. Prompt caching with Azure OpenAI in Azure AI Foundry Models. h/t_tps://learn.microso/f_t.com/en-us/azure/ai-foundry/openai/how-to/prompt-caching

2025
[47]

T. Nguyen. 2016. MS MARCO: A Human Generated Machine Reading Comprehension Dataset. arXiv:1611.09268

2016 arXiv
[48]

Chengyi Nie, Rodrigo Fonseca, and Zhenhua Liu. 2024. Al- addin: Joint Placement and Scaling for SLO-Aware LLM Serving. arXiv:2405.06856

2024 arXiv
[49]

Li, Ryan McElroy, Mike Paleczny, Daniel Peek, Paul Saab, David Staﬀord, Tony Tung, and Venkateshwaran Venkatara- mani

Rajesh Nishtala, Hans Fugal, Steven Grimm, Marc Kwiatkowsk i, Her- man Lee, Harry C. Li, Ryan McElroy, Mike Paleczny, Daniel Peek, Paul Saab, David Staﬀord, Tony Tung, and Venkateshwaran Venkatara- mani. 2013. Scaling Memcache at Facebook. In Proceedings of the Symposium on Ne...

2013
[50]

NVIDIA. 2023. TensorRT-LLM: A TensorRT Tool- box for Optimized Large Language Model Inference. h/t_tps://github.com/NVIDIA/TensorRT-LLM

2023
[51]

NVIDIA Corporation. 2025. NVIDIA DGX H100/H200 System User Guide. h/t_tps://docs.nvidia.com/dgx/dgxh100-user-guide/index.html

2025
[52]

Ramya Prabhu, Ajay Nayak, Jayashree Mohan, Ramachandran Ram- jee, and Ashish Panwar. 2025. vAttention: Dynamic Memory Man- agement for Serving LLMs without PagedAttention. In Proceedings of the ACM International Conference on Architectural Suppo rt for Pro- gramming Languages ...

2025
[53]

Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2025. Mooncake: Trading More Storage for Less Computation — A KVCache-centric Architecture for Serving LLM Chatbot. In Proceedings of the USENIX Conference on File ...

2025
[54]

Nils Reimers. 2019. Sentence-Transformers Fast Clustering Al go- rithm. h/t_tps://github.com/UKPLab/sentence-transformers

2019
[55]

Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the Conference on Empirical Methods in Natural Language Proces sing

2019
[56]

Lütﬁ Kerem Şenel, Ihsan Utlu, Veysel Yücesoy, Aykut Koc, and Tolga Cukur. 2018. Semantic Structure and Interpretability of Word Em- beddings. IEEE/ACM Transactions on Audio, Speech, and Language Processing (2018)

2018
[57]

Sentence-Transformers. 2021. all-distilroberta-v1. h/t_tps://huggingface.co/sentence-transformers/all-distilroberta-v1

2021
[58]

Sentence-Transformers. 2021. all-mpnet-base-v2. h/t_tps://huggingface.co/sentence-transformers/all-mpnet-base-v2

2021
[59]

Sentence-Transformers. 2021. multi-qa-distilbert-cos-v1 . h/t_tps://huggingface.co/sentence-transformers/multi-qa-distilbert-cos-v1

2021
[60]

Sentence-Transformers. 2021. paraphrase-albert-small -v2. h/t_tps://huggingface.co/sentence-transformers/paraphrase-albert-small-v2

2021
[61]

Haichen Shen, Lequn Chen, Yuchen Jin, Liangyu Zhao, Bingyu Kong, Matthai Philipose, Arvind Krishnamurthy, and Ravi Sundaram. 2019. Nexus: A GPU Cluster Engine for Accelerating DNN-based Video Analysis. In Proceedings of the Symposium on Operating Systems Prin- ciples

2019
[62]

Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. 2023. FlexGen: High-throughput Generative Inference of Large Language Models with A Single GPU. In Proceedings of the In- ternational Conferenc...

2023
[63]

Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen. 2024. PowerIn- fer: Fast Large Language Model Serving with a Consumer-grade GPU. In Proceedings of the ACM SIGOPS Symposium on Operating System s Principles

2024
[64]

Tianxiang Sun, Xiaotian Zhang, Zhengfu He, Peng Li, Qinyuan Cheng, Xiangyang Liu, Hang Yan, Yunfan Shao, Qiong Tang, Shiduo Zhang, Xingjian Zhao, Ke Chen, Yining Zheng, Zhejian Zhou, Ruix- iao Li, Jun Zhan, Yunhua Zhou, Linyang Li, Xiaogui Yang, Lingling Wu, Zhangyue Yin, Xuan...
[65]

ShareGPT Team. 2023. ShareGPT. h/t_tps://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered. 14 Rethinking Caching for LLM Serving Systems: Beyond Traditi onal Heuristics

2023
[66]

Gomez, Łukasz Kaiser, and Illia Polosukhin

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. At - tention is All You Need. In Proceedings of the International Conference on Neural Information Processing Systems

2017
[67]

Jiahao Wang, Jinbo Han, Xingda Wei, Sijie Shen, Dingyan Zhang, Chenguang Fang, Rong Chen, Wenyuan Yu, and Haibo Chen. 2025. KVCache Cache in the Wild: Characterizing and Optimizing KV- Cache Cache at a Large Cloud Provider. In Proceedings of the USENIX Annual Technical Conference

2025
[68]

Ma- chine Intelligence Research (2024)

MOSS: An Open Conversational Large Language Model. Ma- chine Intelligence Research (2024)

2024
[69]

Yuxin Wang, Yuhan Chen, Zeyu Li, Xueze Kang, Zhenheng Tang, Xin He, Rui Guo, Xin Wang, Qiang Wang, Amelie Chi Zhou, and Xiaowen Chu. 2024. BurstGPT: A Real-world Workload Dataset to Optimiz e LLM Serving Systems. arXiv:2401.17644

2024 arXiv
[70]

Haojun Xia, Zhen Zheng, Xiaoxia Wu, Shiyang Chen, Zhewei Yao, Stephen Youn, Arash Bakhtiari, Michael Wyatt, Donglin Zhuang, Zhongzhu Zhou, Olatunji Ruwase, Yuxiong He, and Shuaiwen Leon Song. 2024. Quant-LLM: Accelerating the Serving of Large Lan- guage Models via FP6-Centric ...

2024
[71]

Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang. 2025. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion. In Proceedings of the European Conference on Computer Systems

2025
[72]

Yiding Wang, Kai Chen, Haisheng Tan, and Kun Guo. 2023. Tabi: An Eﬃcient Multi-Level Inference System for Large Language Models. In Proceedings of the European Conference on Computer Systems

2023
[73]

Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A Distributed Serving System for Transformer-Based Generative Models. In Proceedings of the Sympo- sium on Operating Systems Design and Implementation

2022
[74]

Shan Yu, Jiarong Xing, Yifan Qiao, Mingyuan Ma, Yangmin Li, Yang Wang, Shuo Yang, Zhiqiang Xie, Shiyi Cao, Ke Bao, Ion Stoica, Harry Xu, and Ying Sheng. 2025. Prism: Unleashing GPU Sharing for Cost- Eﬃcient Multi-LLM Serving. arXiv:2505.04021

2025 arXiv
[75]

Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. 2024. Wildchat: 1m ChatGPT Interaction Logs in the Wild. arXiv:2405.01470

2024 arXiv
[76]

Yiling-J. 2019. cacheme: Asyncio cache framework with multipl e cache storages. h/t_tps://github.com/Yiling-J/cacheme

2019
[77]

Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xu- anzhe Liu, Xin Jin, and Hao Zhang. 2024. DistServe: Disaggregating Preﬁll and Decoding for Goodput-optimized Large Language Model Serving. In Proceedings of the USENIX Conference on Operating Sys- tems Design and ...

2024
[78]

Ruihang Lai Wuwei Lin Yineng Zhang Stephanie Wang Tianqi Chen Baris Kasikci Vinod Grover Arvind Krishnamurthy Luis Ceze Zi- hao Ye, Lequn Chen. 2025. FlashInfer: Eﬃcient and Customizable Attention Engine for LLM Inference Serving. In Proceedings of Ma- chine Learning and Syste...

2025
[80]

Gonzalez, Clark Barrett, and Ying Sheng

Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeﬀ Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. 2024. SGLang: Eﬃcient Execution of Structured Language Model Programs. In Ad- vances in Neural Informa...

2024
[2017]

h/t_tps://www.quora.com/q/quoradata/First-/Q_uora-Dataset-Release-/Q_uestion-Pairs

First Quora Dataset Release: Question Pairs. h/t_tps://www.quora.com/q/quoradata/First-/Q_uora-Dataset-Release-/Q_uestion-Pairs
[2023]

In Proceedings of the Symposium on Operat- ing Systems Principles

Eﬃcient Memory Management for Large Language Model Serv- ing with PagedAttention. In Proceedings of the Symposium on Operat- ing Systems Principles
[2024]

arXiv:2406.17565

Memserve: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool. arXiv:2406.17565

arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. 2024. Taming Throughput-Latency Tradeoﬀ in LLM Infer- ence with Sarathi-Serve. In Proceedings of the Symposium on Operat- ing Systems Design and Implementation

2024

[2] [2]

Friedman, Thomas Williams, Ramesh K

Sohaib Ahmad, Hui Guan, Brian D. Friedman, Thomas Williams, Ramesh K. Sitaraman, and Thomas Woo. 2024. Proteus: A High- Throughput Inference-Serving System with Accuracy Scaling. In Pro- ceedings of the Conference on Architectural Support for Pro gramming Languages and Operating Systems

2024

[3] [3]

Amazon Web Services, Inc. 2025. Amazon EC2 P5 instances. h/t_tps://aws.amazon.com/ec2/instance-types/p5/. Accessed 2025-08- 19

work page 2025

[4] [4]

Mihael Ankerst, Markus M Breunig, Hans-Peter Kriegel, and Jörg Sander. 1999. OPTICS: Ordering Points to Identify the Clustering Structure. ACM Special Interest Group on Management of Data Record (1999)

work page 1999

[5] [5]

Diochnos, and György Turán

Vanda Balogh, Gábor Berend, Dimitrios I. Diochnos, and György Turán. 2020. Understanding the Semantic Content of Sparse Word Embeddings Using a Commonsense Knowledge Base. In Proceedings of the Association for the Advancement of Artiﬁcial Intelli gence

work page 2020

[6] [6]

Fu Bang. 2023. GPTCache: An Open-Source Semantic Cache for LL M Applications Enabling Faster Answers and Cost Savings. In Proceed- ings of the Workshop for Natural Language Processing Open Source Soft- ware

work page 2023

[7] [7]

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeﬀrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin,...

work page 2020

[8] [8]

Ricardo JGB Campello, Davoud Moulavi, and Jörg Sander. 2013. Density-based Clustering Based on Hierarchical Density Estimates. In Proceedings of the Conference on Knowledge Discovery and Da ta Mining

work page 2013