Pith. sign in

REVIEW 2 major objections 4 minor 3 cited by

Auditing Prompt Caching in Language Model APIs

T0 review · 2 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper develops a statistical audit that detects prompt caching in eight of seventeen LLM API providers and shows that in seven of them, including OpenAI, the cache is shared globally across users, so an attacker can infer from fast…

desk verdict Solid, important audit of prompt-caching side channels; the missing control for the victim-priming burst is a real gap, but not a fatal one. read the letter →

arxiv 2502.07776 v2 pith:YAWIU6OS submitted 2025-02-11 cs.CL cs.CRcs.LG

Chenchen Gu , Xiang Lisa Li , Rohith Kuditipudi , Percy Liang , Tatsunori Hashimoto This is my paper · ORCID
classification cs.CLcs.CRcs.LG
keywords promptcachingKVcachereusetimingsidechannelstatisticalhypothesistestingsharingauditLLMAPIprivacydecoder-onlyTransformerinferencetimetofirsttoken
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper investigates a privacy risk created by prompt caching in LLM APIs: because cached prompts are processed faster, response times become a side channel. The authors build a statistical audit that compares time-to-first-token distributions between a procedure that tries to trigger a cache hit (one user primes a prompt, another user sends it or a prefix variant) and one that only produces cache misses; under no caching the distributions are identical, so any difference is evidence of caching. Applied to 17 API providers in late 2024, the audit detects caching in 8 and global cross-user cache sharing in 7, including OpenAI's embedding API. If these observations are correct, an attacker with API access can detect that another user recently sent a prompt sharing a long prefix with a candidate prompt, a real privacy leak. The same timing signal also reveals architecture: prefix caching with different suffixes is only possible in decoder-only Transformers, and the authors use it to infer that OpenAI's text-embedding-3-small is decoder-only.

What carries the argument

The load-bearing object is the two-procedure timing audit built on a two-sample Kolmogorov-Smirnov hypothesis test. Procedure Hit: a 'victim' account sends a prompt once or several times to populate the cache, then an 'attacker' account sends the same prompt (or a prompt sharing 95% of its prefix) and the time to first token is recorded. Procedure Miss: a random prompt with negligible prefix-overlap probability is sent and timed. Under the null of no caching the two distributions are equal; the one-sided KS test rejects when cache-hit times are systematically faster. Cache-sharing levels are isolated by choosing how far apart the victim and attacker accounts are (same user, same organization, different organizations), and the architecture inference rides on the observation that a cache hit with different suffixes requires decoder-only attention where each token attends only to previous tokens.

What would settle it

Run the same two-procedure audit with the priming request sent after the attacker's measurement, or with victim and attacker accounts swapped so no cross-account cache can have been populated; if time-to-first-token distributions still separate at the paper's significance level, the detections could be routing artifacts rather than shared prompt caches.

Watch

Extended reading notes

Core claim

The central claim is that prompt caching can be detected and its scope measured remotely through response-time statistics, and that doing so reveals real cross-user privacy leakage in deployed APIs. Concretely, the paper reports detecting prompt caching in 8 of 17 providers (Anthropic, Azure, Deep Infra, Fireworks, Lepton, OpenAI, Perplexity, Replicate) and global cache sharing—cache hits across users in different organizations—in 7 of them. For the OpenAI and Azure text-embedding-3-small embedding APIs, global sharing means another user's prompt can leave a timing trace. The paper also claims that observing cache hits when two prompts share a prefix but differ in their suffixes identifies the model as a decoder-only Transformer; applied to text-embedding-3-small, this gives evidence that OpenAI's embedding model is decoder-only, previously not publicly known. The reported p-values lie far below a significance threshold of $\alpha = 10^{-8}$, and the average precision for classifying cache-hit from cache-miss times is mostly around 0.8; after responsible disclosure, at least five providers are reported to have changed their caching behavior.

Load-bearing premise

The audit assumes that the only systematic difference between the cache-hit and cache-miss procedures is the cache itself: if a provider routes the two procedures to different servers or otherwise makes latency depend on which account primed a prompt, the statistical test could fire for reasons unrelated to caching.

Editorial extensions

If this is right

  • Global cache sharing means a remote attacker can learn whether another user recently sent any prompt whose long prefix matches an attacker-chosen candidate, using only response-time measurements from a single API key.
  • Per-user caching fully removes this privacy leak, while per-organization caching confines it to users within the same organization; the paper's detection of per-organization sharing in Anthropic and OpenAI matches their public documentation.
  • Because prefix caching with different suffixes implies decoder-only attention, the method converts a latency measurement into an architecture fingerprint for proprietary models such as text-embedding-3-small.
  • The same audit can be rerun at any time to check whether providers changed their cache sharing after disclosure; at least five providers are reported to have done so.
  • Attackers can classify cache hits from misses with average precision around 0.8 in affected APIs, meaning identification of whether a cached prompt shares a prefix with a victim prompt is practical even with one priming request in most providers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the two-procedure design transfers to other latency-changing caches (semantic response caches, router caches), because the null hypothesis only requires equality of the two timing distributions; testing near-duplicate prompts would extend the audit without new machinery.
  • Editorial inference: the paper's unsuccessful token-by-token extraction does not bound membership-inference risk for structured prompts (e.g., known templates with a user-inserted secret); a targeted probe that checks whether a template prefix with a guessed secret is cached would be a cheaper, testable attack.
  • Editorial inference: the architecture fingerprinter could be applied to other black-box embedding or reranking services; observing prefix-sharing latency differences is evidence for causal or decoder-only architectures and could become a standard black-box test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The manuscript develops and applies a statistical audit for prompt caching in commercial LLM APIs. The audit frames caching as a deviation from the null hypothesis that cache-hit and cache-miss procedures yield identical time-to-first-token distributions, then tests that null with randomized ordering, one-sided two-sample Kolmogorov-Smirnov tests, adaptive NUMVICTIM_REQUESTS with Bonferroni correction, and a significance level of 1e-8. On 17 providers, the authors report detecting prompt caching in 8 and global cross-user cache sharing in 7, with average precisions around 0.7-1.0, and they further infer that OpenAI's text-embedding-3-small is a decoder-only Transformer from observed same-prefix/different-suffix caching. The paper includes ablations, full p-value tables, responsible disclosure, and released code and data.

Significance. If the main detections are causally attributable to cross-user prompt caching, the result is significant: it demonstrates a practical timing side channel in deployed APIs and identifies a previously unknown architecture property of a proprietary embedding model. The paper's statistical workflow is a model for such audits: pre-specified significance level, one-sided tests, Bonferroni corrections across NUMVICTIM_REQUESTS and timing sources, randomized ordering, full reporting of p-values, negative controls that agree with documented per-organization policies, and public code and data. The principal weakness is the lack of a control that varies the priming structure independently of cache state; until that is supplied, the causal reading of the seven global-sharing detections remains defensible but not fully established.

major comments (2)
  1. [Section 3.2, cache-hit and cache-miss procedures] The two procedures differ not only in whether a matching prefix is cached, but also in the request history. The cache-hit procedure follows NUMVICTIM_REQUESTS (up to 25) consecutive victim requests with the measured attacker request, while the cache-miss procedure sends a single request with no preceding burst. Under H0 as stated in Section 3.1, the two procedures are assumed to differ only in cache state, so any provider-side effect of the burst - model or instance warm-up, dynamic routing, per-key or per-account scheduling - can make Dhit faster than Dmiss even with no cross-user prompt caching. This confound applies to every level in Section 4 and to the Section 5 architecture inference. The global non-detections at Anthropic and OpenAI GPT-4o mini are reassuring but do not isolate the burst factor, since the burst structure is not varied independently of cache state. A sham-priming control (victim sends a non-matching random prompt before the attacker's measurement) or a role-swap control would directly test this alternative; without it, the abstract's claim that global cache sharing was detected in seven providers is not uniquely identified by the reported comparisons.
  2. [Section 5, architecture inference] The decoder-only inference for text-embedding-3-small rests on the same timing comparison as the main audit and therefore inherits the Section 3.2 confound. The additional observation that the returned embedding changes when the suffix changes rules out whole-prompt output caching, but it does not establish that the timing difference is caused by prefix-level KV-cache reuse rather than by the priming burst. Please either add a control that produces a prefix-matching attacker request without the victim burst, or soften the architecture claim to a conditional inference.
minor comments (4)
  1. [Section 3.2, cache miss paragraph] As written, the cache-miss procedure says 'uvictim sends a random prompt x ~ P', whereas Section 3.1 and the surrounding logic require the attacker to send it; if taken literally this would compare times from different accounts, and in any case the user attribution should be corrected.
  2. [Table 1] Average precision values are reported as point estimates without uncertainty; adding bootstrap confidence intervals would better convey sampling variability, especially for values around 0.7-0.8.
  3. [Sections 1 and 8 / Impact Statement] The responsible disclosure text appears nearly verbatim in the introduction and in the Impact Statement; consider consolidating it into a single statement.
  4. [Appendix C, Figure 7] The colors in Figure 7 are not identified in the figure itself; adding a legend or a note referencing the Figure 5 legend would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the audit is an empirical measurement against external APIs, with hit/miss distributions defined independently of the conclusions and no fitted parameter renamed as a prediction.

full rationale

The paper's central claims are empirical findings about external API providers, not derivations from definitions or self-citations. The null hypothesis H0 : Dhit = Dmiss is stated independently of the conclusion, and the audit compares timing distributions from two procedures that are explicitly defined in Section 3.2. No parameter is fitted to produce the reported p-values; the KS test is applied with fixed configuration and Bonferroni corrections. The cache-hit procedure includes a victim-priming burst, which is a potential threat to the validity of the comparison, but that is an experimental confound, not a circular reduction: the paper does not define 'caching detected' as 'the cache-hit procedure is faster than the cache-miss procedure' by construction. The architecture claim for OpenAI's text-embedding-3-small is explicitly conditional on Transformer-based assumptions and is not derived from the cache-sharing results in a self-referential way. The only relevant self-citation is to Gim et al. (2024) on prompt caching mechanics, which appears in the background section and is not load-bearing for the audit methodology or conclusions. Therefore, no circular step meeting the evidence standard is present.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The central detection claim rests on the timing-signal assumptions in Section 2.1 and the statistical setup in Section 3; none of the audit configuration parameters is fitted to produce the reported p-values. The architecture inference additionally depends on the Transformer premise stated in Section 5.

free parameters (5)
  • PROMPT_LENGTH = 5000 tokens
    Length of random prompts used in the audit; ablations in Section 4.3 show average precision degrades for short prompts, so this choice affects detection power. It is a hand-chosen configuration value, not fitted to provider results.
  • PREFIX_FRACTION = 0.95
    Fraction of prompt tokens that must match for the same-prefix different-suffix test in levels 2 to 4; Section 4.3 ablates this value.
  • NUMSAMPLES = 250
    Number of TTFT samples per procedure; affects statistical power.
  • NUM_VICTIM_REQUESTS = 1, 5, or 25 with early stopping
    Number of consecutive victim requests used to populate the cache; tested in increasing order with Bonferroni correction for three tests.
  • significance level alpha = 1e-8 before Bonferroni correction
    Per-test threshold chosen to control false positives across providers, cache levels, and timing sources.
assumptions (6)
  • domain assumption Cache hits are faster than cache misses after accounting for prompt length.
    Stated as property 2 in Section 2.1; the entire timing audit relies on this latency gap being measurable through API response times.
  • domain assumption Random prompts drawn from P have negligible probability of sharing a long prefix with any cached prompt.
    Section 3.2 argues the probability of sharing a 15-token prefix is below 1e-25, so cache miss timings are uncontaminated.
  • domain assumption All audited API tokenizers split on whitespace, so all prompts in P are exactly PROMPT_LENGTH tokens, and formatting or special tokens add a small constant number of tokens.
    Section 3.2 uses this to control prompt lengths in the cache-hit procedure.
  • domain assumption Setting maximum tokens to 1 makes the total response time equal to the time to first token.
    Section 2.2, used to measure prefill latency.
  • domain assumption text-embedding-3-small is Transformer-based.
    Section 5 explicitly states this assumption before concluding the architecture is decoder-only from prefix caching.
  • domain assumption Prefix caching with different suffixes is possible only in decoder-only Transformers, not in encoder-only or encoder-decoder models with bidirectional attention.
    Section 5 uses this architectural premise to convert the observed prefix caching into an architecture disclosure.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Auditing Prompt Caching in Language Model APIs." pith.science (2026). https://pith.science/paper/YAWIU6OS

@misc{pith2026250207776,
  author       = {Pith},
  title        = {Pith review of: Auditing Prompt Caching in Language Model APIs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YAWIU6OS}},
  note         = {Machine review of arXiv:2502.07776}
}
read the original abstract

Prompt caching in large language models (LLMs) results in data-dependent timing variations: cached prompts are processed faster than non-cached prompts. These timing differences introduce the risk of side-channel timing attacks. For example, if the cache is shared across users, an attacker could identify cached prompts from fast API response times to learn information about other users' prompts. Because prompt caching may cause privacy leakage, transparency around the caching policies of API providers is important. To this end, we develop and conduct statistical audits to detect prompt caching in real-world LLM API providers. We detect global cache sharing across users in seven API providers, including OpenAI, resulting in potential privacy leakage about users' prompts. Timing variations due to prompt caching can also result in leakage of information about model architecture. Namely, we find evidence that OpenAI's embedding model is a decoder-only Transformer, which was previously not publicly known.

Figures

Figures reproduced from arXiv: 2502.07776 by the authors.

Figure 1
Figure 1. An example illustrating prompt caching. (1) A victim sends a prompt to the API, which then becomes cached. (2) An attacker sends a new prompt, resulting in a cache miss and slow response time. (3) An attacker sends a prompt that shares a prefix with the victim’s prompt, resulting in a cache hit. From the fast response time, the attacker can infer that a cache hit occurred, which potentially reveals information about… view at source ↗
Figure 2
Figure 2. Organizations contain users, and the global level con￾tains all users and organizations of an API. tions about how prompts are routed to servers. A prompt may be randomly routed, or it may be intentionally routed to a server where the prompt is already cached. 2.2. API Assumptions We assume that it is possible to send arbitrary prompts to the API (possibly subject to some maximum length) and mea￾sure the TTFT. The T… view at source ↗
Figure 3
Figure 3. Histograms of response times from the cache hit and cache miss procedures in APIs where we detected caching. The distributions of times are clearly distinguishable, with cache hits tending to be faster. Each histogram title states the API provider, model, level of cache sharing (per-org. or global), timing source (client-side or server-side timing), and the NUMVICTIMREQUESTS used, denoted V. response tokens used is … view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Selected precision-recall curves for distinguishing be￾tween times from the cache hit and cache miss procedures. Cache hits are the positive class. The curves show that cache hits can be detected with near perfect precision up to moderate recall scores [PITH_FULL_IMAG…
Figure 5
Figure 5. Figure 5: Ablations on the effects of PROMPTLENGTH, PREFIXFRACTION, and model size on the average precision. In (a)–(c), as the prompt length or prefix match length decreases, the average precision decreases to random chance. In (d), we detect caching across all model sizes, wit…
Figure 6
Figure 6. Figure 6: Precision-recall curves for distinguishing between times produced by the cache hit and cache miss procedures in APIs where we detected caching in our audits ( [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Ablations on the effects of PROMPTLENGTH, PREFIXFRACTION, and model size on the audit p-values. Each test is run using NUMSAMPLES = 250. The top and bottom rows display the p-values on linear and logarithmic scales, respectively. In (a)–(c), as the prompt length or pre…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Leaky Language Models: Stealing Architecture and Inference Optimizations via Per-Token Timing

    cs.CR 2026-07 conditional novelty 7.0 of 10

    Per-token generation timing leaks speculative decoding and draft-model context length from Gemini, and recovers layer count and hidden size of Llama-family models with top-5 accuracy up to 65% when both are unknown.

  2. Cache-Aware Prompt Compression:A Two-Tier Cost Model for LLM API Caching

    cs.LG 2026-07 conditional novelty 7.0 of 10

    CAPC combines query-agnostic compression with prompt caching and is reported cheapest in 16/16 LongBench configurations, but the empirical support has unresolved specification and consistency gaps.

  3. From Similarity to Vulnerability: Key Collision Attack on LLM Semantic Caching

    cs.CR 2026-01 conditional novelty 6.0 of 10

    An attacker can force false cache-key collisions in LLM semantic caching and hijack responses or agent tool calls, with 86% hit rate in experiments.

Reference graph

Works this paper leans on

47 extracted references · 24 canonical work pages · cited by 3 Pith papers

  1. [1]

    Prompt caching (beta)

    Anthropic. Prompt caching (beta). https://web.archive.org/web/20241204121302/https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching, 2024 a

    arXiv 2024

  2. [2]

    Prompt caching with claude

    Anthropic. Prompt caching with claude. https://web.archive.org/web/20240814170229/https://www.anthropic.com/news/prompt-caching, 2024 b

    arXiv 2024

  3. [3]

    GPTC ache: An open-source semantic cache for LLM applications enabling faster answers and cost savings

    Bang, F. GPTC ache: An open-source semantic cache for LLM applications enabling faster answers and cost savings. In Tan, L., Milajevs, D., Chauhan, G., Gwinnup, J., and Rippeth, E. (eds.), Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023), pp.\ 212--218, Singapore, December 2023. Association for Computatio...

    doi:10.18653/v1/2023.nlposs-1.24 2023

  4. [4]

    Bernstein, D. J. Cache-timing attacks on aes. 2005. URL https://cr.yp.to/antiforgery/cachetiming-20050414.pdf

    work page 2005

  5. [5]

    and Mironov, I

    Bonneau, J. and Mironov, I. Cache-collision timing attacks against aes. In Cryptographic Hardware and Embedded Systems-CHES 2006: 8th International Workshop, Yokohama, Japan, October 10-13, 2006. Proceedings 8, pp.\ 201--215. Springer, 2006

    work page 2006

  6. [6]

    and Boneh, D

    Bortz, A. and Boneh, D. Exposing private information by timing web applications. In Proceedings of the 16th international conference on World Wide Web, pp.\ 621--628, 2007

    work page 2007

  7. [7]

    and Nasr, M

    Carlini, N. and Nasr, M. Remote timing attacks on efficient language model inference. arXiv preprint arXiv:2410.17175, 2024

    arXiv 2024

  8. [8]

    D., Steinke, T., Hayase, J., Cooper, A

    Carlini, N., Paleka, D., Dvijotham, K. D., Steinke, T., Hayase, J., Cooper, A. F., Lee, K., Jagielski, M., Nasr, M., Conmy, A., Wallace, E., Rolnick, D., and Tram\` e r, F. Stealing part of a production language model. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Proceedings of the 41st Int...

    work page 2024

Show all 47 references
  1. [9]

    Accelerating large language model decoding with speculative sampling

    Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, L., and Jumper, J. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023

    2023 arXiv

  2. [10]

    Deepseek api introduces context caching on disk, cutting prices by an order of magnitude

    DeepSeek. Deepseek api introduces context caching on disk, cutting prices by an order of magnitude. https://web.archive.org/web/20241120201047/https://api-docs.deepseek.com/news/news0802, 2024

    2024

  3. [11]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  4. [12]

    Felten, E. W. and Schneider, M. A. Timing attacks on web privacy. In Proceedings of the 7th ACM Conference on Computer and Communications Security, pp.\ 25--32, 2000

    2000

  5. [13]

    Logits of API -protected LLM s leak proprietary information

    Finlayson, M., Ren, X., and Swayamdipta, S. Logits of API -protected LLM s leak proprietary information. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=oRcYFm8vyB

    2024

  6. [14]

    Prompt caching

    Fireworks. Prompt caching. https://web.archive.org/web/20250118003034/https://docs.fireworks.ai/guides/prompt-caching, 2024

    2024

  7. [15]

    A new algorithm for data compression

    Gage, P. A new algorithm for data compression. The C Users Journal, 12 0 (2): 0 23--38, 1994

    1994

  8. [16]

    Prompt cache: Modular attention reuse for low-latency inference

    Gim, I., Chen, G., Lee, S.-s., Sarda, N., Khandelwal, A., and Zhong, L. Prompt cache: Modular attention reuse for low-latency inference. Proceedings of Machine Learning and Systems, 6: 0 325--338, 2024

    2024

  9. [17]

    Cache games--bringing access-based cache attacks on aes to practice

    Gullasch, D., Bangerter, E., and Krenn, S. Cache games--bringing access-based cache attacks on aes to practice. In 2011 IEEE Symposium on Security and Privacy, pp.\ 490--505. IEEE, 2011

    2011

  10. [18]

    The significance probability of the smirnov two-sample test

    Hodges Jr, J. The significance probability of the smirnov two-sample test. Arkiv f \"o r matematik , 3 0 (5): 0 469--486, 1958

    1958

  11. [19]

    Epic: Efficient position-independent context caching for serving large language models

    Hu, J., Huang, W., Wang, H., Wang, W., Hu, T., Zhang, Q., Feng, H., Chen, X., Shan, Y., and Xie, T. Epic: Efficient position-independent context caching for serving large language models. arXiv preprint arXiv:2410.15332, 2024

    arXiv

  12. [20]

    Y., R \'e , C., and Mirhoseini, A

    Juravsky, J., Brown, B., Ehrlich, R., Fu, D. Y., R \'e , C., and Mirhoseini, A. Hydragen: High-throughput llm inference with shared prefixes. arXiv preprint arXiv:2402.05099, 2024

    2024 arXiv

  13. [21]

    Spectre attacks: Exploiting speculative execution

    Kocher, P., Horn, J., Fogh, A., Genkin, D., Gruss, D., Haas, W., Hamburg, M., Lipp, M., Mangard, S., Prescher, T., Schwarz, M., and Yarom, Y. Spectre attacks: Exploiting speculative execution. In 2019 IEEE Symposium on Security and Privacy (SP), pp.\ 1--19, 2019. doi:10.1109/S...

    2019

  14. [22]

    H., Gonzalez, J., Zhang, H., and Stoica, I

    Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pp.\ 611--626, 2023

    2023

  15. [23]

    Fast inference from transformers via speculative decoding

    Leviathan, Y., Kalman, M., and Matias, Y. Fast inference from transformers via speculative decoding. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of ...

    2023

  16. [24]

    Meltdown: Reading kernel memory from user space

    Lipp, M., Schwarz, M., Gruss, D., Prescher, T., Haas, W., Fogh, A., Horn, J., Mangard, S., Kocher, P., Genkin, D., Yarom, Y., and Hamburg, M. Meltdown: Reading kernel memory from user space. In 27th USENIX Security Symposium ( USENIX Security 18) , 2018

    2018

  17. [25]

    Liu, F., Yarom, Y., Ge, Q., Heiser, G., and Lee, R. B. Last-level cache side-channel attacks are practical. In 2015 IEEE symposium on security and privacy, pp.\ 605--622. IEEE, 2015

    2015

  18. [26]

    MTEB : Massive text embedding benchmark

    Muennighoff, N., Tazi, N., Magne, L., and Reimers, N. MTEB : Massive text embedding benchmark. In Vlachos, A. and Augenstein, I. (eds.), Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp.\ 2014--2037, Dubrovnik, Cro...

    2014 doi

  19. [27]

    Prompt caching

    OpenAI. Prompt caching. https://platform.openai.com/docs/guides/prompt-caching, 2024 a

    2024

  20. [28]

    Prompt caching in the api

    OpenAI. Prompt caching in the api. https://web.archive.org/web/20241003095307/https://openai.com/index/api-prompt-caching/, 2024 b

    2024

  21. [29]

    A., Shamir, A., and Tromer, E

    Osvik, D. A., Shamir, A., and Tromer, E. Cache attacks and countermeasures: the case of aes. In Topics in Cryptology--CT-RSA 2006: The Cryptographers’ Track at the RSA Conference 2006, San Jose, CA, USA, February 13-17, 2005. Proceedings, pp.\ 1--20. Springer, 2006

  22. [30]

    Cache missing for fun and profit

    Percival, C. Cache missing for fun and profit. BSDCan Ottawa, 2005. URL https://www.daemonology.net/papers/htt.pdf

    2005

  23. [31]

    Mooncake: A kvcache-centric disaggregated architecture for llm serving

    Qin, R., Li, Z., He, W., Zhang, M., Wu, Y., Zheng, W., and Xu, X. Mooncake: A kvcache-centric disaggregated architecture for llm serving. arXiv preprint arXiv:2407.00079, 2024

    2024 arXiv

  24. [32]

    Neural machine translation of rare words with subword units

    Sennrich, R., Haddow, B., and Birch, A. Neural machine translation of rare words with subword units. In Erk, K. and Smith, N. A. (eds.), Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 1715--1725, Berlin, Ge...

    2016 doi

  25. [33]

    The early bird catches the leak: Unveiling timing side channels in llm serving systems

    Song, L., Pang, Z., Wang, W., Wang, Z., Wang, X., Chen, H., Song, W., Jin, Y., Meng, D., and Hou, R. The early bird catches the leak: Unveiling timing side channels in llm serving systems. arXiv preprint arXiv:2409.20002, 2024

    2024

  26. [34]

    A., and Shamir, A

    Tromer, E., Osvik, D. A., and Shamir, A. Efficient cache attacks on aes, and countermeasures. Journal of Cryptology, 23: 0 37--71, 2010

    2010

  27. [35]

    The clock is still ticking: Timing attacks in the modern web

    Van Goethem, T., Joosen, W., and Nikiforakis, N. The clock is still ticking: Timing attacks in the modern web. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, pp.\ 1382--1393, 2015

    2015

  28. [36]

    N., Kaiser, L

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. Attention is all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Informatio...

    2017

  29. [37]

    E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., van der Walt , S

    Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., van der Walt , S. J., Brett, M., Wilson, J., Millman, K. J., Mayorov, N., Nelson, A. R. J., Jones, E., Kern, R., Larson, E., Carey, C. J...

    2020

  30. [38]

    Privacy risks of speculative decoding in large language models

    Wei, J., Abdulrazzag, A., Zhang, T., Muursepp, A., and Saileshwar, G. Privacy risks of speculative decoding in large language models. arXiv preprint arXiv:2411.01076, 2024

    2024

  31. [39]

    What was your prompt? a remote keylogging attack on AI assistants

    Weiss, R., Ayzenshteyn, D., and Mirsky, Y. What was your prompt? a remote keylogging attack on AI assistants. In 33rd USENIX Security Symposium (USENIX Security 24), pp.\ 3367--3384, Philadelphia, PA, August 2024. USENIX Association. ISBN 978-1-939133-44-1. URL https://www.use...

    2024

  32. [40]

    Cacheblend: Fast large language model serving for rag with cached knowledge fusion

    Yao, J., Li, H., Liu, Y., Ray, S., Cheng, Y., Zhang, Q., Du, K., Lu, S., and Jiang, J. Cacheblend: Fast large language model serving for rag with cached knowledge fusion. In Proceedings of the Twentieth European Conference on Computer Systems, EuroSys '25, pp.\ 94–109, New Yor...

    2025

  33. [41]

    and Falkner, K

    Yarom, Y. and Falkner, K. \ FLUSH+ RELOAD \ : A high resolution, low noise, l3 cache \ Side-Channel \ attack. In 23rd USENIX security symposium (USENIX security 14), pp.\ 719--732, 2014

    2014

  34. [42]

    Cachebleed: a timing attack on openssl constant-time rsa

    Yarom, Y., Genkin, D., and Heninger, N. Cachebleed: a timing attack on openssl constant-time rsa. Journal of Cryptographic Engineering, 7: 0 99--112, 2017

    2017

  35. [43]

    C hunk A ttention: Efficient self-attention with prefix-aware KV cache and two-phase partition

    Ye, L., Tao, Z., Huang, Y., and Li, Y. C hunk A ttention: Efficient self-attention with prefix-aware KV cache and two-phase partition. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V...

    2024 doi

  36. [44]

    Cascade inference: Memory bandwidth efficient shared prefix batch decoding, February 2024 b

    Ye, Z., Lai, R., Lu, B.-R., Lin, C.-Y., Zheng, S., Chen, L., Chen, T., and Ceze, L. Cascade inference: Memory bandwidth efficient shared prefix batch decoding, February 2024 b . URL https://flashinfer.ai/2024/02/02/cascade-inference.html

    2024

  37. [45]

    H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J

    Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., Barrett, C., and Sheng, Y. SGL ang: Efficient execution of structured language model programs. In The Thirty-eighth Annual Conference on Neural Information Processi...

    2024

  38. [46]

    Inputsnatch: Stealing input in llm services via timing side-channel attacks

    Zheng, X., Han, H., Shi, S., Fang, Q., Du, Z., Guo, Q., and Hu, X. Inputsnatch: Stealing input in llm services via timing side-channel attacks. arXiv preprint arXiv:2411.18191, 2024 b

    2024 arXiv

  39. [47]

    Recall, precision and average precision

    Zhu, M. Recall, precision and average precision. Department of Statistics and Actuarial Science, University of Waterloo, Waterloo, 2 0 (30): 0 6, 2004

    2004

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.