Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

Semantic Caching of Contextual Summaries for Efficient Question-Answering with Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that caching query-specific document summaries and reusing them for similar questions cuts LLM input tokens in half while preserving answer quality in retrieval-augmented generation.

desk verdict Useful extension of semantic caching to intermediate summaries, with honest self-flagging, but the 'comparable accuracy' claim is under-supported by the cosine-similarity proxy. read the letter →

arxiv 2505.11271 v1 pith:E2R3WHMJ submitted 2025-05-16 cs.CL cs.AIcs.IRcs.LG

Camille Couturier , Spyros Mastorakis , Haiying Shen , Saravan Rajmohan , Victor Rühle This is my paper · ORCID
classification cs.CLcs.AIcs.IRcs.LG
keywords semanticcachingcontextualsummarieslargelanguagemodelsquestionansweringretrieval-augmentedgenerationcachehitratecosinesimilarityefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language model question-answering systems waste computation when every similar question re-processes the same retrieved document. This paper proposes storing the intermediate step—a query-aware contextual summary—and reusing it when a new question is semantically close to one already seen. On three public QA benchmarks, the authors report that at a 0.8 cosine-similarity threshold, input-token counts fall by at least half on two of the datasets while answer utility stays comparable to answering from the full document. The contribution is a middle design point between expensive fresh processing and cheap but weaker non-contextual summaries.

What carries the argument

The central object is the contextual summary, written by the LLM from the document plus the user's question ($x^{\mathrm{sum}} = f(x^{\mathrm{doc}}, x^{\mathrm{que}})$), as opposed to a non-contextual summary that ignores the question. The summaries are cached in a vector index keyed by question embeddings, and retrieval is decided by cosine similarity between the incoming question and previously cached questions; a threshold, 0.8 in the headline runs, separates reuse from fresh summarization. The design is deliberately selective and order-independent: only the dynamic question participates in the similarity match, so two requests can share a summary regardless of where the question appears in the prompt. That mechanism turns question similarity directly into reduced LLM input tokens, since the expensive document-to-summary step is the one skipped on a hit.

What would settle it

For a fixed document, identify many question pairs whose embedding cosine similarity is just above the deployment threshold, answer the second question using only the first question's cached contextual summary, and compare utility against a query-agnostic summary baseline; if the cached summary is no better across such pairs, the central assumption fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that a contextual summary is a reusable intermediate artifact in LLM question answering. Once a summary has been generated for a document and a specific question, it can answer later questions whose embeddings are sufficiently close to the original question—above a cosine threshold such as 0.8—without reprocessing the document. This reuse preserves the utility advantage of query-aware compression while avoiding the cost of regenerating it, and it outperforms both non-contextual summaries, which are cheaper but weaker, and end-to-end answer caching, which can return the same answer to different questions. The evidence is that, as question sequences grow, cache hit rates reach 0.2–0.3 for diverse question sets and 0.5–0.6 for clustered ones, and input-token usage drops below half of full-document processing on two benchmarks while utility stays comparable.

Load-bearing premise

The system rests on the premise that questions whose embeddings are close under cosine similarity need essentially the same facts from the document, so a contextual summary written for one will serve the other.

Editorial extensions

If this is right

  • On workloads with semantically clustered questions, half or more of the summarization calls can be skipped once the cache warms up, with cache hit rates between 0.5 and 0.6.
  • Input-token consumption drops to at most half of full-document processing on the two benchmarks with high question similarity, directly lowering the dominant cost of LLM API calls.
  • Answer utility from cached summaries is comparable to full-document answers, and on at least one benchmark the focused summary answers slightly more accurately than the full document.
  • Raising the similarity threshold improves answer relevance at the price of fewer cache hits; the paper finds a candidate sweet spot near 0.85–0.9 for one of the benchmarks.
  • Repeated queries to the same document make the system progressively cheaper and faster, because cache hits skip the summarization API call that dominates end-to-end latency.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the cache is keyed on question embeddings rather than on document chunks or token positions, the same mechanism could be adapted to group queries into clusters per document and precompute which cached summaries cover which clusters, turning threshold tuning into a clustering problem.
  • The design is compatible with prompt and KV caching rather than competing with it: contextual summaries shrink the long document context, while KV caching accelerates the fixed instruction portions that remain, so the two savings could stack.
  • A natural stress test is multi-document synthesis queries, which the paper does not evaluate; since the current system caches one document at a time, cross-document questions may need a second-level cache over combined summaries.
  • If the answer-quality metric is replaced by human judgments, the threshold sweet spots may shift; the paper's use of cosine similarity between generated and ground-truth answers could be validated against exact-match or human ratings in a follow-up.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes semantic caching of intermediate contextual summaries for RAG-based question answering. When a new question arrives, the system computes the cosine similarity between the question embedding and embeddings of previous questions for the same document; if the similarity exceeds a threshold, it reuses the cached query-aware summary instead of regenerating a summary from the full document. The method is compared with five baselines (full document, non-contextual summary, fresh contextual summary, full API input/output caching, and no retrieval) on NaturalQuestions, TriviaQA, and a synthetic ArXiv dataset. Evaluation metrics are utility (cosine similarity between generated and reference answers), cache hit rate, token usage, and latency. The authors report cache hit ratios of 0.2-0.6 and input-token reductions of at least 50% at a similarity threshold of 0.8 with utility 'very close' to that of full-document processing.

Significance. If the accuracy claim held, this would be a practically useful contribution: it caches intermediate, compressed contextual summaries, is order-independent, supports dynamic document updates, and avoids the storage and order-dependency issues of KV-cache and end-to-end response caching. The cost-side evidence is concrete and internally consistent, the evaluation uses public datasets, and the paper is admirably candid in disclosing the ArXiv reference artifact in the Figure 7 note. The idea is timely for edge/cloud LLM deployment. However, the accuracy component of the central claim is not yet established: the only quality metric is cosine similarity of short answers, and on the synthetic ArXiv dataset the reference answers are generated by the full-document method itself, so the 'comparable to full document' claim is partly circular on that dataset.

major comments (4)
  1. [4.5, Figure 7, Abstract] The central claim 'maintaining answer accuracy comparable to full document processing' is not actually tested as accuracy. Section 4.5 defines utility as cosine similarity between the generated answer and the ground-truth answer, and all comparisons in Figure 7 and Table 3 use this metric. For two-to-three-word answers, cosine similarity is a weak proxy for correctness: a topically related but incorrect answer can score highly, and the metric does not penalize missing or extraneous information. On the ArXiv dataset, the ground truth is generated by the full-document method itself (Figure 7 note), so utility there measures agreement with full-document outputs, not correctness. The authors should add exact-match or token-level F1 (or human evaluation) on at least NaturalQuestions and TriviaQA, and use independent references for the ArXiv dataset.
  2. [3.1, 5.4, Table 3] The similarity threshold is a free parameter and is selected from the same evaluation data. Section 5.4 recommends a threshold between 0.85 and 0.9 based on the NaturalQuestions utility curve, and Table 3 reports results at 0.6 and 0.8, but there is no held-out split or a priori selection rule. Because the threshold directly controls cache hit rate and which summaries are reused, the headline token-savings and utility numbers may be optimistic. The authors should either fix the threshold by a stated rule on a training portion of the data or report the full trade-off curve as a function of threshold, with confidence intervals, and show that the main cost-quality conclusions are robust across a range of thresholds.
  3. [3.1, Figure 9] The operational assumption (Insight 1) is that cosine similarity between question embeddings predicts whether a cached contextual summary, generated for an earlier question, contains the information needed to answer the new question. The paper does not test this directly; it only correlates question similarity with answer-embedding similarity. Figure 9 shows that for low-similarity questions cached contextual summaries perform worse than non-contextual summaries, which is precisely the regime where the assumption needs validation. A direct test would be to measure, for cache-hit pairs, whether the cached summary contains the ground-truth answer tokens and to compare end-task accuracy (exact match or F1) on hits versus misses. Without such evidence, the 50-60% token savings cannot be firmly tied to 'maintaining answer accuracy.'
  4. [5.1, Table 3] The token-reduction claim of 'at least 50% lower on average for NaturalQuestions and ArXiv' is plausible from Figure 7, but Table 3 does not include the full-document token baseline, so the reader cannot verify the percentage directly from the table. Also, utility values are reported as means with standard deviations but no significance tests; several comparisons (e.g., TriviaQA utility 0.70 versus 0.71 for 200 versus 400 words) are within noise. Please add the full-document baseline columns to Table 3 and include appropriate significance or effect-size measures for the utility comparisons that support the central claim.
minor comments (6)
  1. [4.3] The text says two sentence encoders were tried (all-MiniLM-L6-v2 and all-mpnet-base-v2), but the results do not state which encoder was used for Figures 5, 7, and 9. Please specify the encoder used for the main experiments and state whether the conclusions are stable across the two encoders.
  2. [5.2, Figure 8] The claim that total latency is 'predominantly (>99%) due to LLM API calls' would be easier to assess if the figure axes for encoding time and search cache time indicated their scales, and if the measurement setup (same hardware, number of runs, API endpoint) were stated.
  3. [5.5] The statement that 'the question is the most influential component in answer generation' is supported only by 'exploratory experiments' with no details. Either provide the supporting experiment or explicitly label this as a hypothesis.
  4. [2.2, 5.7] GPTCache is criticized for 'static similarity thresholds requiring manual tuning per domain,' but the proposed method also requires manual threshold tuning (acknowledged in Section 5.7). The contrast in Table 1 and the related-work text should be softened or clarified.
  5. [Figure 6] The top panel of Figure 6 combines 'ratios' and 'lengths'; please make the y-axis labels explicit and indicate whether the distributions are shown for TriviaQA only or for all datasets.
  6. [Abstract, 5.1] The phrase 'reduces redundant computations by up to 50-60%' is used while the measured quantity is input-token count. Please rephrase to 'input token usage' unless wall-clock computation or FLOPs are reported.

Circularity Check

1 steps flagged · score 2.0 of 10

No global circularity; the only self-referential element is the disclosed use of full-document answers as ground truth on the synthetic ArXiv dataset.

  1. self definitional [Section 4.5 (Evaluation metrics) and Figure 7 note (ArXiv panel)]
    "Utility: Measured as cosine similarity between generated and ground truth answers. ... Note: For the ArXiv dataset, full document answers serve as the reference, creating artificially high utility values for this method (shown as dotted lines)."

    The abstract's central claim is 'maintaining answer accuracy comparable to full document processing.' On the synthetic ArXiv dataset, the evaluation's 'ground truth answers' are themselves the output of full-document processing, and utility is defined as cosine similarity to that ground truth. Therefore, for that dataset, the measured property 'accuracy comparable to full document' is literally cosine similarity to the full-document answer: the comparison target and the reference are the same quantity by construction. This is a local self-definitional evaluation; it is disclosed and does not affect the NaturalQuestions/TriviaQA results, whose ground truth is human-provided.

full rationale

The paper is an empirical systems evaluation rather than a formal derivation, so there is no derivation chain in which a prediction reduces to a fitted input. The 50-60% token-reduction claim is supported by direct measurements of LLM input tokens across three datasets, so that part is externally grounded. The accuracy component is weaker: Section 4.5 defines utility as cosine similarity to ground-truth answers, and for the synthetic ArXiv dataset the ground truth is generated by the full-document method itself (Figure 7 note), making the 'comparable to full document' claim on that dataset partly self-referential. This is a real but local circularity, disclosed by the authors. It does not by itself make the whole paper circular because NaturalQuestions and TriviaQA use human-provided answers. The self-citations (e.g., LLMLingua-2 [21] for prompt-compression notation, [24] for deployment context) are not load-bearing. Threshold selection on the same datasets is a methodological overfitting risk, not a circular-derivation risk.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central numbers of the paper are the similarity threshold and summary length, both selected by the authors and varied in experiments. The threshold is explicitly tuned to the observed utility/hit-rate curve in Section 5.4, which makes it a free parameter rather than an input derived from theory. The dataset filter (documents with at least 10 questions and at least 800 words) is a selection choice that favors high cache hit rates.

free parameters (3)
  • Similarity threshold = 0.8 in main experiments; 0.85-0.9 recommended in Section 5.4
    Chosen from the observed utility/hit-rate trade-off on the same NaturalQuestions data (Section 5.4), not a held-out tuning set. It directly controls cache hit rate and the central cost-quality claim.
  • Summary length = 200 words in main experiments; 100 and 400 words varied
    Selected based on average document length and the compression/information trade-off (Section 4.1, Figure 6). Affects both token savings and answer utility.
  • Dataset selection filter = Documents with at least 10 questions and at least 800 words
    Applied in Section 4.2 to all datasets. Choosing documents with many related questions raises achievable cache hit rates, so the reported hit ratios may not generalize to sparser query workloads.
assumptions (4)
  • domain assumption Cosine similarity between question embeddings captures semantic relatedness that predicts whether a cached summary is useful.
    Section 3.1, Insight 1, cites sentence-embedding studies; the entire cache-hit decision relies on this.
  • domain assumption A contextual summary generated for one query preserves enough document information to answer related future queries.
    Section 3.2 defines query-aware summaries as functions of document and query; the reuse claim assumes information transfer to similar questions.
  • domain assumption Cosine similarity between a generated short answer and the ground-truth short answer is a valid measure of answer utility.
    Section 4.5 defines utility this way; no human or exact-match validation is provided.
  • domain assumption Documents with at least 10 questions and 800 words are representative of real assistant workloads.
    Section 4.2 applies these filters to all datasets; they also raise achievable cache hit rates.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Semantic Caching of Contextual Summaries for Efficient Question-Answering with Language Models." pith.science (2026). https://pith.science/paper/E2R3WHMJ

@misc{pith2026250511271,
  author       = {Pith},
  title        = {Pith review of: Semantic Caching of Contextual Summaries for Efficient Question-Answering with Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E2R3WHMJ}},
  note         = {Machine review of arXiv:2505.11271}
}
read the original abstract

Large Language Models (LLMs) are increasingly deployed across edge and cloud platforms for real-time question-answering and retrieval-augmented generation. However, processing lengthy contexts in distributed systems incurs high computational overhead, memory usage, and network bandwidth. This paper introduces a novel semantic caching approach for storing and reusing intermediate contextual summaries, enabling efficient information reuse across similar queries in LLM-based QA workflows. Our method reduces redundant computations by up to 50-60% while maintaining answer accuracy comparable to full document processing, as demonstrated on NaturalQuestions, TriviaQA, and a synthetic ArXiv dataset. This approach balances computational cost and response quality, critical for real-time AI assistants.

Figures

Figures reproduced from arXiv: 2505.11271 by the authors.

Figure 1
Figure 1. Simplified diagram of a chat-based assistant system with document retrieval: a user [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Instructions used to prompt the LLM to generate a general, non-contextual summary [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Instructions used to prompt the LLM to generate a query-aware, contextual summary [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Instructions used to prompt the LLM to generate an answer using a reference (a [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: NaturalQuestions dataset: cosine similarity between pairs of questions asked about a [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Ratios (compared to the document length) and lengths (in terms of words) of different [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Comparative analysis of retrieval methods across TriviaQA, NaturalQuestions, and [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: LLM API call time, search cache time, encoding time, and total latency of different [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: NaturalQuestions dataset: End-to-end utility using various versions of the retrieved [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Mobility-Aware Cache Framework for Scalable LLM-Based Human Mobility Simulation

    cs.AI 2026-02 conditional novelty 6.0 of 10

    A latent-space reasoning cache with a lightweight decoder cuts the cost of LLM-based human mobility simulation by roughly 40-90% while keeping trajectory quality comparable.

  2. From Similarity to Vulnerability: Key Collision Attack on LLM Semantic Caching

    cs.CR 2026-01 conditional novelty 6.0 of 10

    An attacker can force false cache-key collisions in LLM semantic caching and hijack responses or agent tool calls, with 86% hit rate in experiments.

Reference graph

Works this paper leans on

28 extracted references · 13 canonical work pages · cited by 2 Pith papers

  1. [1]

    GPTCache: An open-source semantic cache for llm applications enabling faster answers and cost savings

    Fu Bang. GPTCache: An open-source semantic cache for llm applications enabling faster answers and cost savings. In Proceedings of the 3rd Workshop for Natural Language Pro- cessing Open Source Software (NLP-OSS 2023) , pages 212–218, 2023

    work page 2023

  2. [2]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    arXiv 2020

  3. [3]

    Chan, Chao-Ting Chen, Jui-Hung Cheng, and Hen-Hsen Huang

    Brian J. Chan, Chao-Ting Chen, Jui-Hung Cheng, and Hen-Hsen Huang. Don’t Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks, February 2025. arXiv:2412.15605 [cs]

    arXiv 2025

  4. [4]

    codefuse-ai/ModelCache, June 2024

    CodeFuse. codefuse-ai/ModelCache, June 2024. original-date: 2023-11-01T01:56:10Z

    work page 2024

  5. [5]

    Langchain: Build applications with llms through composability

    LangChain Contributors. Langchain: Build applications with llms through composability. Available at: https://github.com/langchain-ai/langchain, 2022

    work page 2022

  6. [6]

    Model caches in LangChain

    LangChain Contributors. Model caches in LangChain. Available at: https://python.langchain.com/docs/integrations/llm caching/, 2025

    work page 2025

  7. [7]

    Semantic cache for RAG using FAISS

    Hugging Face. Semantic cache for RAG using FAISS. https://huggingface.co/blog/ semantic-cache-rag, 2023

    work page 2023

  8. [8]

    Privacy-aware semantic cache for large language models

    Waris Gill, Mohamed Elidrisi, Pallavi Kalapatapu, Ali Anwar, and Muhammad Ali Gulzar. Privacy-aware semantic cache for large language models. arXiv preprint arXiv:2403.02694, 2024

    arXiv 2024

Show all 28 references
  1. [9]

    Prompt cache: Modular attention reuse for low-latency inference

    In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong. Prompt cache: Modular attention reuse for low-latency inference. arXiv preprint arXiv:2311.04934, 2023

    2023 arXiv

  2. [10]

    EPIC: Efficient Position-Independent Context Caching for Serving Large Language Models, February 2025

    Junhao Hu, Wenrui Huang, Haoyi Wang, Weidong Wang, Tiancheng Hu, Qin Zhang, Hao Feng, Xusheng Chen, Yizhou Shan, and Tao Xie. EPIC: Efficient Position-Independent Context Caching for Serving Large Language Models, February 2025. arXiv:2410.15332 [cs]

    2025 arXiv

  3. [11]

    LLMLingua: Compressing prompts for accelerated inference of large language models, December 2023

    Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. LLMLingua: Compressing prompts for accelerated inference of large language models, December 2023. arXiv:2310.05736 [cs]. 13

    2023 arXiv

  4. [12]

    LongLLMLingua: Accelerating and enhancing llms in long context scenarios via prompt compression, October 2023

    Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. LongLLMLingua: Accelerating and enhancing llms in long context scenarios via prompt compression, October 2023. arXiv:2310.06839 [cs]

    2023 arXiv

  5. [13]

    RAGCache: Efficient knowledge caching for retrieval-augmented generation.arXiv preprint arXiv:2404.12457, 2024

    Chao Jin, Zili Zhang, Xuanlin Jiang, Fangyue Liu, Xin Liu, Xuanzhe Liu, and Xin Jin. RAGCache: Efficient knowledge caching for retrieval-augmented generation.arXiv preprint arXiv:2404.12457, 2024

    2024 arXiv

  6. [14]

    Billion-scale similarity search with gpus

    Jeff Johnson, Matthijs Douze, and Herv´ e J´ egou. Billion-scale similarity search with gpus. arXiv preprint arXiv:1702.08734 , 2017

    2017 arXiv

  7. [15]

    Weld, and Luke Zettlemoyer

    Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension, May 2017

    2017

  8. [16]

    Dai, Jakob Uszko- reit, Quoc Le, and Slav Petrov

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszko- reit, Quoc Le, and Sl...

    2019

  9. [17]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient Memory Management for Large Language Model Serving with PagedAttention, September 2023. arXiv:2309.06180 [cs]

    2023 arXiv

  10. [18]

    Scalm: Towards semantic caching for automated chat services with large language models

    Jiaxing Li, Chi Xu, Feng Wang, Isaac M von Riedemann, Cong Zhang, and Jiangchuan Liu. Scalm: Towards semantic caching for automated chat services with large language models. arXiv preprint arXiv:2406.00025 , 2024

    2024 arXiv

  11. [19]

    Context-based Semantic Caching for LLM Applications

    Ramaswami Mohandoss. Context-based Semantic Caching for LLM Applications . In 2024 IEEE Conference on Artificial Intelligence (CAI) , pages 371–376, Los Alamitos, CA, USA, June 2024. IEEE Computer Society

    2024

  12. [20]

    Gpt-3.5 turbo model on OpenAI Platform

    OpenAI. Gpt-3.5 turbo model on OpenAI Platform. Available at: https://platform.openai.com/docs/models/gpt-3-5-turbo, 2023

    2023

  13. [21]

    Vicky Zhao, Lili Qiu, and Dongmei Zhang

    Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor R¨ uhle, Yuqing Yang, Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, and Dongmei Zhang. LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression, March 20...

    2024 arXiv

  14. [22]

    Sentence-bert: Sentence embeddings using siamese bert-networks, 2019

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks, 2019

    2019

  15. [23]

    Sentence-BERT: Sentence embeddings using siamese bert-networks

    Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019

    2019

  16. [24]

    Hybrid-RACA: Hybrid retrieval-augmented composition assistance for real- time text prediction, 2024

    Menglin Xia, Xuchao Zhang, Camille Couturier, Guoqing Zheng, Saravan Rajmohan, and Victor Ruhle. Hybrid-RACA: Hybrid retrieval-augmented composition assistance for real- time text prediction, 2024

    2024

  17. [25]

    CacheBlend: Fast large language model serving with cached knowledge fusion

    Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang. CacheBlend: Fast large language model serving with cached knowledge fusion. arXiv preprint arXiv:2405.16444 , 2024. 14

    2024 arXiv

  18. [26]

    Yu, and Jiawei Zhang

    Haopeng Zhang, Philip S. Yu, and Jiawei Zhang. A systematic survey of text summa- rization: From statistical methods to large language models. arXiv:2406.11289v1 [cs.CL], 2024

    2024 arXiv

  19. [27]

    Hammerla

    Vitalii Zhelezniak, Aleksandar Savkov, April Shen, and Nils Y. Hammerla. Correlation coefficients and semantic textual similarity, 2019

    2019

  20. [28]

    Instcache: A predictive cache for llm serving, 2024

    Longwei Zou, Tingfeng Liu, Kai Chen, Jiangang Kong, and Yangdong Deng. Instcache: A predictive cache for llm serving, 2024. 15 2 4 6 8 10 0.2 0.4 0.6 0.8 Utility [how close to ground truth] Dataset: TQA 2 4 6 8 10 Dataset: NQ 5 10 15 Dataset: ARXIV 2 4 6 8 10 0.0 0.2 0.4 0.6 0...

    2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.