{"id":"39e81550-c7a9-4b96-93b4-fb9c5c2e099d","arxiv_id":"2505.11271","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Caching query-aware LLM document summaries and reusing them for similar questions cuts input token use by roughly half while keeping answer similarity close to full-document processing.","lead":"This paper stores the document summaries an LLM creates for one question and reuses them when a later question is semantically similar, cutting token use in half. It shows on three QA datasets that answer quality stays close to processing the full document, offering a cost lever for real-time assistants.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'comparable accuracy' claim rests on a cosine-similarity utility metric, and on ArXiv the reference answers are generated by the full-document method itself; accuracy is never directly measured.","rationale":"The accuracy half of the central claim is load-bearing because the method's value proposition is a cost-quality trade-off: token savings only matter if quality does not degrade. The paper's only quality signal is cosine similarity of answer embeddings. On ArXiv, the reference is produced by the same full-document pipeline being compared against, so high utility for cached summaries is expected whenever the summary retains enough text to reproduce the full-document answer; this does not demonstrate that answers are correct. On the other two datasets, cosine similarity of very short answers is not a reliable correctness measure and may be inflated by shared style or common tokens. The reader's verdict of CONDITIONAL already reflects these concerns; my stress-test sharpens the condition: the accuracy comparison must be re-run with a direct correctness metric before the central claim can be accepted. I do not see an internal inconsistency in the method itself, and the token-reduction numbers are plausible, so the appropriate verdict remains CONDITIONAL/UNCHANGED rather than REJECT.","tokens_in":11832,"tokens_out":5123,"duration_ms":53906,"concrete_test":"Re-score all generated answers from Figure 7 with exact-match and token-level F1 against the human gold answers for NaturalQuestions and TriviaQA. For ArXiv, collect human judgments (or a blinded LLM judge) on correctness of full-document answers and cached-summary answers, and compare the two. If cached-summary answers remain within a small tolerance of full-document answers on these accuracy metrics across all three datasets, the concern is resolved; if not, the 'comparable accuracy' claim should be weakened or restricted to token savings.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in the abstract—'maintaining answer accuracy comparable to full document processing'—is not actually tested as accuracy. Section 4.5 defines utility as the cosine similarity between the generated answer and the ground-truth answer, and all accuracy comparisons in Figure 7 and Table 3 use this metric. For the synthetic ArXiv dataset, the ground-truth answers are generated by the full-document method (Figure 7 note: 'full document answers serve as the reference'), so the cached-summary method's utility on ArXiv measures agreement with full-document outputs, not correctness; this makes the 'comparable to full document' claim partly circular on that dataset. On NaturalQuestions and TriviaQA, the ground truth is human-provided, but cosine similarity of embeddings of 2-3 word answers is a weak proxy for answer accuracy: it can rate a topically related but wrong answer as highly similar and does not penalize missing or extraneous information. Consequently, the evidence for the central claim's accuracy component is not yet established, even though the token-reduction component is plausible.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes semantic caching of intermediate contextual summaries for RAG-based question answering. When a new question arrives, the system computes the cosine similarity between the question embedding and embeddings of previous questions for the same document; if the similarity exceeds a threshold, it reuses the cached query-aware summary instead of regenerating a summary from the full document. The method is compared with five baselines (full document, non-contextual summary, fresh contextual summary, full API input/output caching, and no retrieval) on NaturalQuestions, TriviaQA, and a synthetic ArXiv dataset. Evaluation metrics are utility (cosine similarity between generated and reference answers), cache hit rate, token usage, and latency. The authors report cache hit ratios of 0.2-0.6 and input-token reductions of at least 50% at a similarity threshold of 0.8 with utility 'very close' to that of full-document processing.","tokens_in":12007,"tokens_out":4896,"duration_ms":45903,"significance":"If the accuracy claim held, this would be a practically useful contribution: it caches intermediate, compressed contextual summaries, is order-independent, supports dynamic document updates, and avoids the storage and order-dependency issues of KV-cache and end-to-end response caching. The cost-side evidence is concrete and internally consistent, the evaluation uses public datasets, and the paper is admirably candid in disclosing the ArXiv reference artifact in the Figure 7 note. The idea is timely for edge/cloud LLM deployment. However, the accuracy component of the central claim is not yet established: the only quality metric is cosine similarity of short answers, and on the synthetic ArXiv dataset the reference answers are generated by the full-document method itself, so the 'comparable to full document' claim is partly circular on that dataset.","major_comments":[{"comment":"The central claim 'maintaining answer accuracy comparable to full document processing' is not actually tested as accuracy. Section 4.5 defines utility as cosine similarity between the generated answer and the ground-truth answer, and all comparisons in Figure 7 and Table 3 use this metric. For two-to-three-word answers, cosine similarity is a weak proxy for correctness: a topically related but incorrect answer can score highly, and the metric does not penalize missing or extraneous information. On the ArXiv dataset, the ground truth is generated by the full-document method itself (Figure 7 note), so utility there measures agreement with full-document outputs, not correctness. The authors should add exact-match or token-level F1 (or human evaluation) on at least NaturalQuestions and TriviaQA, and use independent references for the ArXiv dataset.","section":"4.5, Figure 7, Abstract"},{"comment":"The similarity threshold is a free parameter and is selected from the same evaluation data. Section 5.4 recommends a threshold between 0.85 and 0.9 based on the NaturalQuestions utility curve, and Table 3 reports results at 0.6 and 0.8, but there is no held-out split or a priori selection rule. Because the threshold directly controls cache hit rate and which summaries are reused, the headline token-savings and utility numbers may be optimistic. The authors should either fix the threshold by a stated rule on a training portion of the data or report the full trade-off curve as a function of threshold, with confidence intervals, and show that the main cost-quality conclusions are robust across a range of thresholds.","section":"3.1, 5.4, Table 3"},{"comment":"The operational assumption (Insight 1) is that cosine similarity between question embeddings predicts whether a cached contextual summary, generated for an earlier question, contains the information needed to answer the new question. The paper does not test this directly; it only correlates question similarity with answer-embedding similarity. Figure 9 shows that for low-similarity questions cached contextual summaries perform worse than non-contextual summaries, which is precisely the regime where the assumption needs validation. A direct test would be to measure, for cache-hit pairs, whether the cached summary contains the ground-truth answer tokens and to compare end-task accuracy (exact match or F1) on hits versus misses. Without such evidence, the 50-60% token savings cannot be firmly tied to 'maintaining answer accuracy.'","section":"3.1, Figure 9"},{"comment":"The token-reduction claim of 'at least 50% lower on average for NaturalQuestions and ArXiv' is plausible from Figure 7, but Table 3 does not include the full-document token baseline, so the reader cannot verify the percentage directly from the table. Also, utility values are reported as means with standard deviations but no significance tests; several comparisons (e.g., TriviaQA utility 0.70 versus 0.71 for 200 versus 400 words) are within noise. Please add the full-document baseline columns to Table 3 and include appropriate significance or effect-size measures for the utility comparisons that support the central claim.","section":"5.1, Table 3"}],"minor_comments":[{"comment":"The text says two sentence encoders were tried (all-MiniLM-L6-v2 and all-mpnet-base-v2), but the results do not state which encoder was used for Figures 5, 7, and 9. Please specify the encoder used for the main experiments and state whether the conclusions are stable across the two encoders.","section":"4.3"},{"comment":"The claim that total latency is 'predominantly (>99%) due to LLM API calls' would be easier to assess if the figure axes for encoding time and search cache time indicated their scales, and if the measurement setup (same hardware, number of runs, API endpoint) were stated.","section":"5.2, Figure 8"},{"comment":"The statement that 'the question is the most influential component in answer generation' is supported only by 'exploratory experiments' with no details. Either provide the supporting experiment or explicitly label this as a hypothesis.","section":"5.5"},{"comment":"GPTCache is criticized for 'static similarity thresholds requiring manual tuning per domain,' but the proposed method also requires manual threshold tuning (acknowledged in Section 5.7). The contrast in Table 1 and the related-work text should be softened or clarified.","section":"2.2, 5.7"},{"comment":"The top panel of Figure 6 combines 'ratios' and 'lengths'; please make the y-axis labels explicit and indicate whether the distributions are shown for TriviaQA only or for all datasets.","section":"Figure 6"},{"comment":"The phrase 'reduces redundant computations by up to 50-60%' is used while the measured quantity is input-token count. Please rephrase to 'input token usage' unless wall-clock computation or FLOPs are reported.","section":"Abstract, 5.1"}],"recommendation":"major_revision","confidential_remarks":"This is a borderline paper. The cost-side evidence (token counts, latency, cache hit rates) is solid and the idea is practically relevant, but the accuracy evidence is not yet sufficient for the central claim. If the authors add exact-match or F1 metrics and a proper threshold-selection protocol, I would support acceptance. The paper's own disclosure of the ArXiv reference artifact (Figure 7 note) is a point in its favor."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper does something genuinely new—caching intermediate contextual summaries in RAG QA, keyed on query embeddings—and measures the obvious win: fewer input tokens and lower latency on three datasets. The token savings are plausible and internally consistent. I'd believe the 50-60% reduction for workloads with repeated similar queries. The paper also earns credit for flagging its own artifact: on the synthetic ArXiv set, the 'ground truth' answers are generated by the full-document method, so the utility numbers there measure agreement with that method, not correctness. The Figure 7 note says exactly this.\n\nThe soft spots are real but not fatal. First, 'accuracy comparable to full document processing' is never directly tested. Utility is cosine similarity between the generated answer embedding and the ground-truth answer embedding. For 2-3 word answers, embedding similarity can rate a topically related but wrong answer as similar, and it doesn't penalize missing or extra content. That weakens the central claim, especially since the similarity threshold (0.8, then recommended 0.85-0.9) is tuned on the same evaluation data. That's not circular in a derivational sense, but it means the headline accuracy number may be optimistic. Second, the experimental setup filters for documents with at least 10 questions, which inflates cache hit rates relative to many real workloads. The paper doesn't hide this, but the abstract's 'up to 50-60%' could mislead. Third, there is no direct comparison with GPTCache or SCALM, so we don't learn whether caching summaries beats caching whole responses on the same workloads. The related-work table is qualitative.\n\nWhat's solid: the system design is sensible, the latency breakdown shows where time goes, and the authors are candid about cold starts, threshold tuning, and privacy. The writing is clear. The math is simple but used correctly, and the numbers in Table 3 are internally consistent.\n\nWho this is for: people building RAG-based assistants or edge LLM services who want a cheap way to cut redundant computation for similar queries. It's a conference-level empirical contribution, not a field-changer. I'd send it to peer review—the idea is worth refereeing, and the weaknesses are addressable in revision. If I were the editor, I'd ask for direct accuracy evaluation on a small human-labeled sample and a comparison against at least GPTCache on the same datasets.","headline":"Useful extension of semantic caching to intermediate summaries, with honest self-flagging, but the 'comparable accuracy' claim is under-supported by the cosine-similarity proxy.","tokens_in":12562,"tokens_out":2121,"would_cite":true,"duration_ms":19966,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that caching query-specific document summaries and reusing them for similar questions cuts LLM input tokens in half while preserving answer quality in retrieval-augmented generation.","keywords":["semantic caching","contextual summaries","large language models","question answering","retrieval-augmented generation","cache hit rate","cosine similarity","efficiency"],"falsifier":"For a fixed document, identify many question pairs whose embedding cosine similarity is just above the deployment threshold, answer the second question using only the first question's cached contextual summary, and compare utility against a query-agnostic summary baseline; if the cached summary is no better across such pairs, the central assumption fails.","tokens_in":11620,"feed_emoji":"⚡","tokens_out":9878,"duration_ms":93688,"temperature":0.7,"pith_summary":"Large language model question-answering systems waste computation when every similar question re-processes the same retrieved document. This paper proposes storing the intermediate step—a query-aware contextual summary—and reusing it when a new question is semantically close to one already seen. On three public QA benchmarks, the authors report that at a 0.8 cosine-similarity threshold, input-token counts fall by at least half on two of the datasets while answer utility stays comparable to answering from the full document. The contribution is a middle design point between expensive fresh processing and cheap but weaker non-contextual summaries.","feed_headline":"Cached summaries halve LLM question-answering cost","feed_subtitle":"A query-specific document summary, cached and reused for similar questions, matches full-document accuracy while cutting input tokens.","key_machinery":"The central object is the contextual summary, written by the LLM from the document plus the user's question ($x^{\\mathrm{sum}} = f(x^{\\mathrm{doc}}, x^{\\mathrm{que}})$), as opposed to a non-contextual summary that ignores the question. The summaries are cached in a vector index keyed by question embeddings, and retrieval is decided by cosine similarity between the incoming question and previously cached questions; a threshold, 0.8 in the headline runs, separates reuse from fresh summarization. The design is deliberately selective and order-independent: only the dynamic question participates in the similarity match, so two requests can share a summary regardless of where the question appears in the prompt. That mechanism turns question similarity directly into reduced LLM input tokens, since the expensive document-to-summary step is the one skipped on a hit.","core_discovery":"The paper's central claim is that a contextual summary is a reusable intermediate artifact in LLM question answering. Once a summary has been generated for a document and a specific question, it can answer later questions whose embeddings are sufficiently close to the original question—above a cosine threshold such as 0.8—without reprocessing the document. This reuse preserves the utility advantage of query-aware compression while avoiding the cost of regenerating it, and it outperforms both non-contextual summaries, which are cheaper but weaker, and end-to-end answer caching, which can return the same answer to different questions. The evidence is that, as question sequences grow, cache hit rates reach 0.2–0.3 for diverse question sets and 0.5–0.6 for clustered ones, and input-token usage drops below half of full-document processing on two benchmarks while utility stays comparable.","pith_inferences":["Because the cache is keyed on question embeddings rather than on document chunks or token positions, the same mechanism could be adapted to group queries into clusters per document and precompute which cached summaries cover which clusters, turning threshold tuning into a clustering problem.","The design is compatible with prompt and KV caching rather than competing with it: contextual summaries shrink the long document context, while KV caching accelerates the fixed instruction portions that remain, so the two savings could stack.","A natural stress test is multi-document synthesis queries, which the paper does not evaluate; since the current system caches one document at a time, cross-document questions may need a second-level cache over combined summaries.","If the answer-quality metric is replaced by human judgments, the threshold sweet spots may shift; the paper's use of cosine similarity between generated and ground-truth answers could be validated against exact-match or human ratings in a follow-up."],"forward_implications":["On workloads with semantically clustered questions, half or more of the summarization calls can be skipped once the cache warms up, with cache hit rates between 0.5 and 0.6.","Input-token consumption drops to at most half of full-document processing on the two benchmarks with high question similarity, directly lowering the dominant cost of LLM API calls.","Answer utility from cached summaries is comparable to full-document answers, and on at least one benchmark the focused summary answers slightly more accurately than the full document.","Raising the similarity threshold improves answer relevance at the price of fewer cache hits; the paper finds a candidate sweet spot near 0.85–0.9 for one of the benchmarks.","Repeated queries to the same document make the system progressively cheaper and faster, because cache hits skip the summarization API call that dominates end-to-end latency."],"supporting_citations":[{"why":"Supplies the sentence-embedding encoder used to turn questions into the vectors whose cosine similarity drives cache hits.","marker":"[23]"},{"why":"Provides the vector-index technology used to store and search cached summaries by question embedding.","marker":"[14]"},{"why":"Supplies one of the three public QA datasets used to measure utility, hit rate, and token savings.","marker":"[16]"},{"why":"Supplies a second public QA dataset used in the same evaluation.","marker":"[15]"},{"why":"Defines the full-response semantic-cache baseline that the intermediate-summary design is compared against.","marker":"[1]"},{"why":"Represents the order-sensitive prompt-cache approach whose limitations motivate the order-independent summary cache.","marker":"[9]"},{"why":"Shows an alternative RAG-oriented KV-cache reuse whose dynamic-update constraints the summary approach avoids.","marker":"[25]"},{"why":"Provides another KV-cache RAG baseline contrasted with caching intermediate summaries.","marker":"[13]"},{"why":"Supports the finding that compressed, focused context can match or beat full-context answer quality.","marker":"[12]"},{"why":"Supplies evidence that cosine similarity of embeddings tracks semantic closeness between questions.","marker":"[27]"}],"fun_headline_variants":["Semantic summary caching halves LLM QA compute","Cached contextual summaries cut LLM QA costs in half","Reuse query-aware summaries to halve QA compute","Summary caching: 50% less compute for LLM QA","Cached summaries: same QA accuracy, half compute"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The system rests on the premise that questions whose embeddings are close under cosine similarity need essentially the same facts from the document, so a contextual summary written for one will serve the other.","fun_headline_variants_meta":{"raw":{"variants":["Semantic summary caching halves LLM QA compute","Cached contextual summaries cut LLM QA costs in half","Reuse query-aware summaries to halve QA compute","Summary caching: 50% less compute for LLM QA","Cached summaries: same QA accuracy, half compute"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00037,"raw_usage":{"total_tokens":1926,"prompt_tokens":834,"completion_tokens":1092,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":450,"completion_tokens_details":{"reasoning_tokens":1014}},"tokens_in":450,"tokens_out":1092,"duration_ms":10460,"temperature":1.0,"reasoning_tokens":1014,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:53:51.699983+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"For a fixed document, identify many question pairs whose embedding cosine similarity is just above the deployment threshold, answer the second question using only the first question's cached contextual summary, and compare utility against a query-agnostic summary baseline; if the cached summary is no better across such pairs, the central assumption fails.","supporting_citations":[{"cited_title":"Sentence-BERT: Sentence embeddings using siamese bert-networks","cited_arxiv_id":null,"evidence_quote":"Supplies the sentence-embedding encoder used to turn questions into the vectors whose cosine similarity drives cache hits."},{"cited_title":"Dai, Jakob Uszko- reit, Quoc Le, and Slav Petrov","cited_arxiv_id":null,"evidence_quote":"Supplies one of the three public QA datasets used to measure utility, hit rate, and token savings."},{"cited_title":"Weld, and Luke Zettlemoyer","cited_arxiv_id":null,"evidence_quote":"Supplies a second public QA dataset used in the same evaluation."},{"cited_title":"GPTCache: An open-source semantic cache for llm applications enabling faster answers and cost savings","cited_arxiv_id":null,"evidence_quote":"Defines the full-response semantic-cache baseline that the intermediate-summary design is compared against."},{"cited_title":"Hammerla","cited_arxiv_id":null,"evidence_quote":"Supplies evidence that cosine similarity of embeddings tracks semantic closeness between questions."}],"review_version":1}