Pith. sign in

REVIEW 14 cited by

RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2404.12457 v2 pith:JRNDNKXZ submitted 2024-04-18 cs.DC cs.CLcs.LG

Chao Jin , Zili Zhang , Xuanlin Jiang , Fangyue Liu , Xin Liu , Xuanzhe Liu , Xin Jin This is my paper
classification cs.DCcs.CLcs.LG
keywords knowledgeragcachecachinggenerationinferencefaissintermediatelanguage
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Retrieval-Augmented Generation (RAG) has shown significant improvements in various natural language processing tasks by integrating the strengths of large language models (LLMs) and external knowledge databases. However, RAG introduces long sequence generation and leads to high computation and memory costs. We propose RAGCache, a novel multilevel dynamic caching system tailored for RAG. Our analysis benchmarks current RAG systems, pinpointing the performance bottleneck (i.e., long sequence due to knowledge injection) and optimization opportunities (i.e., caching knowledge's intermediate states). Based on these insights, we design RAGCache, which organizes the intermediate states of retrieved knowledge in a knowledge tree and caches them in the GPU and host memory hierarchy. RAGCache proposes a replacement policy that is aware of LLM inference characteristics and RAG retrieval patterns. It also dynamically overlaps the retrieval and inference steps to minimize the end-to-end latency. We implement RAGCache and evaluate it on vLLM, a state-of-the-art LLM inference system and Faiss, a state-of-the-art vector database. The experimental results show that RAGCache reduces the time to first token (TTFT) by up to 4x and improves the throughput by up to 2.1x compared to vLLM integrated with Faiss.

Discussion (0). Sign in to comment.

Forward citations

Cited by 14 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Understanding Is Done Early: A Depth Division of Labor in Large Language Models and Its Use for Unbounded-Context Memory

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Long-context memory can be organized by caching intermediate residual states and recomputing only query-conditioned upper layers over a bounded retrieved pack.

  2. FinCacheServe: Dependency-Consistent Answer Reuse for Cost-Efficient RAG Serving over Mutable Enterprise Documents

    cs.IR 2026-07 conditional novelty 6.0 of 10

    Dependency-checked answer reuse lets FinCacheServe skip about 53% of LLM calls in hosted SEC-derived RAG traces without serving any observed stale outputs.

  3. Functional Cache Grafting: Robust and Rapid Code-Policy Synthesis for Embodied Agents

    cs.PL 2026-06 unverdicted novelty 6.0 of 10

    FCGraft synthesizes code policies for embodied agents by grafting KV caches from a library of validated functions, claiming 18.31% higher success rate and 2.3x faster synthesis than prompt-level caching.

  4. Mobility-Aware Cache Framework for Scalable LLM-Based Human Mobility Simulation

    cs.AI 2026-02 conditional novelty 6.0 of 10

    A latent-space reasoning cache with a lightweight decoder cuts the cost of LLM-based human mobility simulation by roughly 40-90% while keeping trajectory quality comparable.

  5. HedraRAG: Coordinating LLM Generation and Database Retrieval in Heterogeneous RAG Serving

    cs.DB 2025-07 conditional novelty 6.0 of 10

    HedraRAG uses a graph abstraction and dynamic transformations to pipeline generation and retrieval stages, achieving 1.5x to 5x speedups in heterogeneous RAG serving.

  6. KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows

    cs.DC 2025-07 conditional novelty 6.0 of 10

    KVFlow uses workflow-aware eviction priorities and overlapped KV prefetching to cut cache-miss latency in LLM multi-agent serving.

  7. KAP: Bridging the Knowledge Selection-Runtime Consumption Gap in LLM Systems

    cs.LG 2026-07 conditional novelty 5.0 of 10

    KAP compiles structured knowledge priors into runtime access plans that cut proposal-time KV access to 5.5% at 128K while matching full-context QA quality and yielding up to 1.19× decode throughput.

  8. ConceptBot: Enhancing Robot's Autonomy through Task Decomposition with Large Language Models and Knowledge Graph

    cs.RO 2025-08 conditional novelty 5.0 of 10

    Using ConceptNet-augmented prompts, ConceptBot reports 87% vs 31% success on implicit tasks and 76% vs 15% on risk-aware tasks over a re-implemented SayCan baseline, with an 80% SafeAgentBench score.

  9. Beyond Isolated Capabilities: Bridging Long CoT Reasoning and Long-Context Understanding

    cs.CL 2025-07 conditional novelty 5.0 of 10

    DeepSeek-R1-distilled models show higher multi-document QA accuracy than their base counterparts and flatter position-bias curves, especially with 50-80 documents.

  10. WebANNS: Fast and Efficient Approximate Nearest Neighbor Search in Web Browsers

    cs.IR 2025-07 conditional novelty 5.0 of 10

    A new browser-native ANNS engine uses WebAssembly, lazy loading, and a memory optimizer to cut P99 query latency by up to 743.8x and memory use by up to 39% compared to Mememo.

  11. A Survey of LLM $\times$ DATA

    cs.DB 2025-05 conditional novelty 5.0 of 10

    A comprehensive survey of the bidirectional links between LLMs and data management, organized as DATA4LLM and LLM4DATA with a new 'IaaS' data-quality framework.

  12. MultiFluxAI Enhancing Platform Engineering with Advanced Agent-Orchestrated Retrieval Systems

    cs.AI 2025-08 reject novelty 4.0 of 10

    The authors claim their MultiFluxAI orchestration framework achieves 95% accuracy and 0-10 ms responses by combining rule-based routing, caching, and graph knowledge stores for multi-service RAG queries.

  13. A Survey on Proactive Defense Strategies Against Misinformation in Large Language Models

    cs.IR 2025-07 reject novelty 3.0 of 10

    A survey claims proactive defenses against LLM misinformation outperform post-hoc detection by up to 63%, but no meta-analysis details are provided to support the claim.

  14. Retrieval-Augmented Generation: A Comprehensive Survey of Architectures, Enhancements, and Robustness Frontiers

    cs.IR 2025-05 conditional novelty 2.0 of 10

    A survey of retrieval-augmented generation methods, organized by architecture, with a comparative analysis of published QA results.

Pith tools