REVIEW 6 cited by
EPIC: Efficient Position-Independent Caching for Serving Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large Language Models (LLMs) show great capabilities in a wide range of applications, but serving them efficiently becomes increasingly challenging as requests (prompts) become more complex. Context caching improves serving performance by reusing Key-Value (KV) vectors, the intermediate representations of tokens that are repeated across requests. However, existing context caching requires exact prefix matches across requests, limiting reuse cases in settings such as few-shot learning and retrieval-augmented generation, where immutable content (e.g., documents) remains unchanged across requests but is preceded by varying prefixes. Position-Independent Caching (PIC) addresses this issue by enabling modular reuse of the KV vectors regardless of prefixes. We formalize PIC and advance prior work by introducing EPIC, a serving system incorporating our new LegoLink algorithm, which mitigates the inappropriate "attention sink" effect at every document beginning, to maintain accuracy with minimal computation. Experiments show that EPIC achieves up to 8x improvements in Time-To-First-Token (TTFT) and 7x throughput gains over existing systems, with negligible or no accuracy loss.
Forward citations
Cited by 6 Pith papers
-
Low-Scaling Many-Body Green's Function Calculations for Molecular Systems via Interacting-Bath Dynamical Embedding Theory
Molecular ibDET assembles full-system GW/EOM-CCSD Green's functions from small atom-centered impurity problems and recovers IPs/EAs within about 0.1 eV of full-system results.
-
Auditing Prompt Caching in Language Model APIs
Timing audits of 17 LLM APIs show prompt caching in eight providers, global cross-user cache sharing in seven, and evidence that OpenAI's embedding model is decoder-only.
-
Compute Globally, Materialize Locally: The Memory Contract of Sparse Event-KV
Source-omitted event KV rows can carry compact upstream state (semantic materialization); deliberate answer-free carriers raise recovery from 6% to 51% on Qwen3-8B, while passive natural mentions do not.
-
Functional Cache Grafting: Robust and Rapid Code-Policy Synthesis for Embodied Agents
FCGraft synthesizes code policies for embodied agents by grafting KV caches from a library of validated functions, claiming 18.31% higher success rate and 2.3x faster synthesis than prompt-level caching.
-
Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Serving
Pattern-aware speculative tool execution cuts agent end-to-end latency by roughly half and observed tool latency by about 1.8× by overlapping predicted tools with LLM generation.
-
InfoFlow KV: Information-Flow-Aware KV Recomputation for Long Context
Selecting tokens for KV recomputation by prompt-attention norms, computed under inference-consistent global RoPE positions, beats CacheBlend and EPIC on long-context QA in LLMs and VLMs.
Discussion (0). Continue with ORCID to comment.