Pith. sign in

REVIEW 15 cited by

Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2407.18003 v4 pith:KDVJUZN7 submitted 2024-07-25 cs.CL

Luohe Shi , Hongyi Zhang , Yao Yao , Zuchao Li , Hai Zhao This is my paper · ORCID
classification cs.CL
keywords methodscachereviewvariouslanguagephaseefficiencygithub
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large Language Models (LLMs), epitomized by ChatGPT's release in late 2022, have revolutionized various industries with their advanced language comprehension. However, their efficiency is challenged by the Transformer architecture's struggle with handling long texts. KV Cache has emerged as a pivotal solution to this issue, converting the time complexity of token generation from quadratic to linear, albeit with increased GPU memory overhead proportional to conversation length. With the development of the LLM community and academia, various KV Cache compression methods have been proposed. In this review, we dissect the various properties of KV Cache and elaborate on various methods currently used to optimize the KV Cache space usage of LLMs. These methods span the pre-training phase, deployment phase, and inference phase, and we summarize the commonalities and differences among these methods. Additionally, we list some metrics for evaluating the long-text capabilities of large language models, from both efficiency and capability perspectives. Our review thus sheds light on the evolving landscape of LLM optimization, offering insights into future advancements in this dynamic field. Links to the papers mentioned in this review can be found in our Github Repo https://github.com/zcli-charlie/Awesome-KV-Cache.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 15 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. FastTPS: An Optimized Method for LLM Token Phase for AI accelerators

    cs.LG 2026-07 conditional novelty 6.0 of 10

    FastTPS accelerates LLM token-phase inference via reloading-free static KV-cache management, tiled fused RoPE attention, and interlaced-weight MLP fusion, yielding up to 6× speedup at 93% bandwidth on AMD NPUs.

  2. EvolKV: Evolutionary KV Cache Compression for LLM Inference

    cs.LG 2025-09 conditional novelty 6.0 of 10

    CMA-ES search over per-layer KV cache budgets beats uniform and pyramidal compression heuristics on LongBench, NIAH, RULER, and GSM8K, and edges past the full cache on one code dataset at 1.5% of the budget.

  3. Adaptive KV-Cache Compression without Manually Setting Budget

    cs.DB 2025-09 conditional novelty 6.0 of 10

    GVote removes hand-set KV-cache budgets by synthesizing future queries from the hidden state distribution and keeping the union of keys they vote for, yielding around 2x memory savings at comparable accuracy.

  4. A Distributed Learned Hash Table

    cs.NI 2025-08 conditional novelty 6.0 of 10

    A learned order-preserving hash inside a Chord-style DHT lets range queries finish in roughly the same cost as single-key lookups, cutting latency and messages by 80-90%+ in tests.

  5. MadaKV: Adaptive Modality-Perception KV Cache Eviction for Efficient Multimodal Long-Context Inference

    cs.LG 2025-06 conditional novelty 6.0 of 10

    MadaKV adaptively splits the KV cache budget by attention-head modality preference and compensates across layers, cutting cache memory by 80-95% and speeding decoding by 1.3-1.5x with small accuracy loss.

  6. Top-Theta Attention: Sparsifying Transformers by Compensated Thresholding

    cs.CL 2025-02 conditional novelty 6.0 of 10

    Static per-head thresholds calibrated on a small dataset can replace top-k selection in transformer attention, achieving 3 to 10x sparsity with negligible accuracy loss.

  7. EgoPrune: Efficient Token Pruning for Egomotion Video Reasoning in Embodied Agent

    cs.CV 2025-07 conditional novelty 5.0 of 10

    EgoPrune prunes egomotion video tokens by homography-based frame alignment and MMR selection, keeping accuracy close to the full-token baseline while reducing compute.

  8. DAC: A Dynamic Attention-aware Approach for Task-Agnostic Prompt Compression

    cs.CL 2025-07 conditional novelty 5.0 of 10

    A dynamic attention-aware metric for task-agnostic prompt compression that combines entropy and attention scores, with iterative entropy recalculation, outperforms prior entropy-based compressors on multiple benchmarks.

  9. SpindleKV: A Novel KV Cache Reduction Method Balancing Both Shallow and Deep Layers

    cs.CL 2025-07 conditional novelty 5.0 of 10

    SpindleKV compresses LLM KV cache by evicting low-attention tokens in deep layers and replacing near-duplicate key and value vectors in shallow layers with a shared codebook, while preserving benchmark accuracy at 15 ...

  10. Semantic Scheduling for LLM Inference

    cs.LG 2025-06 conditional novelty 5.0 of 10

    A semantic scheduler for LLM inference uses urgency labels and estimated remaining compute to cut waiting times for urgent requests, tested on emergency medical data.

  11. PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inference

    cs.LG 2025-09 conditional novelty 4.0 of 10

    PagedEviction prunes the KV cache in whole blocks using a key-value norm ratio, speeding up long-context LLM inference in vLLM while staying close to full-cache accuracy.

  12. Infinite Sampling: Efficient and Stable Grouped RL Training for Large Language Models

    cs.LG 2025-06 conditional novelty 4.0 of 10

    A GRPO decoding framework that cuts memory via micro-batched KV-cache reuse and improves decoding-round efficiency with predicted-length scheduling, at the cost of serialization.

  13. Effective and Efficient Schema-aware Information Extraction Using On-Device Large Language Models

    cs.CL 2025-05 conditional novelty 4.0 of 10

    DLISC, a dual-LoRA two-stage schema-aware extraction method with incremental schema caching, reports better F1 and lower latency than three RAG baselines on two IE datasets, though the comparison lacks error bars and code.

  14. Voice-based AI Agents: Filling the Economic Gaps in Digital Health Delivery

    cs.AI 2025-07 reject novelty 3.0 of 10

    A position paper with a 33-patient pilot argues LLM phone agents can make routine monitoring cheaper, but the savings are assumed rather than measured.

  15. DeepSeek: Paradigm Shifts and Technical Evolution in Large AI Models

    cs.AI 2025-07 unverdicted

    A survey of DeepSeek's V3 and R1 models covering MLA, MoE, MTP, GRPO, and training engineering, with no new experimental results.

Pith tools