Pith. sign in

REVIEW 4 cited by

Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.09398 v2 pith:QGEUKG3Z submitted 2024-02-14 cs.LG cs.AI

Harry Dong , Xinyu Yang , Zhenyu Zhang , Zhangyang Wang , Yuejie Chi , Beidi Chen This is my paper · ORCID
classification cs.LGcs.AI
keywords cachelesscomputationaldecodingefficientlargememorymethods
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Many computational factors limit broader deployment of large language models. In this paper, we focus on a memory bottleneck imposed by the key-value (KV) cache, a computational shortcut that requires storing previous KV pairs during decoding. While existing KV cache methods approach this problem by pruning or evicting large swaths of relatively less important KV pairs to dramatically reduce the memory footprint of the cache, they can have limited success in tasks that require recollecting a majority of previous tokens. To alleviate this issue, we propose LESS, a simple integration of a (nearly free) constant sized cache with eviction-based cache methods, such that all tokens can be queried at later decoding steps. Its ability to retain information throughout time shows merit on a variety of tasks where we demonstrate LESS can help reduce the performance gap from caching everything, sometimes even matching it, all while being efficient. Relevant code can be found at https://github.com/hdong920/LESS.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving

    cs.DC 2026-06 accept novelty 6.5 of 10

    KV-cache serving systems concentrate into five archetypes under a four-axis taxonomy, with ownership explaining residual distributed design variance and seven measurement gaps blocking next steps.

  2. Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generation

    cs.DC 2025-02 conditional novelty 6.0 of 10

    Cache-Craft reuses stored attention states for repeated RAG text chunks with selective recomputation, cutting redundant LLM prefill computation by 51% over prefix caching while retaining roughly 90% of answer quality.

  3. GLIDE: Guided Layerwise Hybrid Attention for Efficient LLM Inference

    cs.AI 2026-06 reject novelty 4.0 of 10

    GLIDE allocates softmax attention only to early/middle transformer layers and linearizes later layers, cutting KV-cache I/O by up to ~62× while retaining 92–96% of baseline accuracy.

  4. Climber: Toward Efficient Scaling Laws for Large Recommendation Models

    cs.IR 2025-02 conditional novelty 4.0 of 10

    Climber reports that splitting user sequences by behavior type, adding adaptive temperature, and co-designed batching enable more efficient Transformer scaling in recommender systems.

Pith tools