REVIEW 6 cited by
KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Long context capability is a crucial competency for large language models (LLMs) as it mitigates the human struggle to digest long-form texts. This capability enables complex task-solving scenarios such as book summarization, code assistance, and many more tasks that are traditionally manpower-intensive. However, transformer-based LLMs face significant challenges with long context input due to the growing size of the KV cache and the intrinsic complexity of attending to extended inputs; where multiple schools of efficiency-driven approaches - such as KV cache quantization, token dropping, prompt compression, linear-time sequence models, and hybrid architectures - have been proposed to produce efficient yet long context-capable models. Despite these advancements, no existing work has comprehensively benchmarked these methods in a reasonably aligned environment. In this work, we fill this gap by providing a taxonomy of current methods and evaluating 10+ state-of-the-art approaches across seven categories of long context tasks. Our work reveals numerous previously unknown phenomena and offers insights - as well as a friendly workbench - for the future development of long context-capable LLMs. The source code is available at https://github.com/henryzhongsc/longctx_bench.
Forward citations
Cited by 6 Pith papers
-
CommVQ: Commutative Vector Quantization for KV Cache Compression
CommVQ compresses LLM key-value caches to roughly one bit per value using additive quantization with a codebook designed to commute with rotary position embeddings, preserving most benchmark accuracy.
-
Deterministic Inference across Tensor Parallel Sizes That Eliminates Training-Inference Mismatch
Tree-Based Invariant Kernels fix the floating-point reduction order across GPUs, making LLM logits and sampled tokens bitwise identical for tensor-parallel sizes 1/2/4/8 and exactly matching vLLM (TP=4) with FSDP (TP=1).
-
SALE : Low-bit Estimation for Efficient Sparse Attention in Long-context LLM Prefilling
SALE is a training-free sparse attention method that uses 4-bit quantized query-key estimates and a relative attention score to skip unimportant blocks, achieving over 3.36x prefill speedup on 64K+ token contexts with...
-
ProxSparse: Regularized Learning of Semi-Structured Sparsity Masks for Pretrained LLMs
ProxSparse learns 2:4 semi-structured sparsity masks for pretrained LLMs via regularized proximal-gradient optimization, outperforming heuristic baselines on seven models.
-
A Survey on Large Language Model Acceleration based on KV Cache Management
A survey that classifies KV cache management techniques for faster LLM inference into token-level, model-level, and system-level categories, with benchmark resources.
-
A Survey of RWKV
A review of the RWKV architecture, its versions, applications, benchmarks, and open-source ecosystem; it presents no new experimental results.
Discussion (0). Continue with ORCID to comment.