REVIEW 8 cited by
SparQ Attention: Bandwidth-Efficient LLM Inference
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The computational difficulties of large language model (LLM) inference remain a significant obstacle to their widespread deployment. The need for many applications to support long input sequences and process them in large batches typically causes token-generation to be bottlenecked by data transfer. For this reason, we introduce SparQ Attention, a technique for increasing the inference throughput of LLMs by utilising memory bandwidth more efficiently within the attention layers, through selective fetching of the cached history. Our proposed technique can be applied directly to off-the-shelf LLMs during inference, without requiring any modification to the pre-training setup or additional fine-tuning. We show that SparQ Attention brings up to 8x savings in attention data transfers without substantial drops in accuracy, by evaluating Llama 2 and 3, Mistral, Gemma and Pythia models on a wide range of downstream tasks.
Forward citations
Cited by 8 Pith papers
-
Unveiling Simplicities of Attention: Adaptive Long-Context Head Identification
A second-moment Gaussian approximation of bulk key distributions lets a cheap query-adaptive test label attention heads as local or long-context, matching a full-attention oracle and outperforming static head pruning.
-
From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving
KV-cache serving systems concentrate into five archetypes under a four-axis taxonomy, with ownership explaining residual distributed design variance and seven measurement gaps blocking next steps.
-
Cartridges: Lightweight and general-purpose long context representations via self-study
A per-corpus trained KV cache, called a Cartridge, matches full-context in-context learning quality on long-document benchmarks while using up to 38.6x less serving memory.
-
HATA: Trainable and Hardware-Efficient Hash-Aware Top-k Attention for Scalable Large Model Inference
HATA learns per-head binary hash codes for queries and keys and selects top-k tokens by Hamming distance, achieving up to 7.2x decoding speedup over full attention with near-baseline accuracy on tested benchmarks.
-
SALE : Low-bit Estimation for Efficient Sparse Attention in Long-context LLM Prefilling
SALE is a training-free sparse attention method that uses 4-bit quantized query-key estimates and a relative attention score to skip unimportant blocks, achieving over 3.36x prefill speedup on 64K+ token contexts with...
-
Top-Theta Attention: Sparsifying Transformers by Compensated Thresholding
Static per-head thresholds calibrated on a small dataset can replace top-k selection in transformer attention, achieving 3 to 10x sparsity with negligible accuracy loss.
-
AQUA: Attention via QUery mAgnitudes for Memory and Compute Efficient Inference in LLMs
A training-free method that prunes low-magnitude dimensions of projected query/key vectors in attention, cutting dot-product cost by 25% with small benchmark degradation.
-
Learn from the Past: Fast Sparse Indexing for Large Language Model Decoding
LFPS predicts which KV cache positions a decoder will attend to by reusing historical vertical and slash patterns plus local expansion, then runs exact Top-k only on the predicted candidate set.
Discussion (0). Continue with ORCID to comment.