REVIEW 8 cited by
Loki: Low-rank Keys for Efficient Sparse Attention
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Inference on large language models (LLMs) can be expensive in terms of the compute and memory costs involved, especially when long sequence lengths are used. In particular, the self-attention mechanism used in LLM inference contributes significantly to these costs, which has sparked an interest in approximating the self-attention computation to reduce such costs. In this work, we propose to approximate self-attention by focusing on the dimensionality of key vectors computed in the attention block. Our analysis reveals that key vectors lie in a significantly lower-dimensional space, consistently across several datasets and models. Exploiting this observation, we propose Loki, a novel sparse attention method that ranks and selects tokens in the KV-cache based on attention scores computed in low-dimensional space. Our evaluations show that Loki is able to speed up the attention computation due to reduced data movement (load/store) and compute costs while maintaining the efficacy of the models better than other popular approximation methods.
Forward citations
Cited by 8 Pith papers
-
LOCKS: Page-Local Compact Key Summaries for Efficient Long-Context Decoding
Page-local rank-8 spectral key summaries let sparse decode selection track the exact mass oracle and match FullKV quality at ~2% attended tokens with 2× latency cut at 1M context.
-
COBS: Cumulant Order Block Sparse Attention
A compressed within-block key covariance lets block-sparse attention recover most of dense long-context retrieval quality at roughly first-order selector traffic.
-
vAttention: Verified Sparse Attention
vAttention is a sparse attention method that mixes heavy-hitter tokens with a statistically sized random sample to provide (ε, δ)-guaranteed approximation of full attention.
-
HATA: Trainable and Hardware-Efficient Hash-Aware Top-k Attention for Scalable Large Model Inference
HATA learns per-head binary hash codes for queries and keys and selects top-k tokens by Hamming distance, achieving up to 7.2x decoding speedup over full attention with near-baseline accuracy on tested benchmarks.
-
Hardware-Efficient Attention for Fast Decoding
Grouped-Tied Attention and Grouped Latent Attention reduce KV-cache memory and speed up LLM decoding by up to 2x while matching the quality of GQA and MLA at up to 1.47B parameters.
-
AQUA: Attention via QUery mAgnitudes for Memory and Compute Efficient Inference in LLMs
A training-free method that prunes low-magnitude dimensions of projected query/key vectors in attention, cutting dot-product cost by 25% with small benchmark degradation.
-
GraphKV: Breaking the Static Selection Paradigm with Graph-Based KV Cache Eviction
GraphKV adds a similarity-based decay penalty to existing top-k KV cache eviction scores, improving retention diversity and accuracy on LongBench and needle-in-a-haystack tasks.
-
Exploiting Sparsity for Long Context Inference: Million Token Contexts on Commodity GPUs
A top-k attention mechanism backed by CPU vector search allows million-token LLM contexts to run on a 16GB GPU while keeping over 95% of dense-attention performance, though the 2% sparsity claim does not hold at the l...
Discussion (0). Continue with ORCID to comment.