REVIEW 9 cited by
Eigen Attention: Attention in Low-Rank Space for KV Cache Compression
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large language models (LLMs) represent a groundbreaking advancement in the domain of natural language processing due to their impressive reasoning abilities. Recently, there has been considerable interest in increasing the context lengths for these models to enhance their applicability to complex tasks. However, at long context lengths and large batch sizes, the key-value (KV) cache, which stores the attention keys and values, emerges as the new bottleneck in memory usage during inference. To address this, we propose Eigen Attention, which performs the attention operation in a low-rank space, thereby reducing the KV cache memory overhead. Our proposed approach is orthogonal to existing KV cache compression techniques and can be used synergistically with them. Through extensive experiments over OPT, MPT, and Llama model families, we demonstrate that Eigen Attention results in up to 40% reduction in KV cache sizes and up to 60% reduction in attention operation latency with minimal drop in performance. Code is available at https://github.com/UtkarshSaxena1/EigenAttn.
Forward citations
Cited by 9 Pith papers
-
SAKI: Score-Aware Low-Rank Key Indexing with Random-Matrix Noise Correction for KV Retrieval
Score-aware low-rank key compression (SAKI) minimizes expected attention-score distortion and beats PCA-based KV indexes on four LLMs.
-
PhyCheck: Fine-Grained Evidence-Grounded Dataset for Physical Law Understanding in Video-LLMs
PhyCheck is a 69,825-pair video QA benchmark that tests and improves Video-LLMs' ability to judge whether events obey physical laws, with fine-grained evidence questions and a context-sensitivity pilot.
-
KV-Latent: Dimensional-level KV Cache Reduction with Frequency-aware Rotary Positional Embedding
By downsampling key and value head dimensions and retraining with distillation, KV-Latent cuts KV cache memory by about 50% on 7-8B LLMs while keeping average benchmark scores within about 1 point of the base model.
-
Cartridges: Lightweight and general-purpose long context representations via self-study
A per-corpus trained KV cache, called a Cartridge, matches full-context in-context learning quality on long-document benchmarks while using up to 38.6x less serving memory.
-
Hardware-Efficient Attention for Fast Decoding
Grouped-Tied Attention and Grouped Latent Attention reduce KV-cache memory and speed up LLM decoding by up to 2x while matching the quality of GQA and MLA at up to 1.47B parameters.
-
LatentLLM: Attention-Aware Joint Tensor Compression
LatentLLM compresses pretrained LLMs and multimodal models with attention-aware joint low-rank tensor decomposition, outperforming SVD-based baselines on OPT perplexity and LLaVA ScienceQA.
-
AQUA: Attention via QUery mAgnitudes for Memory and Compute Efficient Inference in LLMs
A training-free method that prunes low-magnitude dimensions of projected query/key vectors in attention, cutting dot-product cost by 25% with small benchmark degradation.
-
Beyond Homogeneous Attention: Memory-Efficient LLMs via Fourier-Approximated KV Cache
Compressing the less context-sensitive dimensions of the LLM KV cache with a truncated Fourier transform preserves long-context accuracy on LongBench and needle-in-a-haystack tests, while the claimed memory savings ar...
-
$\mu$-MoE: Test-Time Pruning as Micro-Grained Mixture-of-Experts
Test-time Wanda pruning, reframed as a mixture of micro-experts, adapts the sparse weight mask to each prompt and improves perplexity and VQA accuracy over static pruning baselines.
Discussion (0). Continue with ORCID to comment.