Pith. sign in

REVIEW 9 cited by

LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.03111 v1 pith:LWUTVNCK submitted 2024-10-04 cs.LG cs.AIcs.CL

Rongzhi Zhang , Kuang Wang , Liyuan Liu , Shuohang Wang , Hao Cheng , Chao Zhang , Yelong Shen This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords compressioncachellmsstagesapproachexistingextensivelow-rank
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

The Key-Value (KV) cache is a crucial component in serving transformer-based autoregressive large language models (LLMs), enabling faster inference by storing previously computed KV vectors. However, its memory consumption scales linearly with sequence length and batch size, posing a significant bottleneck in LLM deployment. Existing approaches to mitigate this issue include: (1) efficient attention variants integrated in upcycling stages, which requires extensive parameter tuning thus unsuitable for pre-trained LLMs; (2) KV cache compression at test time, primarily through token eviction policies, which often overlook inter-layer dependencies and can be task-specific. This paper introduces an orthogonal approach to KV cache compression. We propose a low-rank approximation of KV weight matrices, allowing for plug-in integration with existing transformer-based LLMs without model retraining. To effectively compress KV cache at the weight level, we adjust for layerwise sensitivity and introduce a progressive compression strategy, which is supported by our theoretical analysis on how compression errors accumulate in deep networks. Our method is designed to function without model tuning in upcycling stages or task-specific profiling in test stages. Extensive experiments with LLaMA models ranging from 8B to 70B parameters across various tasks show that our approach significantly reduces the GPU memory footprint while maintaining performance.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SAKI: Score-Aware Low-Rank Key Indexing with Random-Matrix Noise Correction for KV Retrieval

    cs.LG 2026-08 conditional novelty 7.0 of 10

    Score-aware low-rank key compression (SAKI) minimizes expected attention-score distortion and beats PCA-based KV indexes on four LLMs.

  2. XQuant: Breaking the Memory Wall for LLM Inference with KV Cache Rematerialization

    cs.LG 2025-08 conditional novelty 7.0 of 10

    XQuant caches quantized layer inputs and rematerializes Keys and Values on the fly, achieving up to 12.5x KV cache memory savings with about 0.1 perplexity degradation.

  3. VarRate: Training-Free Variable-Rate KV Cache Compression for Long-Context LLMs

    cs.CL 2026-07 conditional novelty 6.0 of 10

    A training-free codec that allocates per-token low-rank budget by query salience matches the uncompressed model within 0.8 points at 20% KV memory and loses only 3.5-5.5 points under cache reuse where eviction methods...

  4. Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    Sleep-time Knowledge Seeding plus Dreaming lets LLMs expand capacity, distill fragile in-context memories into stable parameters, and self-improve without human labels.

  5. KV-Latent: Dimensional-level KV Cache Reduction with Frequency-aware Rotary Positional Embedding

    cs.CL 2025-07 conditional novelty 6.0 of 10

    By downsampling key and value head dimensions and retraining with distillation, KV-Latent cuts KV cache memory by about 50% on 7-8B LLMs while keeping average benchmark scores within about 1 point of the base model.

  6. Cartridges: Lightweight and general-purpose long context representations via self-study

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A per-corpus trained KV cache, called a Cartridge, matches full-context in-context learning quality on long-document benchmarks while using up to 38.6x less serving memory.

  7. MadaKV: Adaptive Modality-Perception KV Cache Eviction for Efficient Multimodal Long-Context Inference

    cs.LG 2025-06 conditional novelty 6.0 of 10

    MadaKV adaptively splits the KV cache budget by attention-head modality preference and compensates across layers, cutting cache memory by 80-95% and speeding decoding by 1.3-1.5x with small accuracy loss.

  8. TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inference

    cs.LG 2025-08 conditional novelty 5.0 of 10

    TPLA splits the latent KV cache across tensor-parallel GPUs while keeping every head's full view, yielding 1.79x and 1.93x decode speedups on DeepSeek-V3 and Kimi-K2 at 32K context with modest accuracy loss.

  9. DynaCalKV: Key-Value Cache Compression via Head Grouping and Adaptive Rank Allocation

    cs.LG 2026-07 conditional novelty 4.0 of 10

    Dynamic CKA head grouping plus adaptive rank allocation reduces Key-cache parameters versus ReCalKV while preserving competitive accuracy on MHA models, but hurts GQA long-context performance.

Pith tools