Pith. sign in

REVIEW 9 cited by

ThinK: Thinner Key Cache by Query-Driven Pruning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2407.21018 v3 pith:WHEZHNLK submitted 2024-07-30 cs.CL cs.AI

Yuhui Xu , Zhanming Jie , Hanze Dong , Lei Wang , Xudong Lu , Aojun Zhou , Amrita Saha , Caiming Xiong
show 1 more author
Doyen Sahoo
This is my paper
classification cs.CLcs.AI
keywords cachememorythinkpruningacrossattentionlanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large Language Models (LLMs) have revolutionized the field of natural language processing, achieving unprecedented performance across a variety of applications. However, their increased computational and memory demands present significant challenges, especially when handling long sequences. This paper focuses on the long-context scenario, addressing the inefficiencies in KV cache memory consumption during inference. Unlike existing approaches that optimize the memory based on the sequence length, we identify substantial redundancy in the channel dimension of the KV cache, as indicated by an uneven magnitude distribution and a low-rank structure in the attention weights. In response, we propose ThinK, a novel query-dependent KV cache pruning method designed to minimize attention weight loss while selectively pruning the least significant channels. Our approach not only maintains or enhances model accuracy but also achieves a reduction in KV cache memory costs by over 20% compared with vanilla KV cache eviction and quantization methods. For instance, ThinK integrated with KIVI can achieve a 2.8x reduction in peak memory usage while maintaining nearly the same quality, enabling up to a 5x increase in batch size when using a single GPU. Extensive evaluations on the LLaMA and Mistral models across various long-sequence datasets verified the efficiency of ThinK, establishing a new baseline algorithm for efficient LLM deployment without compromising performance. Our code has been made available at https://github.com/SalesforceAIResearch/ThinK.

Discussion (0). Sign in to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. WitCert: Sound Runtime Risk Observability and Gating for KV-Cache Quantization

    cs.AR 2026-07 conditional novelty 7.0 of 10

    A per-request sound upper bound on KV-cache quantization error, implemented as a runtime meter that gates and repairs compression to restore quality.

  2. High-accuracy Low-Bit KV-Cache Quantization via Local Distribution Restoration

    cs.LG 2026-06 conditional novelty 6.0 of 10

    A learned top-K distribution-restoration layer recovers near-FP16 long-context accuracy under 1-bit KV-cache quantization.

  3. Token Sparse Attention: Efficient Long-Context Inference with Interleaved Token Selection

    cs.CL 2026-02 unverdicted novelty 6.0 of 10

    Token Sparse Attention uses dynamic per-head token compression and decompression during attention to achieve up to 3.23x speedup at 128K context with under 1% accuracy loss.

  4. EffiVLM-BENCH: A Comprehensive Benchmark for Evaluating Training-Free Acceleration in Large Vision-Language Models

    cs.CL 2025-05 conditional novelty 6.0 of 10

    EffiVLM-Bench is a benchmark study showing token compression is task- and model-dependent, KV cache methods are more loyal, and parameter compression preserves accuracy better at typical ratios.

  5. StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understanding

    cs.CV 2025-08 conditional novelty 5.0 of 10

    A training-free, query-agnostic KV cache compression method for streaming video MLLMs, using chat-template attention as a saliency proxy, matches or beats prior streaming methods at a fixed 6K memory budget.

  6. Breaking the Boundaries of Long-Context LLM Inference: Adaptive KV Management on a Single Commodity GPU

    cs.OS 2025-06 conditional novelty 5.0 of 10

    LeoAM reports a 3.46x average latency speedup for long-context LLM inference on one commodity GPU by adaptively chunking KV data and loading compact key abstracts from disk instead of full KV values.

  7. Titanus: Enabling KV Cache Pruning and Quantization On-the-Fly for LLM Acceleration

    cs.AR 2025-05 conditional novelty 5.0 of 10

    Titanus is a proposed accelerator that compresses the KV cache with cascade pruning and quantization and reports large simulated energy and throughput gains over GPU and FPGA baselines.

  8. HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMs

    cs.CL 2025-07 conditional novelty 4.0 of 10

    HCAttention combines key quantization, CPU value offloading, and cumulative-attention eviction to run long-context LLMs with 12.5% to 25% of the GPU KV cache while keeping LongBench accuracy close to full attention.

  9. CE-LSLM: Efficient Large-Small Language Model Inference and Communication via Cloud-Edge Collaboration

    cs.NI 2025-05 reject novelty 4.0 of 10

    CE-LSLM reports lower edge inference latency by reusing cloud LLM KV caches in edge SLMs, but the cross-model reuse mechanism is asserted, not demonstrated.

Pith tools