Pith. sign in

REVIEW 3 major objections 5 minor 59 references

Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read AQUA-KV predicts each layer's attention cache from the previous layer and stores only the residual, achieving near-lossless inference at 2–2.5 bits per value.

desk verdict Solid KV compression method with real gains at 2-bit, but the abstract's '<1% relative error' claim is contradicted by the paper's own perplexity numbers. read the letter →

arxiv 2501.19392 v4 pith:ABSH3VJS submitted 2025-01-31 cs.LG

Alina Shutova , Vladimir Malinovskii , Vage Egiazarian , Denis Kuznedelev , Denis Mazur , Nikita Surkov , Ivan Ermakov , Dan Alistarh This is my paper · ORCID
classification cs.LG
keywords KVcachecompressionquantizationlinearpredictorsresidualvectorLLMinferencelong-contextLLMscross-layerdependencies
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models store every token's attention keys and values in a cache that can consume tens of gigabytes, and compressing that cache to 2 bits usually costs accuracy. AQUA-KV is built on the observation that, in modern transformers, the key-value cache of one layer is strongly and approximately linearly predictable from the cache of the previous layer, because residual connections make adjacent hidden states close. The method trains one tiny linear regressor per layer—keys from the previous reconstructed keys, values from previous reconstructed values plus the current reconstructed keys—and stores only the quantized residual that the predictor cannot recover. On Llama 3.2 models the paper reports under 1% relative error in WikiText-2 perplexity and LongBench scores at 2–2.5 bits per value, roughly matching the uncompressed model while cutting cache memory from tens of gigabytes to a few. The calibration is one-shot and light: training the predictors for a 70B model takes 1–6 hours on a single GPU.

What carries the argument

The load-bearing object is the per-layer predictor pair $(f_{\text{key}}, f_{\text{value}})$ trained by Algorithm 1: $f_{\text{key}}$ is a linear regressor mapping the previous layer's reconstructed keys $\hat{K}_{l-1}$ to the current keys $K_l$, and $f_{\text{value}}$ maps the concatenation $[\hat{V}_{l-1}; \hat{K}_l]$ to the values $V_l$. The predictors are trained layer by layer using the de-quantized outputs of earlier layers, so they are robust to the reconstruction noise they will meet at inference. The remaining signal is compressed by a backbone quantizer—by default HIGGS, which applies a randomized Hadamard transform and then rounds groups of normally-distributed coefficients onto optimized lattice grids, or Quanto for uniform round-to-nearest—and only those residual codes are stored. Two practical safeguards carry weight: the first layer is kept at 3–4 bits because nothing predicts it, and the first few tokens are left uncompressed because attention sinks distort the input distribution the predictors rely on. Compression happens before rotary position embeddings, since linear predictors cannot be rotation-equivariant.

What would settle it

Measure the explained variance of the Algorithm 1 linear predictors on held-out sequences from domains far from RedPajama, such as code or mathematics, and at sequence lengths well beyond 8192 tokens; if keys or values drop below the roughly 0.75–0.89 explained-variance range that the paper equates with 1–2 bit quantization, the claimed near-lossless 2-bit compression would fail.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the information in Key-Value caches is substantially redundant across transformer layers, and that this redundancy can be harvested with simple linear models. Linear-probe measurements on Llama 3.2 3B show that the previous layer's keys already predict the current layer's keys about as well as a 2-bit quantizer, and previous-layer values explain more than half the variance of current values; same-layer keys also carry information for values. AQUA-KV turns this into an inference algorithm: each layer's keys are reconstructed as $f_{\text{key}}(K_{l-1})$, its values as $f_{\text{value}}([V_{l-1}; K_l])$, and only the residuals are quantized with a fast vector quantizer such as HIGGS. Trained sequentially on 256 RedPajama sequences so that predictors see reconstructed inputs exactly as at inference, the method reports WikiText-2 perplexity 7.03 versus 6.98 uncompressed and LongBench average 44.30 versus 44.61 on Llama 3.2 3B at about 2.16 bits, with similar near-lossless behavior across Llama 3.x and Qwen 2.5 models at 2, 3, and 4 bits.

Load-bearing premise

The method assumes that the dependency between consecutive layers' key-value caches is stable and mostly linear across model families and contexts, so that a small regressor trained on 256 calibration sequences can predict each layer's cache well enough that the remaining residual is small.

Editorial extensions

If this is right

  • At 2-bit compression, AQUA-KV roughly matches the quality of 3-bit baselines, so the same accuracy can be delivered on a cache budget about a third smaller; for a full-length Llama 3.1 70B sequence the paper reports the cache footprint dropping from 40 GiB in BF16 to about 5.7 GiB.
  • The method is one-shot and cheap to calibrate: predictors for a 70B model fit on a single GPU in 1–6 hours using about 256 RedPajama sequences, so it can be applied to a new model without retraining the LLM.
  • AQUA-KV is orthogonal to token pruning: combined with H2O keeping 20% of tokens, LongBench average drops by less than a point relative to H2O alone on Llama 3.x 3B and 8B models.
  • At inference the added prediction costs roughly 3% throughput: 8B generation runs at 23.31 versus 24.02 tokens per second and 70B at 5.76 versus 5.91 tokens per second in the paper's measurements.
  • The same predictor framework works with uniform quantization (Quanto), Hadamard-based vector quantization (HIGGS), and a rotated variant (QuaRot), so the redundancy extraction is independent of the quantizer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Extension the paper leaves implicit: the same layer-to-layer redundancy could be exploited at training time, for instance by initializing or regularizing adjacent KV projections toward shared structure, which would shrink the cache at the source rather than through inference-time predictors.
  • Testable extension: apply Algorithm 1 to multi-head attention models without Grouped Query Attention; the predictor's parameter count in the paper is small precisely because GQA keeps the number of KV heads low, so the speed and memory overhead might grow for MHA architectures.
  • Testable extension: allocate bit-widths per layer from the measured explained variance of each predictor, a direction the paper lists as future work; layers with higher predictability could drop below 2 bits and push the average cache size lower without retraining.
  • Cross-domain risk: the calibration uses RedPajama English web text, so the margin over baselines should be re-measured on code, mathematics, or multilingual long-context tasks, where token statistics differ and the paper's own ablations do not cover this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes AQUA-KV, a one-shot calibration method for KV-cache compression that trains compact linear predictors across Transformer layers to exploit inter-layer and key-value dependencies, and then quantizes only the residual error with a vector-quantization backbone (HIGGS). It reports large perplexity and LongBench improvements over KIVI, KVQuant, and over the HIGGS backbone alone at 2-4 bits per value, together with extensive ablations on predictor architectures, first-layer handling, attention sinks, and integration with H2O pruning.

Significance. If the empirical results hold, this is a practically valuable contribution: it demonstrates near-lossless KV-cache compression at 2-2.5 bits on several Llama and Qwen models, with a lightweight calibration procedure and a released reference implementation. The core idea of predict-then-quantize-residual across layers is simple, clearly motivated by linear-probe analysis, and orthogonal to pruning. The paper also ships per-task LongBench results and detailed ablations, which strengthens reproducibility. However, the headline quantitative claim in the abstract is overstated relative to Table 2, and several numerical inconsistencies between the main table and the appendix must be resolved before the results can be taken at face value.

major comments (3)
  1. [Abstract; §4.2, Table 2] The abstract claims 'under 1% relative error in perplexity and LongBench scores' at 2-2.5 bits, but Table 2 contradicts this for perplexity on two of the five evaluated models. For Llama 3.1 8B, WikiText-2 PPL is 5.72 vs 5.61 uncompressed, a relative increase of 1.96%; for Llama 3.1 70B, 2.62 vs 2.54 is a relative increase of 3.15%. LongBench drops are below 1% for these models, but the abstract explicitly commits to both metrics. The paper should either re-scope the claim (e.g., 'under 3% relative perplexity error and under 1% LongBench degradation'), restrict it to the Llama 3.2 3B model where it holds, or define 'relative error' in a way that these numbers satisfy.
  2. [§4.2, Table 2; Appendix E] The 'Quant. bits' column in Table 2 lists a single bitrate per method, but the appendix shows that the bitrate varies substantially by model. For Qwen 2.5 3B and 7B, AQUA-KV uses 2.44 and 2.48 bits respectively (because the first block is not quantized), while Table 2 implies 2.09 bits for all models. Similarly, Table 1 reports AQUA-KV (HIGGS) at 2.16 bits for Llama 3.2 3B, whereas Table 6 reports 2.09 bits for the same configuration. This inconsistent bitrate accounting obscures the true compression cost and makes the comparison to KIVI (2.25 bits) and KVQuant (2.33 bits) appear more favorable than it is. The tables should report per-model bitrates consistently and state explicitly how the recent-token buffer, attention sinks, and first-layer quantization are counted.
  3. [§4.2, Table 2 vs Appendix E, Tables 6 and 8] There are direct numerical contradictions for KVQuant LongBench averages between the main table and the per-model appendix tables. Table 2 lists KVQuant 2-bit LongBench for Llama 3.2 3B as 20.56 and for Llama 3.1 70B as 46.14, while Table 6 reports 18.28 for 3B and Table 8 reports 31.39 for 70B. The 3-bit KVQuant row for 3B also differs (41.40 in Table 2 vs 23.85 in Table 6). At least one of each pair is a transcription error. Because the paper's central claim is that AQUA-KV 'substantially improves over prior works', these baseline numbers must be reconciled and corrected.
minor comments (5)
  1. [Footnote 1; §4.2] The text '217 tokens (≈131K)' and 'sequence length 217 (131K)' should read '2^17 tokens (≈131K)', since 2^17 = 131072. There is also a missing space in 'GBper sequence'.
  2. [Appendix A] There are typos: 'sequenes' should be 'sequences', and 'infereece' should be 'inference'.
  3. [Table 1 caption] The caption states that the right panel reports per-task LongBench scores, but the main-text version of the table only shows the left panel; the per-task results appear only in Appendix D. This should be clarified to avoid confusing the reader.
  4. [§3.3] The phrase 'at least500× less floating point operations' should be 'at least 500× fewer floating point operations' (with a space and correct comparison).
  5. [Abstract and §4.2] The paper never defines 'relative error'. It should specify the formula, e.g., (compressed - uncompressed)/uncompressed, and state whether it applies to perplexity directly or to some transform.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: calibration and evaluation are held out, and the HIGGS self-citation is controlled by same-backbone ablations.

full rationale

The derivation chain is not circular. AQUA-KV fits linear predictors by least squares on 224 RedPajama calibration sequences (Algorithm 1), and every reported result, including WikiText-2 perplexity and LongBench averages in Tables 1-3 and the appendices, is evaluated on data outside that calibration set, so the perplexity and LongBench 'predictions' are genuine out-of-sample measurements rather than reconstructions of fitted values. The Section 3.1 heuristic that a probe's explained variance 'saves approximately this many bits' is explicitly hedged in the text ('While this is not a strict guarantee, we found that it holds well for real-world LLMs, as can be seen in Sections 4.1 and 4.2'), and the paper checks it empirically instead of deriving it from the probes themselves. The only same-group citation entering the method is the HIGGS backbone (Malinovskii et al., 2024b), and it is not load-bearing: AQUA-KV also improves over the Quanto and QuaRot backbones (Table 5), and the headline comparisons hold the backbone fixed (HIGGS alone 7.47 vs AQUA-KV over HIGGS 7.03 perplexity at about 2 bits on Llama 3.2 3B), so the reported gain is not the citation's own result by construction. No uniqueness theorem, ansatz-via-citation, or renaming step is present; the paper explicitly distinguishes itself from RVQ and evaluates layer-sharing baselines (KVSharer) directly. Two items are flagged as non-circularity concerns: (i) Section 3.1 provides no formal bound linking explained variance to final quantization error, a limitation the authors acknowledge ('not a strict guarantee'); and (ii) the abstract's 'under 1% relative error' statement conflicts with Table 2's 1.96% and 3.15% perplexity increases for Llama 3.1 8B and 70B at 2 bits if read across all tested models, which is an overstatement and correctness matter rather than a circular reduction. The score of 2 reflects only the minor, non-load-bearing self-citation of the backbone.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

AQUA-KV is an empirical compression method. Its central claim rests on a handful of hand-set hyperparameters (buffer size, sink tokens, group size, calibration size) and on the domain assumption that linear inter-layer dependencies persist across models and long contexts. There are no invented physical entities. The predictor modules are learned components, not new postulated phenomena.

free parameters (5)
  • Recent token buffer size r = 128 tokens
    Recent tokens are stored uncompressed. This hyperparameter affects both accuracy and effective bitrate and is set once for all experiments.
  • Uncompressed attention sink tokens = 4 tokens
    The first 4 tokens are kept uncompressed to protect attention sinks. The paper ablated 4 versus 64 and uses 4 as default.
  • HIGGS quantization group size = 1024
    The backbone HIGGS quantizer is configured with group size 1024 for the main results.
  • Linear probe regularization = 1e-3
    Closed-form linear regressors in Section 3.1 use regularization rate 1e-3.
  • Calibration set size = 256 sequences of 8192 tokens
    224 sequences train predictors and 32 are used for hyperparameter selection. This choice is a hand-set experimental knob.
assumptions (5)
  • domain assumption Adjacent transformer layers have approximately linear KV dependencies due to residual connections
    Used to justify linear predictors. Cited to Razzhigaev et al. and measured on Llama 3.2 3B in Section 3.1, but assumed to transfer to other models and contexts.
  • domain assumption Predictors trained on reconstructed previous-layer inputs generalize to inference-time reconstructed inputs
    Algorithm 1 trains sequentially on reconstructions, but long-context inference can accumulate quantization error across layers. The paper tests LongBench at full length, so the assumption is empirically checked but not formally guaranteed.
  • ad hoc to paper Linear least-squares regression is a sufficient predictor family
    The method deliberately uses linear predictors because MLPs give only marginal gains. This is a design choice that restricts the method and is not derived from first principles.
  • standard math Scale-independent quantization lets explained variance be interpreted as approximate bit savings
    The paper uses this to connect probe performance to compression headroom. The connection is intuitive and empirically supported, not a proven bound.
  • domain assumption Attention sinks are important and can be handled by keeping the first tokens uncompressed
    The method relies on this known LLM behavior, following prior KV quantization work such as KVQuant.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models." pith.science (2026). https://pith.science/paper/ABSH3VJS

@misc{pith2026250119392,
  author       = {Pith},
  title        = {Pith review of: Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ABSH3VJS}},
  note         = {Machine review of arXiv:2501.19392}
}
abstract

Efficient real-world deployments of large language models (LLMs) rely on Key-Value (KV) caching for processing and generating long outputs, reducing the need for repetitive computation. For large contexts, Key-Value caches can take up tens of gigabytes of device memory, as they store vector representations for each token and layer. Recent work has shown that the cached vectors can be compressed through quantization, pruning or merging, but these techniques often compromise quality towards higher compression rates. In this work, we aim to improve Key & Value compression by exploiting two observations: 1) the inherent dependencies between keys and values across different layers, and 2) high-compression mechanisms for internal network states. We propose AQUA-KV, an adaptive quantization for Key-Value caches that relies on compact adapters to exploit existing dependencies between Keys and Values, and aims to "optimally" compress the information that cannot be predicted. AQUA-KV significantly improves compression rates, while maintaining high accuracy on state-of-the-art LLM families. On Llama 3.2 LLMs, we achieve near-lossless inference at 2-2.5 bits per value with under $1\%$ relative error in perplexity and LongBench scores. AQUA-KV is one-shot, simple, and efficient: it can be calibrated on a single GPU within 1-6 hours, even for 70B models.

Figures

Figures reproduced from arXiv: 2501.19392 by the authors.

Figure 1
Figure 1. Comparison of AQUA-KV to alternative Key-Value Cache compression methods for Llama 3.x models in terms of average LongBench score on 14 english tasks (see Section 4). caching is typically employed, where keys and values are saved for later use. Unfortunately, KV-caching comes with its own pitfalls: KV caches are large, especially when han￾dling long sequences (Bai et al., 2023; Xiao et al., 2023). Thus, the memory f… view at source ↗
Figure 3
Figure 3. An intuitive scheme of the AQUA-KV inference. Only the quantized residuals are saved for each block. methods that are scale-independent 3 (Horvath et al. ´ , 2023) this would mean that the resulting quantization will also have roughly 10 times smaller error. Note that not all predictors will be practical for Key-Value compression. For instance, if a predictor uses a subsequent block or future token KV vectors as inp… view at source ↗
Figure 4
Figure 4. Additional Mean Explained Variance Ratios by linear probes from previous blocks (L), tokens (T) and role on Llama-3.2-3B. 0 5 10 15 20 25 Transformer block index (Llama-3.2-3B) 0.00 0.25 0.50 0.75 KL 1 -> KL VL 1 -> VL KL -> VL Explained Variance Ratio from Chosen Dependencies Across Transformer Blocks [PITH_FULL_IMAGE:figures/full_fig_p014_4.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Explained Variance Ratios per Transformer Block for chosen sets of linear probes on Llama-3.2-3B. C. On LongBench Evaluation on non-Instruct models As we discuss in Section 4, we only evaluate Instruct model variants on LongBench tasks. Non-Instruct models treats the p…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

59 extracted references · 9 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints

    Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., and Sanghai, S. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245, 2023. URL https://arxiv.org/abs/2305.13245

    arXiv 2023

  3. [3]

    and Bengio, Y

    Alain, G. and Bengio, Y. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644, 2016

    arXiv 2016

  4. [4]

    L., Li, B., Jaggi, M., Alistarh, D., Hoefler, T., and Hensman, J

    Ashkboos, S., Mohtashami, A., Croci, M. L., Li, B., Jaggi, M., Alistarh, D., Hoefler, T., and Hensman, J. Quarot: Outlier-free 4-bit inference in rotated llms. arXiv preprint arXiv:2404.00456, 2024

    arXiv 2024

  5. [5]

    Longbench: A bilingual, multitask benchmark for long context understanding

    Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., et al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508, 2023

    arXiv 2023

  6. [6]

    M., Gebru, T., McMillan-Major, A., and Shmitchell, S

    Bender, E. M., Gebru, T., McMillan-Major, A., and Shmitchell, S. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT '21, pp.\ 610–623, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450383097. doi:10.1145/3442188.344592...

    work page arXiv 2021

  7. [7]

    Palu: Compressing kv-cache with low-rank projection

    Chang, C.-C., Lin, W.-C., Lin, C.-Y., Chen, C.-Y., Hu, Y.-F., Wang, P.-S., Huang, N.-C., Ceze, L., and Wu, K.-C. Palu: Compressing kv-cache with low-rank projection. arXiv preprint arXiv:2407.21118, 2024

    arXiv 2024

  8. [8]

    Prefixquant: Eliminating outliers by prefixed tokens for large language models quantization, 2025

    Chen, M., Liu, Y., Wang, J., Bin, Y., Shao, W., and Luo, P. Prefixquant: Eliminating outliers by prefixed tokens for large language models quantization, 2025. URL https://arxiv.org/abs/2410.05265

    arXiv 2025

Show all 59 references
  1. [9]

    Risk taxonomy, mitigation, and assessment benchmarks of large language model systems

    Cui, T., Wang, Y., Fu, C., Xiao, Y., Li, S., Deng, X., Liu, Y., Zhang, Q., Qiu, Z., Li, P., Tan, Z., Xiong, J., Kong, X., Wen, Z., Xu, K., and Li, Q. Risk taxonomy, mitigation, and assessment benchmarks of large language model systems. ArXiv, abs/2401.05778, 2024. URL https://...

    2024 arXiv

  2. [10]

    Qaq: Quality-adaptive quantization for llm kv cache

    Dong, S., Cheng, W., Qin, J., and Wang, W. Qaq: Quality-adaptive quantization for llm kv cache. arXiv preprint arXiv:2403.04643, 2024

    2024 arXiv

  3. [11]

    Skvq: Sliding-window key and value cache quantization for large language models

    Duanmu, H., Yuan, Z., Li, X., Duan, J., Zhang, X., and Lin, D. Skvq: Sliding-window key and value cache quantization for large language models. arXiv preprint arXiv:2405.06219, 2024

    2024 arXiv

  4. [12]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  5. [13]

    Towards measuring the representation of subjective global opinions in language models

    Durmus, E., Nyugen, K., Liao, T., Schiefer, N., Askell, A., Bakhtin, A., Chen, C., Hatfield-Dodds, Z., Hernandez, D., Joseph, N., Lovitt, L., McCandlish, S., Sikder, O., Tamkin, A., Thamkul, J., Kaplan, J., Clark, J., and Ganguli, D. Towards measuring the representation of sub...

    2023 arXiv

  6. [14]

    Extreme compression of large language models via additive quantization

    Egiazarian, V., Panferov, A., Kuznedelev, D., Frantar, E., Babenko, A., and Alistarh, D. Extreme compression of large language models via additive quantization. arXiv preprint arXiv:2401.06118, 2024

    2024 arXiv

  7. [15]

    Gptq: Accurate post-training quantization for generative pre-trained transformers

    Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323, 2022

    2022 arXiv

  8. [16]

    Model tells you what to discard: Adaptive kv cache compression for llms

    Ge, S., Zhang, Y., Liu, L., Zhang, M., Han, J., and Gao, J. Model tells you what to discard: Adaptive kv cache compression for llms. arXiv preprint arXiv:2310.01801, 2023

    2023 arXiv

  9. [17]

    Gray, R. M. and Neuhoff, D. L. Vector quantization. IEEE Transactions on Information Theory, 44 0 (6): 0 2325--2383, 1998

    1998

  10. [18]

    Fast matrix multiplications for lookup table-quantized llms

    Guo, H., Brandon, W., Cholakov, R., Ragan-Kelley, J., Xing, E., and Kim, Y. Fast matrix multiplications for lookup table-quantized llms. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp.\ 12419--12433, 2024

    2024

  11. [19]

    Zipcache: Accurate and efficient kv cache quantization with salient token identification

    He, Y., Zhang, L., Wu, W., Liu, J., Zhou, H., and Zhuang, B. Zipcache: Accurate and efficient kv cache quantization with salient token identification. arXiv preprint arXiv:2405.14256, 2024

    2024 arXiv

  12. [20]

    W., Shao, Y

    Hooper, C., Kim, S., Mohammadzadeh, H., Mahoney, M. W., Shao, Y. S., Keutzer, K., and Gholami, A. Kvquant: Towards 10 million context length llm inference with kv cache quantization. arXiv preprint arXiv:2401.18079, 2024

    2024 arXiv

  13. [21]

    Stochastic distributed learning with gradient quantization and double-variance reduction

    Horv \'a th, S., Kovalev, D., Mishchenko, K., Richt \'a rik, P., and Stich, S. Stochastic distributed learning with gradient quantization and double-variance reduction. Optimization Methods and Software, 38 0 (1): 0 91--106, 2023

    2023

  14. [22]

    Optimum-quanto: A pytorch quantization backend for optimum

    HuggingFace. Optimum-quanto: A pytorch quantization backend for optimum. https://github.com/huggingface/optimum-quanto, 2024. Accessed: 2025-01-28

    2024

  15. [23]

    Gear: An efficient kv cache compression recipe for near-lossless generative inference of llm

    Kang, H., Zhang, Q., Kundu, S., Jeong, G., Liu, Z., Krishna, T., and Zhao, T. Gear: An efficient kv cache compression recipe for near-lossless generative inference of llm. arXiv preprint arXiv:2403.05527, 2024

    2024 arXiv

  16. [24]

    H., Gonzalez, J., Zhang, H., and Stoica, I

    Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pp.\ 611--626, 2023

    2023

  17. [25]

    A survey on large language model acceleration based on kv cache management

    Li, H., Li, Y., Tian, A., Tang, T., Xu, Z., Chen, X., Hu, N., Dong, W., Li, Q., and Chen, L. A survey on large language model acceleration based on kv cache management. arXiv preprint arXiv:2412.19442, 2024 a

    2024 arXiv

  18. [26]

    Snapkv: Llm knows what you are looking for before generation

    Li, Y., Huang, Y., Yang, B., Venkitesh, B., Locatelli, A., Ye, H., Cai, T., Lewis, P., and Chen, D. Snapkv: Llm knows what you are looking for before generation. arXiv preprint arXiv:2404.14469, 2024 b

    2024 arXiv

  19. [27]

    H., Li, D., Gao, J., Yang, Y., et al

    Li, Y., Jiang, H., Wu, Q., Luo, X., Ahn, S., Zhang, C., Abdi, A. H., Li, D., Gao, J., Yang, Y., et al. Scbench: A kv cache-centric analysis of long-context methods. arXiv preprint arXiv:2412.10319, 2024 c

    2024 arXiv

  20. [28]

    Awq: Activation-aware weight quantization for llm compression and acceleration

    Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., and Han, S. Awq: Activation-aware weight quantization for llm compression and acceleration. arXiv preprint arXiv:2306.00978, 2023

    2023 arXiv

  21. [29]

    Minicache: Kv cache compression in depth dimension for large language models

    Liu, A., Liu, J., Pan, Z., He, Y., Haffari, G., and Zhuang, B. Minicache: Kv cache compression in depth dimension for large language models. arXiv preprint arXiv:2405.14366, 2024 a

    2024 arXiv

  22. [30]

    Intactkv: Improving large language model quantization by keeping pivot tokens intact

    Liu, R., Bai, H., Lin, H., Li, Y., Gao, H., Xu, Z., Hou, L., Yao, J., and Yuan, C. Intactkv: Improving large language model quantization by keeping pivot tokens intact. arXiv preprint arXiv:2403.01241, 2024 b

    2024 arXiv

  23. [31]

    Kivi: A tuning-free asymmetric 2bit quantization for kv cache

    Liu, Z., Yuan, J., Jin, H., Zhong, S., Xu, Z., Braverman, V., Chen, B., and Hu, X. Kivi: A tuning-free asymmetric 2bit quantization for kv cache. arXiv preprint arXiv:2402.02750, 2024 c

    2024 arXiv

  24. [32]

    Pv-tuning: Beyond straight-through estimation for extreme llm compression

    Malinovskii, V., Mazur, D., Ilin, I., Kuznedelev, D., Burlachenko, K., Yi, K., Alistarh, D., and Richtarik, P. Pv-tuning: Beyond straight-through estimation for extreme llm compression. arXiv preprint arXiv:2405.14852, 2024 a

    2024 arXiv

  25. [33]

    Pushing the limits of large language model quantization via the linearity theorem

    Malinovskii, V., Panferov, A., Ilin, I., Guo, H., Richt \'a rik, P., and Alistarh, D. Pushing the limits of large language model quantization via the linearity theorem. arXiv preprint arXiv:2411.17525, 2024 b

    2024 arXiv

  26. [34]

    Pointer sentinel mixture models

    Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016

    2016 arXiv

  27. [35]

    PyTorch : An imperative style, high-performance deep learning library

    Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. PyTorch : An imper...

    2019

  28. [36]

    Your transformer is secretly linear

    Razzhigaev, A., Mikhalchuk, M., Goncharova, E., Gerasimenko, N., Oseledets, I., Dimitrov, D., and Kuznetsov, A. Your transformer is secretly linear. arXiv preprint arXiv:2405.12250, 2024

    2024 arXiv

  29. [37]

    Reinsel, G. C. and Velu, R. D. Multivariate Reduced-Rank Regression. Springer, New York, 1998. ISBN 978-1-4757-2853-8. doi:10.1007/978-1-4757-2853-8. URL https://link.springer.com/book/10.1007/978-1-4757-2853-8

    1998 doi

  30. [38]

    Societal biases in language generation: Progress and challenges

    Sheng, E., Chang, K.-W., Natarajan, P., and Peng, N. Societal biases in language generation: Progress and challenges. ArXiv, abs/2105.04054, 2021. URL https://api.semanticscholar.org/CorpusID:234337004

    2021 arXiv

  31. [39]

    Roformer: Enhanced transformer with rotary position embedding

    Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09864, 2021

    2021 arXiv

  32. [40]

    Qwen2.5: A party of foundation models, September 2024

    Team, Q. Qwen2.5: A party of foundation models, September 2024. URL https://qwenlm.github.io/blog/qwen2.5/

    2024

  33. [41]

    Llama: Open and efficient foundation language models

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi \`e re, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  34. [42]

    Quip\#: Even better llm quantization with hadamard incoherence and lattice codebooks

    Tseng, A., Chee, J., Sun, Q., Kuleshov, V., and De Sa, C. Quip\#: Even better llm quantization with hadamard incoherence and lattice codebooks. arXiv preprint arXiv:2402.04396, 2024 a

    2024 arXiv

  35. [43]

    Tseng, A., Chee, J., Sun, Q., Kuleshov, V., and Sa, C. D. Quip\#: Even better llm quantization with hadamard incoherence and lattice codebooks, 2024 b

    2024

  36. [44]

    Tseng, A., Sun, Q., Hou, D., and Sa, C. D. QTIP : Quantization with trellises and incoherence processing. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024 c . URL https://openreview.net/forum?id=7sdkLVuYCU

    2024

  37. [45]

    Gptvq: The blessing of dimensionality for llm quantization

    van Baalen, M., Kuzmin, A., Nagel, M., Couperus, P., Bastoul, C., Mahurin, E., Blankevoort, T., and Whatmough, P. Gptvq: The blessing of dimensionality for llm quantization. arXiv preprint arXiv:2402.15319, 2024

    2024 arXiv

  38. [46]

    Attention is all you need

    Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

  39. [47]

    Weber, M., Fu, D. Y., Anthony, Q., Oren, Y., Adams, S., Alexandrov, A., Lyu, X., Nguyen, H., Yao, X., Adams, V., Athiwaratkun, B., Chalamala, R., Chen, K., Ryabinin, M., Dao, T., Liang, P., Ré, C., Rish, I., and Zhang, C. Redpajama: an open dataset for training large language ...

    2024

  40. [48]

    Weidinger, L., Mellor, J. F. J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., Kenton, Z., Brown, S. M., Hawkins, W. T., Stepleton, T., Biles, C., Birhane, A., Haas, J., Rimell, L., Hendricks, L. A., Isaac, W. S., Legassick,...

    2021 arXiv

  41. [49]

    Weidinger, L., Uesato, J., Rauh, M., Griffin, C., Huang, P.-S., Mellor, J. F. J., Glaese, A., Cheng, M., Balle, B., Kasirzadeh, A., Biles, C., Brown, S. M., Kenton, Z., Hawkins, W. T., Stepleton, T., Birhane, A., Hendricks, L. A., Rimell, L., Isaac, W. S., Haas, J., Legassick,...

    2022

  42. [50]

    Huggingface's transformers: State-of-the-art natural language processing

    Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019

    1910 arXiv

  43. [51]

    Efficient streaming language models with attention sinks

    Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453, 2023

    2023 arXiv

  44. [52]

    Qwen2 technical report

    Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., Dong, G., Wei, H., Lin, H., Tang, J., Wang, J., Yang, J., Tu, J., Zhang, J., Ma, J., Xu, J., Zhou, J., Bai, J., He, J., Lin, J., Dang, K., Lu, K., Chen, K., Yang, K., Li, M., Xue, M.,...

    2024 arXiv

  45. [53]

    Y., Kim, B., Bae, J., Kwon, B., Park, G., Yang, E., Kwon, S

    Yang, J. Y., Kim, B., Bae, J., Kwon, B., Park, G., Yang, E., Kwon, S. J., and Lee, D. No token left behind: Reliable kv cache compression via importance-aware mixed precision quantization. arXiv preprint arXiv:2402.18096, 2024 b

    2024 arXiv

  46. [54]

    Kvsharer: Efficient inference via layer-wise dissimilar kv cache sharing

    Yang, Y., Cao, Z., Chen, Q., Qin, L., Yang, D., Zhao, H., and Chen, Z. Kvsharer: Efficient inference via layer-wise dissimilar kv cache sharing. arXiv preprint arXiv:2410.18517, 2024 c

    2024 arXiv

  47. [55]

    Wkvquant: Quantizing weight and key/value cache for large language models gains more

    Yue, Y., Yuan, Z., Duanmu, H., Zhou, S., Wu, J., and Nie, L. Wkvquant: Quantizing weight and key/value cache for large language models gains more. arXiv preprint arXiv:2402.12065, 2024

    2024 arXiv

  48. [56]

    Qjl: 1-bit quantized jl transform for kv cache quantization with zero overhead

    Zandieh, A., Daliri, M., and Han, I. Qjl: 1-bit quantized jl transform for kv cache quantization with zero overhead. arXiv preprint arXiv:2406.03482, 2024

    2024 arXiv

  49. [57]

    and Shen, H

    Zhang, Z. and Shen, H. Zero-delay qkv compression for mitigating kv cache and network bottlenecks in llm inference. arXiv preprint arXiv:2408.04107, 2024

    2024 arXiv

  50. [58]

    H2o: Heavy-hitter oracle for efficient generative inference of large language models

    Zhang, Z., Sheng, Y., Zhou, T., Chen, T., Zheng, L., Cai, R., Song, Z., Tian, Y., R \'e , C., Barrett, C., et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36: 0 34661--34710, 2023

    2023

  51. [59]

    Y., Huang, Y., Chen, C., and Xing, Z

    Zhuo, T. Y., Huang, Y., Chen, C., and Xing, Z. Red teaming chatgpt via jailbreaking: Bias, robustness, reliability and toxicity, 2023. URL https://arxiv.org/abs/2301.12867

    2023 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.