Pith. sign in

REVIEW 3 major objections 4 minor 48 references

XQuant: Breaking the Memory Wall for LLM Inference with KV Cache Rematerialization

T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read XQuant caches layer inputs instead of keys and values, cutting LLM inference memory up to 12.5x.

desk verdict XQuant's accuracy numbers are real, but its own roofline shows the 'breaking the memory wall' speedup claim does not hold for long contexts on current hardware. read the letter →

arxiv 2508.10395 v1 pith:PFLLN7U2 submitted 2025-08-14 cs.LG

Aditya Tomar , Coleman Hooper , Minjae Lee , Haocheng Xi , Rishabh Tiwari , Wonjun Kang , Luca Manolache , Michael W. Mahoney
show 2 more authors
Kurt Keutzer Amir Gholami
This is my paper
classification cs.LG
keywords KVcachequantizationLLMinferenceactivationrematerializationcross-layercompressionmemorywalllow-bitgroupedqueryattentionrooflineanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

XQuant aims to break the LLM inference memory bottleneck by replacing the key-value (KV) cache with a much smaller cache of quantized layer input activations X, and recomputing keys and values on the fly when attention needs them. Since X is one tensor per layer while the KV cache holds two, this immediately halves the cache footprint, and the paper shows X survives 2–4 bit uniform quantization more gracefully than keys or values do. The variant XQuant-CL goes further by quantizing the small differences between successive layers' inputs, exploiting the residual-stream structure of Transformers, and claims up to 12.5x memory savings relative to the FP16 baseline with only 0.1 perplexity degradation, outperforming state-of-the-art KV cache quantization on perplexity. The paper also extends the scheme to grouped-query attention models through an offline SVD down-projection of the K/V projection matrices. The motivating claim is that as compute capability outpaces memory bandwidth, paying extra FLOPs to shrink memory traffic is the right trade.

What carries the argument

The load-bearing mechanism is the quantized X cache with on-the-fly KV rematerialization: store $e$-bit $X$ per layer, then compute $K=XW_k$ and $V=XW_v$ at attention time, using the roofline ridge point (peak FLOPs / peak bandwidth) to argue the added FLOPs are hidden when the kernel is memory-bound. For cross-layer compression, the accumulator identity $\hat{X}_i=X_0+\sum_{j=1}^{i}\Delta\hat{X}_j$ with $\Delta\hat{X}_j=Q(X_j-\hat{X}_{j-1})$ lets each layer load only the running accumulator and one quantized delta. For GQA, the SVD factorizations $W_k=U_k\Sigma_k B_k^T$, $W_v=U_v\Sigma_v B_v^T$ supply a $d/g$-dimensional latent space $XU_k$ whose first channel concentrates all outliers, ena

What would settle it

On an H100, run Llama-2-7B autoregressive decoding with 2-bit XQuant at a 32K-token sequence and measure tokens/second and perplexity against the FP16 baseline. If end-to-end throughput is not higher than baseline (or is lower), or if perplexity degrades by more than ~0.1, the central memory-for-compute tradeoff is not realized on current hardware.

Watch

Extended reading notes

Core claim

The central claim: cache the quantized post-norm layer input $X$ instead of keys and values, and regenerate $K=XW_k$, $V=XW_v$ on the fly. Caching one tensor per layer versus two halves memory, and $X$ quantizes to 2–4 bits far more cleanly than $K$ or $V$. At matched memory, XQuant reports <0.1 perplexity degradation with $7.7\times$ savings vs FP16 on Llama-2-7B/13B, beating KVQuant and KIVI. XQuant-CL further caches quantized inter-layer deltas $\Delta\hat{X}_i=Q(X_i-\hat{X}_{i-1})$ via the accumulator $\hat{X}_i=X_0+\sum_j\Delta\hat{X}_j$, giving $10\times$ savings at 3-bit (0.01 degradation) and $12.5\times$ at 2-bit (0.1). For GQA, offline SVD projects $X$ to dimension $d/g$, matching

Load-bearing premise

The approach stands or falls on the premise that the extra computation needed to regenerate keys and values can be hidden behind the memory traffic saved—a crossover the paper's own roofline analysis places at roughly 2.3K tokens of sequence length for a 7B multi-head model on an H100 at 2-bit, so for short sequences or memory-rich hardware the trade can go the other way.

Editorial extensions

If this is right

  • At the reported compression, the KV cache for a 32K-token sequence fits in a few hundred megabytes at 2-bit, enabling longer context and larger batch on existing GPUs.
  • Simple uniform quantization of X is claimed to beat non-uniform, calibration-based KV quantization, removing the need for calibration datasets and outlier-aware sparse storage.
  • The cross-layer delta view suggests the residual stream of Transformers is highly compressible between adjacent layers, which may inform other memory-reduction schemes.
  • For GQA models, the SVD projection makes XQuant directly applicable to current Llama/Mistral/Qwen families, with 6.7–7.1x memory savings at near-FP16 accuracy.
  • As compute-to-bandwidth ratios grow, the rematerialization overhead shrinks relative to memory savings, making the method progressively more attractive on future hardware.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • On today's H100-class parts, the paper's own Equation (3) places the crossover near 2.3K tokens for a 7B MHA model at 2-bit; below that length the extra FLOPs may exceed the memory-time saved, so the end-to-end latency benefit is hardware- and length-dependent.
  • The appendix's outlier analysis could be pushed further: if the first row of $B_v^T$ reliably predicts Key outlier channels, KV quantization could be configured purely offline from weights, with no calibration corpus at all.
  • The X-cache idea is orthogonal to weight quantization and speculative decoding; combining them could compound memory savings beyond what the paper evaluates.
  • The same accumulator-delta scheme could be applied to any residual network for inference-time activation compression, not just Transformer LLMs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. XQuant proposes to reduce the KV cache memory footprint of LLM inference by caching a quantized version of the per-layer input activation X (or, in XQuant-CL, quantized cross-layer deltas of X) and rematerializing the Keys and Values on the fly during decoding. For MHA models this immediately gives a 2x memory reduction relative to quantizing K and V; for GQA models the authors add an offline SVD-based down-projection to keep the cached representation in the KV-projection latent space. The paper reports perplexity on WikiText-2/C4 for Llama-2-7B/13B, Llama-3.1-8B, and Mistral-7B, plus LongBench and GSM8K evaluations, and claims up to 12.5x memory savings relative to FP16 KV cache with about 0.1 perplexity degradation. A system-level roofline analysis in Section 3.4 argues that the extra rematerialization compute can be hidden under memory operations due to the growing compute/memory gap.

Significance. If the memory-savings and accuracy results are taken at face value, the paper makes a useful contribution: it shows that a simple uniform-quantization scheme applied to layer inputs can outperform substantially more complex KV-cache quantization methods (KVQuant, KIVI) at ultra-low bit widths, and the cross-layer delta compression idea is interesting. The SVD-based GQA extension and the off-line analysis of Key outlier channels are also plausible and potentially useful. The core memory-capacity arithmetic is sound, and the perplexity tables are extensive. However, the paper's headline framing—'breaking the memory wall' and 'speed up inference'—is not supported by any end-to-end latency or throughput measurement; the paper's own roofline model, as written, shows that for the flagship MHA configuration the method becomes compute-bound for sequence lengths above roughly 2.3K, which undermines the central performance claim.

major comments (3)
  1. [Section 3.4, Eq. (3)] The roofline analysis used to justify the central claim actually shows the opposite for the headline MHA configuration. With P=378 (H100), d=4K, e=2, the authors solve Eq. (3) and get l≈2.3K as the maximum sequence length before rematerialization compute becomes the bottleneck. Thus for any sequence length above 2.3K — exactly the long-context regime where KV-cache memory matters — XQuant is compute-bound, not memory-bound. This contradicts the abstract's claim of 'eliminate the memory bottleneck' and the conclusion's claim of 'speeding up inference.' No kernel-level or end-to-end latency/throughput measurement is reported anywhere in the paper. To make the central claim load-bearing, the authors should either provide real measurements on a target GPU (prefill and decode, with the reported bit widths and sequence lengths) or substantially qualify the claim to memory-capacity savings rath
  2. [Section 3.4, Eqs. (3)–(4) and surrounding text] The analysis assumes that KV rematerialization can be perfectly overlapped with loading model weights ('we assume that we can overlap the KV cache recomputation with loading the model weights'). This assumption is exactly the question at issue: whether the extra compute can be hidden under the memory stream. The manuscript provides no profiling, no kernel implementation, and no demonstration that the memory system remains the bottleneck throughout the decode step. Moreover, the roofline uses theoretical peak FLOPs and peak bandwidth; real GEMM kernels for the relevant shapes (l×d by d×d, with small l per token) typically achieve far below peak FLOPs, which would make the compute-bound regime begin at even shorter sequence lengths. The analysis needs to be either validated with measurements or replaced by a conservative model that does not assume perfect overlap.
  3. [Section 3.2 / Section 3.4 (XQuant-CL accumulator)] For XQuant-CL the text states that computing Xhat_i 'only requires loading the accumulator and a single delta,' and Section 3.4 counts an additional eb/8·l·d memory operations per layer for the accumulator. The accumulator is an l×d tensor kept in higher precision (eb=4 bits in the experiments). At any non-trivial sequence length (e.g., 4K tokens, d=4K), this is a multi-megabyte tensor that cannot reside in on-chip SRAM, so it must be stored in DRAM. The memory-capacity figures in Table 4 appear to count only the cached deltas and the high-precision base layer, not the accumulator working set or its traffic. If the accumulator must be stored in DRAM, the '12.5x memory savings' claims are for the persistent cache only, not for peak memory footprint, and the claim that XQuant-CL is suitable for memory-constrained scenarios is not established. The paper should clarify how the accumulator is
minor comments (4)
  1. [Figure 2] The label '2X Memory Savings!' in the figure applies to MHA models only; for GQA the paper later requires SVD down-projection and does not get 2x over KV quantization at the same bit width. The figure caption should note this limitation.
  2. [Section 3.4] The text interchangeably uses 'memory operations' and 'bytes' (e.g., '2·12·d² additional memory operations' for weight loading). Clarify that these are byte counts; otherwise Eq. (3) is dimensionally confusing.
  3. [Section 3.3.2, displayed equation] The lossless-reconstruction equation is correct, but the notation Q(ΔXiUkv)·UT_kv is easy to misread: the up-projection by UT_kv is applied after quantization. Add a sentence clarifying the order of operations.
  4. [Abstract and Conclusion] The Limitations section correctly states that rematerialization 'may increase latency on particular hardware platforms.' This caveat should be reflected in the abstract and conclusion, which currently state without qualification that the method 'eliminates the memory bottleneck' and 'speeds up inference.'

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: XQuant's memory and accuracy results are measured against external baselines; the derivation is self-contained, and the self-citations are motivational rather than load-bearing.

full rationale

The central derivation is not circular. XQuant stores quantized layer inputs X instead of K and V; the 2x memory reduction is definitional (one tensor instead of two), but it is a design property, and all accuracy claims are empirical, measured against an FP16 baseline and the external KIVI*/KVQuant baselines. No parameter is fitted to the target perplexity: quantization is standard uniform quantization with fixed group size, and the per-channel/per-token choices are applied consistently to baselines. The XQuant-CL cross-layer delta compression is justified by an empirical observation (Figure 3) and evaluated on held-out perplexity; it is not a tautology, since residual connections do not by themselves guarantee small deltas. The GQA outlier-channel 'prediction' is a genuine external check: Key outlier channels predicted from the offline SVD weight matrix B^T_v are compared against activation-derived ground truth (Table B.2), which is the opposite of circular. The system-level roofline (Eq. 3) is self-contained, using H100 specifications and model dimensions; the lack of end-to-end latency measurements and the compute-bound crossover for long contexts is a correctness/evidence concern, explicitly acknowledged in the Limitations section ('may increase latency on particular hardware platforms'), not a circular derivation. The authors' self-citation [11] for the compute-vs-memory scaling trend is motivational only; the central memory-savings and accuracy results do not reduce to that citation.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No new physical or model entities are introduced; the latent XUk and delta caches are representations of existing activations.

free parameters (4)
  • initial_layer_precision_count = 3 layers kept in 4-bit (base layer = layer 3)
    Post-hoc selection in Table 4 to mitigate perplexity degradation; directly affects the reported memory/accuracy tradeoff.
  • quantization_group_size = 128
    Fixed hyperparameter for all uniform quantization, carried over from prior KV quantization work; affects quantization error.
  • accumulator_precision_bits = 4
    Higher-precision running sum in XQuant-CL, chosen by hand; contributes to memory operations but not cache capacity.
  • outlier_channel_fp16 = 1 channel of XUk kept in FP16 (appendix only)
    Optional enhancement for GQA 2-bit XQuant; improves C4 perplexity by about 0.2, showing dependence on this hand-chosen detail.
assumptions (4)
  • domain assumption LLM inference is memory-bandwidth bound, so compute can be traded for memory bandwidth
    Section 2.1; underpins the whole rematerialization motivation and depends on hardware ridge point and workload regime.
  • domain assumption Rematerialization compute can be perfectly overlapped with loading model weights
    Used in the roofline balance of Section 3.4 (Eq 3 and 4); no measured overlap is demonstrated.
  • domain assumption Consecutive layer inputs X are highly similar (residual stream refinement)
    Basis for XQuant-CL delta compression; supported by Figure 3 and [17], but is an empirical property assumed to hold across models.
  • domain assumption Quantization error in the latent/delta space remains small and does not cascade across layers
    The sequential delta accumulator propagates approximations through all layers; the paper checks this empirically via perplexity but gives no error bound.

how reviews work

0 comments
Cite this review

Pith. "Pith review of XQuant: Breaking the Memory Wall for LLM Inference with KV Cache Rematerialization." pith.science (2026). https://pith.science/paper/PFLLN7U2

@misc{pith2026250810395,
  author       = {Pith},
  title        = {Pith review of: XQuant: Breaking the Memory Wall for LLM Inference with KV Cache Rematerialization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PFLLN7U2}},
  note         = {Machine review of arXiv:2508.10395}
}
abstract

Although LLM inference has emerged as a critical workload for many downstream applications, efficiently inferring LLMs is challenging due to the substantial memory footprint and bandwidth requirements. In parallel, compute capabilities have steadily outpaced both memory capacity and bandwidth over the last few decades, a trend that remains evident in modern GPU hardware and exacerbates the challenge of LLM inference. As such, new algorithms are emerging that trade increased computation for reduced memory operations. To that end, we present XQuant, which takes advantage of this trend, enabling an order-of-magnitude reduction in memory consumption through low-bit quantization with substantial accuracy benefits relative to state-of-the-art KV cache quantization methods. We accomplish this by quantizing and caching the layer input activations X, instead of using standard KV caching, and then rematerializing the Keys and Values on-the-fly during inference. This results in an immediate 2$\times$ memory savings compared to KV caching. By applying XQuant, we achieve up to $\sim 7.7\times$ memory savings with $<0.1$ perplexity degradation compared to the FP16 baseline. Furthermore, our approach leverages the fact that X values are similar across layers. Building on this observation, we introduce XQuant-CL, which exploits the cross-layer similarity in the X embeddings for extreme compression. Across different models, XQuant-CL attains up to 10$\times$ memory savings relative to the FP16 baseline with only 0.01 perplexity degradation, and 12.5$\times$ memory savings with only $0.1$ perplexity degradation. XQuant exploits the rapidly increasing compute capabilities of hardware platforms to eliminate the memory bottleneck, while surpassing state-of-the-art KV cache quantization methods and achieving near-FP16 accuracy across a wide range of models.

Figures

Figures reproduced from arXiv: 2508.10395 by the authors.

Figure 1
Figure 1. Perplexity degradation (lower is better) versus memory compression factor (higher is better) evaluated using Llama-2-7B on WikiText-2 for state-of-the-art KV cache quantization methods and for our XQUANT, across {4,3,2}-bit widths. The top right edge of the plot represents the optimal configuration that attains the most memory compression and the least perplexity degradation. Memory compression factor and perplexity… view at source ↗
Figure 2
Figure 2. A visualization of how XQUANT reduces the memory footprint by caching the input embedding (X) instead of the KV cache. We use the cached input to rematerialize the Keys and Values in order to compute attention. This increases the amount of computation required when computing attention. However, since LLM inference is typically memory bandwidth-bound, we can accelerate inference by reducing memory operations, even at… view at source ↗
Figure 3
Figure 3. Comparison of the post-norm input embeddings X, pre-RoPE Keys, and Values for successive layers in the Llama-3.1-8B model. The distributions were collected using a test sample with 2K sequence length from Wikitext-2. Although the Keys and Values exhibit distinct differences across successive layers, the X embeddings bear remarkable similarity. We exploit this similarity using cross-layer compression in XQUANT-CL. al… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Illustration of XQUANT-CL algorithm during decoding. Besides Layer 0, the input to all other layers is a cross layer approximation, computed using the deltas of all previous layers and the input of Layer 0. The input of Layer 0 is summed with each layer’s delta so it c…
Figure 5
Figure 5. Figure 5: A diagram outlining how we apply XQUANT for GQA-based models. GQA down-projects the input embedding (X) to a smaller d/g dimension when computing the Keys and Values. Hence, if we naively quantize the input X rather than the KV cache, this will potentially have greater…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

48 extracted references · 21 canonical work pages

  1. [1]

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4895–4901, 2023

    work page 2023

  2. [2]

    Longbench: A bilingual, multitask benchmark for long context understanding

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. In Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3119–3137, 2024

    work page 2024

  3. [3]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  4. [4]

    xkv: Cross-layer svd for kv-cache compression

    Chi-Chih Chang, Chien-Yu Lin, Yash Akhauri, Wei-Cheng Lin, Kai-Chiang Wu, Luis Ceze, and Mohamed S Abdelfattah. xkv: Cross-layer svd for kv-cache compression. arXiv preprint arXiv:2503.18893, 2025

    arXiv 2025

  5. [5]

    Palu: Compressing kv-cache with low-rank projection

    Chi-Chih Chang, Wei-Cheng Lin, Chien-Yu Lin, Chong-Yan Chen, Yu-Fang Hu, Pei-Shuo Wang, Ning-Chi Huang, Luis Ceze, and Kai-Chiang Wu. Palu: Compressing kv-cache with low-rank projection. In Proceedings of International Conference on Learning Representations (ICLR), April 2025. 14

    work page 2025

  6. [6]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1–113, 2023

    2023

  7. [7]

    Training verifiers to solve math word problems, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021

    2021

  8. [8]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

Show all 48 references
  1. [9]

    The language model evaluation harness, 07 2024

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024

  2. [10]

    Fast state restoration in llm serving with hcache

    Shiwei Gao, Youmin Chen, and Jiwu Shu. Fast state restoration in llm serving with hcache. In Proceedings of the Twentieth European Conference on Computer Systems, pages 128–143, 2025

    2025

  3. [11]

    Ai and memory wall

    Amir Gholami, Zhewei Yao, Sehoon Kim, Coleman Hooper, Michael W Mahoney, and Kurt Keutzer. Ai and memory wall. IEEE Micro, 2024

    2024

  4. [12]

    Slim attention: cut your context memory in half without loss–k-cache is all you need for mha

    Nils Graef and Andrew Wasielewski. Slim attention: cut your context memory in half without loss–k-cache is all you need for mha. arXiv preprint arXiv:2503.05840, 2025

    2025 arXiv

  5. [13]

    Polarquant: Quantizing kv caches with polar transformation

    Insu Han, Praneeth Kacham, Amin Karbasi, Vahab Mirrokni, and Amir Zandieh. Polarquant: Quantizing kv caches with polar transformation. arXiv preprint arXiv:2502.02617, 2025

    2025 arXiv

  6. [14]

    Zipcache: Accurate and efficient kv cache quantization with salient token identification

    Yefei He, Luoming Zhang, Weijia Wu, Jing Liu, Hong Zhou, and Bohan Zhuang. Zipcache: Accurate and efficient kv cache quantization with salient token identification. Advances in Neural Information Processing Systems, 37:68287–68307, 2024

    2024

  7. [15]

    Kvquant: Towards 10 million context length llm inference with kv cache quantization.Advances in Neural Information Processing Systems, 37:1270–1303, 2024

    Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Yakun S Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length llm inference with kv cache quantization.Advances in Neural Information Processing Systems, 37:1270–1303, 2024

    2024

  8. [16]

    Checkmate: Breaking the memory wall with optimal tensor rematerialization

    Paras Jain, Ajay Jain, Aniruddha Nrusimha, Amir Gholami, Pieter Abbeel, Joseph Gonzalez, Kurt Keutzer, and Ion Stoica. Checkmate: Breaking the memory wall with optimal tensor rematerialization. Proceedings of Machine Learning and Systems, 2:497–511, 2020

    2020

  9. [17]

    Residual connections encourage iterative inference

    Stanisław Jastrzebski, Devansh Arpit, Nicolas Ballas, Vikas Verma, Tong Che, and Yoshua Bengio. Residual connections encourage iterative inference. In International Conference on Learning Representations, 2018

    2018

  10. [18]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    2023 arXiv

  11. [19]

    Mahoney, Sophia Shao, and Amir Gholami

    Sehoon Kim, Coleman Hooper, Thanakul Wattanawong, Minwoo Kang, Ruohan Yan, Hasan Genc, Grace Dinh, Qijing Huang, Kurt Keutzer, Michael W. Mahoney, Sophia Shao, and Amir Gholami. Full stack optimization of transformer inference. In Architecture and System Support for Transforme...

    2023

  12. [20]

    Squeezellm: Dense-and-sparse quantization

    Sehoon Kim, Coleman Richard Charles Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. Squeezellm: Dense-and-sparse quantization. In International Conference on Ma- chine Learning, pages 23901–23923. PMLR, 2024

    2024

  13. [21]

    Efficient llm inference with activation checkpointing and hybrid caching

    Sanghyeon Lee, Hongbeen Kim, Soojin Hwang, Guseul Heo, Minwoo Noh, and Jaehyuk Huh. Efficient llm inference with activation checkpointing and hybrid caching. arXiv preprint arXiv:2501.01792, 2025. 15

    2025

  14. [22]

    Intactkv: Improving large language model quantization by keeping pivot tokens intact

    Ruikang Liu, Haoli Bai, Haokun Lin, Yuening Li, Han Gao, Zhengzhuo Xu, Lu Hou, Jun Yao, and Chun Yuan. Intactkv: Improving large language model quantization by keeping pivot tokens intact. In Findings of the Asso- ciation for Computational Linguistics ACL 2024, pages 7716–7741, 2024

    2024

  15. [23]

    Kivi: A tuning-free asymmetric 2bit quantization for kv cache

    Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: A tuning-free asymmetric 2bit quantization for kv cache. In International Conference on Machine Learning, pages 32332–32344. PMLR, 2024

    2024

  16. [24]

    Pointer sentinel mixture models, 2016

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016

    2016

  17. [25]

    Llama 3.1: https://ai.meta.com/blog/meta-llama-3-1 , 2024

    Meta. Llama 3.1: https://ai.meta.com/blog/meta-llama-3-1 , 2024

    2024

  18. [26]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv e-prints, 2019

    2019

  19. [27]

    Magicdec: Breaking the latency-throughput tradeoff for long context generation with speculative decoding

    Ranajoy Sadhukhan, Jian Chen, Zhuoming Chen, Vashisth Tiwari, Ruihang Lai, Jinyuan Shi, Ian En-Hsu Yen, Avner May, Tianqi Chen, and Beidi Chen. Magicdec: Breaking the latency-throughput tradeoff for long context generation with speculative decoding. In The Thirteenth Internati...

  20. [28]

    Eigen attention: Attention in low-rank space for kv cache compression

    Utkarsh Saxena, Gobinda Saha, Sakshi Choudhary, and Kaushik Roy. Eigen attention: Attention in low-rank space for kv cache compression. In Findings of the Association for Computational Linguistics: EMNLP 2024 , pages 15332–15344, 2024

    2024

  21. [29]

    Loki: Low-rank keys for efficient sparse attention

    Prajwal Singhania, Siddharth Singh, Shwai He, Soheil Feizi, and Abhinav Bhatele. Loki: Low-rank keys for efficient sparse attention. Advances in Neural Information Processing Systems, 37:16692–16723, 2024

    2024

  22. [30]

    Roformer: Enhanced trans- former with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced trans- former with rotary position embedding. Neurocomputing, 568:127063, 2024

    2024

  23. [31]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalk- wyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  24. [32]

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Per- rin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon...

    2025

  25. [33]

    Quantspec: Self-speculative decoding with hierarchical quantized kv cache

    Rishabh Tiwari, Haocheng Xi, Aditya Tomar, Coleman Hooper, Sehoon Kim, Maxwell Horton, Mahyar Najibi, Michael W Mahoney, Kurt Keutzer, and Amir Gholami. Quantspec: Self-speculative decoding with hierarchical quantized kv cache. In International Conference on Machine Learning, 2025

    2025

  26. [34]

    LLaMA: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Bap- tiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  27. [35]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  28. [36]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems , pages 5998–6008, 2017

    2017

  29. [37]

    Roofline: an insightful visual performance model for multicore architectures

    Samuel Williams, Andrew Waterman, and David Patterson. Roofline: an insightful visual performance model for multicore architectures. Communications of the ACM, 52(4):65–76, 2009

    2009

  30. [38]

    Polarquant: Leveraging polar transformation for efficient key cache quantization and decoding acceleration

    Songhao Wu, Ang Lv, Xiao Feng, Yufei Zhang, Xun Zhang, Guojun Yin, Wei Lin, and Rui Yan. Polarquant: Leveraging polar transformation for efficient key cache quantization and decoding acceleration. arXiv preprint arXiv:2502.00527, 2025

    2025 arXiv

  31. [39]

    Efficient streaming language models with attention sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations

  32. [40]

    On layer normalization in the transformer architecture

    Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture. InInternational conference on machine learning, pages 10524–10533. PMLR, 2020

    2020

  33. [41]

    Recalkv: Low-rank kv cache compression via head reordering and offline calibration

    Xianglong Yan, Zhiteng Li, Tianao Zhang, Linghe Kong, Yulun Zhang, and Xiaokang Yang. Recalkv: Low-rank kv cache compression via head reordering and offline calibration. arXiv preprint arXiv:2505.24357, 2025

    2025

  34. [42]

    El- attention: Memory efficient lossless attention for generation

    Yu Yan, Jiusheng Chen, Weizhen Qi, Nikhil Bhendawade, Yeyun Gong, Nan Duan, and Ruofei Zhang. El- attention: Memory efficient lossless attention for generation. In International Conference on Machine Learning, pages 11648–11658. PMLR, 2021

    2021

  35. [43]

    Qwen3 technical report, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jia...

    2025

  36. [44]

    No token left behind: Reliable kv cache compression via importance-aware mixed precision quantization

    June Yong Yang, Byeongwook Kim, Jeongin Bae, Beomseok Kwon, Gunho Park, Eunho Yang, Se Jung Kwon, and Dongsoo Lee. No token left behind: Reliable kv cache compression via importance-aware mixed precision quantization. arXiv preprint arXiv:2402.18096, 2024. 17

    2024 arXiv

  37. [45]

    Llm inference unveiled: Survey and roofline model insights

    Zhihang Yuan, Yuzhang Shang, Yang Zhou, Zhen Dong, Zhe Zhou, Chenhao Xue, Bingzhe Wu, Zhikai Li, Qingyi Gu, Yong Jae Lee, et al. Llm inference unveiled: Survey and roofline model insights. arXiv preprint arXiv:2402.16363, 2024

    2024 arXiv

  38. [46]

    Root mean square layer normalization

    Biao Zhang and Rico Sennrich. Root mean square layer normalization. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pages 12381–12392, 2019

    2019

  39. [47]

    Lorc: Low-rank compression for llms kv cache with a progressive compression strategy

    Rongzhi Zhang, Kuang Wang, Liyuan Liu, Shuohang Wang, Hao Cheng, Chao Zhang, and Yelong Shen. Lorc: Low-rank compression for llms kv cache with a progressive compression strategy. arXiv preprint arXiv:2410.03111, 2024. 18 A Prefill for XQ UANT-CL Here we include a visualizatio...

    2024 arXiv

  40. [48]

    run models on calibration datasets, which allows them to note which channels tend to be outliers across diverse data samples. Having observed the outlier behavior of the first channel in XUk, we explore if outlier channels in the Keys can be identified by inspecting the SVD de...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.