Pith. sign in

REVIEW 3 major objections 6 minor 2 cited by

QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cache

T0 review · 3 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read QuantSpec proposes a self-speculative decoder whose draft pass reads only the upper 4 bits of a hierarchical KV cache while the verification pass reads all 8 bits, reporting up to about 2.5x speedup for long-context generation.

desk verdict Clever hierarchical KV-cache trick and credible speedups, but the undefined verification step leaves the 'no quality loss' claim unproven. read the letter →

arxiv 2502.10424 v1 pith:MJJN7I5Q submitted 2025-02-05 cs.LG cs.AI

Rishabh Tiwari , Haocheng Xi , Aditya Tomar , Coleman Hooper , Sehoon Kim , Maxwell Horton , Mahyar Najibi , Michael W. Mahoney
show 2 more authors
Kurt Keutzer Amir Gholami
This is my paper · ORCID
classification cs.LGcs.AI
keywords speculativedecodingself-speculativeKVcachequantizationhierarchicallong-contextLLMinference4-bitacceptancerateGPUmemory
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

QuantSpec is a self-speculative decoding scheme for long-context LLM inference. The draft model is the same network as the target model but runs with 4-bit weights and reads only the upper 4 bits of a hierarchically quantized KV cache, while the target model reads the full 8-bit reconstruction from the same cache. The paper argues that this design eliminates the need for a separate draft KV cache, keeps acceptance rates near or above 90%, and delivers up to about 2.5x end-to-end speedup with about 1.3x less GPU memory than sparse-KV self-speculative baselines. The central claim is that quantizing the cache is a better way to build a fast, accurate draft model than evicting tokens from it.

What carries the argument

The load-bearing object is the hierarchical 4-bit KV cache: each key or value group of size $G$ is quantized into an upper INT4 tensor $C_U$ and an error-residual lower INT4 tensor $C_L$, with $C^{\mathrm{INT8}} = 2^4 C_U^{\mathrm{INT4}} + C_L^{\mathrm{INT4}}$ and the scaling and zero-point shared by both representations. This lets the draft model dequantize only the upper 4 bits while the target model adds the lower residual to reach INT8 precision, so no second cache copy exists. The companion double full-precision buffer, two halves of size $G$, absorbs newly generated tokens and lets rejected speculative tokens be dropped before quantization, so quantization happens only once every $G$ decoding steps.

What would settle it

Run a distribution-level check: generate several thousand tokens from QuantSpec and from the target model on identical prompts and seeds, and compare next-token histograms; any systematic divergence (e.g., a KL divergence much larger than sampling noise) would show the verifier is not preserving the target distribution. The direct check is the code path of Algorithm 1: VERIFY and CORRECT are never defined in the paper, so if CORRECT does not sample from the residual distribution $p_i - q_i$ (or a normalized version) when a token is rejected, QuantSpec is not exact speculative decoding.

Watch

Extended reading notes

Core claim

The paper's discovery is that an INT8 KV cache can be physically stored as two INT4 tensors, an upper part and a lower residual, so one allocation serves both draft and target: the draft model's attention loads only the upper 4 bits, and the target model's attention loads both parts and reconstructs a value within about one bit of the full INT8 cache. A double full-precision buffer of size $2G$ keeps the most recent tokens unquantized until the target has verified them, which removes repeated quantization and dequantization caused by rejected tokens. On Llama-2-7B-32K-Instruct and LWM-Text-Chat-128k, QuantSpec reports end-to-end speedups from $1.35\times$ to $2.49\times$ over autoregressive decoding, peak memory as low as $25.84$ GB at 32k context, and acceptance rates mostly above 90%.

Load-bearing premise

The paper's accuracy claim rests on Algorithm 1's VERIFY and CORRECT routines, which it never defines; if those routines do not implement rejection sampling from the target distribution, QuantSpec's output is not guaranteed to match the target model.

Editorial extensions

If this is right

  • If the verification step is exact, QuantSpec's output distribution matches the target model's, so the speedup does not change generation quality.
  • Long-context decoding no longer needs a second KV cache for the draft model, saving about 1.3x memory relative to sparse-KV self-speculative methods.
  • The same framework can favor weight quantization at short contexts and KV quantization at long contexts, based on the measured arithmetic-intensity regime.
  • Acceptance rates stay above 90% even with larger speculation lengths, so the method can take advantage of drafting several tokens per verification pass.
  • Combining QuantSpec with sparse-KV drafting is left for future work but explicitly anticipated, so additional speedups may be available by stacking the two approaches.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural generalization of the 4/8 split is storing the top $k$ bits for the draft and all bits for the verifier; the paper only demonstrates $k=4$, so the trade-off curve between draft speed and acceptance is left open.
  • Quantization keeps every token in the context rather than evicting any, so the approach should be particularly strong on tasks where important information is distributed across the whole input, such as multi-document summarization; the acceptance-rate figures on Multi-LexSum are consistent with that expectation.
  • The reported speedups are for single-sequence decoding on a single GPU. How the KV-cache bandwidth savings transfer to batched or multi-GPU serving is not measured, and that is the next test for production usefulness.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes QuantSpec, a self-speculative decoding framework in which the draft model shares the target's architecture but uses a hierarchical INT4 KV cache (upper 4 bits) and INT4 weights, while the target reads the full INT8 KV cache reconstructed from upper and lower 4-bit components. The authors argue that the hierarchical representation lets the draft and target share one KV cache, saving memory, and that the quantized draft achieves high acceptance rates (>90%) and end-to-end speedups up to ~2.5x over autoregressive decoding, outperforming sparse-KV self-speculative baselines. The paper includes an arithmetic-intensity analysis, kernel benchmarks, and ablations. The central accuracy-preservation claim relies on the speculative verification step, which is not fully specified.

Significance. If the claims hold, QuantSpec is a practical contribution to long-context inference: it introduces a memory-efficient hierarchical KV cache that avoids a separate draft cache, and it reports consistent speedups across two models and multiple context lengths. The paper provides a clear high-level idea, measured kernel-level speedups, and a hyperparameter search disclosure. However, because the verification routines are undefined and the long-context quality preservation is not directly measured, the central 'without compromising accuracy' claim is not yet supported. The method is plausible and the issues are fixable, so the significance is moderate conditional on those clarifications.

major comments (3)
  1. [Algorithm 1 (Appendix, lines 11-16)] The algorithm calls VERIFY(g_i, p_i, q_i) and CORRECT(p_i, q_i) but never defines these routines. In standard speculative decoding, the output distribution equals the target distribution only if verification uses rejection sampling with acceptance probability min(1, p_i(g_i)/q_i(g_i)) and, on rejection, correction samples from the residual distribution proportional to (p_i - q_i)_+. If VERIFY/CORRECT are instead implemented as greedy argmax matching and target resampling, the reported acceptance rates and speedups do not reflect generation from the target distribution. Since the abstract and Section 6 claim the method works 'without compromising accuracy,' please specify VERIFY and CORRECT explicitly (equations or pseudocode) and state that they implement distribution-preserving rejection sampling, citing the standard guarantee (e.g., Leviathan et al. 2023, Chen et al. 2023).
  2. [Algorithm 1 (Inputs, lines 2 and 7)] The algorithm takes 'Sensitive Layer Number LS' as an input and uses it in the QUANTIZE and DRAFT calls, but LS is never defined anywhere in the main text or appendix. There is no description of what a sensitive layer is, how LS is chosen, or how it affects the hierarchical cache. This is a reproducibility gap: a reader cannot reimplement the method without knowing the role of LS. Please define the sensitive-layer concept and state the selection procedure or default value.
  3. [Table 2 and Section 6] The 'without compromising accuracy' claim is validated only by perplexity on WikiText-2 and C4 in Table 2, with no context length reported; these are standard short-context evaluations. The long-context experiments in Table 3 report no generation-quality metrics (e.g., ROUGE or task accuracy) for QuantSpec against the FP16 target. Since the target model itself uses an INT8 KV cache, the accuracy-preservation claim for 16k-128k contexts is not directly supported. Please report long-context quality metrics (or a strong argument that the INT8 target preserves quality at these lengths) or qualify the claim.
minor comments (6)
  1. [Abstract and Table 3] The abstract states that QuantSpec 'maintains high acceptance rates (>90%),' but Table 3 reports 89.88% for PG19 at 8k context on Llama-2-7B. Please adjust the abstract or note the exception to avoid an overclaim.
  2. [Section 5.2.1 and Table 4] The text says 'for a context length of 128k, our INT4 attention kernel is ~2.88x faster,' but Table 4 reports only 64k and 256k. Moreover, the listed speedup factors do not match the latency ratios: at 64k, 3.07/0.54 = 5.69x for INT4, not 2.88x; at 256k, 6.16/2.15 = 2.87x. Please correct the table labels, the speedup values, or the text so the reported kernel speedups are consistent.
  3. [Section 4.2 and Table 2] The 'residual length R' is used in Table 2 and Section 5.1 but never formally defined. It appears to equal the double full-precision buffer size 2G (with G=128, R=256), but please state the definition explicitly.
  4. [Table 3] No error bars or standard deviations are reported even though the abstract says results are averaged over 10 examples. Please include variance or at least the range across examples.
  5. [Algorithm 1, line 19] The line 'x_{N+γ+1} ← p_{γ+1}' uses p_{γ+1}, which is never computed (only p_1,...,p_γ are produced by TARGET). This is likely a typo; please clarify the intended token selection after accepting all γ draft tokens.
  6. [Algorithm 1 caption] The function list includes 'VERITY' (a typo for 'VERIFY'), and there is a minor typo 'Moreoever' in Section 4.3.1. Please proofread.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: QuantSpec's headline speedups and acceptance rates are measured against external baselines rather than derived from its own assumptions.

full rationale

QuantSpec does not fit a parameter and then relabel it as a prediction. The acceptance rates, GPU memory, and end-to-end speedups in Table 3 are measured values on PG19, ∞BENCH Sum, and Multi-LexSum, and the kernel speedups in Table 4 are measured against a standard FP16 FlashAttention kernel; none of these quantities is defined as the output of the quantization scheme itself. The hierarchical KV cache, expressed as CINT8 = 24CINT4U + CINT4L, is an explicit representational construction rather than a first-principles law whose conclusion is its own input. The only tuned quantity, the speculation length γ, is disclosed as a hyperparameter search in Appendix G, with the chosen values reported in Table 6, so this is ordinary model selection rather than a hidden fit. The citation to Anonymous (2025) is used only to select the StreamingLLM and SnapKV baseline configurations and is not load-bearing for QuantSpec's own claims. The one genuine weakness is a completeness gap, not circularity: Algorithm 1 (Appendix, lines 11-16) calls VERIFY and CORRECT without defining their acceptance and rejection semantics, so the "without compromising accuracy" guarantee is not established in the text. However, nothing in the paper shows that these routines are defined in terms of, or fitted to, the reported acceptance rates or speedups. This gap is a correctness and rigor concern and does not raise the circularity score.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The method adds design hyperparameters but no trained parameters or new physical entities. The main burden is on the empirical claim that quantized cache precision is sufficient at long contexts and that the unspecified verification procedure preserves target quality.

free parameters (4)
  • Speculation length gamma = 1-6 depending on dataset/model, chosen by hyperparameter search at 8k context
    Tuned per dataset-method pair in Appendix G; directly affects acceptance rate and end-to-end speedup.
  • Quantization group size G = 128
    Set by hand in Section 5.1; controls quantization granularity and the size of the full-precision buffer.
  • Residual length R = 256
    Chosen in Section 5.1; controls how many recent KV entries remain in full precision in the hierarchical cache.
  • Sensitive layer number LS = not reported
    Passed through Algorithm 1 but never defined or given a value; appears to control which layers use full precision.
assumptions (5)
  • standard math FlashAttention does not materialize the B*SL^2 attention scores matrix, so attention memory operations are O(B*SL).
    Used in Section 3.1.1 to derive the asymptotic arithmetic intensity table.
  • domain assumption The NVIDIA A6000 roofline ridge point used to classify regimes is representative of the hardware where speedups are claimed.
    Section 3.1.2; if deployment hardware differs, the regime classification and hence the quantization guidance may change.
  • domain assumption The INT8 reconstruction from upper and lower 4-bit parts preserves target-model quality at long contexts.
    Justified only by Table 2 perplexity on Llama-2-7B with WikiText-2 and C4, not by long-context task accuracy.
  • domain assumption The VERIFY/CORRECT steps in Algorithm 1 implement a distribution-preserving speculative-decoding accept/reject scheme.
    Never defined in the paper; needed for the claim that generation quality is not compromised.
  • domain assumption Per-group uniform 4-bit quantization of weights and KV cache is computationally cheap enough to not erase end-to-end gains.
    Supported only by the custom kernel benchmarks, which are not released.

how reviews work

0 comments
Cite this review

Pith. "Pith review of QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cache." pith.science (2026). https://pith.science/paper/MJJN7I5Q

@misc{pith2026250210424,
  author       = {Pith},
  title        = {Pith review of: QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cache},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MJJN7I5Q}},
  note         = {Machine review of arXiv:2502.10424}
}
abstract

Large Language Models (LLMs) are increasingly being deployed on edge devices for long-context settings, creating a growing need for fast and efficient long-context inference. In these scenarios, the Key-Value (KV) cache is the primary bottleneck in terms of both GPU memory and latency, as the full KV cache must be loaded for each decoding step. While speculative decoding is a widely accepted technique to accelerate autoregressive decoding, existing methods often struggle to achieve significant speedups due to inefficient KV cache optimization strategies and result in low acceptance rates. To address these challenges, we propose a novel self-speculative decoding framework, QuantSpec, where the draft model shares the architecture of the target model but employs a hierarchical 4-bit quantized KV cache and 4-bit quantized weights for acceleration. QuantSpec maintains high acceptance rates ($>$90%) and reliably provides consistent end-to-end speedups upto $\sim2.5\times$, outperforming other self-speculative decoding methods that use sparse KV cache for long-context LLM inference. QuantSpec also reduces the memory requirements by $\sim 1.3\times$ compared to these alternatives.

Figures

Figures reproduced from arXiv: 2502.10424 by the authors.

Figure 1
Figure 1. Throughput in tokens/sec of various decoding methods. QuantSpec achieves > 1.78× speedup over the autoregressive baseline across several context lengths. Benchmarked on LWM￾Text-Chat-128k. tails across extended sequences. However, long-context inference presents significant challenges in terms of effi￾ciency and scalability. For example, token eviction (Zhang et al., 2024c; Ge et al., 2023; Liu et al., 2024b) and KV… view at source ↗
Figure 2
Figure 2. Breakdown of how arithmetic intensity changes during decoding as the context length and batch size are scaled logarithmically for linear, attention, and aggregate operations. All regimes lie below the ridge plane and thus are memory-bound. The ridge plane is calculated for an NVIDIA A6000 GPU. The colors for the linear and attention surface plots simply represent the magnitude of the arithmetic intensity. The aggreg… view at source ↗
Figure 3
Figure 3. How our Hierarchical KV Cache works in the speculative decoding setting. the KV cache in the higher INT8 precision. To represent the INT8 KV cache C INT8 as the upper INT4 KV cache C INT4 U and the lower INT4 cache C INT4 L , the INT8 KV cache can be expressed as C INT8 = 24C INT4 U + C INT4 L , where we multiply by 2 4 to align their represented values. The asym￾metric quantization for the KV cache can be represent… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Speedup ratio of QuantSpec compared to autoregressive baseline as we scale the context length. We report QuantSpec with KV cache-only quantization, weight-only quantization, and both. Benchmarked on Llama-2-7B-32k-Instruct using PG-19. 6. Conclusions In this paper, we …
Figure 5
Figure 5. Figure 5: During prefill, all regimes lie above the ridge plane and thus are compute-bound. C.2. Modern GPU Hardware VRAM Size Constraints [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: KV cache memory usage by Llama-2-7B on a single node (8 GPUs) as context length and batch size are scaled logarithmically. The surface plot’s color represents the ratio of KV cache memory to the model weights memory. The dotted-lines represent GPU DRAM capacities for s…
Figure 7
Figure 7. Figure 7: We apply asymmetric and per-group quantization for both the key cache and value cache, along the channel axis and token axis, respectively. This figure describes how it works when only the upper-4 bit cache is applied. Key Cache Value Cache token-wise channel-wise toke…
Figure 9
Figure 9. Figure 9: Acceptance rate of self-speculative decoding methods at different speculation length measured for model LWM-Text-Chat-128k on Multi-LexSum dataset. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. High-accuracy Low-Bit KV-Cache Quantization via Local Distribution Restoration

    cs.LG 2026-06 conditional novelty 6.0 of 10

    A learned top-K distribution-restoration layer recovers near-FP16 long-context accuracy under 1-bit KV-cache quantization.

  2. Speculative Decoding Meets Quantization: Compatibility Evaluation and Hierarchical Framework Design

    cs.CL 2025-05 conditional novelty 6.0 of 10

    EAGLE-2 loses most of its speedup on 4-bit quantized LLMs, and a hierarchical draft-then-sequence scheme restores 1.31x speedup over EAGLE-2 on W4A16 Llama-3-70B.

Reference graph

Works this paper leans on

33 extracted references · 4 canonical work pages · cited by 2 Pith papers

  1. [3]

    D., Chen, D., and Dao, T

    Cai, T., Li, Y ., Geng, Z., Peng, H., Lee, J. D., Chen, D., and Dao, T. Medusa: Simple llm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.10774,

    arXiv

  2. [4]

    Accelerating large language model decoding with speculative sampling

    Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, L., and Jumper, J. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318,

    arXiv

  3. [7]

    Scal- ing fp8 training to trillion-token llms

    Fishman, M., Chmiel, B., Banner, R., and Soudry, D. Scal- ing fp8 training to trillion-token llms. arXiv preprint arXiv:2409.12517,

    arXiv

  4. [8]

    Lazyllm: Dynamic token pruning for efficient long context llm inference

    Fu, Q., Cho, M., Merth, T., Mehta, S., Rastegari, M., and Najibi, M. Lazyllm: Dynamic token pruning for efficient long context llm inference. arXiv preprint arXiv:2407.14057,

    arXiv

  5. [9]

    Model tells you what to discard: Adaptive kv cache compression for llms

    Ge, S., Zhang, Y ., Liu, L., Zhang, M., Han, J., and Gao, J. Model tells you what to discard: Adaptive kv cache compression for llms. arXiv preprint arXiv:2310.01801,

    arXiv

  6. [10]

    Jiang, H., Li, Y ., Zhang, C., Wu, Q., Luo, X., Ahn, S., Han, Z., Abdi, A

    URL https://arxiv.org/abs/ 2410.08391. Jiang, H., Li, Y ., Zhang, C., Wu, Q., Luo, X., Ahn, S., Han, Z., Abdi, A. H., Li, D., Lin, C.-Y ., et al. Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention. arXiv preprint arXiv:2407.02490,

    arXiv

  7. [11]

    Gear: An efficient kv cache compression recipefor near-lossless generative inference of llm

    Kang, H., Zhang, Q., Kundu, S., Jeong, G., Liu, Z., Krishna, T., and Zhao, T. Gear: An efficient kv cache compression recipefor near-lossless generative inference of llm. arXiv preprint arXiv:2403.05527,

    arXiv

  8. [12]

    W., and Keutzer, K

    9 QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cache Kim, S., Hooper, C., Gholami, A., Dong, Z., Li, X., Shen, S., Mahoney, M. W., and Keutzer, K. Squeezellm: Dense-and-sparse quantization. arXiv preprint arXiv:2306.07629, 2023a. Kim, S., Hooper, C., Wattanawong, T., Kang, M., Yan, R., Genc, H., Dinh, G., Huang, Q., Keutzer, K., Mah...

    arXiv

Show all 33 references
  1. [13]

    Snapkv: Llm knows what you are looking for before generation

    Li, Y ., Huang, Y ., Yang, B., Venkitesh, B., Locatelli, A., Ye, H., Cai, T., Lewis, P., and Chen, D. Snapkv: Llm knows what you are looking for before generation. arXiv preprint arXiv:2404.14469, 2024a. Li, Y ., Wei, F., Zhang, C., and Zhang, H. Eagle: Speculative sampling re...

    arXiv

  2. [14]

    Re- trievalattention: Accelerating long-context llm inference via vector retrieval

    Liu, D., Chen, M., Lu, B., Jiang, H., Han, Z., Zhang, Q., Chen, Q., Zhang, C., Ding, B., Zhang, K., et al. Re- trievalattention: Accelerating long-context llm inference via vector retrieval. arXiv preprint arXiv:2409.10516 , 2024a. Liu, L., Qu, Z., Chen, Z., Ding, Y ., and Xie...

    arXiv

  3. [17]

    Park, Y ., Hyun, J., Cho, S., Sim, B., and Lee, J. W. Any-precision llm: Low-cost deployment of multiple, different-sized llms. arXiv preprint arXiv:2402.10517 ,

    arXiv

  4. [18]

    Fp8-lm: Training fp8 large language models

    Peng, H., Wu, K., Wei, Y ., Zhao, G., Yang, Y ., Liu, Z., Xiong, Y ., Yang, Z., Ni, B., Hu, J., et al. Fp8-lm: Training fp8 large language models. arXiv preprint arXiv:2310.18313,

    arXiv

  5. [21]

    Omniquant: Omnidirectionally calibrated quantization for large lan- guage models

    Shao, W., Chen, M., Zhang, Z., Xu, P., Zhao, L., Li, Z., Zhang, K., Gao, P., Qiao, Y ., and Luo, P. Omniquant: Omnidirectionally calibrated quantization for large lan- guage models. arXiv preprint arXiv:2308.13137,

    arXiv

  6. [22]

    Sun, H., Chen, Z., Yang, X., Tian, Y ., and Chen, B

    URL https: //openreview.net/forum?id=z1d8fUiS8Cr. Sun, H., Chen, Z., Yang, X., Tian, Y ., and Chen, B. Tri- force: Lossless acceleration of long sequence generation with hierarchical speculative decoding. arXiv preprint arXiv:2404.11912,

    arXiv

  7. [23]

    E., and Popa, R

    Tan, S., Li, X., Patil, S., Wu, Z., Zhang, T., Keutzer, K., Gonzalez, J. E., and Popa, R. A. Lloco: Learning long contexts offline. arXiv preprint arXiv:2404.07979,

    arXiv

  8. [24]

    Quest: Query-aware sparsity for efficient long-context llm inference

    Tang, J., Zhao, Y ., Zhu, K., Xiao, G., Kasikci, B., and Han, S. Quest: Query-aware sparsity for efficient long-context llm inference. arXiv preprint arXiv:2406.10774,

    arXiv

  9. [26]

    Sirllm: Streaming infinite retentive llm

    Yao, Y ., Li, Z., and Zhao, H. Sirllm: Streaming infinite retentive llm. arXiv preprint arXiv:2405.12528,

    arXiv

  10. [27]

    Helmet: How to evaluate long-context language models effectively and thoroughly

    Yen, H., Gao, T., Hou, M., Ding, K., Fleischer, D., Izsak, P., Wasserblat, M., and Chen, D. Helmet: How to evaluate long-context language models effectively and thoroughly. arXiv preprint arXiv:2410.02694,

    arXiv

  11. [28]

    Sageatten- tion: Accurate 8-bit attention for plug-and-play inference acceleration

    Zhang, J., Zhang, P., Zhu, J., Chen, J., et al. Sageatten- tion: Accurate 8-bit attention for plug-and-play inference acceleration. arXiv preprint arXiv:2410.02367, 2024a. Zhang, X., Chen, Y ., Hu, S., Xu, Z., Chen, J., Hao, M. K., Han, X., Thai, Z. L., Wang, S., Liu, Z., and ...

  12. [29]

    Sir- ius: Contextual sparsity with correction for efficient llms

    Zhou, Y ., Chen, Z., Xu, Z., Lin, V ., and Chen, B. Sir- ius: Contextual sparsity with correction for efficient llms. arXiv preprint arXiv:2409.03856,

    arXiv

  13. [30]

    Attention Module’s Inference Workflow The inference of LLMs can be divided into 2 parts: the prefill stage and the decoding stage

    11 QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cache Appendix A. Attention Module’s Inference Workflow The inference of LLMs can be divided into 2 parts: the prefill stage and the decoding stage. In the prefill stage, for the input sequence X ∈ RB×SL×d,...

    2024

  14. [31]

    incorporates multiple precision levels (e.g., INT2, INT4, and INT8) within a single representation, eliminating the need to store separate KV caches for each precision and allowing the framework to dynamically select the optimal precision based on the complexity of the task. T...

    2023

  15. [33]

    • C4 (Raffel et al., 2020): C4 is a large scale web-crawled language modelling dataset mostly used for pretraining LLMs

    Our KV cache with 2 full precision cache buffers for recent KV cache. • C4 (Raffel et al., 2020): C4 is a large scale web-crawled language modelling dataset mostly used for pretraining LLMs. • PG-19 (Rae et al., 2019): It is a dataset of books from Project Gutenberg, designed ...

    2020

  16. [128]

    Channel-wise quantization for key cache and token-wise quantization for value cache gives the best performance. E. Compatibility with Flash Decoding Our full-precision buffer design, as shown in Figure 8, is fully compatible with Flash Decoding (Dao et al., 2023), a fast atten...

    2023

  17. [2009]

    Coat: Compressing optimizer states and activation for memory-efficient fp8 training, 2024a

    Xi, H., Cai, H., Zhu, L., Lu, Y ., Keutzer, K., Chen, J., and Han, S. Coat: Compressing optimizer states and activation for memory-efficient fp8 training, 2024a. URL https://arxiv.org/abs/2410.19313. Xi, H., Chen, Y ., Zhao, K., Zheng, K., Chen, J., and Zhu, J. Jetfire: Effici...

    arXiv

  18. [2016]

    Nawrot, P., Ła´ncucki, A., Chochowski, M., Tarjan, D., and Ponti, E. M. Dynamic memory compression: Retrofitting llms for accelerated inference. arXiv preprint arXiv:2403.09636,

    arXiv

  19. [2019]

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y ., Li, W., Liu, P

    URL https://arxiv.org/abs/1911.05507. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y ., Li, W., Liu, P. J., et al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(140):1–67,

    1911 arXiv

  20. [2020]

    Flashattention-3: Fast and accurate atten- tion with asynchrony and low-precision

    Shah, J., Bikshandi, G., Zhang, Y ., Thakkar, V ., Ramani, P., and Dao, T. Flashattention-3: Fast and accurate atten- tion with asynchrony and low-precision. arXiv preprint arXiv:2407.08608,

    arXiv

  21. [2021]

    Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time

    Liu, Z., Desai, A., Liao, F., Wang, W., Xie, V ., Xu, Z., Kyril- lidis, A., and Shrivastava, A. Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time. Advances in Neural Information Processing Systems, 36, 2024b. Liu, Z., Y...

    arXiv

  22. [2022]

    Dao, T., Haziza, D., Massa, F., and Sisov, G

    URL https://arxiv.org/abs/ 2205.14135. Dao, T., Haziza, D., Massa, F., and Sisov, G. Flash-decoding for long-context inference: https://crfm.stanford.edu/2023/10/ 12/flashdecoding.html,

    2023 arXiv

  23. [2023]

    Int-flashattention: Enabling flash attention for int8 quantization

    Chen, S., Liu, Z., Wu, Z., Zheng, C., Cong, P., Jiang, Z., Su, L., and Yang, T. Int-flashattention: Enabling flash attention for int8 quantization. arXiv preprint arXiv:2409.16997, 2024a. Chen, Z., May, A., Svirschevski, R., Huang, Y ., Ryabinin, M., Jia, Z., and Chen, B. Sequ...

    arXiv

  24. [2024]

    Brandon, W., Mishra, M., Nrusimha, A., Panda, R., and Kelly, J. R. Reducing transformer key-value cache size with cross-layer attention. arXiv preprint arXiv:2405.12981,

    arXiv

  25. [2025]

    Bhendawade, N., Belousova, I., Fu, Q., Mason, H., Raste- gari, M., and Najibi, M

    URL https:// openreview.net/forum?id=CS2JWaziYr. Bhendawade, N., Belousova, I., Fu, Q., Mason, H., Raste- gari, M., and Najibi, M. Speculative streaming: Fast llm inference without auxiliary models. arXiv preprint arXiv:2402.11131,

    arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.