Pith. sign in

REVIEW 3 major objections 6 minor 45 references

NQKV: A KV Cache Quantization Scheme Based on Normal Distribution Characteristics

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that KV cache blocks are normally distributed, so a fixed 4-bit quantile codebook can quantize them with near-optimal error and negligible model degradation.

desk verdict Plausible 4-bit KV cache quantizer on OPT, but the information-theoretic optimality claim is unsupported; worth refereeing with major revisions. read the letter →

arxiv 2505.16210 v1 pith:RHBC7CNY submitted 2025-05-22 cs.LG cs.AIcs.CL

Zhihang Cai , Xingjun Zhang , Zhendong Tan , Zheng Wei This is my paper
classification cs.LGcs.AIcs.CL
keywords KVcachequantization4-bitnormaldistributionquantileFloatlargelanguagemodelsinferencememorypost-training
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the key-value cache of a large language model can be quantized to 4 bits with almost no loss in output quality, because the numbers inside each block of the cache follow a normal distribution. On that basis, NQKV applies per-block quantile quantization using the NF4 codebook, whose quantiles are placed to minimize error for normally distributed data. The authors report that on OPT models from 125M to 30B, zero-shot accuracy barely moves, and that the memory savings let the KV cache serve a 2x larger batch or a 4x longer context, with up to 9.3x throughput compared to running without the cache. If correct, the scheme offers a fine-tuning-free way to shrink the dominant memory cost of long-context and large-batch inference.

What carries the argument

The load-bearing mechanism is per-block quantile quantization with the 4-bit NormalFloat (NF4) codebook: a fixed set of 16 quantiles of the standard normal distribution, used to map each block's standardized values to the nearest codebook entry. NQKV divides the hidden dimension of each key and value tensor into blocks (block size 256 in the experiments), computes a block-level scale, and stores NF4 indices in the KV cache; dequantization is a table lookup. The padding strategy aligns the token dimension to multiples of 16 so that dequantized tensors can use efficient GEMM kernels, keeping the overhead of the storage-based data type low.

What would settle it

Run the same block-level normality test on KV caches from a different model family (e.g., LLaMA or Mistral) across diverse long-context inputs; if a majority of blocks reject normality at alpha=0.05, or if the fixed NF4 codebook's empirical quantization MSE is substantially worse than a per-block optimized codebook (e.g., Lloyd-Max), then the theoretical basis of NQKV collapses, even if its measured accuracy remains acceptable.

Watch

Extended reading notes

Core claim

The central discovery is that the elements of the KV cache, both within a token and within blocks of a token, conform to a normal distribution, and that this makes a fixed 4-bit quantile codebook (NF4) near-optimal for per-block quantization. NQKV splits each key and value tensor into blocks along the token dimension, quantizes each block independently to NF4 indices, stores only the 4-bit indices, and dequantizes on the fly during attention. Because the codebook's quantiles match the normal distribution, the quantization error is claimed to be information-theoretically optimal for the block distribution, and because errors are confined to a block, they do not propagate across the token. The paper supports this with Q-Q plots, D'Agostino-Pearson tests on 16 blocks of one model, and zero-shot accuracy tables on OPT models, plus throughput and memory measurements.

Load-bearing premise

The paper assumes that KV cache elements within every block are normally distributed; this is supported only by visual Q-Q plots and D'Agostino-Pearson tests on 16 blocks from one model (OPT-6.7B) at a few layers, without a stated sampling protocol or correction for multiple testing, and without checking distributional drift across layers, contexts, or other model families.

Editorial extensions

If this is right

  • KV cache memory can be cut by roughly 75% on OPT-class models with a reported average zero-shot accuracy drop below one percentage point.
  • Under the reported settings, NQKV lets the KV-cache-enabled model run with a 2x larger batch size or a 4x longer sequence length than SmoothQuant's KV cache allows, and delivers up to 9.3x throughput over running without a KV cache on OPT-30B.
  • NQKV is orthogonal to weight-activation quantization: combining it with SmoothQuant's W8A8 keeps average accuracy within about one point on the tested OPT models.
  • Because the method is fine-tuning-free and operates block-wise along the token dimension, it can be plugged into existing inference systems and combined with offloading or paged attention.
  • The padding scheme limits the dequantization overhead so the throughput loss relative to unquantized KV cache stays below 20% in the measured configurations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If block-level normality holds across model families, the same fixed NF4 codebook could be standardized in inference engines, and per-block scaling would replace per-token or per-channel quantization; this is a testable claim the paper does not establish.
  • Because the codebook is fixed and derived from the standard normal, layers or heads whose KV distributions drift (e.g., at very long contexts) could accumulate error; a per-layer or per-head codebook, or an online recalibration of quantiles, is a natural extension the paper leaves open.
  • The reported 9.3x speedup is specific to one OPT-30B configuration where the baseline without KV cache is memory-bound; in settings with ample memory the speedup will be smaller, so the headline number should be read as a ceiling from a memory-constrained case.
  • The dequantization overhead could be amortized by fusing table lookup into the attention kernel rather than materializing FP16 tensors, which would reduce the under-20% throughput loss the authors report.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes NQKV, a 4-bit KV-cache quantization scheme for OPT models. It claims that elements of the KV cache follow a normal distribution both within tokens and within blocks, and uses this to justify a per-block quantile quantizer based on the NF4 codebook. The authors report negligible accuracy loss on seven zero-shot tasks across OPT-125M to OPT-30B, demonstrate orthogonality with SmoothQuant, and report memory savings that allow larger batch sizes and longer sequences, with throughput up to 9.3x higher than a no-KV-cache baseline. The central theoretical claim is that per-block quantile quantization achieves information-theoretically optimal quantization error because the block data are normal.

Significance. If the empirical results are representative, NQKV provides a simple, fine-tuning-free 4-bit KV-cache quantizer with 60-80% memory savings and only small accuracy degradation on the tested OPT models, and it is orthogonal to weight/activation quantization methods such as SmoothQuant. The accuracy tables and memory measurements are concrete empirical contributions, and the streaming append-only design is a sensible practical choice. However, the paper's central theoretical justification, the information-theoretic optimality claim, is not substantiated by the described algorithm, and the normality evidence is too thin to establish the premise across layers, contexts, and model families. The practical method may still be useful even if the optimality claim is removed, but the current manuscript overstates its theoretical foundation. No code or reproducibility artifacts are provided.

major comments (3)
  1. [Section 3.2, Fig. 5] The claim that NQKV achieves 'information-theoretically optimal quantization error' is not supported by the algorithm as described. In the prefill and decoding phases, each block is rescaled by its maximum absolute value and then mapped through a fixed, zero-centered NF4 codebook; there is no per-block mean subtraction or standard-deviation normalization anywhere in the formulation or in Fig. 5. A block drawn from N(mu, sigma^2) with nonzero mu is not a standard normal after absmax scaling, so the fixed quantiles are not the optimal quantizer for that block. Even under the normality assumption, the optimality claim requires a derivation showing that the chosen codebook minimizes the expected quantization error for the actual per-block distributions; no such derivation is provided.
  2. [Section 3.1, Table 1] The normality evidence is insufficient to support the load-bearing premise that every KV-cache block follows a normal distribution. Table 1 reports D'Agostino-Pearson p-values for only 16 blocks from one model (OPT-6.7B), with no description of how tokens or layers were sampled, no multiple-testing correction across the 16 tests, and no coverage of different context lengths, input distributions, or model families. If blocks are heavy-tailed, skewed, or otherwise non-normal, the fixed quantile codebook is no longer justified and the theoretical rationale collapses, even though the empirical accuracy might remain acceptable. The paper should either substantially expand the normality study or revise the theoretical claims to match the evidence.
  3. [Abstract and Section 5] The phrase 'information-theoretically optimal quantization error' appears in the abstract and the conclusion, but no information-theoretic argument or rate-distortion analysis appears anywhere in the manuscript. The connection between quantile quantization of a standard normal source and optimal quantization error is not established, and the actual algorithm's absmax scaling changes the source distribution. This claim should be either rigorously derived under explicit assumptions or removed as unsupported. As written, it overstates what the paper demonstrates.
minor comments (6)
  1. [Table 4] The reported 9.3x throughput improvement is relative to the 'SQ' baseline without KV cache, not to an exact FP16 KV-cache baseline; this should be stated clearly in the table caption or text, as should the dequantization overhead relative to 'SQKV' (the exact KV-cache baseline).
  2. [Table 4, OPT-30B row] In the row for BS=64, SeqLen=512, the 'Saving' column shows an entry of '62.49' followed by a dash, which appears to be a formatting artifact; the memory saving relative to SQ should be reported consistently, and the OOM entry for SQKV should be explained.
  3. [Abstract and Section 4.1] There is a typo in the abstract: 'an 2×' should be 'a 2×' or '2× larger'. Also, Section 4.1 says 'ARC (Easy) [40]' is in the task list, but the abstract and Table 2 list only 'ARC(Challenge)'; the evaluation configuration should be described consistently.
  4. [Section 3.1, Fig. 4] The Q-Q plots in Fig. 4 show only three of the sixteen blocks for a single model, and the caption says 'for the sake of demonstration' without reporting the other thirteen; this is too little evidence for the strong claim that all blocks are normal.
  5. [Section 4.2.1] The accuracy tables report point estimates without error bars or repeated-run variance; for a claim of 'negligible accuracy loss', it would be helpful to know whether differences of 0.1-0.7% are within run-to-run noise.
  6. [Section 2] The concurrent work cited as [29] (KIVI) is described without a complete venue or date; adding the full citation would help readers compare NQKV with the streaming asymmetric quantization approach.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's accuracy and memory-throughput claims are empirical measurements against external benchmarks, and its codebook is imported from external QLoRA work rather than derived from its own conclusions.

full rationale

The paper's central empirical claims — negligible accuracy loss on zero-shot tasks, orthogonality with SmoothQuant, and memory/throughput improvements — are measured against external benchmarks (lm-eval-harness tasks and Wikitext-2 workloads) and compared with FP16/SmoothQuant baselines. These results do not reduce by construction to the paper's normality assumption; they are observed outcomes of a concrete quantization pipeline. The NF4 codebook is not invented or fitted in this paper; it is imported from the external QLoRA work [20], whose information-theoretic optimality claim concerns a standard normal distribution and is independent of the KV-cache results. The D'Agostino-Pearson tests in Table 1 are empirical support for a design premise, not a restatement of the conclusion, so the design is not self-validating. The only notable weakness is a logic/correctness gap: even if each block is normally distributed, the absmax scaling and fixed zero-centered codebook described in Section 3.2 and Fig. 5 do not by themselves produce a standard normal block, so the information-theoretically optimal quantization error claim does not follow from the implemented pipeline as written. That is an invalid or unsupported inference, however, not a circular reduction, and it does not contaminate the measured accuracy, memory, or throughput results. There are no load-bearing self-citations, no fitted parameters relabeled as predictions, and no equation that reduces to its own input.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central claim rests on one hand-chosen hyperparameter (block size 256), the empirical normality premise that motivates the fixed NF4 codebook, and the imported NF4 data type from QLoRA. The paper does not introduce new physical entities; the main burden is on the normality assumption and the unsupported optimality claim.

free parameters (1)
  • block_size = 256
    Chosen by hand in Section 4.1 as the trade-off between accuracy and memory; no sensitivity study is reported across block sizes.
assumptions (3)
  • domain assumption Elements within each block of the KV cache follow a normal distribution.
    Stated in the abstract and Section 3.1 and used to justify the fixed NF4 codebook and the 'information-theoretically optimal' claim. Supported only by a small D'Agostino-Pearson test on 16 blocks of OPT-6.7B, with no protocol for sampling or multiple-comparison correction.
  • domain assumption The NF4 codebook from QLoRA [20] is the correct quantizer for normal data and yields information-theoretically optimal quantization error for block-wise quantization.
    Imported from prior work without proof; the paper does not show that the fixed 16-level codebook minimizes squared error for each block's empirical distribution, nor does it account for per-block scale factors.
  • domain assumption KV cache is append-only, so per-token/block encoding can be appended without re-quantizing previous entries.
    Structural property of the inference loop used in Section 3.2 to justify token-dimension block quantization; reasonable for standard decoder inference.

how reviews work

0 comments
Cite this review

Pith. "Pith review of NQKV: A KV Cache Quantization Scheme Based on Normal Distribution Characteristics." pith.science (2026). https://pith.science/paper/RHBC7CNY

@misc{pith2026250516210,
  author       = {Pith},
  title        = {Pith review of: NQKV: A KV Cache Quantization Scheme Based on Normal Distribution Characteristics},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RHBC7CNY}},
  note         = {Machine review of arXiv:2505.16210}
}
read the original abstract

Large Language Models (LLMs) have demonstrated remarkable proficiency across a wide range of tasks. However, LLMs often require larger batch sizes to enhance throughput or longer context lengths to meet task demands, which significantly increases the memory resource consumption of the Key-Value (KV) cache during inference, becoming a major bottleneck in LLM deployment. To address this issue, quantization is a common and straightforward approach. Currently, quantization methods for activations are limited to 8-bit, and quantization to even lower bits can lead to substantial accuracy drops. To further save space by quantizing the KV cache to even lower bits, we analyzed the element distribution of the KV cache and designed the NQKV algorithm. Since the elements within each block of the KV cache follow a normal distribution, NQKV employs per-block quantile quantization to achieve information-theoretically optimal quantization error. Without significantly compromising model output quality, NQKV enables the OPT model to perform inference with an 2x larger batch size or a 4x longer context length, and it improves throughput by 9.3x compared to when the KV cache is not used.

Figures

Figures reproduced from arXiv: 2505.16210 by the authors.

Figure 1
Figure 1. The memory comsumption of OPT models in differ￾ent scales under various batch size and sequence length con￾figurations. 2.8% 97.2% Weights KV Cache SeqLen 2048, Batch Size 1 1 0.4% 89.6% SeqLen 81 92, Batch Size 1 KV Cache Weights 35.0% 65.0% SeqLen 2048, Batch Size 64 Weights KV Cache 11.9% 88.1% SeqLen 81 92, Batch Size 64 KV Cache Weights [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗
Figure 2
Figure 2. The memory usage percentages of different compo￾nents during inference for the OPT-175B model. As the batch size and sequence length increase, the memory space allocated to the KV cache ignificantly increases. Zhihang Cai et al.: Preprint submitted to Elsevier Page 9 of 9 [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Demonstration of the data distribution of randomly selected tokens in OPT-6.7B decoder layers. Even if the data within each token follows a normal distribution, their standard deviations may differ. Therefore, we standardized the data to make their standard deviations equal to 1, allowing for easy comparison with the standard normal distribution. For ease of observation, we also plotted the probability density funct… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Quantile-Quantile plots of data distribution in tokens and blocks of the OPT-6.7B model. The hidden states size of OPT-6.7B is 4096. With a block size of 256, we can obtain 16 blocks. For the sake of demonstration, only the Quantile-Quantile plots of three of these blo…
Figure 5
Figure 5. Figure 5: Block-wise quantile quantization. For demonstration purposes, let’s assume the hidden states size is 24, input token dimension size is 1024, the block size is 6, and the dimensions of the keys matrix are 1024×24 (ignoring batch size). There￾fore, each token of the keys…
Figure 6
Figure 6. Figure 6: Execution flow of the NQKV algorithm. For ease of description, only the scenario of Key cache is described in the decoding phase, with the situation for Value cache being identical. KV Cache Keys and Values padding padding zeros Keys and Values KV Cache (Already Padded…
Figure 7
Figure 7. Figure 7: Padding the KV cache during computation would result in lower peak memory usage and has no additional com￾putation overhead compared to directly padding the KV cache. 0 40 80 1 20 1 60 0 20 40 60 80 P e a k M e m o r y ( G B ) Batch Size NQKV SQ FP1 6 [PITH_FULL_IMAGE…
Figure 9
Figure 9. Figure 9: For the OPT-6.7B model, NQKV can perform infer￾ence with 2.5 × sequence length compared to a standard FP16 model, and with 1.5 × sequence length compared to Smooth￾Quant. Zhihang Cai et al.: Preprint submitted to Elsevier Page 12 of 9 [PITH_FULL_IMAGE:figures/full_fig…
Figure 8
Figure 8. Figure 8: For the OPT-6.7B model, NQKV can perform infer￾ence with 4 × batch size compared to a standard FP16 model, and with 2 × batch size compared to SmoothQuant. 0 450 900 1350 1800 20 40 60 80 Peak Memory(GB) Sequence Length NQKV SQ FP16 Out of Mmeory [PITH_FULL_IMAGE:figu…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

45 extracted references · 22 canonical work pages

  1. [1]

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhari- wal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al., Zhihang Cai et al.:Preprint submitted to ElsevierPage 7 of 9 NQKV

    work page

  2. [2]

    Yuan, J., Tang, R., Jiang, X., Hu, X., 2023. Llm for patient-trial matching: Privacy-aware data augmentation towards better perfor- mance and generalizability, in: American Medical Informatics As- sociation (AMIA) Annual Symposium

    work page 2023

  3. [3]

    Opt: Open pre-trained transformer language models

    Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X.V., et al., 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068

    arXiv 2022

  4. [4]

    Longlora: Efficient fine-tuning of long-context large language mod- els

    Chen, Y., Qian, S., Tang, H., Lai, X., Liu, Z., Han, S., Jia, J., 2023. Longlora: Efficient fine-tuning of long-context large language mod- els. arXiv preprint arXiv:2309.12307

    arXiv 2023

  5. [5]

    Rptq: Reorder-based post-training quan- tization for large language models

    Yuan,Z.,Niu,L.,Liu,J.,Liu,W.,Wang,X.,Shang,Y.,Sun,G.,Wu, Q., Wu, J., Wu, B., 2023. Rptq: Reorder-based post-training quan- tization for large language models. arXiv preprint arXiv:2304.01089

    arXiv 2023

  6. [6]

    Flexgen: High-throughput generative inference of large language models with a single gpu, in: International Conference on Machine Learning, PMLR

    Sheng,Y.,Zheng,L.,Yuan,B.,Li,Z.,Ryabinin,M.,Chen,B.,Liang, P., Ré, C., Stoica, I., Zhang, C., 2023. Flexgen: High-throughput generative inference of large language models with a single gpu, in: International Conference on Machine Learning, PMLR. pp. 31094– 31116

    work page 2023

  7. [8]

    Fast transformer decoding: One write-head is all you need

    Shazeer, N., 2019. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150

    arXiv 2019

  8. [9]

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints

    Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., Sanghai, S., 2023. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245

    arXiv 2023

Show all 45 references
  1. [10]

    H2o: Heavy-hitteroracle forefficientgenerativeinferenceoflargelanguagemodels

    Zhang, Z., Sheng, Y., Zhou, T., Chen, T., Zheng, L., Cai, R., Song, Z.,Tian,Y.,Ré,C.,Barrett,C.,etal.,2024. H2o: Heavy-hitteroracle forefficientgenerativeinferenceoflargelanguagemodels. Advances in Neural Information Processing Systems 36

    2024

  2. [11]

    arXiv preprint arXiv:2210.17323

    Frantar,E.,Ashkboos,S.,Hoefler,T.,Alistarh,D.,2022.Gptq: Accu- ratepost-trainingquantizationforgenerativepre-trainedtransformers. arXiv preprint arXiv:2210.17323

    2022 arXiv

  3. [12]

    Spqr: A sparse-quantized representation for near-lossless llm weight compression

    Dettmers, T., Svirschevski, R., Egiazarian, V., Kuznedelev, D., Fran- tar, E., Ashkboos, S., Borzunov, A., Hoefler, T., Alistarh, D., 2023. Spqr: A sparse-quantized representation for near-lossless llm weight compression. arXiv preprint arXiv:2306.03078

    2023 arXiv

  4. [13]

    Smoothquant: Accurate and efficient post-training quantization for large language models, in: International Conference on Machine Learning, PMLR

    Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., Han, S., 2023. Smoothquant: Accurate and efficient post-training quantization for large language models, in: International Conference on Machine Learning, PMLR. pp. 38087–38099

    2023

  5. [14]

    Yao, Z., Yazdani Aminabadi, R., Zhang, M., Wu, X., Li, C., He, Y.,

  6. [15]

    Dettmers, T., Lewis, M., Belkada, Y., Zettlemoyer, L., 2022. Gpt3. int8(): 8-bitmatrixmultiplicationfortransformersatscale.Advances in Neural Information Processing Systems 35, 30318–30332

    2022

  7. [16]

    The case for 4-bit precision: k- bit inference scaling laws, in: International Conference on Machine Learning, PMLR

    Dettmers, T., Zettlemoyer, L., 2023. The case for 4-bit precision: k- bit inference scaling laws, in: International Conference on Machine Learning, PMLR. pp. 7750–7774

    2023

  8. [17]

    Attention is all you need

    Vaswani,A.,Shazeer,N.,Parmar,N.,Uszkoreit,J.,Jones,L.,Gomez, A.N., Kaiser, Ł., Polosukhin, I., 2017. Attention is all you need. Ad- vances in neural information processing systems 30

    2017

  9. [18]

    Fp8 quantization: The power of the expo- nent.AdvancesinNeuralInformationProcessingSystems35,14651– 14662

    Kuzmin, A., Van Baalen, M., Ren, Y., Nagel, M., Peters, J., Blankevoort, T., 2022. Fp8 quantization: The power of the expo- nent.AdvancesinNeuralInformationProcessingSystems35,14651– 14662

    2022

  10. [19]

    Integer or floating point? new outlooks for low-bit quantization on large language models

    Zhang, Y., Zhao, L., Cao, S., Wang, W., Cao, T., Yang, F., Yang, M., Zhang, S., Xu, N., 2023. Integer or floating point? new outlooks for low-bit quantization on large language models. arXiv preprint arXiv:2305.12356

    2023 arXiv

  11. [20]

    Qlora: Efficient finetuning of quantized llms

    Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L., 2024. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems 36

    2024

  12. [21]

    A white paper on neural net- work quantization

    Nagel, M., Fournarakis, M., Amjad, R.A., Bondarenko, Y., Van Baalen, M., Blankevoort, T., 2021. A white paper on neural net- work quantization. arXiv preprint arXiv:2106.08295

    2021 arXiv

  13. [22]

    A survey on model compression for large language models

    Zhu, X., Li, J., Liu, Y., Ma, C., Wang, W., 2023. A survey on model compression for large language models. arXiv preprint arXiv:2308.07633

    2023 arXiv

  14. [23]

    arXiv preprint arXiv:1510.00149

    Han,S.,Mao,H.,Dally,W.J.,2015.Deepcompression: Compressing deepneuralnetworkswithpruning,trainedquantizationandhuffman coding. arXiv preprint arXiv:1510.00149

    2015 arXiv

  15. [24]

    Awq: Activation-awareweightquantizationforllmcompressionandaccel- eration

    Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., Han, S., 2023. Awq: Activation-awareweightquantizationforllmcompressionandaccel- eration. arXiv preprint arXiv:2306.00978

    2023 arXiv

  16. [25]

    Squeezellm: Dense-and-sparse quantization

    Kim, S., Hooper, C., Gholami, A., Dong, Z., Li, X., Shen, S., Ma- honey, M.W., Keutzer, K., 2023. Squeezellm: Dense-and-sparse quantization. arXiv preprint arXiv:2306.07629

    2023 arXiv

  17. [26]

    Qdrop: Randomly dropping quantization for extremely low-bit post-training quantiza- tion

    Wei, X., Gong, R., Li, Y., Liu, X., Yu, F., 2022. Qdrop: Randomly dropping quantization for extremely low-bit post-training quantiza- tion. arXiv preprint arXiv:2203.05740

    2022 arXiv

  18. [27]

    Wei, X., Zhang, Y., Li, Y., Zhang, X., Gong, R., Guo, J., Liu, X.,

  19. [28]

    Llm-qat: Data-free quan- tization aware training for large language models

    Liu,Z.,Oguz,B.,Zhao,C.,Chang,E.,Stock,P.,Mehdad,Y.,Shi,Y., Krishnamoorthi, R., Chandra, V., 2023. Llm-qat: Data-free quan- tization aware training for large language models. arXiv preprint arXiv:2305.17888

    2023 arXiv

  20. [29]

    Kivi: Plug-and-play 2bit kv cache quantization with streaming asymmetric quantization

    Liu, Z., Yuan, J., Jin, H., Zhong, S., Xu, Z., Braverman, V., Chen, B., Hu, X., . Kivi: Plug-and-play 2bit kv cache quantization with streaming asymmetric quantization

  21. [30]

    Efficientmemorymanagementfor largelanguagemodelservingwithpagedattention,in: Proceedingsof the 29th Symposium on Operating Systems Principles, pp

    Kwon,W.,Li,Z.,Zhuang,S.,Sheng,Y.,Zheng,L.,Yu,C.H.,Gonza- lez,J.,Zhang,H.,Stoica,I.,2023. Efficientmemorymanagementfor largelanguagemodelservingwithpagedattention,in: Proceedingsof the 29th Symposium on Operating Systems Principles, pp. 611–626

    2023

  22. [31]

    Advances in Neural Information Processing Systems 36

    Jin,Y.,Wu,C.F.,Brooks,D.,Wei,G.Y.,2024.𝑠 3: Increasinggpuuti- lization during generative inference for higher throughput. Advances in Neural Information Processing Systems 36

    2024

  23. [32]

    Effi- cient streaming language models with attention sinks

    Xiao, G., Tian, Y., Chen, B., Han, S., Lewis, M., 2023. Effi- cient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453

    2023 arXiv

  24. [33]

    Probability plotting methods for the analysis of data

    Gnanadesikan, R., Wilk, M.B., 1968. Probability plotting methods for the analysis of data. Biometrika 55, 1–17

    1968

  25. [34]

    Tests for normal distribution in goodness-of-fit techniques

    D’Agostino, R.B., Stephens, M., 1986. Tests for normal distribution in goodness-of-fit techniques. Marcel Decker

    1986

  26. [35]

    Thepile: An 800gb dataset of diverse text for language modeling

    Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang,J.,He,H.,Thite,A.,Nabeshima,N.,etal.,2020. Thepile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027

    2020 arXiv

  27. [36]

    Hugging face, in: Introduction to transformers for NLP: With the hugging face library and models to solve problems

    Jain, S.M., 2022. Hugging face, in: Introduction to transformers for NLP: With the hugging face library and models to solve problems. Springer, pp. 51–67

    2022

  28. [37]

    Piqa: Reasoning about physical commonsense in natural language, in: Proceedings of the AAAI conference on artificial intelligence, pp

    Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al., 2020. Piqa: Reasoning about physical commonsense in natural language, in: Proceedings of the AAAI conference on artificial intelligence, pp. 7432–7439

    2020

  29. [38]

    Wino- grande: An adversarial winograd schema challenge at scale

    Sakaguchi, K., Bras, R.L., Bhagavatula, C., Choi, Y., 2021. Wino- grande: An adversarial winograd schema challenge at scale. Com- munications of the ACM 64, 99–106

    2021

  30. [39]

    Hel- laswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830

    Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., Choi, Y., 2019. Hel- laswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830

    2019 arXiv

  31. [40]

    Thinkyouhavesolvedquestionanswering? try arc, the ai2 reasoning challenge

    Clark,P.,Cowhey,I.,Etzioni,O.,Khot,T.,Sabharwal,A.,Schoenick, C.,Tafjord,O.,2018. Thinkyouhavesolvedquestionanswering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457

    2018 arXiv

  32. [41]

    Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., Bowman, S.R.,

  33. [42]

    Bert: Pre- training of deep bidirectional transformers for language understand- ing

    Devlin, J., Chang, M.W., Lee, K., Toutanova, K., 2018. Bert: Pre- training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.04805 . Zhihang Cai received the B.S. degrees from the school of Xi’an Jiaotong University, Xi’an, China, in201...

    2018 arXiv

  34. [2018]

    arXiv preprint arXiv:1804.07461

    Glue: Amulti-taskbenchmarkandanalysisplatformfornatural language understanding. arXiv preprint arXiv:1804.07461 . Zhihang Cai et al.:Preprint submitted to ElsevierPage 8 of 9 NQKV

    arXiv

  35. [2020]

    Advances in neural information processing systems 33, 1877–1901

    Language models are few-shot learners. Advances in neural information processing systems 33, 1877–1901

    1901

  36. [2022]

    Advances in Neural Information Pro- cessing Systems 35, 27168–27183

    Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. Advances in Neural Information Pro- cessing Systems 35, 27168–27183

  37. [2023]

    arXivpreprint arXiv:2304.09145

    Outliersuppression+: Accuratequantizationoflargelanguage modelsbyequivalentandoptimalshiftingandscaling. arXivpreprint arXiv:2304.09145

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.