Pith. sign in

REVIEW 2 major objections 4 minor 3 cited by

FPTQuant: Function-Preserving Transforms for LLM Quantization

T0 review · 2 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FPTQuant claims static INT4 quantization with up to 3.9× speed-up over FP16 by inserting transforms that reshape activations while provably preserving the transformer's output.

desk verdict Solid FPT paper: correct new transforms, fair experiments, but the no-bias assumption and single-block speedup need flagging. read the letter →

arxiv 2506.04985 v2 pith:ONCLVBIP submitted 2025-06-05 cs.LG

Boris van Breugel , Yelysei Bondarenko , Paul Whatmough , Markus Nagel This is my paper
classification cs.LG
keywords LLMquantizationfunction-preservingtransformsstaticINT4inferenceactivationoutliersRoPEequivarianceresidualscalingstudent-teachertrainingpost-training
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FPTQuant aims to make 4-bit (INT4) inference on large language models both fast and accurate by reshaping the numbers that flow through the transformer before they are quantized. The key idea is that several operations inside a transformer—rotary position embeddings, per-head attention value mixing, and the elementwise gate in the feed-forward block—leave room to insert invertible transforms that change the distribution of activations without changing the network's output. These function-preserving transforms can mostly be folded into existing weight matrices, so inference adds almost no extra work and needs no custom kernels. If the claim holds, LLMs can run on static INT4 quantization with up to 3.9× speed-up over FP16 while matching the accuracy of prior methods that are noticeably slower.

What carries the argument

The load-bearing objects are the four transforms. The pre-RoPE transform applies a block-diagonal matrix of scaled 2×2 rotations to the key weights and its near-inverse to the query weights before rotary embedding; because 2×2 rotations commute with RoPE's per-block rotations, the attention dot product is provably unchanged (Theorem 3.1), yet the queries and keys can be reshaped to quantize better. The value transform inserts an arbitrary invertible matrix per attention head between the value and output projections, exploiting the fact that the attention softmax-product sums over tokens, not over head dimension. The pseudodynamic residual scaling moves the existing RMSNorm (the per-token normalizer) so it also normalizes the residual stream, then rescales each attention and MLP output by the same per-token factor computed recursively from the normalized residuals. The up-projection scaler uses the fact that a diagonal matrix commutes with the elementwise gate product in SwiGLU, so scaling the up projection can be absorbed into the down projection weights. Together these transforms give the quantization grid a much friendlier activation distribution at near-zero inference cost.

What would settle it

Take any transformer whose out or down projection layers contain bias terms (or whose normalization is not scale-invariant), apply only the pseudodynamic residual scaling in full precision, and compare the logits with the original model: any difference beyond floating-point round-off would refute the function-preservation claim for that architecture. A cheaper check is a survey of open-weight model families counting whether any use bias in those layers, which would falsify the paper's stated premise directly.

Watch

Extended reading notes

Core claim

The paper's central claim is that a small set of function-preserving transforms—a pre-RoPE scale-and-rotate transform for queries and keys, per-head invertible transforms for values, a per-channel scaler for the up/down projection pair, and a per-token residual scaler computed from the existing RMSNorms—lets a pretrained transformer be quantized to static INT4 without changing what the model computes. Function preservation is shown exactly: Theorem 3.1 proves that the pre-RoPE transform, built from scaled 2×2 rotations that commute with RoPE's block rotations, leaves attention logits unchanged, and Section 3.1.3 shows the residual scaler is equivalent to normalizing the residual and rescaling each block's output by the same per-token factors. After these transforms are merged into the weights, they are trained locally to shrink outliers and then end-to-end with a student-teacher loss so the quantized model matches the full-precision one. The empirical claim is a speed-accuracy frontier: up to 3.9× faster prefill than FP16, within a few percent of the INT4 upper bound, and accuracy on par with or better than most prior transforms, trailing only a method that is up to 29% slower.

Load-bearing premise

The residual-scaling step keeps the model's function intact only if the out and down projection layers have no bias and the block is invariant to per-token scaling of its input; the paper asserts the no-bias fact without a survey and checks it only on the Llama and Qwen models it tests.

Editorial extensions

If this is right

  • Static INT4 quantization becomes practical without custom kernels or online per-token rescaling, so the speed-up transfers to hardware that lacks dynamic-quantization support.
  • Because the transforms are function-preserving and merged into weights, the inference graph is unchanged from the full-precision model apart from the cheap residual scales.
  • Student-teacher training gives better generalization than next-token prediction for learned transforms, since it resists overfitting to the training text.
  • The per-head value transform is more expressive than prior single shared transforms at the same (zero) inference cost, so it should be preferred for value and output-projection quantization.
  • The measured prefill speed-up grows with model size and batch size, approaching within 5-6% of the INT4 upper bound.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The function-preservation guarantee is architectural: any future transformer that keeps the same equivariance structure (no bias in out/down projections, rotation-compatible RoPE) would inherit the transforms unchanged, making this a template for new model families rather than a per-model fix.
  • The residual-scaling trick points to a broader design rule: wherever a pretrained block has an input normalizer and an unnormalized residual, the normalizer can be moved onto the residual and compensated inside the block, which could also help goals such as stabilizing low-precision training.
  • A testable extension would apply the transforms to instruction-tuned or Mixture-of-Experts models, whose outlier patterns may require per-expert transforms; the paper does not report results there.
  • The speed-ups come from single-block prefill benchmarks, so end-to-end token-generation gains could differ: decoding is more memory-bound and spends relatively more time on parts the method does not accelerate.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. FPTQuant proposes four function-preserving transforms for LLM quantization: a mergeable pre-RoPE transform for queries and keys, a mergeable per-head value transform, a mergeable scaling transform for the up projection, and a pseudodynamic per-token residual scaling transform. The transforms exploit equivariances in standard transformer blocks so that, in the absence of quantization, the network function is unchanged; the transform parameters and the quantization grid are then optimized locally and end-to-end with a student-teacher objective. Experiments on Llama-2-7B, Llama-3-8B, Llama-3.2-3B, and Qwen2.5 report competitive WikiText-2 perplexity and zero-shot accuracy under static and dynamic INT4 settings, with block-level speedups of up to 3.9× over FP16.

Significance. If the function-preserving claims hold, FPTQuant is a useful contribution to LLM quantization: it extends the FPT toolbox with mergeable transforms for query/key/value projections and a cheap residual scaling scheme, and it demonstrates that static INT4 quantization can be competitive with more expensive dynamic schemes. Strengths include the clear mathematical derivations, the same-training-budget comparisons across methods, and detailed ablations isolating the contribution of each transform (Tv, Tk, Tu, and the residual scaler). The paper is also transparent about compute costs and limitations. The main risk is that one of the central correctness arguments depends on an empirically asserted but unverified architectural assumption.

major comments (2)
  1. [§3.1.3, footnote 2] The pseudodynamic residual scaling is exactly function-preserving only if the per-token scale can be pushed through out_proj and down_proj, and this requires those linear layers to be bias-free. The derivation in §3.1.3 relies on the statement that 'linear layers without bias' commute with a batch/sequence-dimension scaler, and footnote 2 asserts 'We have not found any modern LLMs that use bias for the out and down projection layers.' This universal claim is not supported by a survey or citation, and it is load-bearing: if a target model has a bias term in out_proj or down_proj, pushing the scale before that layer gives (S⊙x)W + b instead of S⊙(xW + b), so the transformed network is not function-preserving even before quantization. The paper should either verify the no-bias property across a broader set of architectures or explicitly restrict the correctness claim to bias-free layers and provide a bias-correcting variant.
  2. [§4.1, Figure 2, Abstract] The headline speedup ('up to 3.9× over FP') is measured on a single transformer block, not on end-to-end prefill of the model. The paper states this clearly in §4.1, but it then compares these block-level numbers with 'prior reported speedups of QuaRot and FlatQuant' without establishing that those prior numbers use the same protocol. Whole-model prefill includes unquantized embedding, LM head, RMSNorm, RoPE, and memory-bound operations, so the block-level speedup can overstate the true inference gain, especially at small batch sizes. Please report end-to-end speedups or, if that is not feasible, explicitly qualify the abstract and Section 4.1 claims as single-transformer-block speedups and compare only against same-protocol baselines.
minor comments (4)
  1. [Abstract and Section 1] The second paragraph of the abstract says 'three novel' transforms, while the full abstract and Section 1 list four transforms; the body actually contains four novel FPTs. Please unify the count.
  2. [Theorem 3.1, footnote 5] Footnote 5 states that for single-headed attention ¯Tk = T−1k, but with the definitions in Theorem 3.1 this is not correct; the identity used in the proof is ¯Tk Tk^⊤ = I. Please correct the footnote to avoid confusing readers.
  3. [Appendix F.2.2 and Table 13] There are several typographical errors: 'raining' should be 'training', 'alter's' should be 'alters', and Table 13's caption has 'underform' instead of 'underperform'. A copyedit pass is needed.
  4. [§3.1.3] The statement that the LM head starts with an RMSNorm and therefore automatically removes the final scale S_N is model-specific; please state the condition under which Step 3 can be omitted, or explain that a final division by S_N is required when the head is not scale-invariant.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: FPTQuant's transforms are proven function-preserving from stated equivariances, and accuracy and speed claims are benchmarked against external baselines.

full rationale

FPTQuant's central claim is that inserting transforms Tk/Tv/Tu/Sn does not change the full-precision output. This is derived, not assumed: Theorem 3.1 (Appendix C) proves the pre-RoPE transform preserves attention scores using commutation of block 2x2 rotations with RoPE and the identity \bar Tk Tk^T = I; Section 3.1.3 shows residual scaling is function-preserving via the recursion S_n = S_{n-1}/||\tilde Z_{n-1}||_R and rescaling block outputs by S_n; Sections 3.1.2 and 3.1.4 give invertible per-head value transforms and mergeable up/down scaler transforms. The end-to-end objective (Eq. 11) fits the quantized model to the full-precision teacher and is then evaluated on Wikitext-2 and zero-shot tasks, so the accuracy and speed results are not forced by construction. Citations to the authors' prior work (e.g., [30] for the L4 norm, [1,4] for outlier observations) are background or hyperparameter justifications and are not load-bearing. The only unsupported universality claim is footnote 2's no-bias assumption for out/down projections; this is a stated condition of the derivation and a generality and correctness risk, not a circular step. No equation reduces to its own input and no fitted parameter is renamed as a prediction.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim is an empirical method plus equivalence proofs. The proofs rest on standard linear algebra and on domain assumptions about canonical transformer structure (pre-norm RMSNorm, no bias in out/down projections, even head dimension for RoPE). The transforms and quantization grids contain many learned parameters, which are fit to data during local and end-to-end optimization; these are the method's free parameters rather than constants in a derivation. No new physical entities are introduced.

free parameters (5)
  • Pre-RoPE transform parameters (Tk) = learned, not reported
    Per-head scale vector s^(h) and set of 2x2 rotation matrices R_n^(h) for each key head; optimized locally and end-to-end to reduce quantization error.
  • Value transform matrices (Tv) = learned, not reported
    Per-head invertible d_head x d_head matrices, merged into Wv and Wo; trained to reduce quantization error.
  • Up-projection scaler (Tu) = learned, not reported
    Per-channel scaling vector applied to up projection and merged into down projection; optimized to reduce down-projection input outliers.
  • Quantization grid scale and offset = learned, not reported
    Scale and offset for each quantized tensor are learnable parameters, initialized with L3 range setting and updated end-to-end.
  • Hyperparameters (p=4, L3 range setting, learning rate, steps) = p=4, L3, lr=1e-3 (2e-4 dynamic), 1024 steps, batch 16, seq 2048
    Chosen by hand or via validation perplexity; they affect the reported accuracy but are standard ML choices rather than parameters of the FPT construction.
assumptions (5)
  • domain assumption RMSNorm and LayerNorm are invariant to per-token positive scaling of the input
    Needed for the pseudodynamic residual scaling (Section 3.1.3) to preserve the block output; holds for canonical pre-norm transformers.
  • domain assumption Out and down projection layers have no bias
    Assumed in Section 3.1.3 and footnote 2 to allow per-token scalers to be pushed into these linear layers; paper states no modern LLMs use bias there, without a systematic survey.
  • domain assumption Head dimension d_head is even and RoPE is block-diagonal with 2x2 rotation blocks
    Required for the pre-RoPE transform construction in Theorem 3.1; true for Llama-family models but not universal.
  • standard math Matrix multiplication, linear layers without bias, and batched matmul commute with per-token scaling on the batch/sequence dimension
    Used throughout Section 3.1.3 to push the residual scaling Sn into the block outputs.
  • domain assumption Grouped-query attention bookkeeping: transforms repeat across query heads that share a key head
    Used in Eqs. (1)-(6) to define Tk and Tv for GQA architectures.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FPTQuant: Function-Preserving Transforms for LLM Quantization." pith.science (2026). https://pith.science/paper/ONCLVBIP

@misc{pith2026250604985,
  author       = {Pith},
  title        = {Pith review of: FPTQuant: Function-Preserving Transforms for LLM Quantization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ONCLVBIP}},
  note         = {Machine review of arXiv:2506.04985}
}
read the original abstract

Large language models (LLMs) require substantial compute, and thus energy, at inference time. While quantizing weights and activations is effective at improving efficiency, naive quantization of LLMs can significantly degrade performance due to large magnitude outliers. This paper describes FPTQuant, which introduces three novel, lightweight, and expressive function-preserving transforms (FPTs) to facilitate quantization of transformers: (1) a mergeable pre-RoPE transform for queries and keys, (2) a mergeable transform for values, and (3) a cheap, dynamic per-token scaling transform. By leveraging the equivariances and independencies inherent to canonical transformer operation, we designed these FPTs to maintain the model's function while shaping the intermediate activation distributions to be more quantization friendly. FPTQuant requires no custom kernels and adds virtually no overhead during inference. The FPTs are trained both locally to reduce outliers, and end-to-end such that the outputs of the quantized and full-precision models match. FPTQuant enables static INT4 quantization with minimal overhead and shows SOTA speed-up of up to 3.9X over FP. Empirically, FPTQuant has an excellent accuracy-speed trade-off -- it is performing on par or exceeding most prior work and only shows slightly lower accuracy compared to a method that is up to 29% slower.

Figures

Figures reproduced from arXiv: 2506.04985 by the authors.

Figure 1
Figure 1. FPTQuant. FPTQuant consists of 6 transform types. (Tk, T¯ k) is a scale-and-rotate transform merged into the query and key weights; (Tv, T¯ v) consists of invertible matrices per head merged into value and output weights; (Tu, T −1 u ) is a per-channel scaler merged into up and down projection weights; transforms {Sn} N n=1 (N = 2×number of transformer blocks for typical LLMs) are per-token scalers applied to the re… view at source ↗
Figure 2
Figure 2. Static INT4 prefill speedup of FPTQuant on a single transformer block of LLaMA models across different sizes (3B, 7B, 8B, 13B, and 70B), and batch sizes (1 and 16). We use a sequence length of 1024. Baselines. We compare FPTQuant against the original floating point model (FP), PTQ using rounding-to-nearest (RTN), RTN with optimizing the ranges (RTN-opt), QuaRot [7], SpinQuant [8] and FlatQuant [47]. Training set-up.… view at source ↗
Figure 4
Figure 4. Local optimization (Section 3.2.1) leads to more stable and faster end-to-end training. We train FPTQuant on Llama 3.2 3B instruct with and without local optimization, for different number of end-to-end training steps. F.2.2 Student-teacher training. We compare the value of end-to-end training in a student teacher fashion (E2E[ST]), versus the original next-token prediction loss (E2E[label]) used in e.g. SpinQuant [… view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Dynamic INT4 prefill speedup of FPTQuant on a single transformer block of LLaMA models across different sizes (3B, 7B, 8B, 13B, and 70B), and batch sizes (1 and 16). We use a sequence length of 1024. that did not make it in the final paper and hyperparameter tuning we …

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. RotateAttention: RoPE-Aware Rotation and Range Rectification for INT4 Quantized Attention in Video Generation

    cs.CV 2026-07 conditional novelty 6.5 of 10

    RoPE-aware orthogonal rotations plus full-range INT4 quantization of attention probabilities deliver mixed-precision FlashAttention that matches FP16 video quality at 1.5–1.68× end-to-end speedup.

  2. Leech Lattice Vector Quantization for Efficient LLM Compression

    cs.LG 2026-03 conditional novelty 6.5 of 10

    Leech-lattice vector quantization with codebook-free indexing and shape-gain search reaches state-of-the-art ~2-bit post-training LLM compression and reduces reliance on Hadamard rotations.

  3. Efficient Reasoning on the Edge

    cs.LG 2026-03 accept novelty 5.5 of 10

    LoRA adapters, budget-forced GRPO, dynamic switching, parallel verification and FPTQuant enable practical chain-of-thought reasoning on quantized Qwen2.5-7B for edge devices.

Reference graph

Works this paper leans on

71 extracted references · 32 canonical work pages · cited by 3 Pith papers

  1. [1]

    Understanding and overcoming the challenges of efficient transformer quantization

    Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Understanding and overcoming the challenges of efficient transformer quantization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7947–7969, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10...

    work page 2021

  2. [2]

    Bert busters: Outlier dimensions that disrupt transformers

    Olga Kovaleva, Saurabh Kulshreshtha, Anna Rogers, and Anna Rumshisky. Bert busters: Outlier dimensions that disrupt transformers. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3392–3405, 2021

    work page 2021

  3. [3]

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. In Advances in Neural Information Processing Systems, 2022

    2022

  4. [4]

    Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing

    Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing. Advances in Neural Information Processing Systems, 2023. URL https://arxiv.org/abs/2306.12929v2

    arXiv 2023

  5. [5]

    Massive activations in large language models

    Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations in large language models. arXiv preprint arXiv:2402.17762, 2024

    arXiv 2024

  6. [6]

    SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models, March 2024

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models, March 2024. URL http://arxiv.org/abs/2211.10438. arXiv:2211.10438 [cs]

    arXiv 2024

  7. [7]

    Croci, Bo Li, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman

    Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs, March 2024. URL https://arxiv.org/abs/2404.00456v1

    arXiv 2024

  8. [8]

    SpinQuant: LLM quan- tization with learned rotations, May 2024

    Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Kr- ishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. SpinQuant: LLM quan- tization with learned rotations, May 2024. URL https://arxiv.org/abs/2405.16406v2

    arXiv 2024

Show all 71 references
  1. [9]

    Quantizing deep convolutional networks for efficient inference: A whitepaper

    Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv preprint arXiv:1806.08342, 2018

    2018 arXiv

  2. [10]

    A white paper on neural network quantization

    Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart Van Baalen, and Tijmen Blankevoort. A white paper on neural network quantization. arXiv preprint arXiv:2106.08295, 2021

    2021 arXiv

  3. [11]

    Post-training 4-bit quantization of convolution networks for rapid-deployment

    Ron Banner, Yury Nahshan, Elad Hoffer, and Daniel Soudry. Post-training 4-bit quantization of convolution networks for rapid-deployment. arXiv preprint arXiv:1810.05723, 2018

    2018 arXiv

  4. [12]

    Zeroq: A novel zero shot quantization framework

    Yaohui Cai, Zhewei Yao, Zhen Dong, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Zeroq: A novel zero shot quantization framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13169–13178, 2020

    2020

  5. [13]

    Low-bit quantization of neural networks for efficient inference

    Yoni Choukroun, Eli Kravchik, Fan Yang, and Pavel Kisilev. Low-bit quantization of neural networks for efficient inference. In ICCV Workshops, pages 3009–3018, 2019

    2019

  6. [14]

    Improving post training neural quantization: Layer-wise calibration and integer programming

    Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry. Improving post training neural quantization: Layer-wise calibration and integer programming. arXiv preprint arXiv:2006.10518, 2020

    2006 arXiv

  7. [15]

    Same, same but different: Recovering neural network quantization error through weight factorization

    Eldad Meller, Alexander Finkelstein, Uri Almog, and Mark Grobman. Same, same but different: Recovering neural network quantization error through weight factorization. In International Conference on Machine Learning, pages 4486–4495. PMLR, 2019

    2019

  8. [16]

    Improving neural net- work quantization without retraining using outlier channel splitting

    Ritchie Zhao, Yuwei Hu, Jordan Dotzel, Chris De Sa, and Zhiru Zhang. Improving neural net- work quantization without retraining using outlier channel splitting. In International conference on machine learning, pages 7543–7552. PMLR, 2019. 10

    2019

  9. [17]

    Data-free quantization through weight equalization and bias correction

    Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. Data-free quantization through weight equalization and bias correction. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019

    2019

  10. [18]

    Up or Down? Adaptive Rounding for Post-Training Quantization, April 2020

    Markus Nagel, Rana Ali Amjad, Mart van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or Down? Adaptive Rounding for Post-Training Quantization, April 2020. URL https: //arxiv.org/abs/2004.10568v2

    2020 arXiv

  11. [19]

    Brecq: Pushing the limit of post-training quantization by block reconstruction

    Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Brecq: Pushing the limit of post-training quantization by block reconstruction. arXiv preprint arXiv:2102.05426, 2021

    2021 arXiv

  12. [20]

    Deep learning with limited numerical precision

    Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In International conference on machine learning , pages 1737–1746. PMLR, 2015

    2015

  13. [21]

    Quantization and training of neural networks for efficient integer-arithmetic-only inference

    Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proceedings of the IEEE Conference on Computer Vision an...

    2018

  14. [22]

    Esser, Jeffrey L

    Steven K. Esser, Jeffrey L. McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S. Modha. Learned step size quantization. In International Conference on Learning Representations (ICLR), 2020

    2020

  15. [23]

    Lsq+: Improving low-bit quantization through learnable offsets and better initialization

    Yash Bhalgat, Jinwon Lee, Markus Nagel, Tijmen Blankevoort, and Nojun Kwak. Lsq+: Improving low-bit quantization through learnable offsets and better initialization. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2020

    2020

  16. [24]

    Overcoming oscillations in quantization-aware training

    Markus Nagel, Marios Fournarakis, Yelysei Bondarenko, and Tijmen Blankevoort. Overcoming oscillations in quantization-aware training. In International Conference on Machine Learning, pages 16318–16330. PMLR, 2022

    2022

  17. [25]

    LLM-QAT: Data-Free Quan- tization Aware Training for Large Language Models

    Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. LLM-QAT: Data-Free Quan- tization Aware Training for Large Language Models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors...

    2024 doi

  18. [26]

    Bitdistiller: Unleashing the potential of sub-4-bit llms via self-distillation

    Dayou Du, Yijia Zhang, Shijie Cao, Jiaqi Guo, Ting Cao, Xiaowen Chu, and Ningyi Xu. Bitdistiller: Unleashing the potential of sub-4-bit llms via self-distillation. arXiv preprint arXiv:2402.10631, 2024

    2024 arXiv

  19. [27]

    Efficientqat: Efficient quantization-aware training for large language models

    Mengzhao Chen, Wenqi Shao, Peng Xu, Jiahao Wang, Peng Gao, Kaipeng Zhang, and Ping Luo. Efficientqat: Efficient quantization-aware training for large language models. arXiv preprint arXiv:2407.11062, 2024

    2024 arXiv

  20. [28]

    Qlora: Efficient finetuning of quantized llms

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024

    2024

  21. [29]

    Qa-lora: Quantization-aware low-rank adaptation of large language models

    Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhensu Chen, Xiaopeng Zhang, and Qi Tian. Qa-lora: Quantization-aware low-rank adaptation of large language models. arXiv preprint arXiv:2309.14717, 2023

    2023 arXiv

  22. [30]

    Low-Rank Quantization- Aware Training for LLMs, September 2024

    Yelysei Bondarenko, Riccardo Del Chiaro, and Markus Nagel. Low-Rank Quantization- Aware Training for LLMs, September 2024. URL http://arxiv.org/abs/2406.06385. arXiv:2406.06385. 11

    2024 arXiv

  23. [31]

    Paretoq: Scaling laws in extremely low-bit llm quantization, 2025

    Zechun Liu, Changsheng Zhao, Hanxian Huang, Sijia Chen, Jing Zhang, Jiawei Zhao, Scott Roy, Lisa Jin, Yunyang Xiong, Yangyang Shi, Lin Xiao, Yuandong Tian, Bilge Soran, Raghuraman Krishnamoorthi, Tijmen Blankevoort, and Vikas Chandra. Paretoq: Scaling laws in extremely low-bit...

    2025

  24. [32]

    Gptq: Accurate post-training quantization for generative pre-trained transformers

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323, 2022

    2022 arXiv

  25. [33]

    Spqr: A sparse-quantized representation for near-lossless llm weight compression

    Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. Spqr: A sparse-quantized representation for near-lossless llm weight compression. arXiv preprint arXiv:2306.03078, 2023

    2023 arXiv

  26. [34]

    Awq: Activation-aware weight quantization for llm compression and acceleration

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq: Activation-aware weight quantization for llm compression and acceleration. arXiv preprint arXiv:2306.00978, 2023

    2023 arXiv

  27. [35]

    Owq: Outlier- aware weight quantization for efficient fine-tuning and inference of large language models

    Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim, and Eunhyeok Park. Owq: Outlier- aware weight quantization for efficient fine-tuning and inference of large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 13355–13364, 2024

    2024

  28. [36]

    Squeezellm: Dense-and-sparse quantization

    Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. Squeezellm: Dense-and-sparse quantization. arXiv preprint arXiv:2306.07629, 2023

    2023 arXiv

  29. [37]

    Slim-llm: Salience-driven mixed-precision quantization for large language models

    Wei Huang, Haotong Qin, Yangdong Liu, Yawei Li, Xianglong Liu, Luca Benini, Michele Magno, and Xiaojuan Qi. Slim-llm: Salience-driven mixed-precision quantization for large language models. arXiv preprint arXiv:2405.14917, 2024

    2024 arXiv

  30. [38]

    Extreme compression of large language models via additive quantization

    Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh. Extreme compression of large language models via additive quantization. arXiv preprint arXiv:2401.06118, 2024

    2024 arXiv

  31. [39]

    A frustratingly easy post-training quantization scheme for llms

    Yongkweon Jeon, Chungman Lee, Kyungphil Park, and Ho-young Kim. A frustratingly easy post-training quantization scheme for llms. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14446–14461, 2023

    2023

  32. [40]

    Flexround: Learnable rounding based on element-wise division for post-training quantization

    Jung Hyun Lee, Jeonghoon Kim, Se Jung Kwon, and Dongsoo Lee. Flexround: Learnable rounding based on element-wise division for post-training quantization. In International Conference on Machine Learning, pages 18913–18939. PMLR, 2023

    2023

  33. [41]

    Long- range zero-shot generative deep network quantization

    Yan Luo, Yangcheng Gao, Zhao Zhang, Jicong Fan, Haijun Zhang, and Mingliang Xu. Long- range zero-shot generative deep network quantization. Neural Networks, 166:683–691, 2023

    2023

  34. [42]

    Quip: 2-bit quantiza- tion of large language models with guarantees

    Jerry Chee, Yaohui Cai, V olodymyr Kuleshov, and Christopher M De Sa. Quip: 2-bit quantiza- tion of large language models with guarantees. Advances in Neural Information Processing Systems, 36, 2024

    2024

  35. [43]

    Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling, October 2023

    Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xianglong Liu. Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling, October 2023. URL http://arxiv.org/abs/ 2304.09145. arXiv:23...

    2023 arXiv

  36. [44]

    OmniQuant: Omnidirectionally Calibrated Quantiza- tion for Large Language Models, March 2024

    Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. OmniQuant: Omnidirectionally Calibrated Quantiza- tion for Large Language Models, March 2024. URL http://arxiv.org/abs/2308.13137. arXiv:2308.13137 [cs]

    2024 arXiv

  37. [45]

    QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks, February

    Albert Tseng, Jerry Chee, Qingyao Sun, V olodymyr Kuleshov, and Christopher De Sa. QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks, February

  38. [46]

    DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs

    Haokun Lin, Haobo Xu, Yichen Wu, Jingzhi Cui, Yingtao Zhang, Linzhan Mou, Linqi Song, Zhenan Sun, and Ying Wei. DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs. In Advances in Neural Information Processing Systems . arXiv, November 2024. do...

    2024 doi arXiv

  39. [47]

    FlatQuant: Flatness Matters for LLM Quantization, October 2024

    Yuxuan Sun, Ruikang Liu, Haoli Bai, Han Bao, Kang Zhao, Yuening Li, Jiaxin Hu, Xianzhi Yu, Lu Hou, Chun Yuan, Xin Jiang, Wulong Liu, and Jun Yao. FlatQuant: Flatness Matters for LLM Quantization, October 2024. URL http://arxiv.org/abs/2410.09426. arXiv:2410.09426

    2024 arXiv

  40. [48]

    Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman

    Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. SliceGPT: Compress Large Language Models by Deleting Rows and Columns, February 2024. URL http://arxiv.org/abs/2401.15024. arXiv:2401.15024

    2024 arXiv

  41. [49]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024

    2024

  42. [50]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  43. [51]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  44. [52]

    Pointer sentinel mixture models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017

    2017

  45. [53]

    PIQA: Reasoning about Physical Commonsense in Natural Language

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: Reasoning about Physical Commonsense in Natural Language. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):7432–7439, April 2020. ISSN 2374-3468. doi: 10.1609/ aaai.v34i05.6239...

    2020

  46. [54]

    WinoGrande: an adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: an adversarial winograd schema challenge at scale. Commun. ACM, 64(9):99–106, August

  47. [55]

    HellaSwag: Can a Machine Really Finish Your Sentence?, May 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a Machine Really Finish Your Sentence?, May 2019. URL http://arxiv.org/abs/1905. 07830. arXiv:1905.07830 [cs]

    2019 arXiv

  48. [56]

    Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge, March 2018

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge, March 2018. URL http://arxiv.org/abs/1803.05457. arXiv:1803.05457 [cs]

    2018 arXiv

  49. [57]

    The lambada dataset: Word prediction requiring a broad discourse context

    Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc-Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context. InProceedings of the 54th Annual Meeting of th...

    2016

  50. [58]

    Working with Quantized Types — NVIDIA TensorRT Doc- umentation

    Nvidia Corporation. Working with Quantized Types — NVIDIA TensorRT Doc- umentation. URL https://docs.nvidia.com/deeplearning/tensorrt/latest/ inference-library/work-quantized-types.html#dynamic-quantization . Version 10.10.0

  51. [59]

    Quantization — PyTorch AO documentation

    PyTorch. Quantization — PyTorch AO documentation. URL https://docs.pytorch.org/ docs/stable/quantization.html. Version 2.7.0

  52. [60]

    AI Engine Direct SDK documentation

    Qualcomm. AI Engine Direct SDK documentation. URL https://docs.qualcomm.com/ bundle/publicresource/topics/80-63442-50/quantization.html . 13

  53. [61]

    TensorRT operators documentation: DynamicQuantize not supported on DLA

    Nvidia. TensorRT operators documentation: DynamicQuantize not supported on DLA. URL https://docs.nvidia.com/deeplearning/tensorrt/10.10.0/_static/ operators/DynamicQuantize.html. Version 10.10.0

  54. [62]

    fast-hadamard-transform

    Dao AI Lab. fast-hadamard-transform. URL https://github.com/Dao-AILab/ fast-hadamard-transform. 14 A Detailed transforms comparison In Table 5 we include the representation and theoretical cost of existing transforms. In Table 6 we review existing works, the transforms they us...

  55. [65]

    double-packed

    that for 4-bit quantization of queries and keys, FPTQuant underperforms baselines due to the more restrictive FPT and less mixing across channels. At W4A8, we find Tk performs on par with baseline FPTs. This experiment clearly shows the expressivity and cost trade-off, P2 vs P...

  56. [66]

    Evaluate quantization error per quantizer placement (e.g

    Explore. Evaluate quantization error per quantizer placement (e.g. Appendix E)

  57. [67]

    Based on step 1, choose which FPTs to add: (a) Attention and FFN input

    Choose transforms. Based on step 1, choose which FPTs to add: (a) Attention and FFN input. R1 (SpinQuant) and Pa, Pd (FlatQuant) are similar trans- forms. The first is shared across all layers of the model, whilst FlatQuant’s are not. However, an orthogonal matrixR1 has about ...

  58. [68]

    Initialize transforms, e.g

    Initialize FPTs. Initialize transforms, e.g. as a Welsh-Hadamard matrix or identity

  59. [69]

    Locally optimizing transforms improves performance and reduces training time, whilst incurring very little cost (Appendix F.2.1)

    Locally optimize FPTs. Locally optimizing transforms improves performance and reduces training time, whilst incurring very little cost (Appendix F.2.1)

  60. [70]

    Set the initial quantization grid, e.g

    Set quantization range. Set the initial quantization grid, e.g. using L3 minimization (Appendix D). It is important to only set the grid now, so that initialized FPTs can be taken into account when choosing this grid

  61. [71]

    Train the FPTs and quantization grid end-to-end, with the unquantized outputs as target

    Train end-to-end. Train the FPTs and quantization grid end-to-end, with the unquantized outputs as target. 25

  62. [2021]

    doi: 10.1145/3474381

    ISSN 0001-0782. doi: 10.1145/3474381. URL https://dl.acm.org/doi/10.1145/ 3474381

    doi

  63. [2024]

    URL https://arxiv.org/abs/2402.04396v2. 12

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.