REVIEW 3 major objections 5 minor 20 references
Low-Rank Correction for Quantized LLMs
T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Adding full-precision low-rank matrices that read the unquantized activations reduces the W4A4 accuracy gap by more than half at 10% rank and closes it at 30%.
desk verdict Genuinely new algorithm for activation quantization, but the main comparison is not bitrate-matched, so the headline accuracy claims are overstated. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the pair of low-rank factors $(U,V)$ together with the decoupled forward computation $\widehat{W}Q_a(X) + UV^\top X$. The factors are chosen not from a residual SVD of the weight error but from the eigen-decomposition of $\Sigma_{1}+\Sigma_{2}-\Sigma_{3}$, where $\Sigma_{1}=WXX^\top W^\top$, $\Sigma_{2}=\widehat{W}YX^\top(XX^\top)^{-1}XY^\top\widehat{W}^\top$, and $\Sigma_{3}=\widehat{W}YX^\top W^\top + WXY^\top\widehat{W}^\top$, with $Y=Q_a(X)$; these are computed from calibration covariances $XX^\top$, $YY^\top$, and $XY^\top$. The $\widehat{W}$ update is equivalent to quantizing $\widetilde{W}=(W-UV^\top)XY^\top(YY^\top)^{-1}$ with any standard layer-wise quantizer. The covariance coupling is what lets the correction target the actual activation error rather than a generic low-rank approximation.
What would settle it
Quantize an LLM with LRC at a 30% rank budget using a calibration set from one domain, then evaluate on tasks from a domain with very different activation statistics, such as code or non-English text; if the average accuracy gap to the FP16 model is substantially larger than the paper's near-lossless result, the calibration-representativeness assumption is falsified.
Extended reading notes
Core claim
The discovery is that activation-quantization error, not weight-quantization error, is the bottleneck at W4A4, and that this error has a low-rank structure that can be corrected by full-precision factors. For each layer the method minimizes $\|W X - \widehat{W} Q_a(X) - U V^\top X\|_2^2$, where $W$ is the original weight, $X$ the calibration activations, $Q_a(X)$ their 4-bit quantized version, $\widehat{W}$ the quantized weight, and $U,V$ the low-rank correction applied to unquantized $X$. The weight update is re-expressed as a standard layer-wise quantization problem on a transformed target, so any existing weight quantizer can be reused; the low-rank update has a closed form given by the top-$k$ eigenvectors of a matrix built from input and output covariances. Running this alternating scheme once or a few times yields, at 10% rank, more than a 50% reduction in the accuracy gap to the FP16 model and, at 30% rank, a completely closed gap on the evaluated tasks.
Load-bearing premise
The method assumes that the 128 text sequences used to calibrate the activation statistics represent the inputs the model will see on every downstream task; if the calibration distribution is unrepresentative, the low-rank correction can overfit and the reported accuracy gains will shrink.
Editorial extensions
If this is right
- At a 10% rank budget, LRC recovers more than half of the accuracy lost when an LLM is quantized to W4A4, improving on the W4A4 baseline by several percentage points on average over six tasks.
- At a 30% rank budget, LRC reaches essentially the original FP16 model's average accuracy on the evaluated tasks, both with and without activation group scaling.
- The weight-quantization step is modular: LRC improves any layer-wise quantizer it is paired with, and the gain is largest when the underlying quantizer is simple, such as round-to-nearest.
- When activations are left in full precision (W4 only), low-rank correction provides no additional accuracy, confirming that the correction is specifically repairing activation-quantization error.
- A 10% rank overhead in FP16 adds roughly 13% to the stored model size, putting the effective bit width near 6.08 bits; the accuracy gain is bought with extra memory and forward-pass latency.
Reading between the lines
- If the calibration covariances can be estimated cheaply at deployment time, LRC could be adapted to domain-specific activation statistics, potentially reducing the rank budget needed for a given accuracy target.
- The closed-form eigenvector update suggests the activation error concentrates in a low-dimensional subspace; an extension would allocate rank adaptively per layer based on the eigenvalue spectrum of $\Sigma$, rather than a flat 10% or 30% of every matrix.
- The paper's own timing numbers imply that the low-rank path must be fused with or run in parallel with the 4-bit GEMM for the accuracy gains to translate into serving speedups; a fused kernel is the natural next engineering step.
- Because weights-only quantization is already near-lossless, the remaining performance ceiling for W4A4 is the activation quantizer itself; improving on-the-fly activation rounding may matter more than larger correction ranks.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes LRC, a post-training quantization method that augments a quantized weight matrix with low-rank full-precision matrices U and V acting on unquantized activations, with the goal of compensating for activation-quantization errors. The method alternates between a GPTQ-style quantization of an effective weight matrix and closed-form updates of U and V obtained from eigen-decompositions of calibration covariances. Experiments on Llama-2, Llama-3, Phi-3, and Mixtral at W4A4 report accuracy gains over QuaRot and over a fixed SVD-based low-rank baseline, and the abstract claims that 10% ranks reduce the accuracy gap by more than 50% and 30% ranks close it completely. The appendix reports calibration-dataset sensitivity, latency measurements, and proofs of the main propositions.
Significance. The central idea is useful: a low-rank correction that is optimized jointly with the quantized weights, rather than fixed by SVD, is a plausible and principled way to reduce quantization error. The derivations in Propositions 3.1, 3.3, and 3.4 appear mathematically correct under the stated full-rank assumptions, and Appendix D provides complete proofs. The paper also contains helpful ablations (rank, quantizer choice, calibration set) and, unusually, explicit latency measurements. However, the headline comparison is confounded by effective bitrate: by the authors' own calculation in Appendix C.2, 10% ranks correspond to 6.08 effective bits, and 30% ranks to roughly 10 effective bits, while the QuaRot baseline is 4-bit. The 'more than 50% gap reduction' claim is also not uniformly supported by the tables. These issues currently limit the strength of the central claims, though they are addressable with additional experiments and revised wording.
major comments (3)
- [Appendix C.2 and Tables 1, 3, 9] The main W4A4 comparison is not bitrate-matched. Appendix C.2 states that 10% ranks make LRC 'effectively at 6.08 bits (4 + 0.13 * 16),' and Table 3 gives LRC model sizes of 2.59 GB (Phi-3), 4.95 GB (Llama-3), and 32.1 GB (Mixtral), versus 1.69, 3.25, and 21.6 GB for 4-bit QuaRot. At 30% ranks (Table 9), the LRC sizes are 4.39, 8.35, and 53 GB, i.e. roughly 10 effective bits. Since the main tables compare LRC only against 4-bit QuaRot, the reported gains could be due to the additional bit budget rather than to the low-rank correction itself. The central claim that LRC 'outperforms all existing methods at W4A4' therefore needs a same-total-memory baseline, for example QuaRot with 6-bit weights and 4-bit activations, or an LRC variant with U and V quantized to lower precision.
- [Abstract and Tables 1-2] The claim that at 10% rank LRC 'reduces the accuracy gap with the original model by more than 50%' is not consistently supported by the reported numbers. Using the average lm-eval accuracy in Table 1, the gap reduction relative to QuaRot for LRC(1) is approximately 68% (Phi-3), 38% (Llama-3), 31% (Mixtral), 45% (Llama-2 7B), and 44% (Llama-2 13B). With group-scaling in Table 2, Mixtral still shows only 27% gap reduction. The abstract either needs to state the specific setting (e.g., with group-scaling, or a pooled metric) or the claim should be weakened; as written, a reader would reasonably expect the 50% figure to hold across the reported models.
- [Section 3.1, Figure 1, and Appendix C.2] The method is a hybrid W4A4 plus FP16 low-rank pipeline rather than a pure 4-bit activation-compute scheme. In Eq. (2), the correction path U V^T X operates on the unquantized activations X and on full-precision U and V, so the full-precision activation sequence must be materialized at every layer in addition to the 4-bit path. The latency tables in Appendix C.2 show this has a noticeable cost (e.g., Table 6: 13.89 ms for the rank-0 int4 kernel versus 18.04 ms at rank 128). This should be stated prominently whenever 'W4A4' is used, and the effective-bitrate caveat should appear in the abstract, not only in an appendix.
minor comments (5)
- [Table 1, Llama-3 rows] LRC(1) and LRC(5) report worse Wikitext-2 perplexity (8.05 and 7.94) than QuaRot (7.78) despite better lm-eval averages; a brief comment on this discrepancy would help readers interpret the perplexity metric.
- [Section 5 and Appendix C.2] The Limitations paragraph states that 'we have not studied the computational costs of adding low-rank computations to the forward pass,' but Appendix C.2 presents a latency experiment with detailed timings; the wording should be reconciled with the appendix.
- [Algorithms 1 and 4, Section 3.2] The experiments report ranks as a percentage of the original weight-matrix size, but Algorithm 1 takes a single integer k and the paper does not specify how k is chosen for non-square layers; please state the per-layer rule explicitly.
- [Propositions 3.3 and 3.4] In the proposition statements, the notation suggests that Sigma_2 and Sigma_3 are constants, but in Proposition 3.3 they depend on the current cW^(t); adding the superscript to the displayed matrices in the statement would improve clarity.
- [Throughout] There are several typos and inconsistencies: 'LLama' (Section 4), 'orignal' (Tables 9 and 10), 'round-to-neatest' (Section 4.2), and 'experiements' (Section 4.1). A careful proofread is needed.
Circularity Check
No load-bearing circularity; LRC's derivation is self-contained, with only a mild in-sample PPL caveat and a non-load-bearing overlap with the QuaRot baseline.
full rationale
LRC's claimed derivation is not circular. The low-rank correction problem (Eq. 2) is a layer-wise reconstruction objective, and Propositions 3.1, 3.3, and 3.4 give algebraic equivalences and closed-form updates for the quantized weights and the low-rank factors from the stated covariance statistics; the proofs in Appendix D establish these as mathematical equivalences rather than assuming the conclusion. The central accuracy claims are evaluated on lm-eval tasks such as PIQA, HellaSwag, Arc, Winogrande, and Lambada, which are external to the Wikitext-2 calibration set used to compute Sigma_x, Sigma_y, and Sigma_xy, so the headline 'reduces the accuracy gap by more than 50%' is not a fitted-input prediction by construction. The only mild in-sample element is the Wikitext-2 PPL column in Table 1, since calibration uses Wikitext-2 sequences; however, PPL is not the optimized objective (the objective is layer-wise squared reconstruction error), so this is an evaluation caveat rather than a derivational circle. The paper's use of QuaRot (Ashkboos et al., 2024), which shares an author with the present paper, is as a preprocessing step and as a baseline; it is an externally published, code-reproduced method and is not invoked as a uniqueness theorem or as the justification for LRC's correctness. No self-citation chain forces the empirical result. The bitrate mismatch noted by a skeptical reader (LRC at 10% rank is 'effectively at 6.08 bits' per Appendix C.2, versus 4-bit QuaRot) is a comparison confound and a correctness concern, but it does not make the derivation equivalent to its inputs. Overall, no load-bearing circularity is present; the score reflects only minor evaluation caveats and a self-citation overlap that does not affect the derivation's independence.
Assumptions & free parameters
free parameters (4)
- Activation quantization scale c =
not reported (searched per layer or globally)
- Rank k (low-rank correction size) =
10% or 30% of matrix size
- Regularization constants eps_x, eps_y =
1e-2/d_in * Tr(XX^T), 1e-2/d_in * Tr(YY^T)
- Number of alternating iterations T =
1 or 5
assumptions (5)
- domain assumption The quantized activation matrix Y = Q_a(X) is full rank (or made so by regularization).
- domain assumption The calibration activations X have at least as many columns as rows (n >= max(d_out, d_in)).
- domain assumption The calibration set (128 Wikitext-2 sequences) yields covariance statistics representative of inference-time activations.
- domain assumption Activation quantization is a simple scale-then-round procedure with max(abs(x)) scaling.
- domain assumption QuaRot's Hadamard rotation preprocessing is applied and preserved.
Cite this review
Pith. "Pith review of Low-Rank Correction for Quantized LLMs." pith.science (2026). https://pith.science/paper/KGAIDZ5K
@misc{pith2026241207902,
author = {Pith},
title = {Pith review of: Low-Rank Correction for Quantized LLMs},
year = {2026},
howpublished = {\url{https://pith.science/paper/KGAIDZ5K}},
note = {Machine review of arXiv:2412.07902}
}
read the original abstract
We consider the problem of model compression for Large Language Models (LLMs) at post-training time, where the task is to compress a well-trained model using only a small set of calibration input data. In this work, we introduce a new low-rank approach to correct for quantization errors of \emph{activations} in LLMs: we propose to add low-rank weight matrices in full precision that act on the \emph{unquantized} activations. We then solve a joint optimization problem over the quantized representation of the weights and additional low-rank weight matrices to quantize both weights and activations. We focus on the case of 4-bit weight-and-activation quantization (W4A4). Using ranks equivalent to 10\% of the original weight matrix size, our approach reduces the accuracy gap with the original model by more than 50\%. Using ranks equivalent to 30\% of the original weight matrix, the accuracy gap is closed completely. We demonstrate our results on four recent LLMs, namely Llama-2, Llama-3, Phi-3 and Mixtral models.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
Phi-3 technical re- port: A highly capable language model locally on your phone
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical re- port: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219,
-
[5]
Extreme compression of large language models via additive quantization
Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Al- istarh. Extreme compression of large language models via additive quantization. arXiv preprint arXiv:2401.06118,
-
[6]
Gptq: Accurate post-training quantization for generative pre-trained transformers
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323,
-
[7]
URL https://zenodo.org/records/ 12608602. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685,
-
[9]
Gear: An efficient kv cache compression recipefor near-lossless generative inference of llm
11 Preprint Hao Kang, Qingru Zhang, Souvik Kundu, Geonhwa Jeong, Zaoxing Liu, Tushar Krishna, and Tuo Zhao. Gear: An efficient kv cache compression recipefor near-lossless generative inference of llm. arXiv preprint arXiv:2403.05527,
-
[10]
Spinquant–llm quantization with learned rotations
Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krish- namoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Spinquant–llm quantization with learned rotations. arXiv preprint arXiv:2405.16406,
-
[11]
Compress- ing large language models using low rank and low precision decomposition
Rajarshi Saha, Naomi Sagan, Varun Srivastava, Andrea J Goldsmith, and Mert Pilanci. Compress- ing large language models using low rank and low precision decomposition. arXiv preprint arXiv:2405.18886,
-
[12]
Omniquant: Omnidirectionally calibrated quantization for large language models
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. arXiv preprint arXiv:2308.13137,
Show all 20 references
-
[13]
Quip#: Even better llm quantization with hadamard incoherence and lattice codebooks
Albert Tseng, Jerry Chee, Qingyao Sun, V olodymyr Kuleshov, and Christopher De Sa. Quip#: Even better llm quantization with hadamard incoherence and lattice codebooks. arXiv preprint arXiv:2402.04396,
-
[14]
Gptvq: The blessing of dimensionality for llm quan- tization
Mart van Baalen, Andrey Kuzmin, Markus Nagel, Peter Couperus, Cedric Bastoul, Eric Mahurin, Tijmen Blankevoort, and Paul Whatmough. Gptvq: The blessing of dimensionality for llm quan- tization. arXiv preprint arXiv:2402.15319,
-
[15]
Understanding int4 quantization for transformer models: Latency speedup, composability, and failure cases
Xiaoxia Wu, Cheng Li, Reza Yazdani Aminabadi, Zhewei Yao, and Yuxiong He. Understanding int4 quantization for transformer models: Latency speedup, composability, and failure cases. arXiv preprint arXiv:2301.12017,
-
[16]
Lqer: Low-rank quantiza- tion error reconstruction for llms
Cheng Zhang, Jianyi Cheng, George A Constantinides, and Yiren Zhao. Lqer: Low-rank quantiza- tion error reconstruction for llms. arXiv preprint arXiv:2402.02446,
-
[17]
The GPTQ algorithm, introduced by Frantar et al
12 Preprint APPENDIX A A DDITIONAL BACKGROUND GPTQ Algorithm. The GPTQ algorithm, introduced by Frantar et al. (2022), is a post-training quantization technique designed to efficiently reduce the precision of weights in large language models (LLMs) while maintaining their perf...
2022
-
[18]
propose to improve the computational scheme of the greedy approach originally pro- posed by LeCun et al. (1989) for pruning, and then adapted for quantization in (Frantar & Alistarh, 2022), by removing the ordering in the greedy quantization process, and applying the algorithm...
1989
-
[19]
that only requires access to the target weight matrix fW (t) and the covariance matrix Y Y⊤. Algorithm 2 Update-Quant(W , U , V , Y Y⊤, XY ⊤, b) 1: Input: Original weight matrix W , low-rank weight matrices U , V , covariance matrix Y Y⊤, cross-covariance matrix XY ⊤, and the ...
2022
-
[20]
Algorithm 5 LRC(W , X, b, a, k) 1: Input: Original weight matrix W , activation X, the bit precision for weightsb, the bit precision for activation a, the rank k, and number of iterations T . 2: Output: Quantized weight matrix cW , low-rank weight matrices U , V 3: Σx ← XX ⊤ +...
-
[2018]
Mixtral of experts
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088,
-
[2022]
Qlora: efficient finetuning of quantized llms (2023)
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: efficient finetuning of quantized llms (2023). arXiv preprint arXiv:2305.14314, 52:3982–3992, 2023a. Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashk- boos...
2023 arXiv
-
[2023]
Quarot: Outlier-free 4-bit inference in rotated llms
Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L Croci, Bo Li, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated llms. arXiv preprint arXiv:2404.00456,
-
[2024]
Towards end-to-end 4-bit inference on generative large language mod- els
Saleh Ashkboos, Ilia Markov, Elias Frantar, Tingxuan Zhong, Xincheng Wang, Jie Ren, Torsten Hoefler, and Dan Alistarh. Towards end-to-end 4-bit inference on generative large language mod- els. arXiv preprint arXiv:2310.09259,
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.