REVIEW 3 major objections 4 minor 14 references
Quantization Damage Is Multiplicative, Not Additive
T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read Quantization shrinks a model's decision margins proportionally, not by adding fixed noise—so damage is per-item predictable but confident decisions are not safe.
desk verdict Strong empirical case that quantization shrinks margins multiplicatively, with an honest but load-bearing caveat about the margin proxy at severe damage. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the paired first-token margin: for each item, the difference between the logits of the chosen first token and the best-alternative first token, measured at full precision and after quantization with the alternative held fixed. The argument is carried by a per-decision-family regression of the quantized margin on the clean margin, $m' = c\,m + b + \varepsilon$, together with a BIC-based comparison against additive competitors (constant noise, drift, and margin-scaled noise) and a parameter-free bound showing that independent error accumulation cannot lose more than half the margin per bit. The same fitted relation is then read conditionally to derive the flip-probability formula, turning a curve fit into a testable model of the channel from clean to quantized decisions.
What would settle it
A decisive test: on a new model and a tool-call or refusal decision family at 3 bits, fit $(c,b,\sigma)$ on clean/quantized margins only, then compare the predicted flip probability from $\Phi(-(cm+b)\operatorname{sign}(m)/\sigma)$ to the observed flip rate on a held-out half; if the median absolute error exceeds the paper's ~1.8 percentage points, or if an additive margin-scaled-noise model wins BIC in a core family, the multiplicative law does not hold there.
Extended reading notes
Core claim
The paper establishes that for a binary decision with clean margin $m$ (score of the chosen option minus the best alternative), the quantized margin is $m' = c\,m + b + \varepsilon$ with $\varepsilon \sim \mathcal{N}(0,\sigma^2)$; the surviving fraction $c$ falls with bit-width (median 0.86 at 4 bits, 0.33 at 3, ~0 at 2). This is called margin shrinkage. Additive-noise accounts never win in damaged tool-call and safety decisions when fitted statistically. Reading the fit as a conditional distribution forces the flip probability $P(\text{flip}\mid m)=\Phi(-(cm+b)\operatorname{sign}(m)/\sigma)$, which matches held-out flip rates to a median error of 1.8 percentage points with no flips used in the fitting, and is calibrated as an uncertainty estimate (expected calibration error 0.004 over 131,758 predictions). The same form holds in every model measured, but the constants are per-model and do not transfer; because the loss is multiplicative, no margin size is safe, and importance-based weight protection fails against cost-matched controls.
Load-bearing premise
The whole measurement chain assumes the first-token margin, with the full-precision model's best wrong option held fixed, faithfully represents the decision the model would actually make in free generation; if later tokens can override the first-token choice, measured margin shrinkages may not track real behavioural damage.
Editorial extensions
If this is right
- A large full-precision margin is not a certificate of safety: at 3 bits even a 20-logit margin loses most of its size, so confident decisions can flip.
- Aggregate benchmarks can stay flat while many decisions break, because they average over margins that individually cross zero; benchmark scores should not be used to sound an alarm.
- Protecting 'important' weights cannot beat a cost-matched control when damage is a proportional loss of the whole margin, and in the forms tested it does not.
- At deployment bit-widths, the decision of whether to call a tool collapses one-directionally (96% of should-call decisions flip at 3 bits) while which-tool selection survives.
- One additional bit of precision recovers a median 30.5% of flipped decisions and, under the paper's cost-matched tests, outperforms every repair it tried.
Reading between the lines
- Extending the paper's per-model constants, a small paired margin calibration on the target model at 4 and 3 bits could predict the bit-width where each decision family collapses, before running expensive generative evaluation.
- The law's rejection of independent error accumulation suggests that per-layer bit allocation decisions interact non-additively; measuring $c$ on random layer subsets would show whether shrinkage compounds rather than sums.
- The tool-result-use exception points to a boundary worth probing: decisions that copy a value from context have margin-growing noise, so retrieval-augmented generation, which is full of such decisions, may show the same exception at scale.
- If shrinkage is a universal channel property, deployment-time monitoring should shift from aggregate loss to a small paired margin probe, since flat benchmarks are exactly the regime where behavioural damage is beginning.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies how post-training quantization changes individual LLM decisions. It defines the decision margin as the logit difference between the model's chosen first token and its best alternative, holds the two alternatives fixed from the full-precision model, and measures the paired margin before and after quantization across 16 models, three quantization methods, and bit-widths from 8 down to 2. The central empirical claim is that the quantized margin obeys m' = c*m + b + epsilon (Eq. 1), with the multiplicative survival fraction c collapsing as bit-width decreases (median 0.86 at 4 bits, 0.33 at 3, 0.00 at 2), a family-specific push b, and residual noise sigma. From this conditional Gaussian model the paper derives a flip probability formula (Eq. 2), tests it on flip rates that were not used in the fit (median error 1.8 percentage points, ECE 0.004), compares the multiplicative account against additive-noise accounts using BIC, reports a one-directional collapse of tool invocation while tool selection survives, measures refusal losses, and evaluates repair strategies. The paper explicitly reports exceptions, unestimable cells, and a detailed limitations section.
Significance. If the multiplicative law holds as a description of the measured decision margins, it is a substantial reinterpretation of quantization damage: a confident margin is not protected by its size, additive-noise defenses are misplaced, and per-cell calibrated flip probabilities become a practical monitoring tool. The paper is unusually candid: it reports unestimable cells, a recurring exception family, a withdrawn earlier claim, and the instrument's boundary at low bit-widths. It also ships a reproducibility-oriented artifact and a parameter-free bound ruling out independent error accumulation. The out-of-sample flip prediction and the item-for-item overlap across two quantization algorithms are concrete, falsifiable successes. The main weakness is that the unit of measurement is a fixed-contrast margin between two designated tokens, and the connection from that unit to actual free-generation behavior is validated only on small auxiliary checks.
major comments (3)
- [§2, §5.2, Limitations] The central decision unit is the margin between the full-precision model's top two options, held fixed under quantization. The Limitations section reports that at 3-bit RTN the model's actual top-1 next token is one of the two designated options in only 30% of tool items, with median rank near 3,500, and 0% at 2 bits. Thus the fitted law, the flip probabilities, and the calibrated 1.8pp forecast describe a conditional preference under grammar-constrained decoding, not the model's unconstrained emission. The paper scopes this in Limitations, but the abstract and introduction claim the paper predicts 'which of the model's decisions will change.' The supporting free-generation evidence is 60 prompts (AUC 0.918/0.816) and a 64-token refusal check on one instrument. This is too thin for a load-bearing generalization from margin flips to behavioral decisions. Either add a substantially larger free-generation evaluation reporting actual tool-call and refusal rates against predicted sign flips across models and bit-widths, or explicitly reframe the central claim as applying to constrained-decision margins and soften the 'decisions' language throughout.
- [§3.2, Table 1, §2] The model comparison supporting 'additive accounts never win' excludes the 196 unestimable cells (25.1% of the audit cells), which the paper itself identifies as the most damaged (median flip rate 0.33 versus 0.11). The BIC comparison is therefore not established in exactly the regime where the multiplicative law is claimed to be strongest, near c ≈ 0. The paper reports flip rates for these cells but does not compare A0/A1/A2 against M there. Since a slope cannot be estimated, a direct comparison may be impossible, but the claim should be explicitly limited to estimable cells, and the possibility that the worst-damaged cells behave differently should be discussed rather than only reported as a flip-rate summary.
- [§4.2, Figure 7, §8] The held-out forecast accuracy is measured by splitting items within each cell and fitting parameters on one half. This validates the conditional shape within a cell but does not demonstrate that a small paired margin set can predict which decisions break for a new model, method, or bit-width. The paper's transfer tests show that constants do not transfer across models (18–33pp error at 3 bits) and that extrapolating from one measured bit-width is worse than measuring every bit-width. Given the practical claim in the abstract and conclusion that a small paired margin set 'estimates which decisions break without full generative evaluation,' the paper should either provide a concrete procedure and evaluation for that use case within a model (e.g., calibrating on one family and predicting held-out families) or state more modestly that the per-cell law is predictive only after per-cell calibration.
minor comments (4)
- [Abstract] There is a typo in the abstract: 'say ishowit hurts' should read 'say is how it hurts.'
- [Figure 1 caption] The caption says the data form 'a fan through the origin,' but the fitted relation includes a nonzero intercept b, and the plotted line for the 3-bit panel is far from passing through the origin; 'through the origin' should be qualified or removed.
- [§2] The sentence 'the regressor itself carries no such noise' is correct for the mean of the full-precision margin, but the full-precision margin is still an estimated logit difference with its own sampling variation across items; the wording could be sharpened to avoid implying the regressor is measured without any uncertainty relevant to prediction.
- [§5.2] The refusal result reports one unresolved disagreement between the two safety instruments for Granite (0.03 versus 0.64), but the text does not say whether this case is included in the later aggregate counts; a clarifying sentence would help.
Circularity Check
No significant circularity: the flip-probability forecast is derived from a fitted conditional margin model and validated on held-out flip rates, with no flip used in the fit.
full rationale
The paper's central derivation is an empirical regression: it fits the conditional distribution m' | m ~ N(c m + b, sigma^2) to paired full-precision and quantized margins, then derives the flip-probability formula P(flip|m) = Phi(-(c m + b) sign(m)/sigma) as a mathematical corollary of that fitted channel. The predictive check is properly held out: the paper states that '(c, b, sigma) are fitted to margins, and Eq. 2 is scored against a flip rate that played no part in the fit', with held-out halves of cells used for the reported median error of 1.8 percentage points. A flip is defined as a sign change of the same margin variable used in the fit, but because the empirical flip rate is not an input to the parameter estimation, the forecast is not forced by construction. The paper's model-selection step (BIC comparison against additive accounts) is also non-circular: additive models are fit and compared on the same data, and the multiplicative form wins on damaged core-family cells. No load-bearing self-citations appear; the references are to external or concurrent work, and the paper explicitly disclaims novelty for previously published phenomena such as the steep drop below 4 bits and attention re-routing, reserving novelty for the per-family parameterization. The Limitations section candidly notes the instrument's boundary at very low bit-widths (e.g., full-vocabulary top-1 is one of the two designated options in only 30% of tool items at 3-bit RTN), but this is a validity caveat about the margin proxy, not a circularity in the derivation. The derivation chain is therefore self-contained against its stated assumptions, and the predictive claims are tested on data not used in fitting.
Assumptions & free parameters
free parameters (4)
- margin survival fraction c =
per cell; e.g., Qwen3-4B RTN b3 whether-to-call c=0.13; median c=0.86 at 4b, 0.33 at 3b, 0.00 at 2b
- decision-family push b =
per cell; e.g., Qwen3-4B RTN b3 whether-to-call b=-13.5
- residual noise sigma =
per cell; not reported individually
- transition width =
1.2 bits
assumptions (5)
- ad hoc to paper m'|m is Gaussian with mean c*m + b and constant variance sigma^2
- domain assumption The margin at the first token, with the alternative fixed at the full-precision best wrong option, faithfully represents the model's decision
- domain assumption Weight quantization error variance falls fourfold per bit (gamma=4) for the accumulation bound
- domain assumption The alternative is the full-precision model's highest-scoring wrong option, held fixed across quantized conditions
- standard math Least squares with intercept is the estimator for each cell
Cite this review
Pith. "Pith review of Quantization Damage Is Multiplicative, Not Additive." pith.science (2026). https://pith.science/paper/BHOC4LF2
@misc{pith2026260806564,
author = {Pith},
title = {Pith review of: Quantization Damage Is Multiplicative, Not Additive},
year = {2026},
howpublished = {\url{https://pith.science/paper/BHOC4LF2}},
note = {Machine review of arXiv:2608.06564}
}
read the original abstract
Quantization is how large language models are actually deployed, and below four bits it is known to hurt. What nobody can say is which of the model's decisions will change at a given bit-width. The damage is silent: a compressed agent stops calling its tools, then loses half its safety refusals, yet benchmark scores barely move. Prior work assumes quantization adds noise of a roughly fixed size, which would make confident decisions safe. We measure the decision itself instead. The margin of a two-way decision is the model's score for the option it picks minus the score of its best alternative; we track it before and after quantization across 16 models from 8 model families, three quantization methods, and bit-widths from 8 down to 2. Quantization does not add fixed-size noise to the margin. It multiplies the margin by a factor that collapses with bit-width (median 0.86 at 4 bits, 0.33 at 3, 0.00 at 2); we call this margin shrinkage. This contraction reduces the protection a large margin affords; the model's own small biases pick the direction of failure: at 3 bits the decision to call a tool collapses toward inaction while the choice of which tool is untouched. In fitted statistical comparison, additive-noise accounts never win on the damaged tool and safety decisions. The fitted relation predicts flip rates within a median of 1.8 percentage points on held-out decisions, though no flip was used in the fit; per decision, the predicted flip probabilities are calibrated uncertainty estimates (expected calibration error 0.004 over 131,758 predictions). The same form holds in every model we measure, but the constants are each model's own and do not transfer. A small paired margin set, measured per model and bit-width, estimates which decisions break without full generative evaluation; under our cost-matched tests, nothing repairs damage more cheaply than one more bit.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
Croci, Bo Li, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman
Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. 2024. QuaRot : Outlier-free 4-bit inference in rotated LLMs . Advances in Neural Information Processing Systems
work page 2024
-
[2]
Vishnu Kabir Chhabra and Mohammad Mahdi Khalili. 2025. Towards understanding and improving refusal in compressed models via mechanistic interpretability. arXiv preprint arXiv:2504.04215
arXiv 2025
-
[3]
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. LLM.int8() : 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems
work page 2022
-
[4]
Peijie Dong, Zhenheng Tang, Xiang Liu, Lujun Li, Xiaowen Chu, and Bo Li. 2025. Can compressed LLMs truly act? an empirical evaluation of agentic capabilities in LLM compression. arXiv preprint arXiv:2505.19433
arXiv 2025
-
[5]
Kazuki Egashira, Mark Vero, Robin Staab, Jingxuan He, and Martin Vechev. 2024. Exploiting LLM quantization. Advances in Neural Information Processing Systems. ArXiv:2405.18137
arXiv 2024
-
[6]
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022. GPTQ : Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323
arXiv 2022
-
[7]
Jiwon Jang, Kisu Yang, Heuiseok Lim, and Hyunwoo Park. 2026. Flat score, amplified failures: How the error budget masks damage in quantized LLM agents. arXiv preprint arXiv:2607.27275
work page Pith review arXiv 2026
-
[8]
Tanishq Kumar, Zachary Ankner, Benjamin F. Spector, Blake Bordelon, Niklas Muennighoff, Mansheej Paul, Cengiz Pehlevan, Christopher R \'e , and Aditi Raghunathan. 2025. Scaling laws for precision. In International Conference on Learning Representations. ArXiv:2411.04330
arXiv 2025
Show all 14 references
-
[9]
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. AWQ : Activation-aware weight quantization for LLM compression and acceleration. In Proceedings of Machine Learning and Systems (MLSys)
2024
-
[10]
Xu Ouyang, Tao Ge, Thomas Hartvigsen, Zhisong Zhang, Haitao Mi, and Dong Yu. 2025. Low-bit quantization favors undertrained LLMs : Scaling laws for quantized LLMs with 100t training tokens. In Proceedings of ACL. ArXiv:2411.17691
2025 arXiv
-
[11]
Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R. Bowman. 2022. BBQ : A hand-built bias benchmark for question answering. In Findings of ACL
2022
-
[12]
Samuel Salfati. 2026. Quantization dominates rank reduction for KV -cache compression. arXiv preprint arXiv:2604.11501
2026 arXiv
-
[13]
Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. 2023. Are emergent abilities of large language models a mirage? Advances in Neural Information Processing Systems
2023
-
[14]
Patil, Ion Stoica, and Joseph E
Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. 2024. Berkeley function-calling leaderboard. https://gorilla.cs.berkeley.edu/leaderboard.html
2024
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.