Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Scaling Law for Quantization-Aware Training

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Low-bit quantized training error follows one power law in model size, training tokens, and quantization group size.

desk verdict A well-costed empirical scaling law for W4A4 QAT that is worth refereeing; the G variable needs pinning down and the quantizer switch confounds part of the granularity trend. read the letter →

arxiv 2505.14302 v1 pith:V42TV3EH submitted 2025-05-20 cs.LG cs.CL

Mengzhao Chen , Chaoyi Zhang , Jing Liu , Yutao Zeng , Zeyue Xue , Zhiheng Liu , Yunshui Li , Jin Ma
show 3 more authors
Jie Huang Xun Zhou Ping Luo
This is my paper
classification cs.LGcs.CL
keywords quantization-awaretrainingscalinglawW4A4quantizationgranularityactivationoutliersmixed-precisionLLMcompressionweightversuserror
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the extra loss introduced by 4-bit weight-and-activation quantization-aware training (W4A4 QAT) is not fixed by model size alone: it follows a single power law in three variables, $\delta_p(N,D,G)=k D^{\gamma_D}(\log_2 G)^{\gamma_G}/N^{\gamma_N}$, where $N$ is the parameter count, $D$ the training tokens, and $G$ the quantization group size. Fitted on 80 W4A4 runs spanning 74M to 595M parameters and 10B to 100B tokens, the law predicts held-out 973M-parameter models and reduces relative prediction error from 8.5% to 4.7% compared with prior model-size-only QAT scaling laws. The experiments show that quantization error grows with more training tokens and coarser quantization granularity, and that the dominant error source is activation quantization at the FC2 projection input, whose outliers persist under QAT. If the law holds, QAT budgets become a design variable alongside model size: how much data a model sees and how finely weights and activations are grouped directly determine how much precision is lost.

What carries the argument

The load-bearing object is the unified QAT error law $\delta_p(N,D,G) = k D^{\gamma_D}(\log_2 G)^{\gamma_G}/N^{\gamma_N}$, fitted to smoothed training-loss gaps between QAT and BFloat16 models. The logarithmic term in $G$ makes the error vanish when there is no grouping, while the exponents $\gamma_N,\gamma_D,\gamma_G$ measure how strongly model size, data volume, and granularity each drive error. The argument is carried by the decomposition $\delta_{W4A4} \approx 0.906\,(\delta_{W4A16}+\delta_{W16A4})$, which converts one aggregate curve into two independently fitted component curves and lets the fitted exponents identify whether weights or activations are the bottleneck under each training regime.

What would settle it

Train W4A4 QAT models at fixed $N$ and $D$ while sweeping activation group size and weight group size independently in opposite directions, such as 32-token/256-channel versus 256-token/32-channel; if a single $G$ cannot reproduce both measured errors, or if the fitted exponents $\gamma_N,\gamma_D,\gamma_G$ change when the two group types are decoupled, the unified single-curve law is falsified.

Watch

Extended reading notes

Core claim

The central discovery is that W4A4 QAT quantization error has a predictable, factorized structure: it decreases as a power of model size, increases as a power of training tokens, and increases as a power of the logarithm of quantization group size. The same functional form fits weight-only quantization (W4A16) and activation-only quantization (W16A4) with different exponents, and their errors add to reproduce the W4A4 error with a fitted coefficient of 0.906. Activation error generally dominates, especially at coarse granularities, and the paper traces this to high-kurtosis outliers in the FC2 Proj input generated by the SwiGLU module. Keeping FC2 inputs at 8-bit precision drops W4A4 quantization error by 20.5% at group size 32 and 42.9% at group size 256, and makes weight and activation errors nearly equal; as the data-to-parameter ratio grows, weight quantization error eventually overtakes activation error.

Load-bearing premise

The law encodes every granularity choice as one number $G$ inside a log term, but activation groups (per token) and weight groups (per channel) can have different sizes, so the single-curve fit is not fully determined unless a mapping from both group types to one $G$ is specified.

Editorial extensions

If this is right

  • Quantization error is predictable from a single curve across model size, data, and granularity, cutting relative prediction error on W4A4 from 8.5% to 4.7% compared with model-size-only QAT laws.
  • More training data makes quantization harder, so a model trained on 100B tokens instead of 10B needs finer quantization groups or higher precision to hold the same error.
  • Activation quantization at the FC2 Proj input is the main W4A4 bottleneck; keeping that input at 8-bit lowers error by 20.5% at group size 32 and 42.9% at group size 256.
  • Weight quantization error grows faster with training tokens than activation error, so at high data-to-parameter ratios weight quantization deserves as much attention as activation outliers.
  • Under the paper's cost assumptions, the effective parameter multiplier of W4A4 stays above 0.5, making 4-bit QAT a better compute-error trade-off than 8-bit QAT across the studied range.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the law was fitted on models up to 595M parameters, so its predictions much beyond 1B parameters are extrapolation; the same exponents may not hold at frontier scale.
  • Editorial extension: the single numeric $G$ is underdetermined when activation groups are per-token and weight groups are per-channel with different sizes; a two-term law with separate granularity exponents for each component could fit equally well and resolve the ambiguity.
  • Editorial extension: the persistent FC2 input outliers suggest testing whether other outlier-prone parts of a transformer, such as early embedding projections or attention outputs, also deserve mixed precision as training data scales.
  • Editorial extension: since error grows with $D$, a QAT training schedule could anneal quantization granularity or introduce mixed precision later in training to counteract the widening gap.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a unified scaling law for quantization-aware training (QAT) of 4-bit LLMs, modeling the quantization error gap δp between QAT and BFloat16 models as δp(N,D,G) = k · D^{γD} · (log2 G)^{γG} / N^{γN}. The authors fit this law on 80 W4A4 runs across model sizes 74–594M, token counts 10–100B, and granularities {32,64,128,256,per-token/channel}, report good fit (R²=0.94), and validate on held-out 973M-parameter models trained on 100B and 200B tokens. They also decompose W4A4 error into weight (W4A16) and activation (W16A4) components, fit separate scaling laws for each, and show that FC2-proj input activations, due to high kurtosis from SwiGLU, are the main bottleneck. A mixed-precision intervention (8-bit FC2 input) reduces this error and balances weight and activation contributions.

Significance. If the central claims hold, the paper contributes a practically useful extension of scaling-law methodology to QAT: it shows that quantization error increases with training tokens and coarser granularity, captures this in a compact three-variable law, and identifies FC2-input activations as the key bottleneck. The empirical base is substantial (268 experiments, 276K GPU-hours), the law is machine-checkable, and the paper includes explicit held-out validation on 973M models, which is a notable strength. The decomposition into W4A16 and W16A4 components is independently measured and provides falsifiable predictions about how weight-versus-activation error balances shift with data-to-parameter ratio. The main risk is that the granularity variable G is not explicitly mapped to a numerical value for the per-token/channel setting and is confounded by a quantizer change, so the single-curve unification claim is not fully determined by the text.

major comments (4)
  1. [§4.1, Eq. (5)] The variable G in Eq. (5) is underdetermined for the per-token/channel granularity. Section 4.1 lists G ∈ {32,64,128,256,per-token/channel}, but per-token activation quantization and per-channel weight quantization have heterogeneous group sizes, and no numerical G is assigned to the per-token/channel condition. The fitted γG in Table 1 therefore depends on an implicit mapping that is not stated; different mappings (e.g., sequence length, hidden dimension, or an arbitrary constant) would produce different predicted δp for the same experiments. Because the central claim is that Eq. (5) unifies all granularities into one curve, the absence of this mapping leaves the fit in Figure 4c and the parameter γG non-reproducible. Please specify the exact numeric G used for each run, or restrict the law to the numeric group sizes if no principled mapping exists.
  2. [§4.1, §4.2, Fig. 4c] The activation quantizer switches at G = 256: AbsMax is used for G < 256 and LAC for G ≥ 256, which includes the per-token/channel setting. Figure 4c and the fitted γG = 0.7779 therefore combine data from two different quantizers, so the observed 'granularity effect' is confounded with quantizer identity. This undermines the attribution of the error trend solely to G. Please provide an ablation where the same quantizer is used across all granularities (e.g., LAC for all G) or otherwise demonstrate that the fitted exponent γG does not change materially when the quantizer is fixed.
  3. [§4.2, validation] The held-out validation on 973M models tests extrapolation in N and D but not in G: both validation points appear to use the same granularity regime, and the text does not report whether they use per-token/channel or a numeric group size. Since the G-encoding issue raised by Major Comment 1 is precisely about which numeric G is used for coarse granularities, the current validation does not discriminate between alternative assignments. Please report the granularity settings of the two 973M runs, and include at least one held-out data point at a new granularity if possible.
  4. [§4.3, Eq. (5) usage] The decomposition δW4A4 ≈ 0.906·(δW4A16 + δW16A4) is presented as an observed correlation, but the paper then fits separate scaling laws for δW4A16 and δW16A4 and uses them to explain trends in δW4A4. This is reasonable as an approximation, yet the paper does not propagate the uncertainty of the 0.906 coefficient or verify that the fitted exponents are consistent with the W4A4 fit when combined. Since the approximation is load-bearing for the bottleneck analysis, please add a brief consistency check (e.g., compare δW4A4 predicted from the two-component fits against the direct W4A4 fit).
minor comments (4)
  1. [§4.2, Evaluation metric] The definition of δW4A4 appears to have a sign error: the text says δW4A4 = loss_bf16 − loss_W4A4, which would be negative when QAT loss is higher, but Figure 4 and the surrounding discussion treat δ as positive. Please correct the sign or the ordering.
  2. [§4.3] In the sentence 'This strong correlation suggests that we can effectively analyze δW4A4 by separately examining the δW4A16 and δW4A4', the second term should read δW16A4.
  3. [§4.1] The phrase 'per-token/channel' is used inconsistently across figures; for example, Figure 4c labels the last x-axis tick 'Token/Channel' while the text and Table 2 use 'per-token/channel.' Please unify the terminology and define it once in the main text.
  4. [§4.4] The claim that 8-bit FC2 inputs 'provide an upper bound on the improvements possible' is somewhat informal, since outlier-suppression methods could in principle exceed 8-bit quantization if 8-bit itself is not perfect; please rephrase to avoid implying a rigorous bound.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Eq. (5) is an empirical scaling-law fit, checked on held-out 973M runs, not a derivation that presupposes its conclusion.

full rationale

The central law delta_p = k * D^gamma_D * (log2 G)^gamma_G / N^gamma_N is proposed as a fitted functional form and calibrated on 80 W4A4 QAT runs, then evaluated on two held-out 973M-parameter runs (Section 4.2, Figure 5). This is a standard fit-and-validate procedure, not a reduction of the prediction to its fitting inputs. The delta_W4A4 ground truth is independently defined as BF16-minus-W4A4 loss, and the weight/activation decomposition in Section 4.3 is an empirical approximation (coefficient 0.906) rather than an identity imposed by construction. The Appendix H EPM expression is a definitional rearrangement of Eqs. (4), (5), and (2), and the paper does not present it as an independent prediction. Self-citations such as LAC [5] and EfficientQAT [6] are background or quantizer choices, not load-bearing evidence for the scaling law. The main unresolved issue is an under-specification of G for per-token/per-channel runs and a quantizer switch at G=256 (Section 4.1), which affects identifiability and should be scored as a correctness risk, not as circularity. The paper acknowledges scope limits in Appendix A (no MoE, only W4A4, limited scale), which further confirms the claims are empirical rather than circular.

Assumptions & free parameters 26 free parameters · 9 assumptions · 0 invented entities

The central claim rests on a fitted four-parameter power law, a Chinchilla loss form, an additive weight/activation error decomposition, and an extrapolation assumption. None of these is derived from first principles. The free parameters are numerous because the paper fits separate scaling laws for W4A4, W4A16, W16A4, and the FC2-8bit variants.

free parameters (26)
  • k (W4A4) = 0.1582
    Multiplicative constant in Eq. (5), fitted to 80 W4A4 QAT runs.
  • gamma_N (W4A4) = 0.2186
    Exponent on model size in Eq. (5), fitted to W4A4 runs.
  • gamma_D (W4A4) = 0.0745
    Exponent on training tokens in Eq. (5), fitted to W4A4 runs.
  • gamma_G (W4A4) = 0.7779
    Exponent on log2(G) in Eq. (5), fitted to W4A4 runs.
  • k (W4A16) = 0.2522
    Multiplicative constant for weight-only quantization error scaling.
  • gamma_N (W4A16) = 0.3589
    Model-size exponent for weight-only quantization error.
  • gamma_D (W4A16) = 0.1610
    Token exponent for weight-only quantization error.
  • gamma_G (W4A16) = 0.3533
    Granularity exponent for weight-only quantization error.
  • k (W16A4) = 0.1004
    Multiplicative constant for activation-only quantization error scaling.
  • gamma_N (W16A4) = 0.1816
    Model-size exponent for activation-only quantization error.
  • gamma_D (W16A4) = 0.0331
    Token exponent for activation-only quantization error.
  • gamma_G (W16A4) = 0.9812
    Granularity exponent for activation-only quantization error.
  • k (W4A4, FC2 input 8-bit) = 0.3519
    Multiplicative constant for W4A4 scaling law after FC2 input is kept at 8-bit.
  • gamma_N (W4A4, FC2 input 8-bit) = 0.2637
    Model-size exponent after FC2 mixed-precision intervention.
  • gamma_D (W4A4, FC2 input 8-bit) = 0.0964
    Token exponent after FC2 mixed-precision intervention.
  • gamma_G (W4A4, FC2 input 8-bit) = 0.3407
    Granularity exponent after FC2 mixed-precision intervention.
  • k (W16A4, FC2 input 8-bit) = 0.1273
    Multiplicative constant for activation-only scaling after FC2 input is quantized to 8-bit.
  • gamma_N (W16A4, FC2 input 8-bit) = 0.2347
    Model-size exponent for activation-only error after FC2 intervention.
  • gamma_D (W16A4, FC2 input 8-bit) = 0.0827
    Token exponent for activation-only error after FC2 intervention.
  • gamma_G (W16A4, FC2 input 8-bit) = 0.4491
    Granularity exponent for activation-only error after FC2 intervention.
  • Chinchilla E = 1.9279
    Fitted irreducible-loss constant in the Chinchilla loss model for the BF16 baselines.
  • Chinchilla A = 237.7042
    Fitted coefficient of N^(-alpha) in the Chinchilla loss model.
  • Chinchilla alpha = 0.3022
    Fitted exponent on model size, set equal to beta.
  • Chinchilla B = 596.2490
    Fitted coefficient of D^(-beta) in the Chinchilla loss model.
  • Chinchilla beta = 0.3022
    Fitted exponent on training tokens, constrained equal to alpha.
  • Decomposition coefficient c = 0.906
    Empirical slope relating delta_W4A4 to delta_W4A16 + delta_W16A4 in Fig. 6.
assumptions (9)
  • domain assumption Smoothed training loss is an unbiased estimate of validation loss.
    Used as the evaluation metric following Chinchilla-style scaling laws, but not demonstrated for quantized models.
  • domain assumption The Chinchilla loss form Eq. (1) with alpha=beta holds for the trained BF16 models.
    Appendix C fits this form to BF16 models and uses it as the baseline for defining quantization error.
  • domain assumption QAT quantization error adds to the Chinchilla loss independently, Eq. (4).
    No interaction or nonlinear coupling between the BF16 loss and the quantization-error term is modeled.
  • domain assumption Weight and activation quantization errors are additive: delta_W4A4 is approximately c times the sum of delta_W4A16 and delta_W16A4.
    Empirical approximation in Sec. 4.3; W4A16 and W16A4 are separately trained models, not isolated components of the same W4A4 model.
  • domain assumption INT4 and FP4 quantization follow the same scaling functional form.
    Hypothesized in Appendix E.1 and tested on a limited FP4 dataset.
  • ad hoc to paper The log2(G) term is the correct granularity dependence, with G=1 giving zero error.
    Chosen for mathematical convenience; per-token/channel granularity has no unique numeric group size.
  • domain assumption The fitted power law remains valid beyond the calibration range, such as 10B parameters and 10T tokens in Fig. 1.
    Standard scaling-law extrapolation assumption, not verified outside 973M parameters and 200B tokens.
  • domain assumption Same hyperparameters are appropriate for BF16 and W4A4 QAT training.
    Supported by the learning-rate insensitivity test in Fig. 3, but not exhaustively validated.
  • ad hoc to paper Switching activation quantizers at G=256 does not confound the granularity effect.
    The paper uses AbsMax for G<256 and LAC for G>=256, so the measured dependence on G mixes quantizer identity with granularity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scaling Law for Quantization-Aware Training." pith.science (2026). https://pith.science/paper/V42TV3EH

@misc{pith2026250514302,
  author       = {Pith},
  title        = {Pith review of: Scaling Law for Quantization-Aware Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V42TV3EH}},
  note         = {Machine review of arXiv:2505.14302}
}
read the original abstract

Large language models (LLMs) demand substantial computational and memory resources, creating deployment challenges. Quantization-aware training (QAT) addresses these challenges by reducing model precision while maintaining performance. However, the scaling behavior of QAT, especially at 4-bit precision (W4A4), is not well understood. Existing QAT scaling laws often ignore key factors such as the number of training tokens and quantization granularity, which limits their applicability. This paper proposes a unified scaling law for QAT that models quantization error as a function of model size, training data volume, and quantization group size. Through 268 QAT experiments, we show that quantization error decreases as model size increases, but rises with more training tokens and coarser quantization granularity. To identify the sources of W4A4 quantization error, we decompose it into weight and activation components. Both components follow the overall trend of W4A4 quantization error, but with different sensitivities. Specifically, weight quantization error increases more rapidly with more training tokens. Further analysis shows that the activation quantization error in the FC2 layer, caused by outliers, is the primary bottleneck of W4A4 QAT quantization error. By applying mixed-precision quantization to address this bottleneck, we demonstrate that weight and activation quantization errors can converge to similar levels. Additionally, with more training data, weight quantization error eventually exceeds activation quantization error, suggesting that reducing weight quantization error is also important in such scenarios. These findings offer key insights for improving QAT research and development.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Efficient Reasoning on the Edge

    cs.LG 2026-03 accept novelty 5.5 of 10

    LoRA adapters, budget-forced GRPO, dynamic switching, parallel verification and FPTQuant enable practical chain-of-thought reasoning on quantized Qwen2.5-7B for edge devices.

Reference graph

Works this paper leans on

46 extracted references · 3 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints.arXiv preprint arXiv:2305.13245, 2023

    Joshua Ainslie, James Lee-Thorp, Michiel De Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sang- hai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints.arXiv preprint arXiv:2305.13245, 2023

    arXiv 2023

  2. [2]

    Systematic outliers in large language models.arXiv preprint arXiv:2502.06415, 2025

    Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. Systematic outliers in large language models.arXiv preprint arXiv:2502.06415, 2025

    arXiv 2025

  3. [3]

    Quarot: Outlier-free 4-bit inference in rotated llms.arXiv preprint arXiv:2404.00456, 2024

    Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L Croci, Bo Li, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated llms.arXiv preprint arXiv:2404.00456, 2024

    arXiv 2024

  4. [4]

    A survey on mixture of experts in large language models.IEEE Transactions on Knowledge and Data Engineering, 2025

    Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. A survey on mixture of experts in large language models.IEEE Transactions on Knowledge and Data Engineering, 2025

    2025

  5. [5]

    Prefixquant: Eliminating outliers by prefixed tokens for large language models quantization.arXiv preprint arXiv:2410.05265, 2024

    Mengzhao Chen, Yi Liu, Jiahao Wang, Yi Bin, Wenqi Shao, and Ping Luo. Prefixquant: Eliminating outliers by prefixed tokens for large language models quantization.arXiv preprint arXiv:2410.05265, 2024

    arXiv 2024

  6. [6]

    Efficientqat: Efficient quantization-aware training for large language models.arXiv preprint arXiv:2407.11062, 2024

    Mengzhao Chen, Wenqi Shao, Peng Xu, Jiahao Wang, Peng Gao, Kaipeng Zhang, and Ping Luo. Efficientqat: Efficient quantization-aware training for large language models.arXiv preprint arXiv:2407.11062, 2024

    arXiv 2024

  7. [7]

    On the meaning and use of kurtosis.Psychological methods, 2(3):292, 1997

    Lawrence T DeCarlo. On the meaning and use of kurtosis.Psychological methods, 2(3):292, 1997

    work page 1997

  8. [8]

    The case for 4-bit precision: k-bit inference scaling laws

    Tim Dettmers and Luke Zettlemoyer. The case for 4-bit precision: k-bit inference scaling laws. InInternational Conference on Machine Learning, pages 7750–7774. PMLR, 2023

    work page 2023

Show all 46 references
  1. [9]

    Learned step size quantization.arXiv preprint arXiv:1902.08153, 2019

    Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S Modha. Learned step size quantization.arXiv preprint arXiv:1902.08153, 2019

    1902 arXiv

  2. [10]

    Scaling fp8 training to trillion-token llms.arXiv preprint arXiv:2409.12517, 2024

    Maxim Fishman, Brian Chmiel, Ron Banner, and Daniel Soudry. Scaling fp8 training to trillion-token llms.arXiv preprint arXiv:2409.12517, 2024

    2024 arXiv

  3. [11]

    Gptq: Accurate post-training quantization for generative pre-trained transformers.arXiv preprint arXiv:2210.17323, 2022

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers.arXiv preprint arXiv:2210.17323, 2022

    2022 arXiv

  4. [12]

    Compression scaling laws: Unifying sparsity and quantization.arXiv preprint arXiv:2502.16440, 2025

    Elias Frantar, Utku Evci, Wonpyo Park, Neil Houlsby, and Dan Alistarh. Compression scaling laws: Unifying sparsity and quantization.arXiv preprint arXiv:2502.16440, 2025

    2025 arXiv

  5. [13]

    Language models scale reliably with over-training and on downstream tasks

    Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, et al. Language models scale reliably with over-training and on downstream tasks. arXiv preprint arXiv:2403.08540, 2024

    2024 arXiv

  6. [14]

    Mathematics of computation.American Mathematical Society, 24:23, 1970

    Donald Goldfarb. Mathematics of computation.American Mathematical Society, 24:23, 1970

    1970

  7. [15]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  8. [16]

    Training compute-optimal large language models

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022

    2022 arXiv

  9. [17]

    Peter J. Huber. Robust Estimation of a Location Parameter.The Annals of Mathematical Statistics, 35(1):73 – 101, 1964. doi: 10.1214/aoms/1177703732. URLhttps://doi.org/10.1214/aoms/1177703732

    1964

  10. [18]

    A study of bfloat16 for deep learning training

    Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, et al. A study of bfloat16 for deep learning training. arXiv preprint arXiv:1905.12322, 2019

    1905 arXiv

  11. [19]

    Jared Kaplan, Sam McCandlish, T. J. Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei. Scaling laws for neural language models.ArXiv, abs/2001.08361, 2020. URL https://api.semanticscholar.org/CorpusID:210861095

    2001 arXiv

  12. [20]

    Scaling laws for precision.arXiv preprint arXiv:2411.04330, 2024

    Tanishq Kumar, Zachary Ankner, Benjamin F Spector, Blake Bordelon, Niklas Muennighoff, Mansheej Paul, Cengiz Pehlevan, Christopher Ré, and Aditi Raghunathan. Scaling laws for precision.arXiv preprint arXiv:2411.04330, 2024. 11

    2024 arXiv

  13. [21]

    Predictable scale: Part i–optimal hyperparameter scaling law in large language model pretraining.arXiv preprint arXiv:2503.04715, 2025

    Houyi Li, Wenzheng Zheng, Jingcheng Hu, Qiufeng Wang, Hanshan Zhang, Zili Wang, Yangshijie Xu, Shuigeng Zhou, Xiangyu Zhang, and Daxin Jiang. Predictable scale: Part i–optimal hyperparameter scaling law in large language model pretraining.arXiv preprint arXiv:2503.04715, 2025

    2025 arXiv

  14. [22]

    Svdqunat: Absorbing outliers by low-rank components for 4-bit diffusion models.arXiv preprint arXiv:2411.05007, 2024

    Muyang Li, Yujun Lin, Zhekai Zhang, Tianle Cai, Xiuyu Li, Junxian Guo, Enze Xie, Chenlin Meng, Jun-Yan Zhu, and Song Han. Svdqunat: Absorbing outliers by low-rank components for 4-bit diffusion models.arXiv preprint arXiv:2411.05007, 2024

    2024

  15. [23]

    Awq: Activation-aware weight quantization for llm compression and acceleration.arXiv preprint arXiv:2306.00978, 2023

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq: Activation-aware weight quantization for llm compression and acceleration.arXiv preprint arXiv:2306.00978, 2023

    2023 arXiv

  16. [24]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  17. [25]

    Quantization hurts reasoning? an empirical study on quantized reasoning models.arXiv preprintarXiv:2504.04823, 2025

    Ruikang Liu, Yuxuan Sun, Manyi Zhang, Haoli Bai, Xianzhi Yu, Tiezheng Yu, Chun Yuan, and Lu Hou. Quantization hurts reasoning? an empirical study on quantized reasoning models.arXiv preprintarXiv:2504.04823, 2025

    2025 arXiv

  18. [26]

    Spinquant: Llm quantization with learned rotations.arXiv preprint arXiv:2405.16406, 2024

    Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Spinquant: Llm quantization with learned rotations.arXiv preprint arXiv:2405.16406, 2024

    2024 arXiv

  19. [27]

    Paretoq: Scaling laws in extremely low-bit llm quantization.arXiv preprint arXiv:2502.02631, 2025

    Zechun Liu, Changsheng Zhao, Hanxian Huang, Sijia Chen, Jing Zhang, Jiawei Zhao, Scott Roy, Lisa Jin, Yunyang Xiong, Yangyang Shi, et al. Paretoq: Scaling laws in extremely low-bit llm quantization.arXiv preprint arXiv:2502.02631, 2025

    2025

  20. [28]

    The era of 1-bit llms: All large language models are in 1.58 bits.arXiv preprint arXiv:2402.17764, 2024

    Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Li Dong, Ruiping Wang, Jilong Xue, and Furu Wei. The era of 1-bit llms: All large language models are in 1.58 bits.arXiv preprint arXiv:2402.17764, 2024

    2024 arXiv

  21. [29]

    Mitigating the impact of outlier channels for language model quantization with activation regularization.arXiv preprint arXiv:2404.03605, 2024

    Aniruddha Nrusimha, Mayank Mishra, Naigang Wang, Dan Alistarh, Rameswar Panda, and Yoon Kim. Mitigating the impact of outlier channels for language model quantization with activation regularization.arXiv preprint arXiv:2404.03605, 2024

    2024 arXiv

  22. [30]

    2 olmo 2 furious.arXiv preprint arXiv:2501.00656, 2024

    Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2 olmo 2 furious.arXiv preprint arXiv:2501.00656, 2024

    2024 arXiv

  23. [31]

    Low-bit quantization favors undertrained llms: Scaling laws for quantized llms with 100t training tokens.arXiv preprint arXiv:2411.17691, 2024

    Xu Ouyang, Tao Ge, Thomas Hartvigsen, Zhisong Zhang, Haitao Mi, and Dong Yu. Low-bit quantization favors undertrained llms: Scaling laws for quantized llms with 100t training tokens.arXiv preprint arXiv:2411.17691, 2024

    2024 arXiv

  24. [32]

    Quest: Stable training of llms with 1-bit weights and activations.arXiv preprint arXiv:2502.05003, 2025

    Andrei Panferov, Jiale Chen, Soroush Tabesh, Roberto L Castro, Mahdi Nikdan, and Dan Alistarh. Quest: Stable training of llms with 1-bit weights and activations.arXiv preprint arXiv:2502.05003, 2025

    2025 arXiv

  25. [33]

    Fp8-lm: Training fp8 large language models.arXiv preprint arXiv:2310.18313, 2023

    Houwen Peng, Kan Wu, Yixuan Wei, Guoshuai Zhao, Yuxiang Yang, Ze Liu, Yifan Xiong, Ziyue Yang, Bolin Ni, Jingcheng Hu, et al. Fp8-lm: Training fp8 large language models.arXiv preprint arXiv:2310.18313, 2023

    2023 arXiv

  26. [34]

    Microscaling data formats for deep learning

    Bita Darvish Rouhani, Ritchie Zhao, Ankit More, Mathew Hall, Alireza Khodamoradi, Summer Deng, Dhruv Choudhary, Marius Cornea, Eric Dellinger, Kristof Denolf, et al. Microscaling data formats for deep learning. arXiv preprint arXiv:2310.10537, 2023

    arXiv

  27. [35]

    Seed1.5-thinking: Advancing superb reasoning models with reinforcement learning.arXiv preprint arXiv:2504.13914, 2025

    ByteDance Seed. Seed1.5-thinking: Advancing superb reasoning models with reinforcement learning.arXiv preprint arXiv:2504.13914, 2025

    2025

  28. [36]

    Omniquant: Omnidirectionally calibrated quantization for large language models.arXiv preprint arXiv:2308.13137, 2023

    Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models.arXiv preprint arXiv:2308.13137, 2023

    2023 arXiv

  29. [37]

    Glu variants improve transformer.arXiv preprint arXiv:2002.05202, 2020

    Noam Shazeer. Glu variants improve transformer.arXiv preprint arXiv:2002.05202, 2020

    2002 arXiv

  30. [38]

    Scaling laws for floating point quantization training.arXiv preprint arXiv:2501.02423, 2025

    Xingwu Sun, Shuaipeng Li, Ruobing Xie, Weidong Han, Kan Wu, Zhen Yang, Yixing Li, An Wang, Shuai Li, Jinbao Xue, et al. Scaling laws for floating point quantization training.arXiv preprint arXiv:2501.02423, 2025

    2025 arXiv

  31. [39]

    Training llms with mxfp4.arXiv preprint arXiv:2502.20586, 2025

    Albert Tseng, Tao Yu, and Youngsuk Park. Training llms with mxfp4.arXiv preprint arXiv:2502.20586, 2025

    2025 arXiv

  32. [40]

    Bitnet a4

    Hongyu Wang, Shuming Ma, and Furu Wei. Bitnet a4. 8: 4-bit activations for 1-bit llms. arXiv preprint arXiv:2411.04965, 2024. 12

    2024 arXiv

  33. [41]

    Optimizing large language model training using fp4 quantization.arXiv preprint arXiv:2501.17116, 2025

    Ruizhe Wang, Yeyun Gong, Xiao Liu, Guoshuai Zhao, Ziyue Yang, Baining Guo, Zhengjun Zha, and Peng Cheng. Optimizing large language model training using fp4 quantization.arXiv preprint arXiv:2501.17116, 2025

    2025 arXiv

  34. [42]

    Smoothquant: Accurate and efficient post-training quantization for large language models

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. InInternational Conference on Machine Learning, pages 38087–38099. PMLR, 2023

    2023

  35. [43]

    Llm inference unveiled: Survey and roofline model insights.arXiv preprint arXiv:2402.16363, 2024

    Zhihang Yuan, Yuzhang Shang, Yang Zhou, Zhen Dong, Zhe Zhou, Chenhao Xue, Bingzhe Wu, Zhikai Li, Qingyi Gu, Yong Jae Lee, et al. Llm inference unveiled: Survey and roofline model insights.arXiv preprint arXiv:2402.16363, 2024

    2024 arXiv

  36. [44]

    Accurate int8 training through dynamic block-level fallback.arXiv preprint arXiv:2503.08040, 2025

    Pengle Zhang, Jia Wei, Jintao Zhang, Jun Zhu, and Jianfei Chen. Accurate int8 training through dynamic block-level fallback.arXiv preprint arXiv:2503.08040, 2025

    2025 arXiv

  37. [45]

    An empirical study of qwen3 quantization.arXiv preprint arXiv:2505.02214, 2025

    Xingyu Zheng, Yuye Li, Haoran Chu, Yue Feng, Xudong Ma, Jie Luo, Jinyang Guo, Haotong Qin, Michele Magno, and Xianglong Liu. An empirical study of qwen3 quantization.arXiv preprint arXiv:2505.02214, 2025. URL https://arxiv.org/abs/2505.02214

    2025 arXiv

  38. [46]

    A survey on efficient inference for large language models.arXiv preprintarXiv:2404.14294, 2024

    Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, et al. A survey on efficient inference for large language models.arXiv preprintarXiv:2404.14294, 2024. 13 Appendix A Limitations This paper proposes a unifie...

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.