Pith. sign in

REVIEW 4 major objections 6 minor 27 references

Taming Sensitive Weights : Noise Perturbation Fine-tuning for Robust LLM Quantization

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read By adding per-channel random noise to outlier weights during a short LoRA fine-tuning, NPFT shrinks their Hessian sensitivity, letting plain round-to-nearest quantization match GPTQ at 4 bits on LLaMA2-7B with no outliers preserved in…

desk verdict Useful empirical recipe for quantizing LLMs without outlier retention, but the paper never shows that the noise perturbation itself is what does the work. read the letter →

arxiv 2412.06858 v2 pith:NHXGOR3L submitted 2024-12-08 cs.LG cs.AIcs.CL

Dongwei Wang , Huanrui Yang This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords LLMquantizationoutlierweightsHessiantracenoiseperturbationfine-tuningLoRApost-traininground-to-nearestperplexity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models hide a small set of outlier weights that, when rounded to low precision, wreck the model's predictions. Existing quantizers protect these outliers by keeping them in floating point, forcing a mixed-precision format that slows GPU inference. This paper claims the outliers can instead be tamed: adding per-channel random noise to their locations during a short LoRA fine-tuning implicitly shrinks the loss Hessian trace at those weights, making them robust to rounding. After this tuning, the simplest round-to-nearest quantizer matches the more elaborate GPTQ method at 4 bits on LLaMA2-7B (7.42 perplexity on C4), and quantized models no longer need FP16 outliers, cutting inference latency by roughly a tenth. The payoff is that hardware-friendly single-precision quantization can replace mixed-precision schemes without giving up quality.

What carries the argument

The carrying object is the identity $\operatorname{Tr}(H) = \mathbb{E}_z[z^T H z]$ for random vectors $z$ with zero mean and identity covariance, joined with the second-order Taylor expansion of the loss. This converts the expensive goal of shrinking the Hessian trace at outlier weights into a cheap one: minimize the expected loss $\mathbb{E}_z L(W+z)$ under random perturbations. The paper instantiates that with per-channel zero-mean uniform noise, scaled to each outlier-containing channel's value range (mimicking the $\pm\Delta/2$ error bound of round-to-nearest quantization), applied only to the top-$\gamma\%$ of weights ranked by the Fisher Information Matrix $F = \|\partial L/\partial w_{i,j}\|_2^2$. LoRA adapters carry the optimization through the objective $\min_{U,V}\big[L(W+z_i+U^T V)+\beta L(W+U^T V)\big]$ and merge back afterward, keeping the whole procedure to about an hour on a 7B model: the regularization is implicit, and no Hessian is ever formed.

What would settle it

On a model small enough to form its Hessian (for example OPT-125M), measure the Hessian trace restricted to the outlier channels before and after NPFT; the mechanism predicts a substantial drop. A second, quantizer-level test: fine-tune two copies of one model, one with NPFT's synthetic uniform noise and one with perturbations drawn from the actual RTN or GPTQ rounding residuals on the calibration set, then compare 4-bit perplexity; if the residual-matched copy does not match or beat the synthetic-noise copy, the surrogate assumption, rather than Hessian-trace reduction itself, is what carried the result.

Watch

Extended reading notes

Core claim

NPFT's central claim is that the sensitivity of outlier weights can be reduced directly, so that special handling is unnecessary. For a converged model, Taylor expansion says the loss increase under a weight perturbation $z$ is approximately $\tfrac{1}{2} z^T H z$, so its expectation over zero-mean, identity-covariance perturbations is proportional to the Hessian trace; since round-to-nearest error is bounded by half a quantization bin, the paper samples zero-mean uniform noise on each channel that contains outliers as a stand-in for that error. Fine-tuning with LoRA to minimize the expected loss under this noise, while a weighted clean-loss term guards base performance, then acts as cheap Hessian regularization, avoiding any higher-order gradient computation. The paper's headline result is that with NPFT the simple RTN quantizer reaches 7.42 PPL on C4 for LLaMA2-7B-4bits, identical to GPTQ, and that NPFT improves both uniform and non-uniform quantizers across OPT and LLaMA models, with sqLLM+NPFT matching the mixed-precision sqLLM at lower latency.

Load-bearing premise

Everything rests on treating random zero-mean uniform noise spread over outlier channels as a faithful stand-in for the real, deterministic rounding errors the quantizer will later make; if those two distributions disagree, making the model robust to the noise will not make it robust to quantization.

Editorial extensions

If this is right

  • The simplest quantizer, RTN, becomes competitive with GPTQ after NPFT on LLaMA2-7B-4bits (7.42 vs. 7.42 PPL on C4), so the costliest parts of post-training quantization may be unnecessary once the weights are noise-regularized.
  • Outlier weights are quantized at the same bit-width as all other weights, eliminating the FP16 sparse matrices whose mixed-precision format slows GPU kernels and wastes memory bandwidth.
  • The gains transfer across quantizer families (uniform RTN and GPTQ, non-uniform sqLLM) and model families (OPT, LLaMA), including unseen datasets such as WikiText and PTB and reasoning tasks such as PIQA, ARC, and Storycloze, so the tuning is not tailored to one quantizer.
  • A single NPFT pass serves multiple bit-widths at once and uses roughly a quarter of EfficientQAT's training time on LLaMA2-7B, lowering the adoption barrier for retraining-based quantization.
  • With the sqLLM quantizer, NPFT's single-precision 4-bit model matches the mixed-precision baseline's perplexity while cutting generation latency by about 10% and slightly reducing GPU memory.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because any zero-mean perturbation whose covariance is proportional to the identity yields the same trace identity, the specific uniform distribution is probably not load-bearing; the paper's own ablation with uniform, Gaussian, and Laplace noise points the same way, and a natural extension is to draw perturbations from the target quantizer's actual rounding residuals instead of from a channel's d
  • The noise is synthesized per channel from that channel's min-max range, so channels with wider ranges receive bigger perturbations and should dominate the training signal; a testable prediction is that outlier sensitivity drops most in exactly those channels.
  • If the mechanism is genuinely Hessian-trace reduction rather than generic LoRA adaptation, then on a small model where the Hessian is computable, the measured trace on outlier channels should fall after NPFT by an amount predictable from the perturbed-loss objective, and a LoRA-only control without noise should show little or no quantization gain; neither check appears in the paper.
  • NPFT is complementary to quantizer design: its benefit is largest for the crudest quantizer (RTN) and smallest for the already sensitivity-aware sqLLM, so stacking it with activation-aware scaling or non-uniform schemes may extend gains into the 3-bit and sub-3-bit regimes that this paper does not reach (its RTN+NPFT still diverges at 3 bits on OPT).
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Noise Perturbation Fine-tuning (NPFT), a parameter-efficient fine-tuning method that adds random perturbations to a small set of outlier weights (identified by Fisher sensitivity) while training LoRA adapters. The stated goal is to reduce the Hessian trace with respect to those outliers, making the model more robust to post-training quantization so that uniform quantization such as RTN can approach the performance of mixed-precision methods. The authors report perplexity gains on OPT and LLaMA models for RTN, GPTQ, and sqLLM quantizers, including a headline claim that RTN+NPFT matches GPTQ on LLaMA2-7B-4bit, plus latency/memory efficiency comparisons on a 4090 GPU.

Significance. If the mechanism were fully established, NPFT would be a practically valuable contribution: it promises a way to avoid hardware-unfriendly mixed-precision outlier retention while improving quantized model quality, with modest training cost. The paper also provides useful engineering data: one-shot fine-tuning that benefits multiple bit-widths, latency/memory profiling against a QAT baseline, and ablations over noise distribution, layer choice, and calibration data. The main empirical observation, that NPFT improves quantized perplexity across models and quantizers, is credible as reported. However, the central causal claim—that the noise perturbation specifically drives the improvement via Hessian-trace regularization—is not supported by the current experimental design, because no no-noise LoRA fine-tuning control is presented.

major comments (4)
  1. [§3.3, Eq. (7); Tables 1–2; Fig. 4] The paper never ablates the noise perturbation itself. The training objective in Eq. (7) contains a clean-loss term βL(W+UᵀV); setting z_i=0 reduces the method to standard LoRA fine-tuning on the same calibration data. Tables 1–2 compare RTN+NPFT against RTN, GPTQ, etc., but not against “LoRA fine-tuning without noise,” and Fig. 4 sweeps the perturbation ratio starting at 0.05% rather than 0. Consequently, the reported PPL gains could in principle come entirely from LoRA adaptation to the calibration data, with the noise contributing little or nothing. This is load-bearing because the paper's title, abstract, and theoretical motivation (Eqs. 4–6) attribute the gains specifically to perturbation-based implicit Hessian regularization. The authors should add a no-noise LoRA control (z_i=0, same hyperparameters) to Tables 1–2 and include a 0% perturbation point in Fig. 4; if the no-noise control matches the NPFT results, the mechanism claim would need to be substantially revised.
  2. [§3.2, Eqs. (5)–(6); §3.3, Algorithm 1] The noise distribution used in the implementation does not match the theoretical derivation. Eq. (6) assumes z is uniform on [−Δ/2, Δ/2], with covariance (Δ²/12)I, so that the expected loss under perturbation is proportional to Tr(H). Algorithm 1, however, samples per-channel noise from the channel's weight range (max−min), centered by subtracting the mean; the noise magnitude is therefore determined by the channel's data range, not by the quantization bin width Δ of the target bit-width. This breaks the proportionality Tr(H) ∝ E_z[L(W+z)−L(W)] because Cov(z) is no longer a constant multiple of the identity (different channels have different scales, and the scale is unrelated to Δ). If the intent is to mimic RTN rounding error, the perturbation should be tied to Δ or to the actual rounding residuals; as written, the link between the synthetic noise and the quantization error that motivates Eqs. (5)–(6) is unsubstantiated.
  3. [§4.3, Fig. 3] The 'theoretical insight verification' is circular. Fig. 3 reports that after NPFT, the Fisher sensitivity F decreases for the selected outlier positions. But F is exactly the statistic used in Algorithm 1 to choose the outliers in the first place. Showing that a quantity decreases after fine-tuning when it was used as a selection criterion does not independently confirm that the mechanism is Hessian-trace reduction; it could simply reflect regression to the mean or the effect of LoRA fine-tuning on the Fisher diagonal. A direct estimate of the Hessian trace (e.g., using the Hutchinson estimator on a subset of weights) before and after NPFT, or a per-layer breakdown of quantized PPL changes, would be more conclusive.
  4. [Introduction (contribution list) and Table 1] The stated improvement for OPT-1.3B-4bit is inconsistent. The introduction claims 'NPFT helps RTN achieve a 3.69 perplexity improvement on OPT-1.3B-4bits,' but Table 1 reports RTN at 24.68 and RTN+NPFT at 21.74 on C4, which is a 2.94 PPL improvement. The Figure 1 caption likewise says 'over 2.9/0.9,' not 3.69. Please reconcile the numbers; if 3.69 refers to a different configuration or dataset, state that clearly. This matters because the large claimed improvement is highlighted in the contribution summary.
minor comments (6)
  1. [§4.4, training efficiency discussion] The text states that NPFT 'enabl[es] us to perform full-parameter fine-tuning of LLaMA2-7B on a single V100 GPU,' but the method uses LoRA adapters, which is parameter-efficient fine-tuning, not full-parameter fine-tuning. Please correct this phrasing or clarify what is meant.
  2. [Table numbering] The text in §4.4 refers to 'shown in Tab. 11' when the training-time comparison appears as Table 4, and an unnumbered Table 11 is placed in the appendix after Table 10. Please renumber the tables consistently and fix the in-text references.
  3. [Algorithm 1, line 11] The pseudocode samples noise as torch.rand_like(W[:, col_idx]) * (max − min) + min, followed by subtracting the mean. This produces a uniform distribution over the channel's data range, not over [−Δ/2, Δ/2] as claimed in §3.2. The pseudocode should be aligned with the theoretical derivation, or the derivation should be revised to cover the actual noise distribution.
  4. [Eq. (3)] Equation (3) writes H ≈ F = ggᵀ = ||∂L/∂w_{i,j}||²₂. The last equality conflates a full outer-product (or its expectation) with a single squared gradient component; the notation should distinguish the Fisher matrix from its diagonal approximation used for sensitivity scoring.
  5. [Table 3 and efficiency claim] The text claims 'a 10% reduction in inference latency' from Table 3, but for LLaMA2-7B-4bits the latency is identical (4.80 s) for sqLLM and sqLLM+NPFT; the 10% figure comes only from OPT-1.3B. Please qualify the claim accordingly.
  6. [References] Reference [11] attributes GPT-3 to Floridi and Chiriatti rather than to the original Brown et al. paper; please use the canonical citation. Also, the 'Preprint at arXiv' entry for the LLaMA paper is incompletely formatted.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the headline quantized-perplexity gains are external benchmark outcomes, and the Hessian-trace-to-perturbation derivation is a standard Taylor/Hutchinson argument rather than an identity with the paper's own inputs.

full rationale

NPFT's main claims are judged on external quantization benchmarks, not on quantities fitted into the method itself. Tables 1-2 report C4 and WikiText2 perplexities of RTN+NPFT, GPTQ+NPFT, and sqLLM+NPFT against the original PTQ baselines, and these numbers are obtained by actually quantizing the fine-tuned model; they are not equal by construction to any calibration loss, perturbation scale, or Hessian estimate in Eqs. (4)-(7) or Algorithm 1. The chain from Hessian trace to random perturbation uses the standard Hutchinson identity Tr(H) = E[z^T H z] and a second-order Taylor expansion, both of which are independent mathematical facts and do not presuppose the paper's conclusion. The use of the authors' prior HERO work [25] is motivational rather than the sole load-bearing support, since HAWQ [7,8] is independently cited for Hessian-based sensitivity, and the NPFT-specific fine-tuning pipeline is evaluated directly on quantized-model benchmarks. The only self-referential aspect is Section 4.3's verification, which measures Fisher sensitivity F, the same statistic used in Eq. (3) and Algorithm 1 to select outlier positions, and the paper does not include a no-noise LoRA control. That is a genuine causal-identification weakness, but it does not reduce any reported prediction to an input by construction: the observed decrease in F at selected positions is not logically guaranteed by the training objective and is not the headline benchmark result. Therefore, under the stated hard rules, there is no significant circularity in the paper's derivation chain.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The method relies on existing tools (randomized trace estimation, Fisher-based sensitivity, LoRA) rather than inventing new physical entities. The main ledger entries are the free hyperparameters (gamma, beta, training schedule) and several approximations about the relationship between synthetic noise, true quantization error, and the Hessian. The most fragile entries are the Fisher-as-Hessian proxy and the surrogate noise distribution.

free parameters (3)
  • Perturbation ratio gamma = 0.5% (OPT), 0.05% (LLaMA)
    Controls the fraction of weights selected as outliers and perturbed; chosen per model family, not derived from theory.
  • Balance coefficient beta = 0.5
    Weights the unperturbed loss term in Eq. (7); set uniformly across all experiments with no sensitivity study.
  • Learning rate and epochs = 5e-6 for 6 epochs (OPT), 5e-5 for 3 epochs (LLaMA)
    Training schedule set by hand; the paper does not study sensitivity to these choices.
assumptions (6)
  • standard math Randomized trace estimation identity Tr(H) = E[z^T H z] holds for zero-mean identity-covariance z.
    Invoked in Eq. (4), cited to Avron-Toledo; the algorithm approximately implements this per channel.
  • domain assumption The pretrained model is near convergence, so the gradient is approximately zero and first-order Taylor terms can be dropped.
    Used in Eqs. (2) and (5) to justify that quantization loss is dominated by the second-order Hessian term; reasonable for trained models but not empirically checked here.
  • domain assumption Hessian can be approximated by the per-sample gradient outer product F = gg^T for outlier identification.
    Used in Eq. (3) and inherited from SqueezeLLM-style sensitivity analysis; this is not a mathematically valid Hessian approximation and is especially questionable near convergence where gradients are small.
  • domain assumption Quantization error can be modeled as zero-mean uniform noise with covariance proportional to identity.
    Used in Eqs. (5)-(6) and Algorithm 1; actual RTN and GPTQ residuals are bounded but not generally zero-mean or uncorrelated across weights.
  • domain assumption Outliers are concentrated in a small number of output channels, so perturbing entire selected channels covers all sensitive weights.
    Borrowed from SpQR and operationalized in Algorithm 1; the paper does not verify this concentration on the exact OPT and LLaMA models used.
  • domain assumption Low-rank LoRA updates can reduce the Hessian trace with respect to outlier directions without excessively damaging base model loss.
    This is the central mechanism of Eq. (7); no convergence guarantee or direct Hessian measurement is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Taming Sensitive Weights : Noise Perturbation Fine-tuning for Robust LLM Quantization." pith.science (2026). https://pith.science/paper/NHXGOR3L

@misc{pith2026241206858,
  author       = {Pith},
  title        = {Pith review of: Taming Sensitive Weights : Noise Perturbation Fine-tuning for Robust LLM Quantization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NHXGOR3L}},
  note         = {Machine review of arXiv:2412.06858}
}
read the original abstract

Quantization is a critical step to enable efficient LLM serving under limited resource. However, previous research observes that certain weights in the LLM, known as outliers, are significantly sensitive to quantization noises. Existing quantization methods leave these outliers as floating points or higher precisions to retain performance, posting challenges on the efficient hardware deployment of the mixed-precision model. This work investigates an alternative way to tame the sensitive weights' impact on the quantization error, by reducing the loss Hessian trace with respect to outliers through an efficient fine-tuning process. We propose Noise Perturbation Fine-tuning (NPFT), which identifies outlier weights and add random weight perturbations on the outliers as the model going through a PEFT optimization. NPFT tames the sensitivity of outlier weights so that the quantized model performance can be improved without special treatment to the outliers. When applied to OPT and LLaMA models, our NPFT method achieves stable performance improvements for both uniform and non-uniform quantizers, while also offering better inference efficiency. Notably, the simplest RTN can achieve performance on par with GPTQ using our NPFT on LLaMA2-7B-4bits benchmark.

Figures

Figures reproduced from arXiv: 2412.06858 by the authors.

Figure 1
Figure 1. (Left) RTN suffers from the degradation caused by quantizing outliers. Preserving 0.5% of [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. (Left) Existing PTQ methods preserve outliers in FP16 to prevent significant performance [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Visualization of outlier sensitivity in OPT-1.3B. The sensitivity is obtained by calculating [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Training loss curves of OPT-1.3B. The perturbed model converges more swiftly with β > 0. number of training steps. When the perturbation ratio is relatively low (less than 2%), the model’s performance does not vary significantly. However, as the ratio increases, it int…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 13 canonical work pages

  1. [1]

    Randomized algorithms for estimating the trace of an implicit symmetric positive semi-definite matrix

    Haim Avron and Sivan Toledo. Randomized algorithms for estimating the trace of an implicit symmetric positive semi-definite matrix. Journal of the ACM (JACM), 58(2):1–34, 2011. 4

    work page 2011

  2. [2]

    Efficientqat: Efficient quantization-aware training for large language models

    Mengzhao Chen, Wenqi Shao, Peng Xu, Jiahao Wang, Peng Gao, Kaipeng Zhang, Yu Qiao, and Ping Luo. Efficientqat: Efficient quantization-aware training for large language models. arXiv preprint arXiv:2407.11062, 2024. 3, 6, 8

    arXiv 2024

  3. [3]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1– 113, 2023. 1

    work page 2023

  4. [4]

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems , 35:30318–30332, 2022. 3

    work page 2022

  5. [5]

    Qlora: Efficient finetuning of quantized llms

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems , 36, 2024. 3

    work page 2024

  6. [6]

    Spqr: A sparse-quantized representation for near-lossless llm weight compression

    Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. Spqr: A sparse-quantized representation for near-lossless llm weight compression. arXiv preprint arXiv:2306.03078 ,

    arXiv

  7. [7]

    Hawq-v2: Hessian aware trace-weighted quantization of neural networks

    Zhen Dong, Zhewei Yao, Daiyaan Arfeen, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Hawq-v2: Hessian aware trace-weighted quantization of neural networks. Advances in neural information processing systems, 33:18518–18529, 2020. 2, 3

    work page 2020

  8. [8]

    Hawq: Hessian aware quantization of neural networks with mixed-precision

    Zhen Dong, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Hawq: Hessian aware quantization of neural networks with mixed-precision. In Proceedings of the IEEE/CVF international conference on computer vision , pages 293–302, 2019. 2, 3

    work page 2019

Show all 27 references
  1. [9]

    Bitdistiller: Unleashing the potential of sub-4-bit llms via self-distillation

    Dayou Du, Yijia Zhang, Shijie Cao, Jiaqi Guo, Ting Cao, Xiaowen Chu, and Ningyi Xu. Bitdistiller: Unleashing the potential of sub-4-bit llms via self-distillation. arXiv preprint arXiv:2402.10631, 2024. 3

    2024 arXiv

  2. [10]

    Glam: Efficient scaling of language models with mixture-of-experts

    Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning , pages 5547–5569. PM...

    2022

  3. [11]

    Gpt-3: Its nature, scope, limits, and consequences

    Luciano Floridi and Massimo Chiriatti. Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines, 30:681–694, 2020. 1

    2020

  4. [12]

    Gptq: Accurate post-training quantization for generative pre-trained transformers

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323, 2022. 1, 3, 5

    2022 arXiv

  5. [13]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 5

    2021 arXiv

  6. [14]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. 1

    2001 arXiv

  7. [15]

    Squeezellm: Dense-and-sparse quantization

    Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. Squeezellm: Dense-and-sparse quantization. arXiv preprint arXiv:2306.07629, 2023. 1, 2, 3, 4, 5, 6, 8

    2023 arXiv

  8. [16]

    Awq: Activation-aware weight quantization for on-device llm compression and acceleration

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87–100,...

    2024

  9. [17]

    Llm-qat: Data-free quan- tization aware training for large language models

    Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. Llm-qat: Data-free quan- tization aware training for large language models. arXiv preprint arXiv:2305.17888 , 2023. 3 10

    2023 arXiv

  10. [18]

    Pointer sentinel mixture models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016. 5

    2016 arXiv

  11. [19]

    Mixed-Precision Training Guide, 2023

    NVIDIA Corporation. Mixed-Precision Training Guide, 2023. Accessed: 2024-11-04. 2

    2023

  12. [20]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020. 5, 6

    2020

  13. [21]

    Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model

    Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model. ...

    2022 arXiv

  14. [22]

    Open and efficient foundation language models

    H Touvron, T Lavril, G Izacard, X Martinet, MA Lachaux, T Lacroix, B Rozière, N Goyal, E Hambro, F Azhar, et al. Open and efficient foundation language models. Preprint at arXiv. https://doi. org/10.48550/arXiv, 2302, 2023. 1, 5

    2023 doi

  15. [23]

    Outlier suppression: Pushing the limit of low-bit transformer language models

    Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fengwei Yu, and Xianglong Liu. Outlier suppression: Pushing the limit of low-bit transformer language models. Advances in Neural Information Processing Systems , 35:17402–17414, 2022. 3

    2022

  16. [24]

    Bloom: A 176b-parameter open-access multilingual language model

    BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili´c, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100,

    arXiv

  17. [25]

    Hero: Hessian-enhanced robust optimization for unifying and improving generalization and quantization performance

    Huanrui Yang, Xiaoxuan Yang, Neil Zhenqiang Gong, and Yiran Chen. Hero: Hessian-enhanced robust optimization for unifying and improving generalization and quantization performance. In Proceedings of the 59th ACM/IEEE Design Automation Conference , pages 25–30, 2022. 2, 3, 4

    2022

  18. [26]

    Opt: Open pre-trained transformer language models

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022. 1, 5 11 A Performances under different noi...

    2022 arXiv

  19. [27]

    Table 10: OPT-1.3B-4bit latency(s) for generating different lengths of tokens

    The results show that the latency speedup of our method increases with longer outputs, achieving over a 3s improvement when generating 2048 tokens on RTX4090. Table 10: OPT-1.3B-4bit latency(s) for generating different lengths of tokens. Method num of tokens latency (s) 128 25...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.