Pith. sign in

REVIEW 4 major objections 5 minor 40 references

Gaussian Weight Sampling for Scalable, Efficient and Stable Pseudo-Quantization Training

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Pseudo-quantization training becomes both FP-friendly and efficient when the added noise is rounded normal: the method matches or beats BF16 pre-training of GPT2-124M and Llama2-134M/1B up to 300B tokens while adding as little as 1.4%…

desk verdict A genuinely new PQT noise distribution and analysis, with a load-bearing unverified bitwise generation claim and no direct FP6/FP8 training—worth refereeing, not yet fully proven. read the letter →

arxiv 2505.11170 v1 pith:FOMZARMH submitted 2025-05-16 cs.LG

Myeonghwan Ahn , Sungjoo Yoo This is my paper · ORCID
classification cs.LG
keywords pseudo-quantizationtrainingroundednormalnoiselow-precisionfloating-pointstochasticprecisionannealinglargelanguagemodelpre-trainingbitwisegenerationFP6GaussWS
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Fully quantized training (FQT) promises cheaper LLM training but suffers from forward/backward inconsistencies and an exponential search over bitwidth assignments, each case requiring over 200 billion tokens to validate. Pseudo-quantization training (PQT) avoids the search by adding noise instead of rounding, yet existing PQT noise is not friendly to floating-point arithmetic. This paper proposes using rounded normal noise, $R = \lfloor N(0,1)/2 \rceil$, as the fake-quantization distribution, which leaves the weight untouched with probability about 0.717 and has minimum nonzero magnitude 1. The paper proves that these two properties support bitwidths up to 9-bit with BF16 operators and weights down to FP6, and it shows empirically that the resulting method, GaussWS, stabilizes pre-training of GPT2-124M and Llama2-{134M,1B} up to 300B tokens, matching or beating BF16 baselines with as little as 1.40% training throughput overhead on A100 GPUs.

What carries the argument

The central object is the rounded normal noise distribution $R = \lfloor N(0,1)/2 \rceil$ (nearest integer to a standard normal divided by 2), and the mechanism it drives is the FP casting of the fake-quantized weight $\hat{w} = w + R \odot \mathrm{broadcast}_{bl}(\max_{bl}(|w|)\cdot 2^{1-b_t})$. The paper reduces the design problem to two parameters: the minimum nonzero magnitude $2^\tau$ and the zero probability $p$. Lemmas 1 and 2 give inequalities showing that $\tau$ must be small enough to keep near-zero weights from underflowing and large enough (coupled with $m$, the mantissa bits) to keep the noise itself from underflowing; Proposition 3 converts those bounds into a required exponent-bit count for $w$ and $\hat{w}$, and Proposition 4 shows that a nonzero $p$ implements stochastic precision annealing. The rounded normal has $\tau=0$ and $p \approx 0.717$, which resolves the tension, supports $b_t$ up to 8 with BF16 operators and FP6 at $b_t \le 4$. Efficient generation replaces the standard normal-sampling transform with bitwise AND/OR combinations of PRNG bits, packed 8 samples to a 32-bit register in a sign-mantissa format.

What would settle it

Run the bitwise noise generator and empirically measure the distribution of R over a large sample: if Pr(R=0) differs from $\approx 0.717$ or the nonzero values are not $\pm1$ and $\pm2$, then the stochastic-precision-annealing bound in Proposition 4 is broken. A complementary test is to pre-train GPT2-124M twice, once with the bitwise R and once with an exact rounded normal sampled via the standard transform, and check whether the loss curves differ systematically.

Watch

Extended reading notes

Core claim

The paper claims that pseudo-quantization training (PQT) — replacing quantized weights with a high-precision weight plus differentiable noise — becomes practical for low-precision floating-point formats once the noise is drawn from a rounded normal distribution, $R = \lfloor N(0,1)/2 \rceil$, i.e., the nearest integer to a standard normal divided by two. The key insight is that the FP casting of $w + \text{noise}$ underflows the smaller of the two, so the noise must have a minimum nonzero magnitude that is large enough to survive (Lemma 1) yet small enough to preserve near-zero weights (Lemma 2). A distribution with a high probability $p$ of $R = 0$ resolves the conflict via stochastic precision annealing (Proposition 4): with probability $p$ the weight is kept exact, and with probability $1-p$, weights below a threshold are masked to zero, training the model to tolerate a reduced dynamic range. The rounded normal has $\tau = 0$ (minimum nonzero magnitude 1) and $p \approx 0.717$, which the paper proves yields FP exponent and mantissa requirements down to FP6_e3m2 for bitwidth 4 and up to 9-bit noise with BF16 operators. The paper then shows the distribution can be generated by bitwise AND/OR of PRNG bits (8 samples packed in a 32-bit register), avoiding normal-sampling arithmetic, and demonstrates stable pre-training of GPT2-124M and Llama2-134M/1B up to 300B tokens, matching or beating BF16 with as little as 1.40% throughput overhead on A100 GPUs.

Load-bearing premise

The bitwise construction of the noise distribution assumes that every bit of the pseudo-random number generator's output is independent and fair; if bits are correlated, the claimed 71.7% chance of zero and the underflow bounds do not hold.

Editorial extensions

If this is right

  • Bitwidth becomes a trainable hyperparameter rather than a search point: with GaussWS, each 32×32 block has its own $b_t$, updated by gradient, reducing the exponential FQT search to O(1) and removing the need to validate each configuration with 200B-token runs.
  • The underflow bounds give a direct recipe for picking FP datatypes: for a target bitwidth $b_t$, weights require $\lceil \log_2(b_t+1) \rceil$ exponent bits and sampled weights $\lceil \log_2(b_t+3) \rceil$, which for $b_t=4$ is FP6_e3m2 and for $b_t=9$ is FP16_e5m10.
  • GaussWS stabilizes pre-training in the two regimes where BF16 diverges: the attention layers around 30B tokens and the feed-forward layers around 200B tokens, and it works with both AdamW and Adam-mini optimizers.
  • Efficiency is not bought at the cost of accuracy: across GPT2-124M and Llama2-134M/1B, GaussWS matches or exceeds the BF16 loss curve (with the 1B model at $b_{target}=4$ slightly behind, and a preliminary run at $b_{target}=6$ closing the gap), while adding only 2 bytes per parameter of GPU memory and a geometric-mean 3.14% throughput overhead versus 22.34% for uniform-noise PQT.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same underflow analysis could be carried over to activations and gradients: if a similar rounded distribution works there, weight-only GaussWS would generalize to a full FQT pipeline with a fixed noise family, removing the per-tensor case-search the paper highlights.
  • The bitwise approximation trades exact normality for speed; a direct head-to-head of the bitwise R against an exact rounded normal (sampled via the standard transform) on the same training runs would quantify how much of the reported behavior depends on the exact distribution shape versus the $\tau=0$, $p\approx0.717$ parameters.
  • The $\approx 71.7\%$ zero probability makes GaussWS act like a stochastic regularizer in weight space; if this intuition holds, the method may also improve generalization of low-precision models, a claim the paper does not test.
  • The paper's own scaling-law discussion implies $b_{target}$ should grow with model size and token count; an immediately testable extension is to anneal $b_{target}$ during training (e.g., from $b_{init}$ to a schedule) rather than fixing it, which could close the Llama2-1B gap at higher token budgets.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper proposes Gaussian weight sampling (GaussWS), a pseudo-quantization training (PQT) method in which a rounded-normal noise distribution R is added to weights before floating-point casting, with a learned blockwise bitwidth. The central claims are that the method (1) supports low-precision FP parameters down to FP6 and noise precision up to 9-bit when using BF16 operators, (2) adds only about 1.40% training throughput overhead on A100 GPUs while storing 2 bytes per parameter, and (3) provides stable pretraining that closely follows or outperforms a BF16 baseline on GPT2-124M, Llama2-134M, and Llama2-1B models up to roughly 300B tokens. The paper derives underflow bounds (Lemmas 1 and 2), an FP exponent-cutoff proposition (Proposition 3), and a stochastic precision annealing result (Proposition 4) for the proposed R, and implements R generation with bitwise operations on PRNG outputs.

Significance. If the results hold, GaussWS would be a useful contribution to low-precision LLM training. The theoretical bounds in Section 3.3 are parameter-free given the proposed R, and the bitwise generation scheme is clever and shows large speedups over a PyTorch baseline in unit benchmarks. The stability improvement over DiffQ on GPT2 and Llama2-134M is interesting and empirically meaningful. However, the central empirical validation is incomplete: experiments use BF16 GEMM with FP32 accumulation rather than actual FP6/FP8 datatypes, the main Llama2-1B configuration degrades relative to the BF16 baseline, and the bitwise construction that underpins all theoretical claims is asserted but not validated or released. These gaps prevent the paper from currently supporting its advertised claims at the stated level.

major comments (4)
  1. [Section 3.4, Eq. (10)] The bitwise construction that is claimed to generate the distribution in Eq. (10) is asserted without pseudocode, a Boolean circuit, or a derivation. The distribution's two key properties, tau = 0 (min nonzero |R| = 1) and Pr(R = 0) ≈ 0.717, directly feed Lemmas 1 and 2, Propositions 3 and 4, the b_t < 9 bound, and the FP6/FP8 compatibility table. If the actual Triton kernel consumes Philox or Romu bits in a way that makes them correlated, or if the realizable probabilities differ from Eq. (10), these bounds shift. The paper should provide the exact construction and validate it empirically with histograms and bit-correlation tests over a large number of samples.
  2. [Section 4 (experimental setup) and Table C.1] All reported training runs use BF16 GEMM with FP32 accumulation; no model is trained with actual FP6 or FP8 weight storage and arithmetic. The claim that GaussWS 'supports low-precision FP parameters down to FP6' is therefore an analytical inference from Table C.1, not a demonstrated capability. The authors should either train with a real FP6/FP8 compute path, or clearly restrict the empirical claim to what is actually measured.
  3. [Section 4.1, Figure 4, and Appendix F] The Llama2-1B result with the default hyperparameters (b_init = 6, b_target = 4) degrades relative to the BF16 baseline in both average and maximum loss, contradicting the abstract's 'stable' claim. Appendix F reports only a preliminary run with higher bitwidths and states that 'results for GaussWS and DiffQ will be prepared before July 2025.' This is an incomplete experiment in a submitted manuscript; the central stability and scalability claims for 1B-scale models are not yet supported. The authors should complete these runs and incorporate them into the main evaluation, or substantially qualify the claims.
  4. [Section 3.4 and Eq. (5)] The independence assumption on individual PRNG bits is load-bearing for the theoretical guarantees but is never checked. Philox and Romu are not automatically guaranteed to produce independent low-order bits, and the paper neither cites evidence nor reports a test. Concretely, if the low-order bits are correlated, the effective value of p in Proposition 4 and the effective value of tau in Lemmas 1 and 2 change, and the Table C.1 datatype assignments are no longer justified. A simple statistical test on the generated R values should be included.
minor comments (5)
  1. [Section 4.2, Table 1] The 70B† row uses only 4 of 80 model layers, and the geometric-mean overhead reported in the text includes this row. The paper should report the geometric mean for the fully trained configurations separately, or at least state clearly that the 70B† numbers are not a full-model measurement.
  2. [Abstract and Section 3.5] The abstract's phrase 'requiring 2 bytes per parameter in GPU memory' should clarify that this is additional memory for storing the sampled weight w_hat, not the total memory footprint per parameter.
  3. [Section 3.3, Eq. (10)] The distribution in Eq. (10) is introduced as an approximation to floor(N(0,1)/2), but the approximation error is not quantified. The paper should state which distribution, exact or approximate, is assumed in Lemmas 1 and 2 and Propositions 3 and 4.
  4. [Appendix F] The statement that additional results 'will be prepared before July 2025' is not appropriate for a submitted manuscript and should be either completed or removed.
  5. [General] No code or kernel source is released. Given that the correctness of Section 3.4 depends on the exact bit operations, releasing the Triton implementation would substantially improve verifiability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the FP-precision bounds are derived consequences of an explicitly chosen noise distribution, not re-labeled fitted inputs.

full rationale

The derivation chain is self-contained. The paper chooses R = floor(N(0,1)/2) as a design input and computes its two relevant constants, tau = 0 and Pr(R = 0) approx 0.717 (Eq. 10). Lemmas 1 and 2 and Propositions 3 and 4 are algebraic consequences of these constants, the FP casting model fpe,m, and the PQT formulation in Eq. 3; none of their constants (m, tau, bt, exponent-bit counts) are fitted to the reported loss curves. The learned bitwidth bt is an output of training (Eq. 11), not an input used to tune the theory, and the GPT2/Llama2 comparisons are external benchmarks against a BF16 baseline rather than predictions derived from fitted values. The only noticeable self-references (DiffQ, NIPQ) are cited as prior PQT formulations to be generalized, and the load-bearing claims do not reduce to those citations. Section 3.4's bitwise construction of R is asserted rather than empirically verified, and the 'assuming that each bit ... independently random' premise would need a correlation test; however, that is an open verification gap in the implementation claim, not a circular step, because the target result (the bounds and annealing behavior) is not used to define the premise. No step in the paper equates a fitted parameter with a predicted quantity or derives a claim from its own conclusion.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The central derivation rests on the chosen noise distribution and on the bitwise independence assumption. The trained bitwidth and its schedule are the main tuned inputs. No exotic physical entities are introduced; the rounded normal distribution is a method component rather than a new force or particle, but it is counted as an invented entity because the paper postulates it for the first time.

free parameters (3)
  • Blockwise bitwidth b_t and its init/target hyperparameters = b_init=6, b_target=4 in main runs; learned b_t reported in range 4-12
    b_t is trained via Equation 4 and guided by weight decay toward b_target; the FP6 support claim requires b_t <= 4, so the experimental support depends on these hand-chosen values.
  • Loss weight lambda for bitwidth regularization = 1e-4 for GPT2, 0 for Llama2
    Equation 12 adds lambda times the distance from b_target; the value is chosen per model and affects the resulting b_t and training loss.
  • Weight decay applied to b_i = 0.1 (same global weight decay)
    The paper states b_t is guided toward b_target through weight decay applied to b_i; the exact strength is not isolated, yet it affects bitwidth outcomes.
assumptions (4)
  • domain assumption Each bit of the random integers produced by the PRNG is independently random.
    Invoked in Section 3.4 to justify the bitwise construction of R; if bits are correlated, the distribution in Equation 10 changes.
  • domain assumption The gradient of the blockwise maximum with respect to w is negligible.
    Equation 4 sets d max_bl(|w|)/dw approximately 0, making the gradient of the sampled weight equal to the gradient of the output; bias may appear if block maxima shift quickly.
  • standard math The approximate rounded normal R with support {-2,-1,0,1,2} and probabilities from Equation 10 has min nonzero magnitude 1 and Pr(R=0) about 0.717.
    These values feed directly into Lemma 1, Proposition 3, and Proposition 4; the paper computes them from the stated distribution.
  • domain assumption BF16 casting with FP32 accumulation during experiments reproduces the properties assumed for target FP datatypes.
    Section 4 states all GEMMs use BF16 with FP32 accumulation; the FP6 and FP8 claims are theoretical projections from Table C.1 and are not verified by training in those formats.
invented entities (1)
  • Rounded normal noise distribution R = floor(N(0,1)/2), approximated by a three-level distribution with support {-2,-1,0,1,2}
    purpose: Serves as the pseudo-quantization noise in Equation 3 to enable FP-friendly fake quantization, provide stochastic precision annealing, and allow efficient bitwise generation.
    The distribution is introduced by this paper and its benefits are demonstrated only in this paper's training runs and theoretical arguments; there is no external benchmark or independent measurement confirming the distribution's superiority.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Gaussian Weight Sampling for Scalable, Efficient and Stable Pseudo-Quantization Training." pith.science (2026). https://pith.science/paper/FOMZARMH

@misc{pith2026250511170,
  author       = {Pith},
  title        = {Pith review of: Gaussian Weight Sampling for Scalable, Efficient and Stable Pseudo-Quantization Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FOMZARMH}},
  note         = {Machine review of arXiv:2505.11170}
}
abstract

Ever-growing scale of large language models (LLMs) is pushing for improved efficiency, favoring fully quantized training (FQT) over BF16. While FQT accelerates training, it faces consistency challenges and requires searching over an exponential number of cases, each needing over 200B tokens to ensure stability. Pseudo-quantization training (PQT) addresses the issues of FQT, although it is not well-studied. We explore the practical implications of PQT in detail and propose a noise distribution $R$ that is floating-point (FP)-friendly, with ideal properties including stochastic precision annealing. As a result, the proposed method serves as an effective theoretical foundation for low-precision FP parameters through PQT, utilizing efficient fake quantization via an addition and subsequent FP casting. We demonstrate that Gaussian weight sampling is (1) scalable: supports low-precision FP parameters down to FP6 and high-precision noise up to 9-bit with BF16 operator. The proposed method is (2) efficient: incurring computational overhead as low as 1.40\% on the A100 GPU in terms of Llama2 training tokens per second, and requiring 2 bytes per parameter in GPU memory. We demonstrate that PQT with Gaussian weight sampling is (3) stable: closely following or even surpassing performance of the BF16 baseline while pre-training GPT2 and Llama2 models with up to 1B parameters and 300B tokens.

Figures

Figures reproduced from arXiv: 2505.11170 by the authors.

Figure 1
Figure 1. Summary of Gaussian weight sampling (GaussWS). [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Example of effective PQN as in Equation 3 with [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Training loss curve of the GPT2-124M model on the OpenWebText dataset. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Training loss curve of the Llama2-{134M, 1B} models on the C4 dataset. First column [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Resulting bitwidth bt. Dots and red lines indicate layerwise mean and standard deviation. Upper and lower solid lines represent layerwise maximum and minimum while dotted lines represent transformer-blockwise maximum and minimum. Lines on 5, 9 and 12-bit divide the par…
Figure 6
Figure 6. Figure 6: Forward pass benchmark results for the PyTorch layer implementing Equation 3 on a [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 12 canonical work pages

  1. [1]

    Bengio, N

    Y . Bengio, N. Léonard, and A. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation, 2013. URL https://arxiv.org/abs/1308.3432

    arXiv 2013

  2. [2]

    G. E. P. Box and M. E. Muller. A Note on the Generation of Random Normal Deviates. The Annals of Mathematical Statistics, 29(2):610–611, 1958. ISSN 0003-4851. doi: 10.1214/aoms/ 1177706645

    doi:10.1214/aoms/ 1958

  3. [3]

    Y . Chen, H. Xi, J. Zhu, and J. Chen. Oscillation-reduced mxfp4 training for vision transformers,

    work page

  4. [4]

    DeepSeek-AI, A. Liu, B. Feng, B. Xue, B. Wang, et al. Deepseek-v3 technical report, 2025. URL https://arxiv.org/abs/2412.19437

    arXiv 2025

  5. [5]

    Dettmers, A

    T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer. Qlora: Efficient finetuning of quantized llms, 2023. URL https://arxiv.org/abs/2305.14314

    arXiv 2023

  6. [6]

    Défossez, Y

    A. Défossez, Y . Adi, and G. Synnaeve. Differentiable model compression via pseudo quantiza- tion noise, 2022. URL https://arxiv.org/abs/2104.09987

    arXiv 2022

  7. [7]

    Fishman, B

    M. Fishman, B. Chmiel, R. Banner, and D. Soudry. Scaling fp8 training to trillion-token llms,

    work page

  8. [8]

    Gokaslan and V

    A. Gokaslan and V . Cohen. Openwebtext corpus. http://Skylion007.github.io/ OpenWebTextCorpus, 2019

    2019

Show all 40 references
  1. [9]

    URL https://arxiv.org/abs/2409.12517

    arXiv

  2. [10]

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, et al. LoRA: Low-Rank Adaptation of Large Language Models. arXiv, 2021. doi: 10.48550/arxiv.2106.09685

    2021 doi arXiv

  3. [11]

    Grattafiori, A

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, et al. The llama 3 herd of models,

  4. [12]

    Kumar, Z

    T. Kumar, Z. Ankner, B. F. Spector, B. Bordelon, N. Muennighoff, et al. Scaling laws for precision, 2024. URL https://arxiv.org/abs/2411.04330

    2024 arXiv

  5. [13]

    Lathrop, J

    S. Lathrop, J. Costa, W. Kramer, J. K. Salmon, M. A. Moraes, et al. Parallel random numbers: As easy as 1, 2, 3.2011 International Conference for High Performance Computing, Networking, Storage and Analysis (SC), pages 1–12, 2011. doi: 10.1145/2063384.2063405

    2011

  6. [14]

    Karpathy

    A. Karpathy. NanoGPT. https://github.com/karpathy/nanoGPT, 2022. Licensed under the MIT License

    2022

  7. [15]

    Loeschcke, M

    S. Loeschcke, M. Toftrup, M. J. Kastoryano, S. Belongie, and V . Snæbjarnarson. LoQT: Low Rank Adapters for Quantized Training. arXiv, 2024. doi: 10.48550/arxiv.2405.16528

    2024 doi arXiv

  8. [16]

    Loshchilov and F

    I. Loshchilov and F. Hutter. Decoupled weight decay regularization, 2019. URL https: //arxiv.org/abs/1711.05101

    2019 arXiv

  9. [17]

    Liang, T

    W. Liang, T. Liu, L. Wright, W. Constable, A. Gu, et al. Torchtitan: One-stop pytorch native solution for production ready llm pre-training, 2024. URL https://arxiv.org/abs/2410. 06511

    2024

  10. [18]

    Micikevicius, D

    P. Micikevicius, D. Stosic, N. Burgess, M. Cornea, P. Dubey, et al. FP8 Formats for Deep Learning. arXiv, 2022. doi: 10.48550/arxiv.2209.05433. 10

    2022 doi arXiv

  11. [19]

    M. A. Overton. Romu: Fast nonlinear pseudo-random number generators providing high quality,

  12. [20]

    Mattson, A

    T. Mattson, A. Muzahid, A. Solar-Lezama, P. Tillet, H. T. Kung, and D. Cox. Triton: an intermediate language and compiler for tiled neural network computations. Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, pages 10–19...

    2019

  13. [21]

    Radford, J

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Lan- guage models are unsupervised multitask learners. OpenAI, 2019. URL https://cdn.openai.com/better-language-models/language_models_are_ unsupervised_multitask_learners.pdf

    2019

  14. [22]

    Raffel, N

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, et al. Exploring the limits of transfer learning with a unified text-to-text transformer, 2023. URL https://arxiv.org/abs/1910. 10683

    2023

  15. [23]

    Rajbhandari, J

    S. Rajbhandari, J. Rasley, O. Ruwase, and Y . He. Zero: Memory optimizations toward training trillion parameter models, 2020. URL https://arxiv.org/abs/1910.02054

    2020 arXiv

  16. [24]

    H. Peng, K. Wu, Y . Wei, G. Zhao, Y . Yang, et al. Fp8-lm: Training fp8 large language models,

  17. [25]

    J. Shin, J. So, S. Park, S. Kang, S. Yoo, and E. Park. Nipq: Noise proxy-based integrated pseudo-quantization, 2023. URL https://arxiv.org/abs/2206.00820

    2023 arXiv

  18. [26]

    X. Sun, S. Li, R. Xie, W. Han, K. Wu, et al. Scaling laws for floating point quantization training,

  19. [27]

    G. Team, A. Kamath, J. Ferret, S. Pathak, N. Vieillard, et al. Gemma 3 technical report, 2025. URL https://arxiv.org/abs/2503.19786

    2025 arXiv

  20. [28]

    Touvron, L

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, et al. Llama 2: Open foundation and fine-tuned chat models, 2023. URL https://arxiv.org/abs/2307.09288

    2023 arXiv

  21. [29]

    B. D. Rouhani, R. Zhao, A. More, M. Hall, A. Khodamoradi, et al. Microscaling data formats for deep learning, 2023. URL https://arxiv.org/abs/2310.10537

    2023 arXiv

  22. [30]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, et al. Attention is all you need, 2023. URL https://arxiv.org/abs/1706.03762

    2023 arXiv

  23. [31]

    R. Wang, Y . Gong, X. Liu, G. Zhao, Z. Yang, et al. Optimizing Large Language Model Training Using FP4 Quantization. arXiv, 2025. doi: 10.48550/arxiv.2501.17116

    2025 doi arXiv

  24. [32]

    URL https://arxiv.org/abs/2501.02423

    arXiv

  25. [33]

    Y . Zhao, A. Gu, R. Varma, L. Luo, C.-C. Huang, et al. Pytorch fsdp: Experiences on scaling fully sharded data parallel, 2023. URL https://arxiv.org/abs/2304.11277

    2023 arXiv

  26. [34]

    Datatype ˆw

    H. Zhu, Z. Zhang, W. Cong, X. Liu, S. Park, et al. APOLLO: SGD-like Memory, AdamW-level Performance. arXiv, 2024. doi: 10.48550/arxiv.2412.05270. 11 A Proof Lemma 2. Proof. Consider adding PQN to{0,±ϵ} as following: {0,±ϵ} +R⊙ broadcastbl max bl (|w|)· 21−bt (13) Without loss ...

    2024 doi arXiv

  27. [35]

    Tseng, T

    A. Tseng, T. Yu, and Y . Park. Training llms with mxfp4, 2025. URLhttps://arxiv.org/ abs/2502.20586

    2025 arXiv

  28. [38]

    Zhang, C

    Y . Zhang, C. Chen, Z. Li, T. Ding, C. Wu, et al. Adam-mini: Use Fewer Learning Rates To Gain More. arXiv, 2024. doi: 10.48550/arxiv.2406.16793

    2024 doi arXiv

  29. [2020]

    URL https://arxiv.org/abs/2002.11331

    2002 arXiv

  30. [2023]

    URL https://arxiv.org/abs/2310.18313

    arXiv

  31. [2024]

    URL https://arxiv.org/abs/2407.21783

    arXiv

  32. [2025]

    URL https://arxiv.org/abs/2502.20853

    arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.