Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Characterization and Mitigation of Training Instabilities in Microscaling Formats

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MX low-precision training diverges because block-scale clamping packs layer-norm weights into one bin, and a hybrid precision mix restores full-precision performance.

desk verdict A well-executed empirical study that identifies layer-norm affine-weight clamping as a plausible mechanism for MX-format training instabilities, with two practical fixes that work in practice—though the causal story is fully proven only in the synthetic proxy, not isolated in the LLM runs. read the letter →

arxiv 2506.20752 v1 pith:RHTRLSIM submitted 2025-06-25 cs.LG cs.AR

Huangyuan Su , Mujin Kwun , Stephanie Gil , Sham Kakade , Nikhil Anand This is my paper
classification cs.LGcs.AR
keywords microscalingformatsMXFP8low-precisiontraininginstabilitygradientbiasblock-scaledquantizationlayernormalizationscalinglaws
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to determine whether the Microscaling (MX) block-scaled precision formats slated for next-generation hardware can be used for training large language models from scratch, and why they sometimes fail. Across nearly a thousand OLMo models spanning compute budgets from $2\times 10^{17}$ to $4.8\times 10^{19}$ FLOPs, it finds that MX training exhibits sharp, stochastic, often unrecoverable loss spikes in almost every weight–activation precision combination, especially for larger, longer-trained models. Using a small synthetic student–teacher MLP that reproduces the behavior, the paper identifies the cause as multiplicative gradient bias: when values in a block are tightly clustered, the shared power-of-two scale forces them into the format's last quantization bin, clamping layer-norm affine weights (and about 1% of activations) to the same code and systematically biasing the gradient. It then shows that keeping activations in bfloat16 or quantizing only the forward pass stabilizes training, with MXFP8 E4M3 weights plus bfloat16 activations matching full-precision baselines. If correct, this gives hardware and training practitioners a concrete explanation and workaround for an instability that would otherwise block low-precision training on Blackwell-class accelerators.

What carries the argument

The key machinery is the multiplicative-noise model of the quantized gradient, $\widetilde g_t = (1+\zeta_t) g_t$, where the deviation $\varepsilon_t = \widetilde g_t - g_t$ is bounded through the operator norm of $\zeta_t$. Combined with a quadratic approximation of the loss, this yields the spectral-radius stability condition that contracting directions become expanding once $\eta_t \|\zeta_t\|_{\mathrm{op}} \lambda_{\max}$ is large; the paper observes divergence as the bound on $\|\zeta_t\|_{\mathrm{op}}$ reaches roughly 2. The mechanism that makes this bound grow is the shared-scale overflow condition for MXFP8 E4M3: after dividing by the block's power-of-two scale, any value with $|v| > 0.875 \times \max_i |V_i|$ is clamped to the largest representable code, so a cluster of tightly packed layer-norm weights collapses onto a single quantization bin, systematically biasing the gradient in every affected block.

What would settle it

In a diverging MX training run, monitor the fraction of layer-norm affine weights that are clamped into the final quantization bin and the estimated gradient-noise operator norm $\|\zeta_t\|_{\mathrm{op}}$ before divergence; if a run diverges while the clamping fraction stays near zero or the noise norm remains well below 2, the proposed mechanism is not the dominant cause.

Watch

Extended reading notes

Core claim

The paper's central claim is that training with MX block-scaled formats is systematically unstable because of a specific quantization artifact: block-scale clamping of tightly clustered tensors. Most of the layer-norm affine weights — and, secondarily, roughly 1% of activations — end up in the same quantization bin after division by the shared scale, and this clamping injects a multiplicative bias into the gradient that can grow until the descent direction is lost. The paper formalizes this with the multiplicative-noise model $\widetilde g_t = (1+\zeta_t) g_t$ and a crude stability condition $|1-\eta_t\lambda_{\max}| + \eta_t \|\zeta_t\|_{\mathrm{op}}\lambda_{\max} \lesssim 1$; empirically, when the operator-norm lower bound of $\zeta_t$ approaches about 2, training diverges. It verifies the mechanism in the synthetic proxy and in OLMo, showing that the layer-norm overflow fraction rises sharply before divergence. The paper further claims that two mitigations — keeping activations (including layer norms) in higher precision, or applying MX quantization only in the forward pass — remove the divergence, and that MXFP8 E4M3 weights paired with bfloat16 activations reproduce full-bfloat16 validation loss across all tested sizes.

Load-bearing premise

The results depend on the synthetic student–teacher MLP faithfully reproducing the cause of instability in real large language models; if the proxy's layer-norm clamping mechanism is not the dominant driver in production-scale LLMs, the mechanistic explanation weakens even if the empirical fixes still work.

Editorial extensions

If this is right

  • Full MX quantization of both forward and backward passes is unreliable for language-model training in the formats and sizes tested; instability becomes more likely as compute budget grows.
  • The synthetic student–teacher proxy reproduces the key failure modes cheaply, so precision schemes can be screened on minutes-long runs before committing to expensive LLM sweeps.
  • Disabling backward-pass quantization, or keeping activations in bfloat16, stabilizes MXFP8 training across E4M3 and E5M2 formats and supports valid empirical scaling-law fits.
  • MXFP8 E4M3 weights with bfloat16 activations match the validation loss of full-bfloat16 baselines at model sizes from roughly 20M to 1.7B parameters.
  • Layer-normalization affine parameters are the highest-risk tensor class for block-scaled quantization because their values cluster tightly; hardware and software stacks should treat them specially.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A scale that tracks both the minimum and maximum within each block, rather than only the block maximum, would likely remove the clamping bias the paper documents; the authors explicitly defer such a scheme to future work.
  • The clamping mechanism is not limited to pretraining: any trained model whose layer-norm weights or activations are tightly clustered would suffer the same bias under MX quantization, so quantization-aware fine-tuning and post-training quantization need the same defensive precision choices.
  • The paper's own evidence implies a cheaper fix than network-wide bfloat16 activations: only layer-norm affine weights and roughly 1% of activations need higher precision, so selectively raising precision in those tensors could match full-precision performance at lower cost.
  • A direct testable consequence is that the fraction of layer-norm blocks clamped to the largest code should predict divergence in any MX run; a monitoring hook computing that fraction in real time could serve as an early-warning system.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper investigates training instabilities in block-scaled Microscaling (MX) low-precision formats during language model pretraining. The authors run a large sweep of OLMo models (nearly one thousand runs) across MXFP6/MXFP8 weight–activation combinations and report sharp, stochastic, often unrecoverable loss spikes, especially in larger models. To explain this, they introduce a synthetic student–teacher residual MLP proxy that reproduces similar instabilities, propose a multiplicative gradient-noise model, and derive a heuristic norm-based stability condition (Eq. 9). They trace the dominant quantization-induced bias to clamping of tightly clustered layer-norm affine weights (and, to a lesser extent, ~1% of activations) when MX block scaling forces them into the last representable bin. In-situ intervention experiments on the proxy support this mechanism. They then apply two mitigation strategies to OLMo—keeping activations in bfloat16 and restricting MX quantization to the forward pass—and report stable training, valid empirical scaling laws, and that MXFP8 E4M3 weights with bfloat16 activations match full-bfloat16 baselines up to the tested scales. Code is released.

Significance. If the mechanistic claim holds, the paper provides a practically useful diagnosis and mitigation for a class of instabilities that will matter for training on next-generation hardware with MX formats. The empirical contribution is substantial: roughly one thousand controlled LLM runs, a reproducible proxy with identical seeds and batch order, controlled in-situ interventions, and public code. The paper also makes falsifiable predictions through overflow-fraction diagnostics. The main caveat is that the causal attribution in the LLM setting is not isolated by any ablation, and the theoretical condition is admittedly heuristic; these limit the strength of the central mechanistic claim as currently presented.

major comments (3)
  1. [Section 7, Fig. 5, Fig. 7] The central claim that OLMo MX instabilities originate primarily from clamping of layer-norm affine weights is underdetermined in the target setting. The proxy in-situ 'No LayerNorm quant' intervention in Fig. 7 establishes causality only in the synthetic model; the only OLMo evidence is the overflow-fraction correlation in the center panel of Fig. 5 and the network-wide mitigations in Table 1, which also change activation quantization (BF16 activations) or backward-pass gradient quantization (forward-only). These interventions do not discriminate the proposed layer-norm mechanism from activation-outlier or backward-gradient-bias mechanisms. Please either add an OLMo ablation that keeps only layer-norm affine parameters in higher precision while leaving the rest of the MX scheme unchanged, or explicitly reframe the LLM causal claim as a hypothesis motivated by the proxy rather than a demonstrated origin.
  2. [Section 5.2, Eq. (9)] The norm-based stability condition is presented in the contributions as a derived condition, but the bound in Eq. (9) is not a rigorous spectral-radius bound as stated: for a general matrix I - eta(1+zeta)H, the spectral radius is not bounded by |1 - eta lambda_max| + eta ||zeta||_op lambda_max without additional assumptions on the commutator or common eigenbasis of H and zeta. The paper also notes that the actual update is Adam, not GD. Since this condition is a load-bearing theoretical contribution, either prove it under explicit assumptions or consistently label it as a heuristic criterion in the abstract and contributions.
  3. [Section 3.2 and Appendix B] The pervasiveness of OLMo instabilities is asserted qualitatively ('consistently observe') and illustrated with selected loss curves, but the paper does not report a quantitative instability-rate table for the full LLM sweep (e.g., fraction of diverged or spiked runs per format and compute budget). Adding such a table, using a predefined spike-detection threshold like the factor-of-100 rule used in Appendix B, would make the universality claim falsifiable and easier to compare with prior work.
minor comments (6)
  1. [Section 2.1, footnote 2] Typo: 'choies' should be 'choices'.
  2. [Appendix B] The spike-detection heuristic says the loss had to be 'a factor of 100 lager' than the previous step; this should read 'larger', and the exact comparison window (e.g., t vs. t-1) should be stated in one place.
  3. [Section 6.1, Fig. 5 caption] The notation 'S 1111 1112' for the reserved NaN code is confusing; standard bit-pattern notation (e.g., 0x7F for the NaN code in the byte representation) would be clearer.
  4. [Section 7] Heading typo: 'Key Takewaways' should be 'Key Takeaways'.
  5. [Title page] The line 'Preprint. Under review.' is not appropriate for a journal submission and should be removed.
  6. [References] Some reference entries are incomplete, notably Xu et al. (2023), which lists 'arXiv preprint arXiv:.'; please update.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the central mechanism is anchored by direct measurements and controlled interventions; the only caveats are a non-load-bearing self-citation and an overclaimed 'prediction' wording.

full rationale

The paper's central derivation chain is not circular. The claim that MX block-scale clamping of tightly clustered layer-norm affine weights introduces gradient bias is anchored by (i) direct overflow-fraction measurements in both the synthetic proxy and OLMo (Figure 5), (ii) controlled in-situ interventions that alter only the quantization recipe while holding seed, model state, and batch order fixed (Figure 7), and (iii) comparisons against full-precision FP32/bfloat16 baselines. None of these measurements is defined in terms of the instability it explains, and the stabilization results in Table 1 are external benchmarks rather than fitted values. The 'norm-based condition' in Section 5 is a heuristic stability bound: the noise operator norm is estimated from the measured gradient deviation, so the claim that it 'predicts' divergence is better described as an empirical correlate than a first-principles prediction. This is an overstatement, not a circular reduction, because the bound and the measurement are not the same object and no parameter is fitted to the divergence outcome. The only self-citation is to Brandfonbrener et al. (2024) for FLOP-accounting code and scaling-law fitting methods; the functional form used is the standard external Hoffmann/Chinchilla form, so this citation is not load-bearing and does not raise the circularity score.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The paper does not introduce new physical or theoretical entities. Its structural assumptions are the proxy-model transfer, the multiplicative-noise model, the Hessian linearization, and the fidelity of the MX software emulation. The main free parameters are the scaling-law fit constants and some heuristic thresholds, all fitted to data and used descriptively.

free parameters (3)
  • Chinchilla scaling-law fit parameters (E, A, B, alpha, beta) = Table 2: e.g. E=0.54, A=2.82e+03, B=2.04e+04, alpha=0.52, beta=0.55 for MXFP8 E4M3 bfloat16
    These five constants are fitted to the validation loss data in Section C; they are descriptive fits, not predictions, and the paper does not claim them as derived constants.
  • Spike-detection threshold = loss factor of 100 larger than previous step
    Heuristic threshold used in Appendix B to count instability spikes; chosen by hand, but only used for qualitative spike counts.
  • Proxy model learning rate eta=5e-4 = 5e-4
    Chosen from a sweep to move away from stochastic-instability regions and into the regime where low-precision instabilities dominate; this is a modeling choice for the proxy, not a claim about LLMs.
assumptions (5)
  • domain assumption The student-teacher residual MLP on Gaussian inputs is a valid proxy for LLM training instability behavior.
    The entire mechanistic analysis is done on this proxy; the paper itself acknowledges in the Limitations section that results may not extrapolate to much larger models or MoE architectures.
  • domain assumption The multiplicative gradient noise model egt = (1 + zeta_t) gbar_t adequately captures the effect of MX quantization.
    Section 5.1 posits this model; the paper estimates ||zeta_t|| from data but does not prove quantization noise is multiplicative, and it notes zeta_t may not be uniquely defined due to weight permutations.
  • domain assumption The Hessian linearization around the optimum, grad w L(w_t) = H (w_t - w*), is valid for the purpose of the stability bound.
    Section 5.2 explicitly ignores terms of order (w_t - w*)^2 and higher and states the resulting bound is not rigorous because the actual update is Adam, not gradient descent.
  • standard math FP32 training of the proxy is a correct skyline for isolating precision effects.
    Used as the baseline in all proxy experiments; the paper controls seed, data, and batch order, and FP32 runs are treated as the 'no precision error' reference.
  • domain assumption The MX PyTorch emulation library faithfully reproduces the arithmetic of hardware MX formats.
    All experiments are software emulations; the paper's Limitations section says real Blackwell hardware may introduce additional rounding, memory-layout, or fused-kernel errors.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Characterization and Mitigation of Training Instabilities in Microscaling Formats." pith.science (2026). https://pith.science/paper/RHTRLSIM

@misc{pith2026250620752,
  author       = {Pith},
  title        = {Pith review of: Characterization and Mitigation of Training Instabilities in Microscaling Formats},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RHTRLSIM}},
  note         = {Machine review of arXiv:2506.20752}
}
abstract

Training large language models is an expensive, compute-bound process that must be repeated as models scale, algorithms improve, and new data is collected. To address this, next-generation hardware accelerators increasingly support lower-precision arithmetic formats, such as the Microscaling (MX) formats introduced in NVIDIA's Blackwell architecture. These formats use a shared scale within blocks of parameters to extend representable range and perform forward/backward GEMM operations in reduced precision for efficiency gains. In this work, we investigate the challenges and viability of block-scaled precision formats during model training. Across nearly one thousand language models trained from scratch -- spanning compute budgets from $2 \times 10^{17}$ to $4.8 \times 10^{19}$ FLOPs and sweeping over a broad range of weight-activation precision combinations -- we consistently observe that training in MX formats exhibits sharp, stochastic instabilities in the loss, particularly at larger compute scales. To explain this phenomenon, we conduct controlled experiments and ablations on a smaller proxy model that exhibits similar behavior as the language model, sweeping across architectural settings, hyperparameters, and precision formats. These experiments motivate a simple model in which multiplicative gradient bias introduced by the quantization of layer-norm affine parameters and a small fraction of activations can trigger runaway divergence. Through \emph{in situ} intervention experiments on our proxy model, we demonstrate that instabilities can be averted or delayed by modifying precision schemes mid-training. Guided by these findings, we evaluate stabilization strategies in the LLM setting and show that certain hybrid configurations recover performance competitive with full-precision training. We release our code at https://github.com/Hither1/systems-scaling.

Figures

Figures reproduced from arXiv: 2506.20752 by the authors.

Figure 1
Figure 1. Shows stable (bfloat16) OLMo training runs (top) compared to lower precision (MXFP8 [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Comparing FP32 with MXFP6 and MXFP8 formats across different choices for the learning [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Shows the comparison between full and low precision training across different activation [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: Shows the bound on the operator norm ∥ζt∥op (as inferred from Equation (4)), and the cosine angle between the low precision gradient and high precision gradient. Dashed line in the lower right plot shows when the bound on ∥ζt∥op is equal to 2. 5.2 A Crude Bound To unde…
Figure 5
Figure 5. Figure 5: Left: relative gap (xt+1−xt)/xt for successive positive FP8 E4M3 codes (sign bit stripped). Within each exponent band the gap decays from 12.5% to 6.6%; the hatched region marks values that would be clamped once the scaled magnitude exceeds the representable limit of 4…
Figure 6
Figure 6. Figure 6: Shows two mitigations (quantization of only forward pass) and activation elements in [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Intervention experiment for a synthetic student-teacher model with [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Scaling law fit for combinations of precision formats of weights and keep the activations in [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Instability spikes measured in training, for different model depths and widths. [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: SGD with and without momentum; a larger learning rate was used [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: Baseline versus using a lower gain Xavier normal weight initialization. [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: Scaling law fits for fixed stable of precision formats of weights and activations quantizing [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]
Figure 13
Figure 13. Figure 13: Scaling law fits for bfloat16-bfloat16 (baseline) and for MXFP6 format. [PITH_FULL_IMAGE:figures/full_fig_p020_13.png]
Figure 14
Figure 14. Figure 14: Train loss and gradient norm when quantizing only the forward pass. [PITH_FULL_IMAGE:figures/full_fig_p021_14.png]
Figure 15
Figure 15. Figure 15: Train loss and gradient norm when activations are kept in high precision (bfloat16). [PITH_FULL_IMAGE:figures/full_fig_p022_15.png]
Figure 16
Figure 16. Figure 16: Unstable MXFP8 combinations of precision formats of weights and activations. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_16.png]
Figure 17
Figure 17. Figure 17: Unstable combinations of precision formats of weights and activations for MXFP6 weights. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_17.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Preventing Learning Stagnation in PPO by Scaling to 1 Million Parallel Environments

    cs.LG 2026-03 conditional novelty 5.0 of 10

    PPO plateaus can be avoided by increasing the number of parallel environments, which reduces both the outer-loop step size and update noise; scaling to 1M environments sustained improvement to 1T transitions.

Reference graph

Works this paper leans on

47 extracted references · 32 canonical work pages · cited by 1 Pith paper

  1. [1]

    Abdolrashidi, A., Wang, L., Agrawal, S., Malmaud, J., Rybakov, O., Leichner, C., and Lew, L. (2021). Pareto-optimal quantized resnet is mostly 4-bit. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) , page 3085–3093. IEEE

    work page 2021

  2. [2]

    Claude 4 system card

    Anthropic (2025). Claude 4 system card. https://www-cdn.anthropic.com/4263b940cabb546aa0e3283f35b686f4f3b2ff47.pdf. Accessed: 2025-06-20

    work page 2025

  3. [3]

    Bondarenko, Y., Nagel, M., and Blankevoort, T. (2023). Quantizable transformers: Removing outliers by helping attention heads do nothing

    work page 2023

  4. [4]

    Brandfonbrener, D., Anand, N., Vyas, N., Malach, E., and Kakade, S. (2024). Loss-to-loss prediction: Scaling laws for all datasets. arXiv preprint arXiv:2411.12925

    arXiv 2024

  5. [5]

    Chen, M., Zhang, C., Liu, J., Zeng, Y., Xue, Z., Liu, Z., Li, Y., Ma, J., Huang, J., Zhou, X., and Luo, P. (2025). Scaling law for quantization-aware training

    work page 2025

  6. [6]

    Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko, S., Maynez, J., Rao, A., Barnes, P., Tay, Y., Shazeer, N., Prabhakaran, V., Reif, E., Du, N., Hutchinson, B., Pope, R., Bradbury, J., Austin, J., Isard, M., Gur-Ari, G., Yin, P., Duke, T., Levska...

    2022

  7. [7]

    Z., and Talwalkar, A

    Cohen, J., Kaur, S., Li, Y., Kolter, J. Z., and Talwalkar, A. (2021). Gradient descent on neural networks typically occurs at the edge of stability. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net

    work page 2021

  8. [8]

    A., Cao, C., Cao, K., Castagné, R., Cendrero, J., Currie, L

    Cohere, T., :, Aakanksha, Ahmadian, A., Ahmed, M., Alammar, J., Alizadeh, M., Alnumay, Y., Althammer, S., Arkhangorodsky, A., Aryabumi, V., Aumiller, D., Avalos, R., Aviv, Z., Bae, S., Baji, S., Barbet, A., Bartolo, M., Bebensee, B., Beladia, N., Beller-Morales, W., Bérard, A., Berneshawi, A., Bialas, A., Blunsom, P., Bobkin, M., Bongale, A., Braun, S., B...

    work page 2025

Show all 47 references
  1. [9]

    Damian, A., Nichani, E., and Lee, J. D. (2023). Self-stabilization: The implicit bias of gradient descent at the edge of stability. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net

    2023

  2. [10]

    Darvish Rouhani, B., Garegrat, N., Savell, T., More, A., Han, K.-N., Zhao, R., and Hall, M. (2023a). Open compute project

    2023

  3. [11]

    Darvish Rouhani, B., Zhao, R., Elango, V., Shafipour, R., Hall, M., Mesmakhosroshahi, M., More, A., Melnick, L., Golub, M., Varatkar, G., et al. (2023b). With shared microexponents, a little shifting goes a long way. In Proceedings of the 50th Annual International Symposium on...

    2023

  4. [12]

    DeepMind, G. (2025). Gemini 2.5 technical report. https://storage.googleapis.com/deepmind-media/gemini/gemini_v2_5_report.pdf. Accessed: 2025-06-20

    2025

  5. [13]

    F., Mahendran, A., Yu, F., Oliver, A., Huot, F., Bastings, J., Collier, M

    Dehghani, M., Djolonga, J., Mustafa, B., Padlewski, P., Heek, J., Gilmer, J., Steiner, A., Caron, M., Geirhos, R., Alabdulmohsin, I., Jenatton, R., Beyer, L., Tschannen, M., Arnab, A., Wang, X., Riquelme, C., Minderer, M., Puigcerver, J., Evci, U., Kumar, M., van Steenkiste, S...

    2023

  6. [14]

    and Zettlemoyer, L

    Dettmers, T. and Zettlemoyer, L. (2023). The case for 4-bit precision: k-bit inference scaling laws. In International Conference on Machine Learning , pages 7750--7774. PMLR

    2023

  7. [15]

    Fishman, M., Chmiel, B., Banner, R., and Soudry, D. (2024). Scaling fp8 training to trillion-token llms. arXiv preprint arXiv:2409.12517

    2024 arXiv

  8. [16]

    Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024). The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  9. [17]

    H., Ivison, H., Magnusson, I., Wang, Y., et al

    Groeneveld, D., Beltagy, I., Walsh, P., Bhagia, A., Kinney, R., Tafjord, O., Jha, A. H., Ivison, H., Magnusson, I., Wang, Y., et al. (2024). Olmo: Accelerating the science of language models. arXiv preprint arXiv:2402.00838

    2024 arXiv

  10. [18]

    R., Pawar, S

    Henry, A., Dachapally, P. R., Pawar, S. S., and Chen, Y. (2020). Query-key normalization for transformers. CoRR , abs/2010.04245

    2020 arXiv

  11. [19]

    Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022). Training compute-optimal large language models. arXiv preprint arXiv:2203.15556

    2022 arXiv

  12. [20]

    L., Anthony, Q

    Ibrahim, A., Th \'e rien, B., Gupta, K., Richter, M. L., Anthony, Q. G., Belilovsky, E., Lesort, T., and Rish, I. (2024). Simple and scalable strategies to continually pre-train large language models. Transactions on Machine Learning Research

    2024

  13. [21]

    Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., and Kalenichenko, D. (2017). Quantization and training of neural networks for efficient integer-arithmetic-only inference

    2017

  14. [22]

    Jastrzebski, S., Szymczak, M., Fort, S., Arpit, D., Tabor, J., Cho, K., and Geras, K. (2020). The break-even point on optimization trajectories of deep neural networks

    2020

  15. [23]

    B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. (2020). Scaling laws for neural language models. arXiv preprint arXiv:2001.08361

    2020 arXiv

  16. [24]

    F., Bordelon, B., Muennighoff, N., Paul, M., Pehlevan, C., R \'e , C., and Raghunathan, A

    Kumar, T., Ankner, Z., Spector, B. F., Bordelon, B., Muennighoff, N., Paul, M., Pehlevan, C., R \'e , C., and Raghunathan, A. (2024). Scaling laws for precision. arXiv preprint arXiv:2411.04330

    2024 arXiv

  17. [25]

    J., and Lee, D

    Lee, J., Bae, J., Kim, B., Kwon, S. J., and Lee, D. (2025). To fp8 and back again: Quantifying reduced precision effects on llm training stability

    2025

  18. [26]

    Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. (2024). Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437

    2024 arXiv

  19. [27]

    Liu, Z., Zhao, C., Huang, H., Chen, S., Zhang, J., Zhao, J., Roy, S., Jin, L., Xiong, Y., Shi, Y., et al. (2025). Paretoq: Scaling laws in extremely low-bit llm quantization. arXiv preprint arXiv:2502.02631

    2025

  20. [28]

    Ma, J., Pei, H., Lausen, L., and Karypis, G. (2025). Understanding silent data corruption in llm training

    2025

  21. [29]

    Meta, A. (2025). The llama 4 herd: The beginning of a new era of natively multimodal ai innovation

    2025

  22. [30]

    Micikevicius, P., Stosic, D., Burgess, N., Cornea, M., Dubey, P., Grisenthwaite, R., Ha, S., Heinecke, A., Judd, P., Kamalu, J., et al. (2022a). Fp8 formats for deep learning. arXiv preprint arXiv:2209.05433

    2022 arXiv

  23. [31]

    Micikevicius, P., Stosic, D., Burgess, N., Cornea, M., Dubey, P., Grisenthwaite, R., Ha, S., Heinecke, A., Judd, P., Kamalu, J., Mellempudi, N., Oberman, S., Shoeybi, M., Siu, M., and Wu, H. (2022b). Fp8 formats for deep learning

    2022

  24. [32]

    Mx pytorch emulation library

    Microsoft (2024). Mx pytorch emulation library

    2024

  25. [33]

    S., Narang, S., Poulton, A., Silva, R., Tang, B., Liskovich, D., Xu, P., Zhang, Y., Kambadur, M., Roller, S., and Zhang, S

    Molybog, I., Albert, P., Chen, M., DeVito, Z., Esiobu, D., Goyal, N., Koura, P. S., Narang, S., Poulton, A., Silva, R., Tang, B., Liskovich, D., Xu, P., Zhang, Y., Kambadur, M., Roller, S., and Zhang, S. (2023). A theory on adam instability in large-scale machine learning

    2023

  26. [34]

    Nvidia blackwell architecture

    NVIDIA (2025). Nvidia blackwell architecture

    2025

  27. [35]

    Gpt-4.5 system card

    OpenAI (2025). Gpt-4.5 system card. https://cdn.openai.com/gpt-4-5-system-card-2272025.pdf. Accessed: 2025-06-20

    2025

  28. [36]

    Ouyang, X., Ge, T., Hartvigsen, T., Zhang, Z., Mi, H., and Yu, D. (2024). Low-bit quantization favors undertrained llms: Scaling laws for quantized llms with 100t training tokens. arXiv preprint arXiv:2411.17691

    2024 arXiv

  29. [37]

    B., Lozhkov, A., Mitchell, M., Raffel, C., Werra, L

    Penedo, G., Kydl \' c ek, H., allal, L. B., Lozhkov, A., Mitchell, M., Raffel, C., Werra, L. V., and Wolf, T. (2024). The fineweb datasets: Decanting the web for the finest text data at scale. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and...

    2024

  30. [38]

    Porian, T., Wortsman, M., Jitsev, J., Schmidt, L., and Carmon, Y. (2025). Resolving discrepancies in compute-optimal scaling of language models

    2025

  31. [39]

    D., Zhao, R., More, A., Hall, M., Khodamoradi, A., Deng, S., Choudhary, D., Cornea, M., Dellinger, E., Denolf, K., et al

    Rouhani, B. D., Zhao, R., More, A., Hall, M., Khodamoradi, A., Deng, S., Choudhary, D., Cornea, M., Dellinger, E., Denolf, K., et al. (2023). Microscaling data formats for deep learning. arXiv preprint arXiv:2310.10537

    2023 arXiv

  32. [40]

    Shao, W., Chen, M., Zhang, Z., Xu, P., Zhao, L., Li, Z., Zhang, K., Gao, P., Qiao, Y., and Luo, P. (2024). Omniquant: Omnidirectionally calibrated quantization for large language models

    2024

  33. [41]

    Shazeer, N. (2020). Glu variants improve transformer

    2020

  34. [42]

    Z., and Liu, Z

    Sun, M., Chen, X., Kolter, J. Z., and Liu, Z. (2024). Massive activations in large language models. arXiv preprint arXiv:2402.17762

    2024 arXiv

  35. [43]

    Tseng, A., Yu, T., and Park, Y. (2025). Training llms with mxfp4

    2025

  36. [44]

    J., Xiao, L., Everett, K

    Wortsman, M., Liu, P. J., Xiao, L., Everett, K. E., Alemi, A. A., Adlam, B., Co - Reyes, J. D., Gur, I., Kumar, A., Novak, R., Pennington, J., Sohl - Dickstein, J., Xu, K., Lee, J., Gilmer, J., and Kornblith, S. (2024). Small-scale proxies for large-scale transformer training ...

    2024

  37. [45]

    Xu, K., Lin, J., Wang, Z., Hu, P., and Zhao, Z. (2023). Improved fully quantized training via rectifying batch normalization. arXiv preprint arXiv:

    2023

  38. [46]

    V., Mihaylov, T., Ott, M., Shleifer, S., Shuster, K., Simig, D., Koura, P

    Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., Mihaylov, T., Ott, M., Shleifer, S., Shuster, K., Simig, D., Koura, P. S., Sridhar, A., Wang, T., and Zettlemoyer, L. (2022). Opt: Open pre-trained transformer language models

    2022

  39. [47]

    Zoph, B., Bello, I., Kumar, S., Du, N., Huang, Y., Dean, J., Shazeer, N., and Fedus, W. (2022). St-moe: Designing stable and transferable sparse expert models

    2022

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.