Pith. sign in

REVIEW 4 major objections 4 minor 32 references

$\mu$nit Scaling: Simple and Scalable FP8 LLM Training

T0 review · 4 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a small set of static scaling rules lets transformer language models train fully in FP8 at up to 13B parameters, matching BF16 quality while cutting training time by up to a third.

desk verdict A practical FP8 training recipe with real empirical support, but the 'equal quality' claim is over-strong and the width-transfer validation skips depth effects. read the letter →

arxiv 2502.05967 v3 pith:RL65NZWJ submitted 2025-02-09 cs.LG

Saaketh Narayan , Abhay Gupta , Mansheej Paul , Davis Blalock This is my paper · ORCID
classification cs.LG
keywords FP8traininglow-precisionhyperparametertransferunitscalinglargelanguagemodelsstaticvariancepreservationRes-Post-LayerNorm
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that FP8 training of large language models fails mainly because standard transformer blocks are not variance-preserving, and that a small set of static interventions fixes this well enough to drop dynamic scaling factors entirely. The proposed method, µnit Scaling (µS), combines unit-variance weight initialization, a static $1/\sqrt{\mathrm{fan\_in}}$ output multiplier on every hidden linear GEMM, Res-Post-LayerNorm placement, and a fixed depth-dependent residual coefficient $\tau$. Under these rules the residual stream keeps stable scale, so all hidden linear layers can run in FP8 (e4m3 weights and activations, e5m2 gradients) with constant scaling. Training models from 1B to 13B parameters, the paper reports final loss and evaluation quality equal to or better than BF16 baselines, hyperparameters that transfer from a width-256 proxy to widths up to 5120 with no re-tuning, and throughput 25–33% higher than BF16.

What carries the argument

The carrying object is a variance-preserving residual stream. Three pieces do the work: (i) Res-Post-LayerNorm, which places LayerNorm at the end of each residual branch so that whatever enters the stream is normalized before summation; (ii) the “fixed” residual modification $x_{l+1} = \sqrt{1-\tau}\,x_l + \sqrt{\tau}\,f(x_l)$, with $\tau$ picked by depth from a fitted curve; and (iii) static GEMM scaling, where weights initialize with unit variance and every hidden linear layer multiplies its output by $1/\sqrt{\mathrm{fan\_in}}$ in both forward and backward passes. A fourth piece, the Adam-like scale-invariance identity (the output of a hidden layer is unchanged when initialization variance, output multiplier, and learning rate are jointly re-scaled), turns these choices into the learning-rate transfer rule $1/\sqrt{\mathrm{fan\_in}}$ and, across widths, $\sqrt{d_{\mathrm{base}}}/\sqrt{d_{\mathrm{new}}}$. Proposition 2.1—that attention output variance is $\propto 1/k$ for iid inputs—motivates the architecture change but is not itself the final fix; the paper abandons a Square-Root Softmax variant because real data correlations break it.

What would settle it

Train a width-8192 µS model using the learning rate predicted from a width-256 base ($\eta$ multiplied by $\sqrt{256/8192}$) and independently sweep $\eta$ on the same model; if the empirically optimal $\eta$ differs from the predicted value by more than the sweep spacing, the scale-invariance identity at the heart of µS is not holding at finite width.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that the apparent need for dynamic scaling factors in FP8 LLM training is an artifact of uncontrolled variance, not a property of the FP8 format. Through Proposition 2.1 it shows that causal self-attention output variance shrinks roughly as $1/k$ with sequence position $k$, and that real text correlations make the effect input-dependent, so no closed-form correction suffices. The discovery is that moving LayerNorm to the end of each residual branch (Res-Post-LayerNorm) and weighting residuals with a fixed $\tau$ makes the residual stream variance-preserving regardless of token position or data statistics. Combined with unit-variance initialization and a static $1/\sqrt{\mathrm{fan\_in}}$ GEMM multiplier in both forward and backward passes, every hidden linear layer can be cast to FP8 with no per-tensor scaling, and the Adam-like scale-invariance identity of prior hyperparameter-transfer theory then gives learning-rate transfer as $\sqrt{d_{\mathrm{base}}}/\sqrt{d_{\mathrm{new}}}$. The paper's central claim is that this package—not dynamic scaling—trains 1B–13B transformers in FP8 with quality equal to or better than BF16 and with zero-shot hyperparameter transfer across width.

Load-bearing premise

The transfer and stability results assume that the infinite-width scale-invariance identity of Adam-like optimizers, which gives the learning-rate rule, continues to hold exactly at finite widths, with the Lion optimizer, and with weight-decay transfer.

Editorial extensions

If this is right

  • FP8 training can be made static at billion-parameter scale: all hidden linear GEMMs run in FP8 with constant scaling factors, removing the forward and backward cost of computing dynamic scale factors.
  • Hyperparameters found on small proxies (width 256) transfer to models up to 20× wider with no re-tuning, which eliminates large-model hyperparameter sweeps.
  • Because training and inference both use FP8 weights and activations (W8A8), the precision mismatch that usually degrades quantized inference is removed.
  • At 1B–13B scale, FP8 with µS trains 25–33% faster than BF16 and 1–6% faster than TransformerEngine's dynamic FP8 under held-constant batch and distributed configurations.
  • µS needs only three hyperparameters (learning rate, weight decay, residual coefficient $\tau$), versus six or seven in prior transfer schemes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the variance-preservation story is right, the same static-scaling principle should extend to other low-precision formats (INT8, FP6) and to other architectures with residual streams, since the mechanism is about controlling tensor scale rather than about FP8 specifically.
  • The paper's Proposition 2.1 identifies a sequence-position-dependent scale collapse in attention; a testable consequence is that µS-trained models should show more uniform token-representation variance across long sequences, which could matter for long-context generalization even though the paper does not test it.
  • The depth–$\tau$ curve was fitted on models up to depth 100 with widths 256–2048 and then transferred to the 1B–13B runs; whether $\tau$ should also shift with width or token budget at even larger scales is left open.
  • The underflow analysis in Appendix A.7 suggests activation functions with a soft zero (GELU, SiLU) underflow in FP8 far more than ReLU; the paper keeps GELU for final-loss reasons, but a less underflow-prone activation or an asymmetric clip could push FP8 quality further.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes µnit Scaling (µS), a recipe for FP8 LLM training with static per-layer scaling factors, unit-variance initialization/output scaling, Res-Post-LayerNorm, a fixed residual coefficient τ, and µP-style learning-rate transfer. The authors derive a variance analysis of causal attention, propose square-root softmax attention but abandon it in favor of Res-Post-LN, and validate the final recipe by training 1B, 3B, 7B, and 13B models in FP8 with no dynamic scaling, reporting loss and downstream-task quality comparable to BF16 baselines plus 25–33% throughput gains. The paper also reports a controlled width sweep at four layers showing that optimal learning rate and weight decay are stable under µS from width 256 to 8192.

Significance. If the transfer and scalability claims hold, this is a practically important contribution: it would show that statically scaled FP8 training can be applied to all hidden linear layers at multi-billion-parameter scale while retaining BF16-quality loss trajectories and enabling zero-shot hyperparameter transfer across widths. The paper is transparent about its empirical scope, includes large-scale (1B–13B) runs with standard evaluation, and openly reports failures such as SP-FP8 divergence at 13B and Unit Scaling divergence at 7B. The central weakness is that the reported width-transfer experiments use only depth-4 proxies, while the large-scale runs use depths 24–40; because the paper itself shows that the residual coefficient must be tuned with depth, the large-scale experiments do not cleanly separate width transfer from depth effects. The derivation for attention variance is internally consistent under its stated iid assumptions, but the observed training behavior deviates from it, so the 'first-principles' narrative is only partially supported.

major comments (4)
  1. [§3.1, §3.2, Table 4] The hyperparameter-transfer claim is validated only at fixed depth four: Figure 6 sweeps widths 256–8192 for four-layer models, while the 1B–13B runs in Table 4 have depths 24, 32, and 40. Because Appendix A.3 shows that the optimal residual coefficient τ* is depth-dependent, depth is not a neutral variable in this setup, yet no sweep of η or λ versus depth is reported. The large-scale loss curves in Figure 7 therefore jointly depend on the width-scaling rule and on unmeasured depth effects, so the statement in §3.2 that these experiments 'validate zero-shot hyperparameter transfer using µS' is not fully supported. I recommend either adding a same-depth transfer experiment (e.g., a depth-24 base model with width 256 transferred to width 2048 at the same depth) or reporting η* and λ* versus depth at fixed width.
  2. [§3.1, Appendix A.3] The value τ=0.4 used for the four-layer models in §3.1 is an extrapolation: the sweep in Appendix A.3 covers depths 20, 40, 60, 80, and 100, not depth 4. Moreover, in the §3.1 sweeps, η and λ are varied while τ is fixed, so the reported optimal η* and λ* are conditioned on a residual coefficient that has not been directly validated for the proxy architecture. Since τ controls the balance between the residual stream and residual branches (Eq. 11), a misspecified τ could shift the apparent optima of η and λ. At minimum, the paper should report a sensitivity analysis of η* and λ* to τ at the base width, or a joint τ–η–λ sweep for depth 4.
  3. [§2.1, Fig. 2] The theoretical analysis of attention variance is not quantitatively consistent with the observations that motivate the final design. Proposition 2.1 predicts a 1/k decay in attention-output variance under iid value tokens, but Figure 2 shows that observed variance is higher than predicted and that the square-root softmax correction (Eq. 9), which is the direct consequence of the variance-preserving argument in Eqs. 7–8, overcorrects during training. The method ultimately adopted, Res-Post-LayerNorm, is an architectural change justified by correlated-value-token statistics (Fig. 3) and ablations (Fig. 4), not derived from Proposition 2.1. This is not a fatal flaw because the empirical case for Res-Post-LN is separate and convincing, but §1.1's claim that the paper identifies 'root causes' overstates the role of the derivation; the narrative should more clearly separate the iid analysis from the heuristic, empirically-motivated fix.
  4. [§2.3, Appendix A.4] The learning-rate transfer rule in Table 2 rests on the exact scale invariance of Eq. 16, which is an infinite-width result for Adam-like optimizers. Appendix A.4 argues that Lion has the same coordinatewise normalization structure as Adam, but it does not address how finite width or fully decoupled weight decay modifies the invariance. Since the paper transfers λ as constant across widths (Table 1) and uses the √(d_base)/√(d_new) rule at widths up to 5120, the theoretical basis would be stronger with either a finite-width analysis or an explicit statement that weight-decay transfer is an empirical finding supported only by the depth-4 sweep in Fig. 6.
minor comments (4)
  1. [§2.1, Eq. 7–8] In the displayed derivation, the text writes σ²_a(k) = ||c||² = √(Σ c_i²) = √(Σ s_i) = 1, but ||c||² is Σ c_i², not its square root; the equality should read ||c||² = Σ c_i² = Σ s_i = 1. This is a typographical error but it makes the variance argument confusing at first reading.
  2. [Fig. 8 caption] The caption in the main text refers to 'further details in Appendix 3.3', but the efficiency discussion is in Section 3.3, not an appendix. Please correct the cross-reference.
  3. [Appendix A.4, Eq. 24] The expression 'sign(c_t) = c_t / c_t²' is incorrect as written; the sign function is c_t / |c_t| = c_t / √(c_t²). The subsequent substitution of s_t^Lion = c_t² suggests the intended identity is c_t / √(c_t²), so the statement should be fixed to avoid a dimensional inconsistency.
  4. [Abstract, §2.2] The abstract says the method requires 'no dynamic scaling factors or special hyperparameters', but the final recipe includes the residual coefficient τ, which is tuned as a function of depth (Appendix A.3). This is a standard and small hyperparameter set, but the wording could be read as claiming τ-free training; I suggest a more precise phrasing such as 'no special hyperparameters beyond a residual-stream coefficient'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the η/λ transfer rule is derived from an explicit invariance relation and validated out-of-sample; the fitted residual coefficient τ is a design hyperparameter, not a disguised prediction.

full rationale

The paper's two load-bearing claims are statically scaled FP8 training and zero-shot hyperparameter transfer. Neither is equivalent to its inputs by construction. The learning-rate transfer rule is derived explicitly in Section 2.3: starting from the µP values (a,b,c)=(1,1/sqrt(fan_in),1/fan_in) and applying the Adam-like scale invariance (Eq. 16) with θ=1/sqrt(fan_in) yields a=1/sqrt(fan_in), b=1, c=1/sqrt(fan_in), which gives the stated 1/sqrt(fan_in) LR scaling. This derivation is reproduced from Yang et al. (2021, Appendix J.2.1), which is prior external work, not a self-citation. The rule is then validated out-of-sample: η* and λ* are measured on four-layer models of widths 256-8192 (Fig. 6) and transferred to 1B-13B models of widths up to 5120 (Table 4, Fig. 7). The residual coefficient τ is admittedly fitted in Appendix A.3 by sweeping depths 20-100 and is used to choose τ for the large models; however, this is ordinary hyperparameter transfer, and the paper does not relabel those fitted values as a prediction of the large-model results. The large-model FP8 runs independently confirm convergence and task quality. The paper's self-reported limitations (Appendix A.6: 'While we do not identify the exact mechanism...'; Appendix A.7: need for activation functions more suitable for FP8) are missing-mechanism or robustness caveats, not circular steps. The skeptic's concern that transfer is only demonstrated at depth 4 while applied to depths 24-40 is an external-validity risk about depth effects, not a by-construction equivalence between the recipe and its validation.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The method leans on four pillars: the scale-invariance theory of µP, the variance-preserving design rules of Unit Scaling, the residual-post-LN architecture from Liu et al. 2022, and an empirically fitted residual coefficient τ. The first three are prior art reused as assumptions; τ is the only component chosen by fitting to sweeps. No new physical or architectural entities are introduced.

free parameters (3)
  • base learning rate η* = swept on width-256 model; specific value not quoted
    Optimal learning rate for the width-256 base model, found by sweep over powers of 2; then transferred to larger models via the 1/sqrt(fan_in) rule.
  • base weight decay λ* = swept on width-256 model; specific value not quoted
    Optimal weight decay for base model, found by joint sweep; λ transfers unchanged.
  • residual coefficient τ* = 0.4 (depth 4); 0.3 (depths 24, 32); 0.2 (depth 40)
    Chosen from empirical depth-τ sweep in Appendix A.3; a coarsely swept hyperparameter used to keep the residual stream variance-preserving.
assumptions (5)
  • standard math Softmax moment approximations in Prop. 2.1: log-normal numerators, first-order Taylor ratio, Cov[n,d]=Var[n] (Sec 2.1, Eqs 2-6, Appendix A.2).
    Used to derive the 1/k variance scaling that motivates the attention discussion; the assumptions (iid standard normal logits) are stated and the result is approximate.
  • domain assumption Adam-like optimizer scale-invariance (Yang et al. Appendix J.2.1, Eq 16) holds for finite-width models and for Lion.
    This is the mathematical basis for the learning-rate transfer rule; the paper applies an asymptotic result without a finite-width proof, though Appendix A.4 argues Lion is Adam-like.
  • domain assumption Variance-preserving residual combination a^2+b^2=1 with chosen τ keeps the residual stream at unit variance (Sec 2.2, Eq 11).
    The fixed-scheme residual is taken from Blake et al. 2023; the specific τ values are chosen empirically.
  • domain assumption Static 1/sqrt(fan_in) output scaling and unit-variance initialization keep weight, activation, and gradient values within FP8 range throughout training (Table 1, Sec 3.3).
    This is the core premise of the no-dynamic-scaling claim; the paper provides activation distribution evidence in Appendix A.6 but does not track weight or gradient norms over time.
  • domain assumption FP8 e4m3/e5m2 formats with clipping are sufficient for all hidden layers at 1B-13B scale.
    Standard format choice; validated empirically by the reported loss curves.

how reviews work

0 comments
Cite this review

Pith. "Pith review of $\mu$nit Scaling: Simple and Scalable FP8 LLM Training." pith.science (2026). https://pith.science/paper/RL65NZWJ

@misc{pith2026250205967,
  author       = {Pith},
  title        = {Pith review of: $\mu$nit Scaling: Simple and Scalable FP8 LLM Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RL65NZWJ}},
  note         = {Machine review of arXiv:2502.05967}
}
abstract

Large Language Model training with 8-bit floating point (FP8) formats promises significant efficiency improvements, but reduced numerical precision makes training challenging. It is currently possible to train in FP8 only if one is willing to tune various hyperparameters, reduce model scale, or accept the overhead of computing dynamic scale factors. We demonstrate simple, scalable FP8 training that requires no dynamic scaling factors or special hyperparameters, even at large model sizes. Our method, $\mu$nit Scaling ($\mu$S), also enables simple hyperparameter transfer across model widths, matched numerics across training and inference, and other desirable properties. $\mu$nit Scaling is straightforward to implement, consisting of a set of minimal interventions based on a first-principles analysis of common transformer operations. We validate our method by training models from 1B to 13B parameters, performing all hidden linear layer computations in FP8. We achieve quality equal to higher precision baselines while also training up to 33% faster.

Figures

Figures reproduced from arXiv: 2502.05967 by the authors.

Figure 1
Figure 1. Comparison of low-precision training methods. Our proposed method, µnit Scaling (µS, bottom row), enables FP8 training and hyperparameter transfer at scale. Unlike existing methods, it does not use dynamic scaling, requires only a small set of hyperparameters, permits FP8 computation for all hidden layers, and makes the model more easily quantizable for inference. cost-effective. We demonstrate hyperparameter transf… view at source ↗
Figure 3
Figure 3. Value tokens in text are highly correlated. Comparison of cosine similarity between observed value tokens in a text data distribution versus value tokens iid∼ N (0, 1). Repeated tokens in the value matrix, an unavoidable result of token frequency in real text data, lead to higher-than-random σ as sequence position increases (cf [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 2
Figure 2. Attention output variance changes over sequence length. For standard attention, σ decreases over sequence po￾sition both when simulated with iid value tokens (light red) and when observed in training (red). Taking the square root of atten￾tion scores keeps σ constant when simulated with iid value tokens (light blue), but during training (blue), causes σ to increase with sequence position. In practice, neither attent… view at source ↗
Figures from the paper (10 more)
Figure 5
Figure 5. Figure 5: Residual modification schemes affect µnit Scaled model convergence. The fixed residual modification (green, Eq. 11) achieves better training convergence for deep transformers than the running-mean residual modification (blue, Eq. 12). The fixed residual coefficient for…
Figure 4
Figure 4. Figure 4: Res-Post-LayerNorm. (a) Pre-LayerNorm transformer architecture versus Res-Post-LayerNorm architecture. Res-Post￾LayerNorm moves the LayerNorm operation from the start of each residual branch to the end (Liu et al., 2022). This ensures con￾sistent variance across tokens…
Figure 6
Figure 6. Figure 6: With µS, optimal learning rate (η ∗ ) and weight de￾cay (λ ∗ ) are stable across widths. Optimal η (left column) and λ (right column) are shown across a range of model widths for models trained with SP (top row) and µS (bottom row). For each curve, the other hyperparam…
Figure 7
Figure 7. Figure 7: µS models successfully train in FP8 at scale. Comparison of training loss curves for standard parametrized (SP) and µnit scaled (µS) models in both FP8 and BF16, across 1B, 3B, 7B, and 13B parameter models. µS models successfully train in FP8 and converge to similar tr…
Figure 8
Figure 8. Figure 8: Training in FP8 with µS achieves state-of-the-art ef￾ficiency. FP8 training with µnit Scaling provides 25-33% higher throughput than BF16 training and 1-6% higher throughput than FP8 training with TransformerEngine (TE), over 1B, 3B, 7B, and 13B model sizes. Models are…
Figure 9
Figure 9. Figure 9: Optimal residual coefficient τ ∗ decreases with depth. The 3 hyperparameters of τ , η, and λ are swept for models of varying widths (256, 512, 1024, 2048) and depths (20, 40, 60, 80, 100). The mean and standard error of τ is shown for the optimal subset of models from …
Figure 10
Figure 10. Figure 10: Unit Scaling (US) vs µnit Scaling (µS) for 7B models. Convergence test loss curves at 7B model scale show that µS converges smoothly while US training diverges early in training. A.6. Activation Outliers We analyze activation distributions taken over 32,768 tokens at …
Figure 11
Figure 11. Figure 11: Different activation functions cause different amounts of FP8 underflow. When casting N (0, 1) or Unif(−128, 128) values from BF16 to FP8 (e4m3), GELU, SiLU, and ReLU (green) erroneously round to zero (underflow) with different probabilties. GELU and SiLU experience s…
Figure 12
Figure 12. Figure 12: Activation function choice impacts FP8 underflow and low-precision convergence error. FP8 underflow of activation function outputs for each block in a 4 layer transformer model during training is shown for GELU, SiLU, and ReLU. Low precision convergence error, defined…
Figure 13
Figure 13. Figure 13: Activation distributions of µS and SP models. Activation distributions for attention and FFN block inputs and outputs are shown for 1B, 3B, 7B, and 13B FP8 models at every 10th layer. µS models lack the notable right tail of activation outliers in block inputs that SP…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 20 canonical work pages

  1. [1]

    Scaling FP 8 training to trillion-token LLM s

    Anonymous. Scaling FP 8 training to trillion-token LLM s. In Submitted to The Thirteenth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=E1EHO0imOb. under review

    work page 2024

  2. [2]

    Calibrating the Mosaic evaluation Gauntlet , 4 2024

    Barton, T. Calibrating the Mosaic evaluation Gauntlet , 4 2024. URL https://www.databricks.com/blog/calibrating-mosaic-evaluation-gauntlet

    work page 2024

  3. [3]

    Unit scaling: Out-of-the-box low-precision training

    Blake, C., Orr, D., and Luschi, C. Unit scaling: Out-of-the-box low-precision training. In International Conference on Machine Learning, pp.\ 2548--2576. PMLR, 2023

    work page 2023

  4. [4]

    Y., Deiseroth, B., Cruz-Salinas, A

    Blake, C., Eichenberg, C., Dean, J., Balles, L., Prince, L. Y., Deiseroth, B., Cruz-Salinas, A. F., Luschi, C., Weinbach, S., and Orr, D. u- p: The unit-scaled maximal update parametrization. In 2nd Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (WANT@ICML 2024), 2024. URL https://openreview...

    work page 2024

  5. [5]

    and Berger, R

    Casella, G. and Berger, R. L. Statistical Inference. Duxbury, Pacific Grove, CA, 2nd edition, 2002. ISBN 978-0-534-24312-8. URL https://pages.stat.wisc.edu/ shao/stat610/Casella_Berger_Statistical_Inference.pdf

    work page 2002

  6. [6]

    Chen, X., Liang, C., Huang, D., Real, E., Wang, K., Pham, H., Dong, X., Luong, T., Hsieh, C.-J., Lu, Y., and Le, Q. V. Symbolic discovery of optimization algorithms. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=ne6zeqLFCZ

    work page 2023

  7. [7]

    LLM .int8(): 8-bit matrix multiplication for transformers at scale

    Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. LLM .int8(): 8-bit matrix multiplication for transformers at scale. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=dXiGWqBoxaD

    work page 2022

  8. [8]

    Blazingly fast LLM evaluation for in-context learning, 2 2023

    Dohmann, J. Blazingly fast LLM evaluation for in-context learning, 2 2023. URL https://www.databricks.com/blog/llm-evaluation-for-icl

    work page 2023

Show all 32 references
  1. [9]

    Flex attention: A programming model for generating optimized attention kernels, 2024

    Dong, J., Feng, B., Guessous, D., Liang, Y., and He, H. Flex attention: A programming model for generating optimized attention kernels, 2024. URL https://arxiv.org/abs/2412.05496

    2024 arXiv

  2. [10]

    Fbgemm: Enabling high-performance low-precision deep learning inference, 2021

    Khudia, D., Huang, J., Basu, P., Deng, S., Liu, H., Park, J., and Smelyanskiy, M. Fbgemm: Enabling high-performance low-precision deep learning inference, 2021. URL https://arxiv.org/abs/2101.05615

    2021 arXiv

  3. [11]

    Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization, 2017. URL https://arxiv.org/abs/1412.6980

    2017 arXiv

  4. [12]

    A large-scale exploration of -transfer, 2024

    Lingle, L. A large-scale exploration of -transfer, 2024. URL https://arxiv.org/abs/2404.05728

    2024 arXiv

  5. [13]

    Swin transformer v2: Scaling up capacity and resolution

    Liu, Z., Hu, H., Lin, Y., Yao, Z., Xie, Z., Wei, Y., Ning, J., Cao, Y., Zhang, Z., Dong, L., et al. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 12009--12019, 2022

    2022

  6. [14]

    Mixed precision training

    Micikevicius, P., Narang, S., Alben, J., Diamos, G., Elsen, E., Garcia, D., Ginsburg, B., Houston, M., Kuchaiev, O., Venkatesh, G., and Wu, H. Mixed precision training. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=r1gs9JgRZ

    2018

  7. [15]

    Fp8 formats for deep learning

    Micikevicius, P., Stosic, D., Burgess, N., Cornea, M., Dubey, P., Grisenthwaite, R., Ha, S., Heinecke, A., Judd, P., Kamalu, J., et al. Fp8 formats for deep learning. arXiv preprint arXiv:2209.05433, 2022

    2022 arXiv

  8. [16]

    I., Alizadeh-Vahid, K., Mehta, S., del Mundo, C

    Mirzadeh, S. I., Alizadeh-Vahid, K., Mehta, S., del Mundo, C. C., Tuzel, O., Samei, G., Rastegari, M., and Farajtabar, M. Re LU strikes back: Exploiting activation sparsity in large language models. In The Twelfth International Conference on Learning Representations, 2024. URL...

    2024

  9. [17]

    Composer

    MosaicML. Composer. https://github.com/mosaicml/composer/, 2021

    2021

  10. [18]

    LLM F oundry

    MosaicML. LLM F oundry. <https://github.com/mosaicml/llm-foundry/>, 2022 a

    2022

  11. [19]

    Streaming

    MosaicML. Streaming. <https://github.com/mosaicml/streaming/>, 2022 b

    2022

  12. [20]

    Asynchronous multiply-and-accumulate instruction: wgmma.mma\_async

    NVIDIA. Asynchronous multiply-and-accumulate instruction: wgmma.mma\_async. URL https://docs.nvidia.com/cuda/parallel-thread-execution/#asynchronous-warpgroup-level-matrix-instructions-wgmma-mma

  13. [21]

    Transformer E ngine, 2023

    NVIDIA. Transformer E ngine, 2023. URL https://github.com/NVIDIA/TransformerEngine

    2023

  14. [22]

    cuBLAS : cublasLtMatmul()

    NVIDIA Corporation . cuBLAS : cublasLtMatmul() . NVIDIA, 2024. URL https://docs.nvidia.com/cuda/cublas/#cublasltmatmul

    2024

  15. [23]

    2 olmo 2 furious

    OLMo, T., Walsh, P., Soldaini, L., Groeneveld, D., Lo, K., Arora, S., Bhagia, A., Gu, Y., Huang, S., Jordan, M., et al. 2 olmo 2 furious. arXiv preprint arXiv:2501.00656, 2024

    2024 arXiv

  16. [24]

    V., Cui, X., Zhang, W., and Gopalakrishnan, K

    Sun, X., Choi, J., Chen, C.-Y., Wang, N., Venkataramani, S., Srinivasan, V. V., Cui, X., Zhang, W., and Gopalakrishnan, K. Hybrid 8-bit floating point (hfp8) training and inference for deep neural networks. In Wallach, H., Larochelle, H., Beygelzimer, A., d Alch\' e -Buc, F., ...

    2019

  17. [25]

    T., and Cox, D

    Tillet, P., Kung, H. T., and Cox, D. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2019, pp.\ 10–19, New York, NY, USA, 2019....

    2019

  18. [26]

    N., Kaiser, L

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. Attention is all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Informatio...

    2017

  19. [27]

    and Aitchison, L

    Wang, X. and Aitchison, L. How to set A dam W 's weight decay as you scale model and dataset size, 2024. URL https://arxiv.org/abs/2405.13698

    2024 arXiv

  20. [28]

    J., Xiao, L., Everett, K

    Wortsman, M., Liu, P. J., Xiao, L., Everett, K. E., Alemi, A. A., Adlam, B., Co-Reyes, J. D., Gur, I., Kumar, A., Novak, R., Pennington, J., Sohl-Dickstein, J., Xu, K., Lee, J., Gilmer, J., and Kornblith, S. Small-scale proxies for large-scale transformer training instabilitie...

    2024

  21. [29]

    J., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W., and Gao, J

    Yang, G., Hu, E. J., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W., and Gao, J. Tuning large neural networks via zero-shot hyperparameter transfer. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Info...

    2021

  22. [30]

    B., and Bernstein, J

    Yang, G., Simon, J. B., and Bernstein, J. A spectral condition for feature learning. arXiv preprint arXiv:2310.17813, 2023

    2023 arXiv

  23. [31]

    Tensor programs VI : Feature learning in infinite depth neural networks

    Yang, G., Yu, D., Zhu, C., and Hayou, S. Tensor programs VI : Feature learning in infinite depth neural networks. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=17pVDnpwwl

    2024

  24. [32]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.