Pith. sign in

REVIEW 4 major objections 5 minor 49 references

The paper claims that many distinct low-precision faults in transformer training converge on a single query-key spectral runaway, so one targeted intervention at the QK locus can contain them all.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 15:15 UTC pith:4CRAALYV

load-bearing objection Worth serious refereeing: a careful, honest study that convincingly shows a QK-channel dissociation within a software-emulated fault model, but the motivating 'bf16 collapse' is not shown on stock hardware. the 4 major comments →

arxiv 2608.02091 v1 pith:4CRAALYV submitted 2026-08-03 cs.LG

One QK Channel, Many Sources: Guarding Low-Precision Attention Collapse

Shuxiao Xie , Shuyang Xie , Yuan Cao , Dezhi Ran , Wei Yang , Tao Xie This is my paper
classification cs.LG
keywords low-precision trainingattention collapsequery-key spectral runawayMXFP4 quantizationstreaming softmaxQK normalizationtraining stabilitybfloat16
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that heterogeneous low-precision numerical faults in bfloat16 transformer training—bf16 streaming-softmax accumulation, post-reduction weight-gradient quantization, and errors on MLP or value gradients—do not each need their own repair because they all converge on a shared failure channel: a spectral runaway in the query-key (QK) attention projections. By moving a controlled error across fault sources while splicing clean gradients into one route at a time, the paper shows a source-channel dissociation: cleaning only the QK gradient route blocks collapse even though the original fault remains active, while cleaning the MLP route does not. A causal probe confirms the QK channel drives the early runaway, and entry into the channel is gated by temporal sign-coherence of the error across optimization steps, not by its aggregate magnitude. The paper then presents QK-Guard, a dormant controller that switches on parameter-free per-head QK normalization when attention-logit saturation begins, and reports it contains every tested runaway and matches always-on QK normalization over 60k steps. If correct, the practical upshot is that low-precision training collapse can be contained at one shared locus instead of requiring per-source fixes.

Core claim

The central discovery is a source-channel dissociation: errors originating inside attention (the streaming-softmax accumulator), outside attention (MLP and value weight-gradient quantization), and at different quantization placements all drive the same query-key spectral runaway, measured as growth in the largest singular value of the query projection, sigma(W_Q). Cleaning only the query-key gradient route suppresses this runaway and keeps training stable while the fault source remains active, whereas cleaning the MLP route does not. Projecting each step's update off the current W_Q and W_K leading three singular directions holds sigma(W_Q) near 11.1 across all tested arms, while removing eq

What carries the argument

The central object is the QK spectral runaway: growth of the largest singular value sigma(W_Q) of the query projection, which acts as the shared failure channel that heterogeneous low-precision faults converge on. Three mechanisms carry the argument: the source-channel dissociation assay, which moves a controlled error across fault sources while splicing a clean gradient into one named route; the causal probe, which projects each realized parameter update off the current W_Q/W_K leading three singular directions (and rescales back to the original norm) to show that on-target removal suppresses the runaway while equal-energy off-target removal does not; and QK-Guard, a dormant closed-loop con

Load-bearing premise

The central results depend on software-emulated low-precision error faithfully reproducing native low-precision training failure, since the MXFP4 quantizer remains software-emulated and the exact-code B200 run does not validate native four-bit arithmetic.

What would settle it

Run the same GPT-2-class training protocol on hardware with native four-bit (MXFP4) arithmetic and test whether (a) quantizing only the MLP or value weight gradients still drives the sigma(W_Q) runaway and collapses training, and (b) cleaning only the QK gradient route, or triggering QK-Guard on attention-logit saturation, still blocks that collapse. If either fails, the source-channel dissociation is an artifact of the software emulator rather than a property of native low-precision training.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Faults originating outside attention can still drive the same QK channel, so source-local repairs such as fp32 accumulation do not generalize across low-precision fault types.
  • Cleaning only the QK gradient route blocks collapse even while the underlying fault remains active, meaning a single intervention locus can substitute for separate repairs at each source within the paper's stated scope.
  • Entry into the QK channel is controlled by temporal sign-coherence across optimizer steps, so per-step error magnitudes or scalar means are insufficient predictors of collapse.
  • Attention-logit saturation provides a working trigger for containment that transfers across architectures, scales, and a second GPU architecture without re-tuning, and QK-Guard matches always-on QK normalization over 60k steps.
  • The QK channel is a causal driver of the early spectral runaway rather than a correlate, since direction-specific removal of update components suppresses the runaway while equal-energy off-target removal does not.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the QK channel is a universal attractor for sign-coherent low-precision errors, then monitoring attention-logit statistics or sigma(W_Q) in production could serve as an early-warning signal, and always-on or triggered QK normalization could become a standard safety layer for low-precision training.
  • The temporal sign-coherence requirement suggests the mechanism is tied to how the optimizer's momentum accumulates signed updates: a testable extension is that collapse should be manipulable by reversing or randomizing the momentum's sign structure rather than only the injected error's signs.
  • The paper's scope explicitly excludes sources that act after the final attention module, so a natural next experiment is to inject a sign-coherent error into a post-attention residual block or final head and check whether it escapes QK-channel containment, which would bound the guard's coverage.
  • Native four-bit hardware arithmetic, which the paper identifies as 'the next test,' is the decisive check: if native rounding produces error with different temporal sign structure, the source-channel dissociation and QK-Guard's trigger may not transfer from software emulation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies abrupt collapse in low-precision transformer training and argues that heterogeneous error sources converge on a single query–key (QK) spectral runaway, so intervention should target the QK locus rather than each source. It first isolates a reproducible GPT-2-class collapse in a custom bf16 streaming-softmax recurrence and shows that fp32 accumulation repairs it (§2). It then moves controlled errors across sources (weight-gradient quantization on QK, MLP, and value gradients) and shows that cleaning only the QK gradient route blocks collapse while the source fault stays active (§3). A causal probe removes each update's component along the current WQ/WK leading singular directions, suppressing the runaway, while equal-energy off-target removal fails (§4.2). Entry is attributed to temporal sign-coherence rather than aggregate deviation (§4.1). Finally, QK-Guard, a dormant controller triggered by attention-logit saturation, applies per-head QK normalization at the QK locus and contains tested runaways (§5). The paper is unusually candid about its boundaries, including software-emulated quantization, small seed counts, and the provenance of the trigger threshold.

Significance. If the central claim holds, the paper makes a valuable conceptual contribution: fault source is not failure channel, and a single locus intervention can contain faults that arise in different modules. The study is also a model of honest empirical practice: matched controls, disclosed exclusions, explicit per-arm seed counts, exact-code cross-hardware portability checks, and clearly stated limitations (e.g., software-emulated MXFP4, threshold chosen by inspection) are included. The direction-removal experiment is a strong causal probe because the equal-energy off-target control addresses the obvious dose confound. The QK-Guard design, separating sensing from action and comparing action loci at the same fire step, is a well-posed test of the paper's thesis. However, the empirical base is entirely built on emulated or custom precision paths, and several load-bearing claims rely on very small seed counts; the significance conditional on native-hardware validation is high, but the current evidence is narrower than the abstract's phrasing suggests.

major comments (4)
  1. [§2, Table 1(a)] The motivating collapse is shown only in the authors' custom re-implementation of the FlashAttention recurrence with bf16 output and statistics carries. The 'FlashAttention-2 forced' and SDPA arms are clean (0/2 through 4900). Real FlashAttention-2 kernels accumulate the running output and softmax statistics in fp32, so the isolated fault is arguably an artifact of the reimplementation's bf16 carries rather than a native bf16-transformer behavior. This directly weakens the opening claim 'A bfloat16 transformer can train normally for many steps and then collapse abruptly.' The paper should either demonstrate the collapse with a native kernel or explicitly scope the entire contribution to custom bf16-carry recurrences.
  2. [§3.3, §6, Appendix D] The source-channel dissociation, entry analysis, and QK-Guard results all rely on the software-emulated MXFP4 quantizer and the custom recurrence. As the paper states, the B200 exact-code run 'does not validate native four-bit arithmetic' and 'native four-bit arithmetic is the next test.' Because native hardware rounding may have different error structure, the central claim that heterogeneous low-precision faults share one QK channel has not been shown for real low-precision hardware. This is a load-bearing limitation, not a presentation issue. The paper should either provide native-hardware validation or visibly restrict the title and abstract claims to emulated low-precision training.
  3. [§5.1, Appendix A.5] The QK-Guard trigger threshold of 30 was chosen by inspecting both healthy and doomed trajectories, and the appendix states that RoPE and 350M behavior had already been observed when the value was fixed. It is therefore not a pre-declared or out-of-sample threshold, and no false-activation rate can be reported. Consequently, the 'transfer' of the threshold in §5.3 is not an independent replication of a calibrated detector. The paper is honest about this, but the strength of the controller claim should be reduced accordingly: the evidence supports 'a fixed early-chosen threshold that transfers in these specific settings,' not 'the threshold generalizes.'
  4. [Tables 1–3, Figure 1, §3.3] Many central contrasts rest on very small seed counts: Table 2 has cells with 1/1 untreated collapses, several route-cleaning cells have 2–3 seeds, Figure 1 uses one seed per arm, and the B200 portability check is one seed per arm. Given the observed seed-dependent variability (e.g., the RoPE V/O row and the seed-variable weight-gradient outcome), the strong claims 'MLP cleaning rescues none' and 'QK-Guard contains every tested runaway' are supported by limited samples. Additional seeds or explicit uncertainty quantification are needed for the broad statements in the abstract and conclusion.
minor comments (5)
  1. [Abstract] The phrase 'fp32 accumulation repairs it' is stated without noting that this holds only in the custom recurrence; consider qualifying as 'in the custom recurrence.'
  2. [Appendix A.5] The sentence 'We cannot report a false-activation rate, and we do not' is good, but the subsequent threshold sweep (one seed per threshold) should be flagged even more prominently in the main text, since it is easy to over-read.
  3. [Figure 2 caption] The caption says 'A causal probe, not a remedy' but the main text later describes the 60k limit in Appendix D. Consider adding a sentence in §4.2 pointing the reader to the long-horizon limit, as it is easy to miss.
  4. [Table S1] The SGD generality rows state that momentum and rank count are not recoverable; this is a serious metadata gap. It is disclosed, but consider moving this disclosure to the main text where SGD is mentioned in §6.
  5. [§5.2] The phrase 'always-on gradient clipping was already active in every collapsed run reported here' is important context; it may deserve a sentence in the main text methods rather than only in the results section.

Circularity Check

0 steps flagged

No significant circularity: the paper's contrasts are controlled interventions, and its acknowledged limits are external-validity concerns, not definitional dependencies.

full rationale

I walked the paper's derivation chain from §2 through §5. The load-bearing claims are all supported by controlled interventions or direct measurements, not by fitting parameters and then reporting them as predictions, and not by importing conclusions from self-citations. The fault-isolation contrast varies accumulator precision while holding tiling fixed; the source-channel dissociation varies the fault generator or the cleaned gradient route while leaving the source mechanism active; the entry-condition experiments manipulate sign structure, mean components, and DC bias at matched magnitude; and QK-Guard is tested with fixed trigger, fixed fire step, and locus controls. No step exhibits the pattern of a quantity being defined in terms of the outcome it is supposed to predict. The causal probe in §4.2 is the closest to a self-referential design: it removes each update's component along the current top singular directions of WQ/WK and then reports that sigma(WQ) stays low. But the paper does not rest the causal claim on that arm alone; it includes an equal-energy off-target control and an isotropic-shrink control, both of which fail. The contrast between on-target removal and these controls is an intervention, not a tautology. The threshold of 30 is admittedly chosen by inspecting healthy and doomed envelopes and is not claimed to be a calibrated detector; the paper explicitly discloses this and provides threshold-transfer sensitivity results. The software-emulated MXFP4 quantizer and the re-implemented FlashAttention recurrence are fidelity limitations that the paper itself states prominently (§3.3, Appendix D, §6), but they affect whether the results transfer to native hardware, not whether the in-paper derivation is circular. I find no quoted reduction of any result to its own inputs, no fitted parameter renamed as a prediction, and no load-bearing self-citation chain. Therefore no circularity steps are reported.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

No new physical or model entities are introduced. The free parameters are protocol constants (threshold, crash rule, probe rank k) rather than fitted scientific constants. The load-bearing assumptions are the representativeness of emulated low precision and of small-scale models.

free parameters (3)
  • QK-Guard trigger threshold tau = 30
    Chosen by inspecting the gap between healthy and doomed maximum-attention-logit envelopes in an early session; the paper states no pre-declared rule yields 30 and no false-activation rate can be reported (Appendix A.5).
  • Sustained-divergence crash rule constants = 1.5 nat above running minimum; 0.75 recovery margin
    Define what counts as every collapse in the paper; no sensitivity analysis is reported for these constants (Appendix A.2).
  • Number of leading singular directions removed (k) = 3
    The causal probe removes three leading singular directions of WQ and WK; no k-sensitivity study is reported, and the causal conclusion depends on this choice (Section 4.2).
axioms (5)
  • domain assumption Software-emulated low-precision arithmetic faithfully represents native low-precision training behavior
    The FlashAttention recurrence is a re-implementation and the MXFP4 quantizer is software-emulated; the paper says the B200 result 'does not validate native four-bit arithmetic' and names native four-bit as the next test (Sections 3.3, 6).
  • domain assumption Collapse in GPT-2-class models on OpenWebText is representative of low-precision LLM training collapse generally
    All main-grid results are GPT-2 small/medium and a 350M model; the paper itself lists scale as a known edge (Section 6).
  • domain assumption The clean-gradient splice and injected-error assays isolate the failure channel without artifacts
    Section 3.1 runs the faulty and fp32 reference backward and splices one route's gradient; this assumes the remaining faulty routes still exercise the real mechanism.
  • ad hoc to paper The sustained-divergence crash rule captures the relevant failure
    This rule and its 1.5/0.75 constants are introduced for this paper and define every collapse verdict (Appendix A.2).
  • standard math Online-softmax recurrence and Cauchy-Schwarz bound standard math
    Used for the QK clip bound and FlashAttention recurrence (Appendix A.4).

pith-pipeline@v1.3.0-daily-deepseek · 22724 in / 14953 out tokens · 95740 ms · 2026-08-04T15:15:31.799756+00:00 · methodology

0 comments
read the original abstract

A bfloat16 transformer can train normally for many steps and then collapse abruptly. Distinct low-precision errors can trigger the same failure, leaving unclear whether each source needs its own repair or one shared route can be blocked. We isolate a reproduced GPT-2-class collapse to the streaming-softmax accumulator, where fp32 accumulation repairs it, and use the fault as an assay for moving controlled errors across sources. Errors placed outside attention still drive the same query-key (QK) spectral runaway, while correcting only QK keeps training stable with the source fault active. This source-channel dissociation shows that fault source is not failure channel. It holds across the tested architectures and scales and reproduces on a second GPU architecture. A causal probe projects each update off the current QK weights' leading three singular directions: the query projection's largest singular value stays at 11.1, whereas removing equal energy elsewhere leaves it at 237. The QK channel therefore drives the early runaway rather than merely tracking it. Entry depends on temporal sign-coherence across steps, not aggregate deviation. QK-Guard closes the channel with a dormant controller that switches on parameter-free QK normalization when attention-logit saturation begins. It contains every tested runaway and matches always-on QK normalization over 60k steps, while non-QK actions at the same trigger fail. The results support intervention at the shared QK locus rather than separate repair at each fault source.

Figures

Figures reproduced from arXiv: 2608.02091 by Dezhi Ran, Shuxiao Xie, Shuyang Xie, Tao Xie, Wei Yang, Yuan Cao.

Figure 1
Figure 1. Figure 1: Source-channel dissociation. (a) one weight-gradient quantizer routed to the 𝑄𝐾, MLP or value weight gradient drives the same 𝜎(𝑊𝑄 ) runaway — matched source locations of one quantizer, not three mechanisms; per-head 𝑄𝐾-norm twins survive with the fault still running, the action capping the attention logit rather than 𝜎 itself (Figure S1, Appendix D). (b) outcomes for the same runs and step axis. Markers n… view at source ↗
Figure 2
Figure 2. Figure 2: Direction-specific removal suppresses the [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: QK-Guard anatomy, one step axis throughout. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

49 extracted references · 31 linked inside Pith

  1. [1]

    Controllingchangestoattentionlogits.arXivpreprintarXiv:2511.21377, 2025

    BenAnsonandLaurenceAitchison. Controllingchangestoattentionlogits.arXivpreprintarXiv:2511.21377, 2025

    arXiv 2025

  2. [2]

    Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818, 2024

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818, 2024

    Pith/arXiv arXiv 2024

  3. [3]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prab- hakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bra...

    Pith/arXiv arXiv 2022

  4. [4]

    Pretraining large language models with MXFP4 on native FP4 hardware, 2026

    Musa Cim, Poovaiah Palangappa, Miro Hodak, Ravi Dwivedula, Meena Arunachalam, and Mahmut Taylan Kandemir. Pretraining large language models with MXFP4 on native FP4 hardware, 2026. URLhttps: //arxiv.org/abs/2605.09825

    Pith/arXiv arXiv 2026

  5. [5]

    FlashAttention-2: Faster attention with better parallelism and work partitioning.arXiv preprint arXiv:2307.08691, 2023

    Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning.arXiv preprint arXiv:2307.08691, 2023

    Pith/arXiv arXiv 2023

  6. [6]

    why low-precision transformer training fails: An analysis on flash atten- tion

    Tri Dao. Comment on “why low-precision transformer training fails: An analysis on flash atten- tion”. GitHub Discussion inDao-AILab/flash-attention, 2025. URLhttps://github.com/Dao- AILab/flash-attention/discussions/1931. Discussion#1931;commentposted13October2025; accessed 28 July 2026

    2025

  7. [7]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness, 2022. URLhttp://arxiv.org/abs/2205.14135v2

    Pith/arXiv arXiv 2022

  8. [8]

    DeepSeek-V3 technical report.arXiv preprint arXiv:2412.19437, 2024

    DeepSeek-AI. DeepSeek-V3 technical report.arXiv preprint arXiv:2412.19437, 2024. 10 One QK Channel, Many SourcesPreprint

    Pith/arXiv arXiv 2024

  9. [9]

    Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, Rodolphe Jenatton, Lucas Beyer, Michael Tschannen, Anurag Arnab, Xiao Wang, Carlos Riquelme, Matthias Minderer, Joan Puigcerver, Utku Evci, Manoj Kumar, Sjoerd van Steenkiste, Gamaleldin F....

    Pith/arXiv arXiv 2023

  10. [10]

    Dissecting outlier dynamics in llm nvfp4 pretraining, 2026

    Peijie Dong, Ruibo Fan, Yuechen Tao, Di Mou, Wenhu Hu, Zhenheng Tang, Yinghao Yu, Jiamang Wang, Wenbo Su, Guodong Yang, Liping Zhang, Xiaowen Chu, Baochun Li, and Bo Li. Dissecting outlier dynamics in llm nvfp4 pretraining, 2026. URLhttp://arxiv.org/abs/2602.02047

    arXiv 2026

  11. [11]

    Rank-aware spectral bounds on attention logits for stable low-precision training,

    Seyed Morteza Emadi. Rank-aware spectral bounds on attention logits for stable low-precision training,

  12. [12]

    Scaling fp8 training to trillion-token llms,

    Maxim Fishman, Brian Chmiel, Ron Banner, and Daniel Soudry. Scaling fp8 training to trillion-token llms,

  13. [13]

    OpenWebText corpus

    Aaron Gokaslan and Vanya Cohen. OpenWebText corpus. http://Skylion007.github.io/ OpenWebTextCorpus, 2019

    2019

  14. [14]

    Is flash attention stable?, 2024

    Alicia Golden, Samuel Hsia, Fei Sun, Bilge Acun, Basil Hosmer, Yejin Lee, Zachary DeVito, Jeff Johnson, Gu-Yeon Wei, David Brooks, and Carole-Jean Wu. Is flash attention stable?, 2024. URLhttp://arxiv. org/abs/2405.02803v1

    Pith/arXiv arXiv 2024

  15. [15]

    Deep learning with limited numerical precision, 2015

    Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision, 2015. URLhttp://arxiv.org/abs/1502.02551

    Pith/arXiv arXiv 2015

  16. [16]

    Query-key normalization for transformers, 2020

    Alex Henry, Prudhvi Raj Dachapally, Shubham Pawar, and Yuxuan Chen. Query-key normalization for transformers, 2020. URLhttp://arxiv.org/abs/2010.04245v1

    Pith/arXiv arXiv 2020

  17. [17]

    Nicholas J. Higham. The accuracy of floating point summation.SIAM Journal on Scientific Computing, 14 (4):783–799, 1993. doi:10.1137/0914050

    doi:10.1137/0914050 1993

  18. [18]

    Mechanism- driven monitors for preemptive detection of llm training instability, 2026

    Ruixuan Huang, Hantao Huang, Yifan Huang, Ansheng You, Zhenxing Zhang, and Shuai Wang. Mechanism- driven monitors for preemptive detection of llm training instability, 2026. URLhttps://arxiv.org/ abs/2606.28116

    Pith/arXiv arXiv 2026

  19. [19]

    Spam: Spike-aware adam with momentum reset for stable llm training, 2025

    Tianjin Huang, Ziquan Zhu, Gaojie Jin, Lu Liu, Zhangyang Wang, and Shiwei Liu. Spam: Spike-aware adam with momentum reset for stable llm training, 2025. URLhttp://arxiv.org/abs/2501.06842v2

    Pith/arXiv arXiv 2025

  20. [20]

    A study of bfloat16 for deep learning training, 2019

    Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, Jiyan Yang, Jongsoo Park, Alexander Heinecke, Evangelos Georganas, Sudarshan Srinivasan, Abhisek Kundu, Misha Smelyanskiy, Bharat Kaul, and Pradeep Dubey. A study of bfloat16 for de...

    Pith/arXiv arXiv 2019

  21. [21]

    Zclip: Adaptive spike mitigation for llm pre-training, 2025

    Abhay Kumar, Louis Owen, Nilabhra Roy Chowdhury, and Fabian Güra. Zclip: Adaptive spike mitigation for llm pre-training, 2025. URLhttp://arxiv.org/abs/2504.02507v1

    Pith/arXiv arXiv 2025

  22. [22]

    Decoupled weight decay regularization, 2017

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2017. URLhttp://arxiv. org/abs/1711.05101v3

    Pith/arXiv arXiv 2017

  23. [23]

    Mixed precision training,

    Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training,

  24. [24]

    Online normalizer calculation for softmax, 2018

    Maxim Milakov and Natalia Gimelshein. Online normalizer calculation for softmax, 2018. URLhttp: //arxiv.org/abs/1805.02867v2

    Pith/arXiv arXiv 2018

  25. [25]

    A theory on adam instability in large-scale machine learning,

    Igor Molybog, Peter Albert, Moya Chen, Zachary DeVito, David Esiobu, Naman Goyal, Punit Singh Koura, Sharan Narang, Andrew Poulton, Ruan Silva, Binh Tang, Diana Liskovich, Puxin Xu, Yuchen Zhang, Melanie Kambadur, Stephen Roller, and Susan Zhang. A theory on adam instability in large-scale machine learning,

  26. [26]

    2 olmo 2 furious, 2024

    OLMo Team. 2 olmo 2 furious, 2024. URLhttp://arxiv.org/abs/2501.00656

    Pith/arXiv arXiv 2024

  27. [27]

    Taming transformer without using learning rate warmup

    Xianbiao Qi, Yelin He, Jiaquan Ye, Chun-Guang Li, Bojia Zi, Xili Dai, Qin Zou, and Rong Xiao. Taming transformer without using learning rate warmup. InInternational Conference on Learning Representations 11 One QK Channel, Many SourcesPreprint (ICLR), 2025

    2025

  28. [28]

    Why low-precision transformer training fails: An analysis on flash attention, 2025

    Haiquan Qiu and Quanming Yao. Why low-precision transformer training fails: An analysis on flash attention, 2025. URLhttp://arxiv.org/abs/2510.04212v4

    Pith/arXiv arXiv 2025

  29. [29]

    Spectral alignment as predictor of loss explosion in neural network training, 2025

    Haiquan Qiu, You Wu, Yingjie Tan, Yaqing Wang, and Quanming Yao. Spectral alignment as predictor of loss explosion in neural network training, 2025. URLhttp://arxiv.org/abs/2510.04202v1

    arXiv 2025

  30. [30]

    Lan- guage models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Lan- guage models are unsupervised multitask learners. OpenAI technical report, 2019. URL https: //cdn.openai.com/better-language-models/language_models_are_unsupervised_ multitask_learners.pdf

    2019

  31. [31]

    Microscaling data formats for deep learning, 2023

    Bita Darvish Rouhani, Ritchie Zhao, Ankit More, Mathew Hall, Alireza Khodamoradi, Summer Deng, Dhruv Choudhary, Marius Cornea, Eric Dellinger, Kristof Denolf, Stosic Dusan, Venmugil Elango, Maximilian Golub, Alexander Heinecke, Phil James-Roxby, Dharmesh Jani, Gaurav Kolhe, Martin Langhammer, Ada Li, Levi Melnick, Maral Mesmakhosroshahi, Andres Rodriguez,...

    Pith/arXiv arXiv 2023

  32. [32]

    Methods of improving LLM training stability.arXiv preprint arXiv:2410.16682, 2024

    Oleg Rybakov, Mike Chrzanowski, Peter Dykas, Jinze Xue, and Ben Lanir. Methods of improving LLM training stability.arXiv preprint arXiv:2410.16682, 2024

    Pith/arXiv arXiv 2024

  33. [33]

    Accumulation bit-width scaling for ultra-low precision training of deep networks, 2019

    Charbel Sakr, Naigang Wang, Chia-Yu Chen, Jungwook Choi, Ankur Agrawal, Naresh Shanbhag, and Kailash Gopalakrishnan. Accumulation bit-width scaling for ultra-low precision training of deep networks, 2019. URLhttp://arxiv.org/abs/1901.06588v1

    Pith/arXiv arXiv 2019

  34. [34]

    Characterization and mitigation of training instabilities in microscaling formats, 2025

    Huangyuan Su, Mujin Kwun, Stephanie Gil, Sham Kakade, and Nikhil Anand. Characterization and mitigation of training instabilities in microscaling formats, 2025. URLhttp://arxiv.org/abs/2506. 20752

    2025

  35. [35]

    Roformer: Enhanced transformer with rotary position embedding, 2021

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2021. URLhttp://arxiv.org/abs/2104.09864

    Pith/arXiv arXiv 2021

  36. [36]

    Kimi Team, Yifan Bai, Yiping Bao, Y. Charles, Cheng Chen, Guanduo Chen, Haiting Chen, Huarong Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, Zhuofu Chen, Jialei Cui, Hao Ding, Mengnan Dong, Angang Du, Chenzhuang Du, Dikang Du, Yulun Du, Yu Fan, Yichen Feng, Kelin Fu, Bofei Gao, Chenxiao Gao, Hongcheng Gao, Peizhong Ga...

  37. [37]

    Llama: Open and efficient foundation language models, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023. URL http://arxiv.org/abs/2302.13971v1. 12 One QK Channel, Many So...

    Pith/arXiv arXiv 2023

  38. [38]

    Adagc: Enhancing llm pretraining stability via adaptive gradient clipping, 2025

    Guoxia Wang, Shuai Li, Congliang Chen, Jinle Zeng, Jiabin Yang, Dianhai Yu, Yanjun Ma, and Li Shen. Adagc: Enhancing llm pretraining stability via adaptive gradient clipping, 2025. URLhttp://arxiv. org/abs/2502.11034v3

    Pith/arXiv arXiv 2025

  39. [39]

    Liu, Lechao Xiao, Katie Everett, Alex Alemi, Ben Adlam, John D

    Mitchell Wortsman, Peter J. Liu, Lechao Xiao, Katie Everett, Alex Alemi, Ben Adlam, John D. Co-Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, Jeffrey Pennington, Jascha Sohl-dickstein, Kelvin Xu, Jaehoon Lee,JustinGilmer,andSimonKornblith. Small-scaleproxiesforlarge-scaletransformertraininginstabilities,

  40. [40]

    Stabilizing transformer training by preventing attention entropy collapse,

    Shuangfei Zhai, Tatiana Likhomanenko, Etai Littwin, Dan Busbridge, Jason Ramapuram, Yizhe Zhang, Jiatao Gu, and Josh Susskind. Stabilizing transformer training by preventing attention entropy collapse,

  41. [41]

    Root mean square layer normalization, 2019

    Biao Zhang and Rico Sennrich. Root mean square layer normalization, 2019. URLhttp://arxiv.org/ abs/1910.07467

    Pith/arXiv arXiv 2019

  42. [42]

    collapse

    Peiyuan Zhang, Matthew Noto, Wenxuan Tan, Chengquan Jiang, Will Lin, Wei Zhou, and Hao Zhang. Attn-qat: 4-bit attention with quantization-aware training, 2026. URLhttp://arxiv.org/abs/2603. 00040. 13 One QK Channel, Many SourcesPreprint Technical Appendix The main text states the claims and their essential evidence. Appendices A–D provide the complete exp...

    2026

  43. [45]

    URLhttp://arxiv.org/abs/2309.14322v2

    Pith/arXiv arXiv

  44. [47]

    URLhttp://arxiv.org/abs/2303.06296v2

    Pith/arXiv arXiv

  45. [2017]

    URLhttp://arxiv.org/abs/1710.03740v3

    Pith/arXiv arXiv

  46. [2023]

    URLhttp://arxiv.org/abs/2304.09871v2

    Pith/arXiv arXiv

  47. [2024]

    URLhttp://arxiv.org/abs/2409.12517v2

    Pith/arXiv arXiv

  48. [2025]

    URLhttp://arxiv.org/abs/2507.20534v2

    Pith/arXiv arXiv

  49. [2026]

    URLhttp://arxiv.org/abs/2602.18851v1

    arXiv