Pith. sign in

REVIEW 4 major objections 5 minor 18 references

DNT: a Deeply Normalized Transformer that can be trained by Momentum SGD

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read By placing four normalizations at specific positions, the authors claim, a transformer can be trained by plain momentum SGDW to within a fraction of a point of AdamW accuracy — something the original ViT and GPT2 cannot do.

desk verdict DNT is a plausible architecture-level fix for the mSGDW-vs-AdamW gap, but the parity claim rests on a single run per cell and the Jacobian theory substitutes linear attention for softmax. read the letter →

arxiv 2507.17501 v1 pith:WE3RS6WX submitted 2025-07-23 cs.LG cs.CLcs.CV

Xianbiao Qi , Marco Chen , Wenjie Xiao , Jiaquan Ye , Yelin He , Chun-Guang Li , Zhouchen Lin This is my paper · ORCID
classification cs.LGcs.CLcs.CV
keywords deeplynormalizedtransformermomentumSGDheavy-tailedgradientsJacobianmatrixnormalizationplacementQKNormoptimizationAdamWcomparison
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's aim is to remove the need for adaptive optimizers when training transformers: it claims that the heavy-tailed gradient distributions that make momentum SGDW fail are an architectural artifact, not an inherent property of transformers. The fix is a set of four normalizations placed at specific points — InputNorm right after the embedding, PreNorm before attention and the feed-forward block, MidNorm after each block function before the residual add, and QKNorm on queries and keys — with PostNorm deliberately omitted because it can shrink gradients. Each placement is argued to control one source of spread in the layer Jacobian's singular values, so gradients concentrate and vanilla mSGDW becomes competitive with AdamW: the reported numbers on the 307M vision model are 81.5% versus 82.1% ImageNet accuracy, and on the 124M language model 2.849 versus 2.863 validation loss on OpenWebText; the same optimizer on the original ViT/GPT2 gives 78.2% and 2.906. If this holds, it matters because it suggests architecture design can substitute for optimizer complexity, roughly halving the memory spent on optimizer state.

What carries the argument

The mechanism is the four normalization placements acting on the layer Jacobian, together with a scale-invariance estimate: for a high-dimensional weight matrix $W$ with i.i.d. entries of variance $\sigma_W^2$ and a random activation vector $x$ with entry variance $\sigma_x^2$, the norm concentrates as $\|Wx\|^2 \approx m n \sigma_W^2 \sigma_x^2$ and the largest singular value as $\sigma_1(W) \approx (\sqrt m + \sqrt n)\sigma_W$, so the normalized matrix $W/\|Wx\|$ has singular values governed by the shape of $W$ alone. MidNorm applies this directly to the FFN Jacobian $W_2 \mathrm{diag}(\mathbf{1}(W_1 x > 0)) W_1/\|W_2 \mathrm{ReLU}(W_1 x)\|$; QKNorm, which normalizes the query and key vectors in attention, applies the same idea to $W_q x_i$ and $W_k x_j$ in the attention logit gradients; InputNorm and PreNorm instead control the norms of activations, which the analysis shows otherwise enter the Jacobian multiplicatively. The named object the reader should keep is the normalized Jacobian: the paper argues each normalization is a knob that removes one factor of weight or activation magnitude from the singular-value spread of $\partial y/\partial x$.

What would settle it

Scale every weight matrix in a trained DNT block by a common factor (say 2 or 10) and measure the gradient norm on a fixed batch without re-training: the paper's scale-invariance argument implies the Jacobian, hence the gradients, are nearly unchanged, so a clear scaling of gradient magnitudes with the factor would falsify the random-matrix justification on real trained weights. A cheaper probe is to track the largest-to-smallest singular-value ratio of each block's Jacobian during mSGDW training; the argument predicts it stays bounded for DNT while growing for the standard ViT.

Watch

Extended reading notes

Core claim

The paper claims that the heavy-tailed gradient problem in transformers comes from layer Jacobians whose singular values are stretched by three interacting factors: the magnitude of the network's input, the magnitudes of activations, and the magnitudes of weight matrices — including the joint product $W_q^\top W_k$ inside attention. The proposed DNT architecture assigns one normalization to each factor: InputNorm fixes the scale of the embedding output; PreNorm keeps the activation input to self-attention in a stable range; MidNorm divides the output of each sub-block by its own norm, so that by a high-dimensional random-matrix estimate the Jacobian's singular values depend on weight shapes but not weight magnitudes; and QKNorm replaces the joint $W_q^\top W_k$ gradient term with separate $W_q$ and $W_k$ terms, removing a known source of training instability. PostNorm is excluded because its Jacobian divides by the potentially large norm of the post-residual signal. The paper reports that the resulting architecture trains with vanilla momentum SGDW to within about half a point of AdamW on ImageNet (81.5% versus 82.1% for the 307M model) and to within 0.014 validation loss on OpenWebText (2.849 versus 2.863 for the 124M model), while the original ViT and GPT2 architectures under the same optimizer trail far behind (78.2% and 2.906).

Load-bearing premise

The proof that the normalizations make layer Jacobians independent of weight magnitude assumes that weight matrices and activations are high-dimensional random vectors with independent, zero-mean entries, so that norms and largest singular values follow the random-matrix formulas $E[\|Wx\|^2] = mn\,\sigma_W^2\sigma_x^2$ and $\sigma_1(W)\approx(\sqrt m+\sqrt n)\sigma_W$; trained weights are correlated and structured, so this scaling is not validated for the actual DNT models.

Editorial extensions

If this is right

  • Transformers can be trained with vanilla momentum SGDW to near-AdamW accuracy once the Jacobian is controlled by these normalization positions; the reported gaps are about 0.6 accuracy points on ImageNet and 0.01–0.02 validation loss on OpenWebText.
  • The same DNT architecture also beats its ViT/GPT2 counterparts when both are trained with AdamW (82.1 versus 81.7 on ImageNet 307M; 2.863 versus 2.867 on OpenWebText 124M), so the placements are not merely a crutch for SGD.
  • The result extends to larger scales in the paper's experiments: 632M vision and 1.4B language models, where mSGDW-DNT stays within about 0.7 accuracy points and 0.012 validation loss of AdamW-DNT.
  • Using mSGDW instead of AdamW approximately halves the optimizer state memory (11.5 GB versus 5.7 GB by the paper's accounting for a 1.4B model).
  • The ablation indicates the components are cumulative: adding QKNorm alone barely changes the standard PreNorm baseline, InputNorm helps, and MidNorm with the others yields the best results, so the claim depends on the combination, not any single normalization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable corollary the paper leaves implicit: scaling a trained DNT block's weights by a constant should leave its gradient distribution nearly unchanged, because the Jacobian is claimed to be magnitude-free; measuring this on real trained weights would test whether the random-matrix assumption survives training.
  • The paper's theory does not address what happens as weights become correlated and structured during training; if the normalization's protection weakens there, larger models or longer schedules than the ones tested could still require an adaptive optimizer.
  • Because the fix is architectural, a natural untested extension is to combine DNT with memory-lean optimizers other than mSGDW, potentially stacking memory savings beyond the roughly 2x reported for the optimizer state.
  • The paper compares against AdamW only; if the concentrated-gradient mechanism is real, DNT should also narrow the gap to second-order or sign-based optimizers on the same setups, which is a direct experiment the paper does not run.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a Deeply Normalized Transformer (DNT) with a specific arrangement of normalization layers—InputNorm, PreNorm, MidNorm, QKNorm, and no PostNorm—and claims that this architecture can be trained with vanilla momentum SGDW (mSGDW) to match the performance of AdamW-trained Transformers. The central empirical evidence is Table 1, which reports ImageNet accuracy and OpenWebText validation loss for ViT/GPT2 baselines and V-DNT/L-DNT variants under both optimizers at several scales; the key comparisons are mSGDW L-DNT-Small val loss 2.849 vs. AdamW L-DNT-Small 2.863 and mSGDW V-DNT-Large accuracy 81.5% vs. AdamW V-DNT-Large 82.1%. The paper also provides Jacobian-based theoretical justifications for each normalization, ablation studies over five configurations, and a GPU-memory comparison.

Significance. If the empirical claim survives replication, this is a significant architecture-level result: it would show that normalization placement alone can remove the dependence of Transformer training on adaptive optimizers, with a meaningful reduction in optimizer memory (roughly 5.7 GB for a 1.4B model, Table 2). The paper is commendably falsifiable: the main comparison is quantified, and the ablation S1–S5 provides an external, component-level test of the design rather than a single end-to-end claim. However, the theoretical apparatus is currently a post hoc explanation built on assumptions that are not validated on trained models, and the central empirical claim lacks the statistical support needed to distinguish parity from seed noise.

major comments (4)
  1. [Section 4.2 / Table 1] The central claim that mSGDW matches AdamW on DNT relies on single-run numbers with no seeds, error bars, or released code. The decisive margins are small: OpenWebText val-loss deltas between mSGDW-DNT and AdamW-DNT are at most 0.022, and ImageNet accuracy deltas are 0.6–0.7 points. At these scales, seed noise alone can reverse the ordering; indeed, mSGDW L-DNT-Small (2.849) is reported as better than AdamW L-DNT-Small (2.863), while mSGDW V-DNT-Large is worse than its AdamW counterpart by 0.6 points. Without repeated runs or a reproducibility artifact, the 'comparable to AdamW' conclusion is not statistically established.
  2. [Appendix A.1 / Proposition 2] Proposition 2 asserts that for X' = g ⊙ X, with the same weights, ∂vec(Y)/∂vec(X) = ∂vec(Y')/∂vec(X') for self-attention. The proof in Appendix A.1 does not establish this: it replaces softmax attention by linear attention, and it concludes only that Y = Y' because PreNorm maps both inputs to the same normalized input. For a nonlinear PreNorm, the Jacobian with respect to the pre-normalization variables includes the chain-rule factor ∂PreNorm(X)/∂X, so the stated Jacobian equality is generally false. Since this proposition is the stated basis for the PreNorm design in Section 3.2.2, this is a load-bearing gap.
  3. [Appendix A.2 / Propositions 3 and 5] The proof of Proposition 3 uses random-matrix formulas for a single matrix W with i.i.d. zero-mean entries: E||Wx||² = mnσ_W²σ_x² and σ₁(W) ≈ (√m+√n)σ_W. Trained DNT weights and activations are strongly correlated and structured, so these scalings do not apply to actual checkpoints. Moreover, the object in Proposition 3 is the product W₂ diag(1(W₁x>0))W₁ divided by ||W₂ReLU(W₁x)||, but the proof analyzes only a single W/||Wx|| and never handles the product structure or the ReLU mask. Proposition 5 then inherits this unvalidated reasoning. The claimed independence of Jacobian spectra from weight magnitudes is therefore not established for the networks actually trained.
  4. [Section 3.1 / Eq. (1)] The paper asserts that the heavy-tailed gradient distribution is caused by 'diversity of the singular values in the Jacobian matrix' and then uses this as the design principle for the normalizations. This is presented as an axiom rather than a demonstrated fact; Eq. (1) only expresses the chain rule and does not by itself connect Jacobian singular-value diversity to heavy-tailed gradients. Since the theoretical propositions are the stated reason for the architecture, the paper should either validate this mechanism empirically (e.g., by measuring Jacobian spectra of DNT checkpoints) or explicitly reframe the Jacobian analysis as motivation rather than proof.
minor comments (5)
  1. [Throughout] There are numerous typographical errors, including 'Propostion', 'Setttings', 'elliviate', and 'mSGDW is not directly to add' in Appendix B; these should be corrected.
  2. [Section 3.2.1 / Figure 2] The text refers to 'Figure 2 (d)' for InputNorm, but the panel labels in Figure 2 are (A)–(E), and InputNorm is panel (A); PostNorm is also described with inconsistent panel references. Please align the text with the figure.
  3. [Appendix A.1 / Eq. after A.1] The proof contains an unresolved reference 'according to Equation ??', and the transition from the linear-attention Jacobian to the claimed softmax-attention result is missing; this should be rewritten.
  4. [Section 3.2.2] Proposition 2 defines X' = g ⊙ X without specifying whether g is a scalar, a vector, or per-token; this makes the statement ambiguous and should be clarified.
  5. [Appendix B / Tables 4–5] The hyperparameter tables list different peak learning rates for GPT2 baselines (6e-4/2.5e-4/1.5e-4) while the text says 'we use 6e-4 for all our L-DNT models'; please clarify which values are used for each baseline row to avoid apparent inconsistencies.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the DNT results rest on external benchmarks and ablations, and the theory propositions are conditional mathematical statements rather than fitted predictions.

full rationale

The paper's central claim is empirical: a specific combination of normalizations (InputNorm, PreNorm, MidNorm, QKNorm) allows mSGDW to reach parity with AdamW on ImageNet and OpenWebText. This is supported by external benchmarks and by an ablation study (S1-S5) run under mSGDW, so no target quantity is fitted and then relabeled as a prediction. The theoretical propositions derive properties of normalized Jacobians under explicit random-matrix and high-dimensional assumptions; they are conditional derivations, not restatements of the empirical result. Equation (5) is attributed to the authors' prior work (Qi et al., 2025a), but the formula is also restated in the paper and is a standard vectorization calculation, not an unverified premise that forces the conclusion; the passing self-citations on spectral energy concentration and normalization surveys are background rather than load-bearing. That the architecture was motivated by observed gradient tails, or that theory followed design, is post hoc explanation rather than circularity. Correctness and evidence-quality concerns—single-seed Table 1, i.i.d.-random-matrix assumptions not validated on trained checkpoints, and the broken internal reference 'according to Equation ??' in the Appendix A.1 proof of Proposition 2—should be weighed as validity issues, not circularity. No load-bearing reduction to the paper's own inputs was found.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central empirical claim rests on standard deep learning hyperparameters (learning rates, weight decays) and on the empirical effectiveness of the five normalization placements. The theoretical scaffolding relies on random-matrix assumptions about trained weights that are not validated.

free parameters (2)
  • mSGDW peak learning rate = 1.0 (L-DNT), 0.5 (V-DNT-Large), 0.1 (V-DNT-Huge)
    Chosen by rough grid search per model; the success of the central mSGDW comparison depends on these values.
  • mSGDW weight decay = 1e-4 (L-DNT), 2e-4 (V-DNT-Large), 1e-3 (V-DNT-Huge)
    Also chosen by rough grid search; coupled with learning rate and affects the final performance.
assumptions (4)
  • standard math Vershynin concentration and almost-orthogonality theorems (Theorems 1 and 2)
    Used to derive norms of residual streams and the behavior of normalized vectors in high dimension.
  • domain assumption Trained weight matrices and activations behave like high-dimensional random vectors with i.i.d. zero-mean entries
    Appendix A.2 derives the MidNorm scaling using E[||Wx||^2] and sigma_1(W) approximately (sqrt(m)+sqrt(n)) sigma_W; this may fail for correlated trained weights.
  • domain assumption Heavy-tailed gradient distribution is the root cause of SGD failure on transformers
    Adopted from Zhang et al. 2020 and Simsekli et al. 2019; motivates the entire architecture design.
  • ad hoc to paper Gradient heavy-tailedness is caused by diversity of Jacobian singular values
    The paper's own hypothesis in Section 3.1, not proven; links weight and activation norms to gradient tails.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DNT: a Deeply Normalized Transformer that can be trained by Momentum SGD." pith.science (2026). https://pith.science/paper/WE3RS6WX

@misc{pith2026250717501,
  author       = {Pith},
  title        = {Pith review of: DNT: a Deeply Normalized Transformer that can be trained by Momentum SGD},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WE3RS6WX}},
  note         = {Machine review of arXiv:2507.17501}
}
read the original abstract

Transformers have become the de facto backbone of modern deep learning, yet their training typically demands an advanced optimizer with adaptive learning rate like AdamW, rather than a momentum SGDW (mSGDW). Previous works show that it is mainly due to a heavy-tailed distribution of the gradients. In this paper, we introduce a Deeply Normalized Transformer (DNT), which is meticulously engineered to overcome this limitation enabling seamless training with vanilla mSGDW while yielding comparable performance to the Transformers trained via AdamW. To be specific, in DNT, we strategically integrate normalization techniques at proper positions in the Transformers to effectively modulate the Jacobian matrices of each layer, balance the influence of weights, activations, and their interactions, and thus enable the distributions of gradients concentrated. We provide both theoretical justifications of the normalization technique used in our DNT and extensive empirical evaluation on two popular Transformer architectures to validate that: a) DNT outperforms its counterparts (\ie, ViT and GPT), and b) DNT can be effectively trained with vanilla mSGDW.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 6 canonical work pages

  1. [1]

    Layer normalization

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450,

    arXiv

  2. [5]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980,

    arXiv

  3. [6]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437,

    arXiv

  4. [10]

    A survey of optimization methods for training dl models: Theoretical perspective on convergence and generalization

    Jing Wang and Anna Choromanska. A survey of optimization methods for training dl models: Theoretical perspective on convergence and generalization. arXiv preprint arXiv:2501.14458,

    arXiv

  5. [11]

    Large batch training of convolutional networks

    13 DNT: a Deeply Normalized Transformer that can be trained by Momentum SGD Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv:1708.03888,

    arXiv

  6. [13]

    Transformers without normalization

    Jiachen Zhu, Xinlei Chen, Kaiming He, Yann LeCun, and Zhuang Liu. Transformers without normalization. arXiv preprint arXiv:2503.10622,

    arXiv

  7. [14]

    In a backpropagation, since we have obtained ∂L ∂vec(Y ), we would like to further analyze ∂L ∂vec(Wq) , ∂L ∂vec(Wk) , ∂L ∂vec(Wv)

    Furthermore, we would like to conduct a deeper analysis of the gradient of the loss with respect to the weights. In a backpropagation, since we have obtained ∂L ∂vec(Y ), we would like to further analyze ∂L ∂vec(Wq) , ∂L ∂vec(Wk) , ∂L ∂vec(Wv). For the weight matrix Wq, we have ∂L ∂vec(Wq) = ∂L ∂vec(Y ) ∂vec(Y ) ∂vec(A) ∂vec(A) ∂vec(P ) ∂vec(P ) ∂vec(Wq) ...

    work page 2012

  8. [18]

    Défossez et al

    provides a memory-efficient 23 DNT: a Deeply Normalized Transformer that can be trained by Momentum SGD adaptive method. Défossez et al. provides a unified formulation for adaptive methods like AdaGrad, Adam, and AdaDelta. The field continues to evolve with recent innovations including MUON (Jordan et al., 2024), LION (Chen et al., 2024), Sophia (Liu et a...

    work page 2025

Show all 18 references
  1. [1983]

    This method augments the gradient direction with a fraction of the update vector from the previous step, allowing faster convergence and helping escape local minima

    addresses the limitations of vanilla SGD by accelerating gradient descent in relevant directions while dampening oscillations. This method augments the gradient direction with a fraction of the update vector from the previous step, allowing faster convergence and helping escap...

    2018

  2. [2012]

    Qwen technical report

    Qwen Team. Qwen technical report. arXiv preprint arXiv:2309.16609,

    arXiv

  3. [2017]

    Mars: Unleashing the power of variance reduction for training large models

    Huizhuo Yuan, Yifeng Liu, Shuang Wu, Xun Zhou, and Quanquan Gu. Mars: Unleashing the power of variance reduction for training large models. arXiv preprint arXiv:2411.10438,

    arXiv

  4. [2018]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901,

    1901

  5. [2019]

    All language models were trained on OpenWebText, using GPT-2 tokenizer

    with bfloat16 precision GPUs, employing a cosine learning rate schedule. All language models were trained on OpenWebText, using GPT-2 tokenizer. The training dataset contains 9B tokens, with a validation set of 4.4M tokens, following the train-validation split from nanoGPT. We...

    2000

  6. [2020]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

  7. [2022]

    Query-key normalization for transformers

    Alex Henry, Prudhvi Raj Dachapally, Shubham Shantaram Pawar, and Yuxuan Chen. Query-key normalization for transformers. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 4246–4253,

    2020

  8. [2023]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288,

    arXiv

  9. [2024]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892–34916, 2023a. Hong Liu, Zhiyuan Li, David Hall, Percy Liang, and Tengyu Ma. Sophia: A scalable stochastic second-order optimizer for ...

    arXiv

  10. [2242]

    optimizer AdamW mSGDW AdamW mSGDW learning rate schedule cosine decay peak learning rate 1e-3 0.5/0.1 1e-3 0.5/0.1 minimum learning rate 1e-8 1e-8 1e-8 1e-8 weight decay 0.1 2e-4/1e-3 0.1 2e-4/1e-3 optimizer momentum β1, β2 = 0.9, 0.99 µ = 0.9 β1, β2 = 0.9, 0.99 µ = 0.9 warmup...

    2000

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.