Why Post-Norm Transformers Collapse: Attention Amplification and Gradient Repair Failure

Qingyu Han; Xiaodong Luo; Xingjian Wang; Yin Zhang

REVIEW 3 major objections 5 minor 59 references

Why Post-Norm Transformers Collapse: Attention Amplification and Gradient Repair Failure

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Post-Norm transformer collapse is driven by two linked mechanisms: causal attention raises token similarity at initialization, and RMSNorm then shrinks gradients to earlier layers so training cannot repair the growing collapse.

desk verdict Real two-mechanism account, but the causal claim outruns the evidence; the sharp transition remains unexplained and the paper should frame the mechanisms as sustaining collapse rather than causing it. read the letter →

arxiv 2608.09417 v2 pith:XHQI52IP submitted 2026-08-10 cs.LG

Xingjian Wang , Qingyu Han , Xiaodong Luo , Yin Zhang This is my paper · ORCID

classification cs.LG

keywords Post-NormrankcollapsetokensimilaritycausalattentionprefixaveragingRMSNormgradientvanishingfrequencydistribution

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that Post-Norm rank collapse is not a single failure but a two-stage process. At initialization, causal attention acts approximately as a prefix-averager, pushing token representations toward identical across depth, while the SwiGLU feed-forward branch only slightly damps this effect. Once representations become highly similar, growing pre-normalization residual norms make the RMSNorm backward factor contractive, so gradients reaching earlier layers decay geometrically and cannot restore token diversity. The paper also characterizes the collapsed state: its best possible output is the frequency distribution, which gives a relatively high loss floor, and parameter gradients vanish in collapsed layers. Together these mechanisms explain why Post-Norm decoders are more collapse-prone than Pre-Norm and why warmup and learning-rate tuning are needed to avoid collapse.

What carries the argument

The central observable is token similarity, $\mathrm{tsim}(X)=\|\Pi_1 X\|_F^2/\|X\|_F^2$, a scalar between $0$ and $1$ that measures how close token representations are to being all identical. The forward derivation relies on two approximations: replacing the random causal attention matrix by the prefix-averaging matrix $C_n$ with $(C_n)_{i,j}=1/i$ for $i\ge j$, and replacing the token Gram matrix by the equal-correlation closure $R_{\mathrm{eq}}(t_1)=d[(1-\rho(t_1))I+n\rho(t_1)\Pi_1]$ with $\rho(t_1)=(nt_1-1)/(n-1)$. These yield closed-form one-step formulas for attention and SwiGLU similarity changes. The backward mechanism is carried by the per-layer gradient contraction factor $c(y,\alpha)=\alpha\sqrt{d}/\|y\|_2$, which multiplies the RMSNorm Jacobian prefactor with the measured sublayer gradient contribution $\alpha$; when this factor is below one, each layer shrinks the gradient norm on its way to earlier layers.

What would settle it

Train the same 48-layer Post-Norm model at learning rate $8\times10^{-4}$ with the causal attention matrix replaced by $P-C_n$ (the intervention in Corollary 3.3 with $\alpha=1$); if token similarity still rises or the run still collapses near the observed transition step, then attention amplification is not the necessary driver of collapse.

Watch

Extended reading notes

Core claim

The central claim is that Post-Norm rank collapse has a forward cause and a backward cause. Forward, under equal-correlation and prefix-averaging approximations, one attention sublayer increases token similarity by $\Delta_{\mathrm{attn}}(s,t_1)=s f_1(t_1)/(1+s f_2(t_1))>0$ for $t_1\in[1/n,1)$, where $t_1$ is the token similarity and $s=nd^2\sigma_W^2/\|X_1\|_F^2$ measures the amount of attention. Since $s$ stays constant across Post-Norm layers but shrinks with depth in Pre-Norm, Post-Norm experiences larger similarity growth at initialization. Backward, at exact collapse the per-layer gradient bound is $\|\partial L/\partial X_l^k\|_F \le c(y,\alpha)\|\partial L/\partial X_{k+1}\|_F$ with $c(y,\alpha)=\alpha\sqrt{d}/\|y\|_2$; when $c<1$, gradients to earlier layers decay geometrically, and residual-norm growth under collapse is shown to push $c$ below one. A collapsed network can do no better than predicting the frequency distribution of labels, giving loss floor $H(p_{\mathrm{freq}})$, and its parameter gradients vanish in all collapsed layers, making the state nearly stationary.

Load-bearing premise

The analysis assumes that how similar tokens are to each other at a layer can be captured by a single average number; if the real pattern of pairwise similarities is more complex, the predicted growth and contraction curves may not hold.

Editorial extensions

If this is right

Post-Norm decoders start training with higher token similarity than Pre-Norm at the same depth, explaining their greater sensitivity to warmup and learning rate at initialization.
Removing or subtracting the prefix-averaging component of causal attention suppresses the initialization-time similarity increase, so attention-matrix deescalation is a direct intervention against the first stage of collapse.
During collapse, the per-layer contraction factor $c(y,\alpha)$ drops below one in most layers, and early-layer gradient norms fall by orders of magnitude, making the high-similarity state difficult to escape.
Collapsed Post-Norm runs settle near the frequency loss, so the gap between training loss and frequency loss serves as a collapse detector.
Warmup does not prevent collapse: warmup runs at higher learning rates still collapse and stay near the frequency-loss curve after the transition window.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If token similarity is truly governed by a one-parameter equal-correlation closure, then interventions that directly penalize the trace of $\Pi_1 X X^\top$ or encourage Gram-matrix spread should be effective at delaying collapse, a testable design extension.
The two-stage account implies that keeping residual norms small specifically before RMSNorm, rather than merely using Pre-Norm, should preserve backward gradients; this predicts that residual-scaling methods could substitute for normalization placement.
Because collapse is characterized by the frequency loss, comparing the training loss to the frequency loss of the current batch in real time could serve as a cheap early-warning signal for the onset of collapse in production training runs.
The paper does not explain what triggers the sharp transition at a specific optimizer step; a plausible extension is to model the transition as a saddle point or loss-landscape instability when token similarity crosses a threshold.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's my read. The paper does something real: it gives closed-form formulas for how causal attention amplifies token similarity at initialization in Post-Norm decoders with SwiGLU (Theorems 3.2 and 3.4), a backward contraction bound for RMSNorm (Theorem 3.6), and a frequency-loss characterization of collapsed states (Theorem 3.8). The experiments on a 48-layer Llama-style model broadly support the forward prediction, and the prefix-averaging removal intervention is a clean direct test. The paper is also refreshingly honest about its approximations.

The soft spot is the causal claim. The introduction says collapse is 'not caused by a single instability, but by the combination of two mechanisms,' and the abstract frames the two stages as explaining collapse. But the paper's own Section 5 concedes it does not explain what causes the sharp rise in similarity and loss at a particular optimizer step. The stress-test note is decisive on ordering: the stable Post-Norm run at LR=6e-4 has the same initialization-time forward amplification yet does not collapse, so Stage I is not sufficient to trigger the transition. Stage II's contraction factor c<1 is shown to cross below 1 only shortly after the transition begins (Figure 4), placing it contemporaneous with or posterior to onset. What the evidence supports is: forward amplification pushes similarity upward at initialization in all runs; backward contraction accompanies collapse and likely makes it hard to reverse once it starts. That is a two-mechanism account of amplification and repair failure, not an account of what triggers the transition. The language should be scaled back.

The other approximations are in proportion. The equal-correlation closure (Assumption 3.1) is a real ansatz, and the quantitative match in Figure 1 shows a visible mismatch, though the sign and scale are correct. The prefix-averaging approximation is validated at about 2% error (Figure 9), so that part is solid. The backward theorem uses measured alpha_k and norms, making it an empirical diagnosis rather than a predictive theory, but the paper uses it fairly.

I'd send this to peer review. With the causal claims softened to 'amplification and sustained repair failure' and the open transition question acknowledged (as Section 5 partly already does), it would be a solid contribution. I'd bring it to a reading group and cite the closed-form results. Recommendation: revise the causal framing, then accept.

Referee Report

3 major / 5 minor

Summary. This paper proposes a two-stage account of Post-Norm Transformer rank collapse. In Stage I (Section 3.1, Theorem 3.2), the authors show that at initialization the causal attention matrix acts approximately as a prefix-averaging operator that increases token similarity at every layer, with a closed-form one-step increment Delta_attn(s,t1) = s f1(t1) / (1 + s f2(t1)), while the SwiGLU branch supplies only a small damping effect (Theorem 3.4). In Stage II (Section 3.2, Theorem 3.6), they prove that once the representation is close to collapse, growing pre-normalization residual norms make the RMSNorm backward factor contractive, giving a per-layer gradient contraction factor c(y,alpha) = alpha * sqrt(d) / ||y||_2, so gradients reaching earlier layers decay geometrically when c < 1. Section 3.3 characterizes collapsed networks: the best achievable predictor is the empirical frequency distribution of the labels (Theorem 3.8(i)), and parameter gradients vanish in collapsed layers (Theorem 3.8(ii)). Experiments on 48-layer decoder-only Transformers trained on C4 measure the initialization-time similarity growth, the transition-window contraction of c(y,alpha), the per-layer gradient drop, and the loss floor near the frequency loss. Section 5 explicitly concedes that the paper does not explain what causes the sharp rise in last-layer token similarity and training loss at a particular optimizer step.

Significance. If the results hold, the paper delivers a useful mechanistic decomposition of Post-Norm collapse: a parameter-free forward formula for initialization-time similarity amplification, a measurable backward contraction criterion, and a clean characterization of the collapse loss floor. The strengths are real: the forward formula is derived rather than fitted, the intervention in Corollary 3.3 (subtracting alpha*C_n suppresses similarity growth) is a genuinely falsifiable test and the experiment confirms it, the backward theorem is reduced to measurable quantities (alpha^k_l, ||y||_2), the appendix proofs are detailed, and the limitation statement in Section 5 is candid. The principal weakness is that the causal framing of the Introduction is not supported by the evidence, which establishes only that the two mechanisms accompany and sustain high-similarity states; the paper's own limitation statement concedes the trigger is unexplained. In addition, the quantitative forward predictions rest on the equal-correlation closure and show a visible mismatch with measurements (Figure 1). With the causal claims rescaled, this is a publishable and valuable contribution.

major comments (3)

[§1 vs. §5; §4.2; App. F.5] The Introduction asserts that 'Post-Norm rank collapse is not caused by a single instability, but by the combination of two mechanisms' (§1), yet Section 5 explicitly concedes that the paper 'does not explain what causes the sharp rise in last-layer token similarity and training loss at a particular optimizer step.' The evidence supports a weaker, non-causal claim: Stage I is an initialization-time property that also holds in the stable Post-Norm run at LR=6e-4 (Appendix F.5), which never collapses, so forward amplification alone cannot trigger collapse; and Stage II is measured only 'shortly after the transition begins' (§4.2, Figure 4), placing the RMSNorm contraction contemporaneous with or posterior to the transition trigger. Because both stages are present in conditions without collapse or appear only after the transition has begun, the causal form of the central claim is not established. Please reframe the central contribution as a partial, mechanistic account of the mechanisms that push toward and sustain high-similarity states (as the Abstract and Conclusion already mostly do), or provide ordering or intervention evidence that would support the causal reading.
[Assumption 3.1; Fig. 1; §4.1] The closed-form Stage I predictions (Theorems 3.2, 3.4, and C.3) rest on the equal-correlation closure, which replaces the entire token Gram matrix by the one-parameter surrogate R_eq(t1). The paper's own Figure 1 reports a 'visible mismatch' between measured and theoretical increments, so the quantitative accuracy of the closure is limited. Because the signs and scales of the increments are the load-bearing content of the qualitative amplification claim, the paper should either quantify the sensitivity of Delta_attn to deviations from equal correlation, or state explicitly that the closed-form curves are schematic and rely on the surrogate. The prefix-averaging approximation is well validated on its own (Figure 9, roughly 2% relative error), which isolates the equal-correlation closure as the fragile premise of the forward analysis.
[Theorem 3.8(i); §4.3; Fig. 6] The theoretical support for the frequency-loss floor is weaker than the text suggests. The bound in Theorem 3.8(i) is L_CE >= L_freq(y) - 2*sqrt(n)*||W_lm||_2 * epsilon; with n = 2048 this gap is enormous unless epsilon is extremely small, so the theorem has limited quantitative predictive power in the near-collapse regime. The experimental claim that collapsed runs 'stay near' the frequency loss (Figure 6) is therefore primarily an empirical finding, and the comparison has a post-hoc degree of freedom: the aggregated-label reference is chosen because it 'matches more closely' (§4.3). Please state these caveats and temper the claim that Theorem 3.8 predicts the observed loss floor.

minor comments (5)

[§2, §3.1, Def. 3.5, Lemma C.5] The symbol alpha is overloaded: it denotes the de-escalation coefficient in P - alpha*C_n (Corollary 3.3), the sublayer gradient contribution factor alpha^k_l (Definition 3.5), and a row-norm parameter in Lemma C.5. Please rename at least one of these to avoid confusion in Section 4.2, where both meanings appear.
[Fig. 1 caption vs. §4.1 text] The Figure 1 caption states that the increment 'is tracked well by Theorem 3.2,' while the text in §4.1 says 'a visible mismatch remains'; please reconcile these descriptions so the reader knows how much quantitative accuracy is being claimed.
[App. F.4; main-text typos] The plot titles in Appendix F.4 read 'sublayer ambplification factor' instead of 'amplification'; additionally, several inline formulas in the main text (e.g., norm and trace expressions in §3.1 and §3.2) are typeset inconsistently with the appendix and should be cleaned in the final version.
[§2; App. E.3] The claim that the stated results 'directly extend to multi-head cases' is asserted without proof in the main text; since Appendix E.3 does handle multi-head attention, please add an explicit pointer when the extension claim is made in Section 2.
[§4.3] The choice to compare against the aggregated-label frequency loss rather than the per-sequence average is justified by which curve 'matches more closely'; please state this explicitly as an empirical modeling choice with its own caveats rather than a consequence of Theorem 3.8.

Circularity Check

0 steps flagged · score 2.0 of 10

No load-bearing circularity: the two-stage theorems are derived from initialization assumptions and measured gradient quantities, and the only self-citation is attributional.

full rationale

Section 3.1's Theorem 3.2 is derived from Assumption 3.1, the prefix-averaging surrogate C_n, and the ratio-of-expectations approximation; its inputs are initialization constants (s, t1) and the closed-form functions f1/f2, not the collapse trajectories being explained. Figure 1 reports a "visible mismatch" between measured and theoretical increments, which is the opposite of a fitted prediction. The backward bound in Theorem 3.6 is a norm inequality: with Definition 3.5, c(y,α)=α√d/||y||2 is just the RMSNorm Jacobian bound combined with the measured sublayer ratio; the paper then measures α, y-norms, and c during a collapsing run and observes c<1 "shortly after the transition begins". That ordering is a causal-evidence limitation, acknowledged in Section 5 ("it does not explain what causes the sharp rise in last-layer token similarity and training loss at a particular optimizer step"), but it is not a circular reduction because the contraction factor is not fitted to force the collapse. The frequency-loss result in Theorem 3.8 is a consequence of constant-row softmax optima and is compared to loss curves using label counts independent of model parameters. The only overlapping-author citation is Yu & Zhang (2026), whose author Yin Zhang is a co-author of this paper; it is used for the token-similarity observable and for prior encoder initialization analysis, but the paper restates tsim in Definition 2.1 and proves its own formulas, so the self-citation is attributional and not load-bearing. Because one minor self-citation exists but is not load-bearing, the score is 2 rather than 0.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central theory is built on explicit modeling approximations rather than fitted parameters or new entities. The main structural assumptions are the equal-correlation closure and the prefix-averaging approximation; these are stated as assumptions and partially validated empirically. No free parameters are fitted to the collapse data, and no new entities are introduced.

free parameters (1)

Sublayer gradient contribution α_l_k = measured per layer; below 2.5 in the collapsing run (Figure 3)
Appears in the backward contraction factor c(y,α) = α√d/‖y‖2 in Theorem 3.6. The theory does not predict α; it is measured from the training run. The contraction conclusion depends on α being small enough relative to the residual norm, so the empirical value is a free input to the backward analysis.

assumptions (4)

domain assumption Equal-correlation closure (Assumption 3.1, Equation 4): the token Gram matrix is replaced by the one-parameter surrogate R_eq(t1) = d[(1-ρ(t1))I + nρ(t1)Π1].
This ansatz fixes the entire distribution of pairwise token correlations by the scalar t1. It is used in Theorems 3.2, 3.4, C.3, and Lemma C.5. The paper acknowledges a visible mismatch with experiments, so the closure is only approximately valid.
domain assumption Prefix-averaging approximation (Assumption C.2): the causal attention matrix P is replaced by the uniform prefix-averaging matrix C_n with (C_n)_{i,j} = 1/i for i ≥ j.
This is validated at initialization with about 2% relative error (Figure 9), but attention during training is not uniform, so the approximation is an initialization-time surrogate.
domain assumption Ratio-of-expectations surrogate (Assumption C.1(iv)): E[tsim(Y)] ≈ E[‖Π1Y‖_F^2] / E[‖Y‖_F^2].
Used to turn the expectation of a ratio into a ratio of expectations. The appendix gives concentration bounds (Appendix C.7) but the substitution remains an approximation, not an identity.
domain assumption Exact collapse condition for the backward theorem (Theorem 3.6): tsim(Y_k) = tsim(X_k) = 1, extended to a near-collapse neighborhood with δ ≤ 1/(4n+1) (Proposition D.1).
The gradient-contraction bound is proven at exact collapse and for very small δ. Real collapsed runs are only approximately rank-one, so the theorem applies only near the collapsed state.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Why Post-Norm Transformers Collapse: Attention Amplification and Gradient Repair Failure." pith.science (2026). https://pith.science/paper/XHQI52IP

@misc{pith2026260809417,
  author       = {Pith},
  title        = {Pith review of: Why Post-Norm Transformers Collapse: Attention Amplification and Gradient Repair Failure},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XHQI52IP}},
  note         = {Machine review of arXiv:2608.09417}
}

read the original abstract

Deep decoder-only Transformers often replace the original Post-Norm architecture with Pre-Norm variants because Post-Norm training is highly sensitive to warmup and learning rate under conventional initialization schemes. Although prior work has identified rank collapse and gradient vanishing as related symptoms, it remains poorly understood how causal attention creates high-similarity representations and why training dynamics fail to repair them. We give a two-stage analysis of Post-Norm rank collapse using token similarity as a scalar state variable. First, at initialization, causal attention acts approximately as a prefix-averaging operator that increases token similarity across depth, while the SwiGLU branch contributes only a smaller damping effect. Second, once training enters a high-similarity regime, growth of pre-normalization residual norms makes the RMSNorm backward factor contractive; under mild conditions, gradients to earlier layers decay geometrically. As a complementary result, we characterize the properties of a collapsed network: its best predictor is frequency distribution with relatively high loss floor, and gradients in collapsed layers vanish at frequency distribution. Experiments on 48-layer decoder-only Transformers trained on C4 dataset match the predicted initialization-time similarity growth and collapse-time gradient contraction, and show that collapsed runs stay near the predicted frequency loss. Together, these results distinguish the forward similarity amplification and backward repair incapacity in Post-Norm collapse, while also characterizing the behavior of collapsed networks.

Figures

Figures reproduced from arXiv: 2608.09417 by the authors.

**Figure 2.** Removing PrefixAveraging Component suppresses initialization-time token-similarity growth. Removing Prefix-Averaging Component. Curve matching alone does not identify which part of attention drives the effect. To test this more directly, we subtract the prefix-averaging component αCn from the attention matrix at initialization, where Cn is the causal prefixaverage matrix with (Cn)i,j = 1/i for i ≥ j, and 0 otherwi… view at source ↗

**Figure 3.** Sublayer gradient contribution in the collapsing Post-Norm run at LR= [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗

Figures from the paper (19 more)

**Figure 4.** Figure 4: √ d/∥y l k ∥2 and c(y l k , αl k ) in the transition window of the Post-Norm run at LR=8 × 10−4 . Dashed vertical line: transition step √ ≈ 2644; blue/red shading: pre/post-transition windows. (a) Attention d/∥y l 1∥2. (b) Attention c(y l 1 , αl 1 ). (c) FFN √ d/∥y l 2…

**Figure 5.** Figure 5: Each curve covers optimizer steps 2640–2650. Gradients to early layers drop sharply, while later [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]

**Figure 6.** Figure 6: Training loss compared with frequency-loss curves. Left: in the no-warmup setting, real training, [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]

**Figure 7.** Figure 7: m(ρ) under standard initialization. The important asymmetry is that the denominator trace depends only on diagonal terms, whereas the equal-correlation closure only enters through the off-diagonal contribution to the projected numerator trace. Substituting these traces…

**Figure 8.** Figure 8: Layerwise validation of the unconditional proxy [PITH_FULL_IMAGE:figures/full_fig_p041_8.png]

**Figure 9.** Figure 9: Initialization-time validation of the prefix-averaging approximation. [PITH_FULL_IMAGE:figures/full_fig_p042_9.png]

**Figure 10.** Figure 10: Sublayer gradient contribution in another collapsing Post-Norm run. Dashed vertical line: tran [PITH_FULL_IMAGE:figures/full_fig_p042_10.png]

**Figure 11.** Figure 11: Transition-window diagnostic for another collapsing run. Dashed vertical line: transition step [PITH_FULL_IMAGE:figures/full_fig_p043_11.png]

**Figure 12.** Figure 12: Sublayer gradient contribution for the collapsed Post-Norm run with LR= [PITH_FULL_IMAGE:figures/full_fig_p043_12.png]

**Figure 13.** Figure 13: Backward-side diagnostic for the collapsed Post-Norm run with LR= [PITH_FULL_IMAGE:figures/full_fig_p044_13.png]

**Figure 14.** Figure 14: Per-layer gradient norm for the collapsed Post-Norm run with LR= [PITH_FULL_IMAGE:figures/full_fig_p044_14.png]

**Figure 15.** Figure 15: Sublayer gradient contribution for the collapsed Post-Norm run with LR= [PITH_FULL_IMAGE:figures/full_fig_p045_15.png]

**Figure 16.** Figure 16: Backward-side diagnostic for the collapsed Post-Norm run with LR= [PITH_FULL_IMAGE:figures/full_fig_p045_16.png]

**Figure 17.** Figure 17: Per-layer gradient norm for the collapsed Post-Norm run with LR= [PITH_FULL_IMAGE:figures/full_fig_p046_17.png]

**Figure 18.** Figure 18: Sublayer gradient contribution for the collapsed Post-Norm run with LR= [PITH_FULL_IMAGE:figures/full_fig_p046_18.png]

**Figure 19.** Figure 19: Backward-side diagnostic for the collapsed Post-Norm run with LR= [PITH_FULL_IMAGE:figures/full_fig_p047_19.png]

**Figure 20.** Figure 20: Per-layer gradient norm for the collapsed Post-Norm run with LR= [PITH_FULL_IMAGE:figures/full_fig_p047_20.png]

**Figure 21.** Figure 21: Sublayer gradient contribution in a non-collapsed Post-Norm run. [PITH_FULL_IMAGE:figures/full_fig_p048_21.png]

**Figure 22.** Figure 22: Backward-side diagnostic for the same non-collapsed Post-Norm run. [PITH_FULL_IMAGE:figures/full_fig_p048_22.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

59 extracted references · 28 linked inside Pith

[1]

International Conference on Learning Representations , volume=

Efficient streaming language models with attention sinks , author=. International Conference on Learning Representations , volume=
[2]

Advances in Neural Information Processing Systems , volume=

Exponential expressivity in deep neural networks through transient chaos , author=. Advances in Neural Information Processing Systems , volume=
[3]

arXiv preprint arXiv:1611.01232 , year=

Deep information propagation , author=. arXiv preprint arXiv:1611.01232 , year=

arXiv
[4]

Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics , pages=

Understanding the difficulty of training deep feedforward neural networks , author=. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics , pages=. 2010 , organization=

2010
[5]

Proceedings of the IEEE International Conference on Computer Vision , pages=

Delving deep into rectifiers: Surpassing human-level performance on imagenet classification , author=. Proceedings of the IEEE International Conference on Computer Vision , pages=
[6]

arXiv preprint arXiv:1312.6120 , year=

Exact solutions to the nonlinear dynamics of learning in deep linear neural networks , author=. arXiv preprint arXiv:1312.6120 , year=

arXiv
[7]

2012 , publisher=

Bayesian learning for neural networks , author=. 2012 , publisher=

2012
[8]

arXiv preprint arXiv:1711.00165 , year=

Deep Neural Networks as Gaussian Processes , author=. arXiv preprint arXiv:1711.00165 , year=

arXiv

Show all 59 references

[9]

arXiv preprint arXiv:1804.11271 , year=

Gaussian Process Behaviour in Wide Deep Neural Networks , author=. arXiv preprint arXiv:1804.11271 , year=

arXiv
[10]

arXiv preprint arXiv:1711.04735 , year=

Resurrecting the Sigmoid in Deep Learning Through Dynamical Isometry: Theory and Practice , author=. arXiv preprint arXiv:1711.04735 , year=

arXiv
[11]

arXiv preprint arXiv:1806.05393 , year=

Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks , author=. arXiv preprint arXiv:1806.05393 , year=

arXiv
[12]

arXiv preprint arXiv:1806.05394 , year=

Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks , author=. arXiv preprint arXiv:1806.05394 , year=

arXiv
[13]

Classic

Why" Classic" Transformers Are Shallow and A Depth-Enabling Technique , author=. Journal of Machine Learning Research , volume=
[14]

Advances in Neural Information Processing Systems , volume=

From condensation to rank collapse: A two-stage analysis of transformer training dynamics , author=. Advances in Neural Information Processing Systems , volume=
[15]

International Conference on Machine Learning , pages=

Attention is not all you need: Pure attention loses rank doubly exponentially with depth , author=. International Conference on Machine Learning , pages=. 2021 , organization=

2021
[16]

How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings , author=. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language P...

2019
[17]

Advances in Neural Information Processing Systems , volume=

Attention is all you need , author=. Advances in Neural Information Processing Systems , volume=
[18]

arXiv preprint arXiv:1607.06450 , year=

Layer normalization , author=. arXiv preprint arXiv:1607.06450 , year=

arXiv
[19]

Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages=

Deep residual learning for image recognition , author=. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages=
[20]

arXiv preprint arXiv:2410.07799 , year=

Mind the gap: a spectral analysis of rank collapse and signal propagation in attention layers , author=. arXiv preprint arXiv:2410.07799 , year=

arXiv
[21]

Advances in Neural Information Processing Systems , volume=

Signal propagation in transformers: Theoretical perspectives and the role of rank collapse , author=. Advances in Neural Information Processing Systems , volume=
[22]

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages=

Understanding the difficulty of training transformers , author=. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages=

2020
[23]

International Conference on Machine Learning , pages=

On layer normalization in the transformer architecture , author=. International Conference on Machine Learning , pages=. 2020 , organization=

2020
[24]

arXiv preprint arXiv:1910.05895 , year=

Transformers without Tears: Improving the Normalization of Self-Attention , author=. arXiv preprint arXiv:1910.05895 , year=

1910 arXiv
[25]

Uncertainty in Artificial Intelligence , pages=

Addressing token uniformity in transformers via singular value transformation , author=. Uncertainty in Artificial Intelligence , pages=. 2022 , organization=

2022
[26]

arXiv preprint arXiv:2010.04245 , year=

Query-Key Normalization for Transformers , author=. arXiv preprint arXiv:2010.04245 , year=

2010 arXiv
[27]

International Conference on Learning Representations , year=

Improving neural language generation with spectrum control , author=. International Conference on Learning Representations , year=
[28]

International Conference on Machine Learning , pages=

Stabilizing transformer training by preventing attention entropy collapse , author=. International Conference on Machine Learning , pages=. 2023 , organization=

2023
[29]

arXiv preprint arXiv:1907.12009 , year=

Representation degeneration problem in training natural language generation models , author=. arXiv preprint arXiv:1907.12009 , year=

1907 arXiv
[30]

arXiv preprint arXiv:1901.09321 , year=

Fixup initialization: Residual learning without normalization , author=. arXiv preprint arXiv:1901.09321 , year=

1901 arXiv
[31]

2002.05202 , archivePrefix=

Noam Shazeer , year=. 2002.05202 , archivePrefix=

2002 arXiv
[32]

IEEE Transactions on Pattern Analysis and Machine Intelligence , volume=

Deepnet: Scaling transformers to 1,000 layers , author=. IEEE Transactions on Pattern Analysis and Machine Intelligence , volume=. 2024 , publisher=

2024
[33]

arXiv preprint arXiv:2012.11747 , year=

RealFormer: Transformer Likes Residual Attention , author=. arXiv preprint arXiv:2012.11747 , year=

2012 arXiv
[34]

arXiv preprint arXiv:2102.11972 , year=

Do Transformer Modifications Transfer Across Implementations and Applications? , author=. arXiv preprint arXiv:2102.11972 , year=

arXiv
[35]

arXiv preprint arXiv:2110.09456 , year=

NormFormer: Improved Transformer Pretraining with Extra Normalization , author=. arXiv preprint arXiv:2110.09456 , year=

arXiv
[36]

arXiv preprint arXiv:2202.08625 , year=

Revisiting over-smoothing in bert from the perspective of graph , author=. arXiv preprint arXiv:2202.08625 , year=

arXiv
[37]

arXiv preprint arXiv:1905.10947 , year=

Graph neural networks exponentially lose expressive power for node classification , author=. arXiv preprint arXiv:1905.10947 , year=

1905 arXiv
[38]

Uncertainty in Artificial Intelligence , pages=

Rezero is all you need: Fast convergence at large depth , author=. Uncertainty in Artificial Intelligence , pages=. 2021 , organization=

2021
[39]

Advances in Neural Information Processing Systems , volume=

Root mean square layer normalization , author=. Advances in Neural Information Processing Systems , volume=
[40]

arXiv preprint arXiv:2203.05962 , year=

Anti-oversmoothing in deep vision transformers via the fourier domain analysis: From theory to practice , author=. arXiv preprint arXiv:2203.05962 , year=

arXiv
[41]

Proceedings of the AAAI Conference on Artificial Intelligence , volume=

Deeper insights into graph convolutional networks for semi-supervised learning , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=
[42]

Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

Going deeper with image transformers , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=
[43]

Findings of the Association for Computational Linguistics: ACL 2023 , pages=

B2t connection: Serving stability and performance in deep transformers , author=. Findings of the Association for Computational Linguistics: ACL 2023 , pages=

2023
[44]

arXiv preprint arXiv:2302.10322 , year=

Deep transformers without shortcuts: Modifying self-attention for faithful signal propagation , author=. arXiv preprint arXiv:2302.10322 , year=

arXiv
[45]

arXiv preprint arXiv:2304.14802 , year=

ResiDual: Transformer with Dual Residual Connections , author=. arXiv preprint arXiv:2304.14802 , year=

arXiv
[46]

arXiv preprint arXiv:2403.09635 , year=

Transformers get stable: An end-to-end signal propagation theory for language models , author=. arXiv preprint arXiv:2403.09635 , year=

arXiv
[47]

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages=

Learning deep transformer models for machine translation , author=. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages=
[48]

Bert: Pre-training of deep bidirectional transformers for language understanding , author=. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages=

2019
[49]

arXiv preprint arXiv:2010.11929 , year=

An image is worth 16x16 words: Transformers for image recognition at scale , author=. arXiv preprint arXiv:2010.11929 , year=

2010 arXiv
[50]

Advances in Neural Information Processing Systems , volume=

Language models are few-shot learners , author=. Advances in Neural Information Processing Systems , volume=
[51]

arXiv preprint arXiv:2001.08361 , year=

Scaling laws for neural language models , author=. arXiv preprint arXiv:2001.08361 , year=

2001 arXiv
[52]

arXiv preprint arXiv:2203.15556 , volume=

Training compute-optimal large language models , author=. arXiv preprint arXiv:2203.15556 , volume=

arXiv
[53]

arXiv preprint arXiv:1711.05101 , year=

Decoupled weight decay regularization , author=. arXiv preprint arXiv:1711.05101 , year=

arXiv
[54]

Journal of Machine Learning Research , volume=

Exploring the limits of transfer learning with a unified text-to-text transformer , author=. Journal of Machine Learning Research , volume=
[55]

arXiv preprint arXiv:2307.09288 , year=

Llama 2: Open foundation and fine-tuned chat models , author=. arXiv preprint arXiv:2307.09288 , year=

arXiv
[56]

arXiv preprint arXiv:2602.18849 , year=

Exact Attention Sensitivity and the Geometry of Transformer Stability , author=. arXiv preprint arXiv:2602.18849 , year=
[57]

arXiv preprint arXiv:2601.19895 , year=

Post-LayerNorm Is Back: Stable, ExpressivE, and Deep , author=. arXiv preprint arXiv:2601.19895 , year=
[58]

Advances in Neural Information Processing Systems , volume=

On the role of attention masks and layernorm in transformers , author=. Advances in Neural Information Processing Systems , volume=
[59]

Improving deep transformer with depth-scaled initialization and merged attention , author=. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages=

2019

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

International Conference on Learning Representations , volume=

Efficient streaming language models with attention sinks , author=. International Conference on Learning Representations , volume=

[2] [2]

Advances in Neural Information Processing Systems , volume=

Exponential expressivity in deep neural networks through transient chaos , author=. Advances in Neural Information Processing Systems , volume=

[3] [3]

arXiv preprint arXiv:1611.01232 , year=

Deep information propagation , author=. arXiv preprint arXiv:1611.01232 , year=

arXiv

[4] [4]

Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics , pages=

Understanding the difficulty of training deep feedforward neural networks , author=. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics , pages=. 2010 , organization=

2010

[5] [5]

Proceedings of the IEEE International Conference on Computer Vision , pages=

Delving deep into rectifiers: Surpassing human-level performance on imagenet classification , author=. Proceedings of the IEEE International Conference on Computer Vision , pages=

[6] [6]

arXiv preprint arXiv:1312.6120 , year=

Exact solutions to the nonlinear dynamics of learning in deep linear neural networks , author=. arXiv preprint arXiv:1312.6120 , year=

arXiv

[7] [7]

2012 , publisher=

Bayesian learning for neural networks , author=. 2012 , publisher=

2012

[8] [8]

arXiv preprint arXiv:1711.00165 , year=

Deep Neural Networks as Gaussian Processes , author=. arXiv preprint arXiv:1711.00165 , year=

arXiv