{"id":"803acc81-98e5-42a9-ab50-68cf91cf45eb","arxiv_id":"2608.09417","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Post-Norm transformer collapse is driven by attention-driven similarity growth at initialization plus a normalization-induced backward gradient contraction that prevents repair, with collapsed networks limited to frequency-based prediction.","lead":"This paper explains why Post-Norm Transformers, an older architecture design, tend to fail during training: causal attention makes token representations more alike at initialization, and the normalization layers then shrink gradients that could reverse this. It also shows that a collapsed model can only match the raw frequency distribution of words, which sets a high loss floor.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unexplained transition: attention amplification and RMSNorm contraction accompany collapse, but are not shown to cause the sharp loss/similarity spike.","rationale":"The reader correctly identifies the equal-correlation closure as the most fragile approximation in the forward theory; the visible mismatch in Figure 1 is honest evidence that the closed-form growth curves are not exact. However, the deeper problem for the paper's central claim is causal, not quantitative. The paper's own Limitations section states that the sharp transition into collapse is unexplained. Stage I is measured only at initialization, and the stable LR=6e-4 run shares the same initialization-time amplification, so forward amplification cannot by itself determine whether a run collapses. Stage II is measured after the transition begins, with the contraction factor c(y,alpha) crossing below 1 'shortly after the transition begins', which suggests it is a response to the spike rather than the trigger. The two mechanisms are real, well-measured phenomena, and the paper's descriptive account of what happens during collapse is valuable. But the Introduction's causal phrasing that collapse is 'caused by the combination' of the two mechanisms is stronger than the evidence supports. I would keep the reader's CONDITIONAL verdict: the causal claim needs to be softened to 'accompanied and sustained by', or the authors need to provide the missing causal link, such as showing that the contraction precedes the spike or that removing one mechanism prevents collapse. This is why I mark agreement as partial: the reader's formal weakest assumption is the equal-correlation closure, while I see the unexplained transition as the more load-bearing concern, even though the reader did mention the missing transition in the rationale.","tokens_in":35025,"tokens_out":11591,"duration_ms":115774,"concrete_test":"On the collapsing LR=8e-4 Post-Norm run (and the Appendix F.4 runs at 1.2e-3, 1.5e-3, 1.8e-3), log last-layer tsim, training loss, and per-layer c(y,alpha) at every optimizer step from step 2000 through 4000. For each layer, record the first step at which c(y,alpha)<1 and the first step at which last-layer tsim begins its sharp rise. If in most layers c<1 is attained only after the tsim/loss spike begins, then the RMSNorm gradient contraction is a consequence of the transition, not its cause, and the central claim must be downgraded to a persistence mechanism; if c<1 systematically precedes the spike, the gradient-repair-failure mechanism is a plausible part of the trigger.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing issue is not the quantitative accuracy of Assumption 3.1 but the causal status of the two stages. Section 5 explicitly concedes that the paper 'does not explain what causes the sharp rise in last-layer token similarity and training loss at a particular optimizer step.' This concession directly conflicts with the Introduction's claim that Post-Norm rank collapse 'is not caused by a single instability, but by the combination of two mechanisms.' Stage I (Theorem 3.2) is measured at initialization only; the stable Post-Norm run at LR=6e-4 (Appendix F.5) has the same initialization distribution and forward amplification yet does not collapse, so forward amplification alone does not trigger collapse. Stage II (Theorem 3.6) is diagnosed in the collapsing run after the transition window has begun; Figure 4 reports c(y,alpha) crossing below 1 'shortly after the transition begins', which places RMSNorm contraction as contemporaneous with or posterior to the transition trigger. The paper therefore establishes two mechanisms that accompany and may sustain collapse, but leaves the learning-rate-dependent onset (LR=8e-4 vs. 6e-4) unexplained. Without evidence on ordering or an intervention on one mechanism, the causal form of the central claim is not supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a two-stage account of Post-Norm Transformer rank collapse. In Stage I (Section 3.1, Theorem 3.2), the authors show that at initialization the causal attention matrix acts approximately as a prefix-averaging operator that increases token similarity at every layer, with a closed-form one-step increment Delta_attn(s,t1) = s f1(t1) / (1 + s f2(t1)), while the SwiGLU branch supplies only a small damping effect (Theorem 3.4). In Stage II (Section 3.2, Theorem 3.6), they prove that once the representation is close to collapse, growing pre-normalization residual norms make the RMSNorm backward factor contractive, giving a per-layer gradient contraction factor c(y,alpha) = alpha * sqrt(d) / ||y||_2, so gradients reaching earlier layers decay geometrically when c < 1. Section 3.3 characterizes collapsed networks: the best achievable predictor is the empirical frequency distribution of the labels (Theorem 3.8(i)), and parameter gradients vanish in collapsed layers (Theorem 3.8(ii)). Experiments on 48-layer decoder-only Transformers trained on C4 measure the initialization-time similarity growth, the transition-window contraction of c(y,alpha), the per-layer gradient drop, and the loss floor near the frequency loss. Section 5 explicitly concedes that the paper does not explain what causes the sharp rise in last-layer token similarity and training loss at a particular optimizer step.","tokens_in":35173,"tokens_out":9863,"duration_ms":92814,"significance":"If the results hold, the paper delivers a useful mechanistic decomposition of Post-Norm collapse: a parameter-free forward formula for initialization-time similarity amplification, a measurable backward contraction criterion, and a clean characterization of the collapse loss floor. The strengths are real: the forward formula is derived rather than fitted, the intervention in Corollary 3.3 (subtracting alpha*C_n suppresses similarity growth) is a genuinely falsifiable test and the experiment confirms it, the backward theorem is reduced to measurable quantities (alpha^k_l, ||y||_2), the appendix proofs are detailed, and the limitation statement in Section 5 is candid. The principal weakness is that the causal framing of the Introduction is not supported by the evidence, which establishes only that the two mechanisms accompany and sustain high-similarity states; the paper's own limitation statement concedes the trigger is unexplained. In addition, the quantitative forward predictions rest on the equal-correlation closure and show a visible mismatch with measurements (Figure 1). With the causal claims rescaled, this is a publishable and valuable contribution.","major_comments":[{"comment":"The Introduction asserts that 'Post-Norm rank collapse is not caused by a single instability, but by the combination of two mechanisms' (§1), yet Section 5 explicitly concedes that the paper 'does not explain what causes the sharp rise in last-layer token similarity and training loss at a particular optimizer step.' The evidence supports a weaker, non-causal claim: Stage I is an initialization-time property that also holds in the stable Post-Norm run at LR=6e-4 (Appendix F.5), which never collapses, so forward amplification alone cannot trigger collapse; and Stage II is measured only 'shortly after the transition begins' (§4.2, Figure 4), placing the RMSNorm contraction contemporaneous with or posterior to the transition trigger. Because both stages are present in conditions without collapse or appear only after the transition has begun, the causal form of the central claim is not established. Please reframe the central contribution as a partial, mechanistic account of the mechanisms that push toward and sustain high-similarity states (as the Abstract and Conclusion already mostly do), or provide ordering or intervention evidence that would support the causal reading.","section":"§1 vs. §5; §4.2; App. F.5"},{"comment":"The closed-form Stage I predictions (Theorems 3.2, 3.4, and C.3) rest on the equal-correlation closure, which replaces the entire token Gram matrix by the one-parameter surrogate R_eq(t1). The paper's own Figure 1 reports a 'visible mismatch' between measured and theoretical increments, so the quantitative accuracy of the closure is limited. Because the signs and scales of the increments are the load-bearing content of the qualitative amplification claim, the paper should either quantify the sensitivity of Delta_attn to deviations from equal correlation, or state explicitly that the closed-form curves are schematic and rely on the surrogate. The prefix-averaging approximation is well validated on its own (Figure 9, roughly 2% relative error), which isolates the equal-correlation closure as the fragile premise of the forward analysis.","section":"Assumption 3.1; Fig. 1; §4.1"},{"comment":"The theoretical support for the frequency-loss floor is weaker than the text suggests. The bound in Theorem 3.8(i) is L_CE >= L_freq(y) - 2*sqrt(n)*||W_lm||_2 * epsilon; with n = 2048 this gap is enormous unless epsilon is extremely small, so the theorem has limited quantitative predictive power in the near-collapse regime. The experimental claim that collapsed runs 'stay near' the frequency loss (Figure 6) is therefore primarily an empirical finding, and the comparison has a post-hoc degree of freedom: the aggregated-label reference is chosen because it 'matches more closely' (§4.3). Please state these caveats and temper the claim that Theorem 3.8 predicts the observed loss floor.","section":"Theorem 3.8(i); §4.3; Fig. 6"}],"minor_comments":[{"comment":"The symbol alpha is overloaded: it denotes the de-escalation coefficient in P - alpha*C_n (Corollary 3.3), the sublayer gradient contribution factor alpha^k_l (Definition 3.5), and a row-norm parameter in Lemma C.5. Please rename at least one of these to avoid confusion in Section 4.2, where both meanings appear.","section":"§2, §3.1, Def. 3.5, Lemma C.5"},{"comment":"The Figure 1 caption states that the increment 'is tracked well by Theorem 3.2,' while the text in §4.1 says 'a visible mismatch remains'; please reconcile these descriptions so the reader knows how much quantitative accuracy is being claimed.","section":"Fig. 1 caption vs. §4.1 text"},{"comment":"The plot titles in Appendix F.4 read 'sublayer ambplification factor' instead of 'amplification'; additionally, several inline formulas in the main text (e.g., norm and trace expressions in §3.1 and §3.2) are typeset inconsistently with the appendix and should be cleaned in the final version.","section":"App. F.4; main-text typos"},{"comment":"The claim that the stated results 'directly extend to multi-head cases' is asserted without proof in the main text; since Appendix E.3 does handle multi-head attention, please add an explicit pointer when the extension claim is made in Section 2.","section":"§2; App. E.3"},{"comment":"The choice to compare against the aggregated-label frequency loss rather than the per-sequence average is justified by which curve 'matches more closely'; please state this explicitly as an empirical modeling choice with its own caveats rather than a consequence of Theorem 3.8.","section":"§4.3"}],"recommendation":"major_revision","confidential_remarks":"The main gating issue is the mismatch between the causal framing in Section 1 and the admitted limitation in Section 5. The two-stage mechanisms are plausibly correct as descriptors of what pushes toward and sustains collapse, but the paper does not establish that they cause the collapse transition, and the stable LR=6e-4 control run shows that Stage I is not sufficient. I believe a careful rescoping of the central claim, plus the requested caveats on the equal-correlation closure and the frequency-loss bound, is fixable within the manuscript's scope; the paper is honest, technically detailed, and contains a genuinely falsifiable intervention experiment (Corollary 3.3). I would not reject over the approximation error. I recommend major revision rather than minor because the introduction's causal claim is load-bearing and currently unsupported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. The paper does something real: it gives closed-form formulas for how causal attention amplifies token similarity at initialization in Post-Norm decoders with SwiGLU (Theorems 3.2 and 3.4), a backward contraction bound for RMSNorm (Theorem 3.6), and a frequency-loss characterization of collapsed states (Theorem 3.8). The experiments on a 48-layer Llama-style model broadly support the forward prediction, and the prefix-averaging removal intervention is a clean direct test. The paper is also refreshingly honest about its approximations.\n\nThe soft spot is the causal claim. The introduction says collapse is 'not caused by a single instability, but by the combination of two mechanisms,' and the abstract frames the two stages as explaining collapse. But the paper's own Section 5 concedes it does not explain what causes the sharp rise in similarity and loss at a particular optimizer step. The stress-test note is decisive on ordering: the stable Post-Norm run at LR=6e-4 has the same initialization-time forward amplification yet does not collapse, so Stage I is not sufficient to trigger the transition. Stage II's contraction factor c<1 is shown to cross below 1 only shortly after the transition begins (Figure 4), placing it contemporaneous with or posterior to onset. What the evidence supports is: forward amplification pushes similarity upward at initialization in all runs; backward contraction accompanies collapse and likely makes it hard to reverse once it starts. That is a two-mechanism account of amplification and repair failure, not an account of what triggers the transition. The language should be scaled back.\n\nThe other approximations are in proportion. The equal-correlation closure (Assumption 3.1) is a real ansatz, and the quantitative match in Figure 1 shows a visible mismatch, though the sign and scale are correct. The prefix-averaging approximation is validated at about 2% error (Figure 9), so that part is solid. The backward theorem uses measured alpha_k and norms, making it an empirical diagnosis rather than a predictive theory, but the paper uses it fairly.\n\nI'd send this to peer review. With the causal claims softened to 'amplification and sustained repair failure' and the open transition question acknowledged (as Section 5 partly already does), it would be a solid contribution. I'd bring it to a reading group and cite the closed-form results. Recommendation: revise the causal framing, then accept.","headline":"Real two-mechanism account, but the causal claim outruns the evidence; the sharp transition remains unexplained and the paper should frame the mechanisms as sustaining collapse rather than causing it.","tokens_in":35797,"tokens_out":2473,"would_cite":true,"duration_ms":23974,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Post-Norm transformer collapse is driven by two linked mechanisms: causal attention raises token similarity at initialization, and RMSNorm then shrinks gradients to earlier layers so training cannot repair the growing collapse.","keywords":["Post-Norm","rank collapse","token similarity","causal attention","prefix averaging","RMSNorm","gradient vanishing","frequency distribution"],"falsifier":"Train the same 48-layer Post-Norm model at learning rate $8\\times10^{-4}$ with the causal attention matrix replaced by $P-C_n$ (the intervention in Corollary 3.3 with $\\alpha=1$); if token similarity still rises or the run still collapses near the observed transition step, then attention amplification is not the necessary driver of collapse.","tokens_in":34754,"feed_emoji":"📉","tokens_out":5166,"duration_ms":47842,"temperature":0.7,"pith_summary":"This paper argues that Post-Norm rank collapse is not a single failure but a two-stage process. At initialization, causal attention acts approximately as a prefix-averager, pushing token representations toward identical across depth, while the SwiGLU feed-forward branch only slightly damps this effect. Once representations become highly similar, growing pre-normalization residual norms make the RMSNorm backward factor contractive, so gradients reaching earlier layers decay geometrically and cannot restore token diversity. The paper also characterizes the collapsed state: its best possible output is the frequency distribution, which gives a relatively high loss floor, and parameter gradients vanish in collapsed layers. Together these mechanisms explain why Post-Norm decoders are more collapse-prone than Pre-Norm and why warmup and learning-rate tuning are needed to avoid collapse.","feed_headline":"Two mechanisms explain why Post-Norm transformers collapse","feed_subtitle":"Causal attention raises token similarity; RMSNorm then starves earlier layers of gradients.","key_machinery":"The central observable is token similarity, $\\mathrm{tsim}(X)=\\|\\Pi_1 X\\|_F^2/\\|X\\|_F^2$, a scalar between $0$ and $1$ that measures how close token representations are to being all identical. The forward derivation relies on two approximations: replacing the random causal attention matrix by the prefix-averaging matrix $C_n$ with $(C_n)_{i,j}=1/i$ for $i\\ge j$, and replacing the token Gram matrix by the equal-correlation closure $R_{\\mathrm{eq}}(t_1)=d[(1-\\rho(t_1))I+n\\rho(t_1)\\Pi_1]$ with $\\rho(t_1)=(nt_1-1)/(n-1)$. These yield closed-form one-step formulas for attention and SwiGLU similarity changes. The backward mechanism is carried by the per-layer gradient contraction factor $c(y,\\alpha)=\\alpha\\sqrt{d}/\\|y\\|_2$, which multiplies the RMSNorm Jacobian prefactor with the measured sublayer gradient contribution $\\alpha$; when this factor is below one, each layer shrinks the gradient norm on its way to earlier layers.","core_discovery":"The central claim is that Post-Norm rank collapse has a forward cause and a backward cause. Forward, under equal-correlation and prefix-averaging approximations, one attention sublayer increases token similarity by $\\Delta_{\\mathrm{attn}}(s,t_1)=s f_1(t_1)/(1+s f_2(t_1))>0$ for $t_1\\in[1/n,1)$, where $t_1$ is the token similarity and $s=nd^2\\sigma_W^2/\\|X_1\\|_F^2$ measures the amount of attention. Since $s$ stays constant across Post-Norm layers but shrinks with depth in Pre-Norm, Post-Norm experiences larger similarity growth at initialization. Backward, at exact collapse the per-layer gradient bound is $\\|\\partial L/\\partial X_l^k\\|_F \\le c(y,\\alpha)\\|\\partial L/\\partial X_{k+1}\\|_F$ with $c(y,\\alpha)=\\alpha\\sqrt{d}/\\|y\\|_2$; when $c<1$, gradients to earlier layers decay geometrically, and residual-norm growth under collapse is shown to push $c$ below one. A collapsed network can do no better than predicting the frequency distribution of labels, giving loss floor $H(p_{\\mathrm{freq}})$, and its parameter gradients vanish in all collapsed layers, making the state nearly stationary.","pith_inferences":["If token similarity is truly governed by a one-parameter equal-correlation closure, then interventions that directly penalize the trace of $\\Pi_1 X X^\\top$ or encourage Gram-matrix spread should be effective at delaying collapse, a testable design extension.","The two-stage account implies that keeping residual norms small specifically before RMSNorm, rather than merely using Pre-Norm, should preserve backward gradients; this predicts that residual-scaling methods could substitute for normalization placement.","Because collapse is characterized by the frequency loss, comparing the training loss to the frequency loss of the current batch in real time could serve as a cheap early-warning signal for the onset of collapse in production training runs.","The paper does not explain what triggers the sharp transition at a specific optimizer step; a plausible extension is to model the transition as a saddle point or loss-landscape instability when token similarity crosses a threshold."],"forward_implications":["Post-Norm decoders start training with higher token similarity than Pre-Norm at the same depth, explaining their greater sensitivity to warmup and learning rate at initialization.","Removing or subtracting the prefix-averaging component of causal attention suppresses the initialization-time similarity increase, so attention-matrix deescalation is a direct intervention against the first stage of collapse.","During collapse, the per-layer contraction factor $c(y,\\alpha)$ drops below one in most layers, and early-layer gradient norms fall by orders of magnitude, making the high-similarity state difficult to escape.","Collapsed Post-Norm runs settle near the frequency loss, so the gap between training loss and frequency loss serves as a collapse detector.","Warmup does not prevent collapse: warmup runs at higher learning rates still collapse and stay near the frequency-loss curve after the transition window."],"supporting_citations":[{"why":"Defines the Transformer architecture that the paper analyzes.","marker":"Vaswani et al., 2017"},{"why":"Establishes Post-Norm training instability and Pre-Norm as a mitigation, the baseline comparison for why Pre-Norm is more stable.","marker":"Xiong et al., 2020"},{"why":"Prior rank-collapse analysis at initialization that motivates the forward signal-propagation perspective.","marker":"Noci et al., 2022"},{"why":"Introduces token similarity as the scalar observable and analyzes encoder Post-Norm rank collapse, which this paper extends to decoders.","marker":"Yu & Zhang, 2026"},{"why":"Shows pure attention drives token uniformity and rank collapse, providing the basis for the prefix-averaging approximation.","marker":"Dong et al., 2021"},{"why":"Defines RMSNorm, whose backward Jacobian is central to the gradient-shrinkage mechanism.","marker":"Zhang & Sennrich, 2019"},{"why":"Introduces SwiGLU feed-forward layers whose damping effect is analyzed in the forward stage.","marker":"Shazeer, 2020"},{"why":"Supplies the Llama-2 style architecture used in the 48-layer experimental setup.","marker":"Touvron et al., 2023"},{"why":"Provides the C4 dataset used for training the decoder-only models in experiments.","marker":"Raffel et al., 2020"}],"fun_headline_variants":["Post-Norm collapse: attention amplifies, gradients starve","Why Post-Norm dies: attention boosts similarity, RMSNorm kills gradients","Post-Norm failure: forward similarity growth, backward gradient loss","Attention amplification plus gradient repair failure explains Post-Norm collapse","Post-Norm transformers collapse: attention grows similarity, gradients can't fix"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The analysis assumes that how similar tokens are to each other at a layer can be captured by a single average number; if the real pattern of pairwise similarities is more complex, the predicted growth and contraction curves may not hold.","fun_headline_variants_meta":{"raw":{"variants":["Post-Norm collapse: attention amplifies, gradients starve","Why Post-Norm dies: attention boosts similarity, RMSNorm kills gradients","Post-Norm failure: forward similarity growth, backward gradient loss","Attention amplification plus gradient repair failure explains Post-Norm collapse","Post-Norm transformers collapse: attention grows similarity, gradients can't fix"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000262,"raw_usage":{"total_tokens":1660,"prompt_tokens":1073,"completion_tokens":587,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":689,"completion_tokens_details":{"reasoning_tokens":512}},"tokens_in":689,"tokens_out":587,"duration_ms":6135,"temperature":1.0,"reasoning_tokens":512,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:15:35.115498+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same 48-layer Post-Norm model at learning rate $8\\times10^{-4}$ with the causal attention matrix replaced by $P-C_n$ (the intervention in Corollary 3.3 with $\\alpha=1$); if token similarity still rises or the run still collapses near the observed transition step, then attention amplification is not the necessary driver of collapse.","supporting_citations":[],"review_version":2}