REVIEW 6 major objections 5 minor 11 references
Attention in Transformers is relevant or irrelevant depending on the correlation length of the input, not on the architecture itself.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
Attention is a 'relevant' perturbation of an MLP fixed point only when the input has long-range structure; for short-range inputs it is 'irrelevant' and the Transformer reverts to MLP behavior.
T0 review reviewed 2026-08-01 challenge →
load-bearing objection Honest, controlled paper: attention's functional role flips with input correlation length (real effect, nice measurements), but the RG mechanism is inferred, not tested, and single-seed runs plus a non-stabilizing long-ξ TFM keep the theory on thin ice. the 6 major comments →
Relevant and Irrelevant: A Renormalization Group Analysis of Transformer Attention
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
The paper argues that attention acts as a perturbation of the MLP residual stack's fixed point x*, with the resulting shift given by δ = −M*⁻¹(a+b), and that the eigenvalues of M* determine whether attention is irrelevant, marginal, relevant, or bifurcating. In short-correlation sequences, attention is irrelevant: the Transformer achieves the same final loss and fixed-point geometry as the MLP and only contracts perturbations faster. In long-correlation sequences, the slow modes make M* nearly singular, the shift diverges, and attention becomes relevant: effective rank expands from 12.9 to 25.4 at layer 1, loss drops 0.195 nats below the marginal distribution, and the first-layer head L0H0 a
What carries the argument
The central object is the fixed-point shift formula δ = −M*⁻¹(a+b), where x* is the MLP residual stack's fixed point, a = A(x*) is the attention output at x*, b = f(x*+a) is the MLP block at the attention-shifted point, and M* = D[b] + D[a] + D[b]D[a] is the composite Jacobian. Its eigenvalues ν_k classify attention as irrelevant (−2 < ν_k < 0), marginal (ν_k = −2), relevant (ν_k > 0 or ν_k < −2), or bifurcating (ν_k = 0, where the shift diverges). The same matrix governs the stability of the new fixed point through M̃* ≈ I + M*, so it simultaneously controls the size of the shift and the fate of perturbations. The paper maps each layer to one coarse-graining step of a renormalization-group
Load-bearing premise
The load-bearing premise is that a trained MLP residual stack really settles at a fixed point x* with f(x*) = 0, so that attention's effect can be computed as a perturbation of that point; the paper's own long-correlation measurements show the Transformer does not cleanly stabilize across depth, threatening the premise exactly where the headline claim lives.
What would settle it
Measure the long-correlation Transformer's forward pass at more depths or with longer training: if inter-layer kernel drift does not decay to near zero, then no fixed point x* exists for the shift formula to reference. Alternatively, estimate the composite Jacobian M* at the plateau and check whether its eigenvalues lie in (−2, 0); a single eigenvalue outside that interval would invalidate the perturbative classification.
If this is right
- On short-correlation inputs, self-attention is an irrelevant perturbation: the Transformer reaches the same final loss and fixed-point geometry as the MLP, so its extra parameters do not change the learned function.
- On long-correlation inputs, attention is relevant: it drives a discontinuous transition to a high-rank attractor, with effective rank roughly doubling at layer 1 and loss falling 0.195 nats below the marginal distribution.
- The relevant operator is concentrated in the first-layer head: ablating L0H0 moves the Transformer's representations about four times closer to the MLP fixed point than ablating any later head.
- Along input spectral modes, the long-correlation Transformer preserves slow modes roughly 5.4× longer than fast modes while the MLP is nearly mode-blind (1.3× spread); in the short-correlation regime the Transformer suppresses every mode faster than the MLP.
- The transition between regimes is marked by near-singularity of M*: as an eigenvalue approaches zero, the fixed-point shift diverges and perturbation theory breaks down, which the paper identifies with the observed phase transition.
Where Pith is reading between the lines
- Beyond the paper: if attention's relevance is set by input correlation length, then the same trained Transformer should flip between irrelevant and relevant behavior when the input distribution is changed (e.g., by truncating or shuffling long-range context) without any architectural modification — a direct, testable consequence for natural language.
- Beyond the paper: the dominance of the first-layer head suggests that the depth-localization of attention's contribution may be largely an input property, motivating allocation strategies that concentrate early-layer capacity or curriculum designs that expose long-range structure early in training.
- Beyond the paper: because injected perturbations excite mixtures of network eigenmodes, the cleanest test of the mode-ordering prediction would measure the Jacobian of the trained residual block directly, rather than relying on Markov-chain eigenmodes, which the paper itself notes are not eigenmodes of M*.
- Beyond the paper: the breakdown of the expansion at ν_k = 0 hints that near-marginal operators may control depth transitions in larger language models, in which case scale-invariant statistics (e.g., power-law decay with layer index) would be more robust observables than layer-specific metrics.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a renormalization-group (RG) treatment of Transformer attention as a perturbation of a trained MLP residual-stack fixed point. It derives a fixed-point shift formula δ = −M*⁻¹(a+b) (Eq. 7), classifies attention as relevant/irrelevant via eigenvalues of M* (Section 3.3), and states four predictions. Experiments on synthetic Markov-chain sequences with controlled correlation length ξ report that attention is irrelevant for short-ξ chains (same loss and geometry as MLP), relevant for long-ξ chains (rank expansion, loss below H(π), L0H0 dominance), and that perturbation decay shows a regime reversal in spectral selectivity. The paper concludes that attention's relevance is determined by the spectral structure of the data-generating process.
Significance. If the central claim were fully established, this would be a valuable controlled demonstration that an RG-style perturbation framework can organize observed Transformer/MLP differences, and it would connect attention's functional role to input correlation length. The mathematical derivations in Appendices A–C are explicit and internally consistent; the synthetic setup is clean; and the stated predictions are falsifiable in principle. The paper also honestly lists several limitations, including the fact that Predictions 1 and 2 are not strict consequences of the theory and that perturbation-mode mixing weakens Prediction 3. However, the empirical support is substantially weaker than the abstract and conclusion suggest, for reasons detailed below.
major comments (6)
- [§3.1, §4.2, Table 2] The central theoretical object is the MLP fixed point x* and the perturbed fixed point x̃* = x* + δ, with stability governed by Eqs. (3)–(5), (11)–(13). The paper infers the MLP fixed point from near-zero kernel drift, but Table 2 shows that in the long-ξ regime the TFM does not stabilize: drift values are 0.233, 0.060, 0.027, 0.028, 0.014, 0.024 from L0→1 through L5→6, and the text states that the TFM 'does not cleanly stabilize across depth.' The headline relevance claim lives in this regime. Without an observed x̃*, and without measuring M* or its eigenvalues, the shift formula (7), the stability condition (13), and the classification of Section 3.3 are not actually tested; they are imposed on a non-equilibrium transient. This is a load-bearing gap rather than a presentational issue.
- [§3.3, §8, Experiment 1] The RG relevance/irrelevance classification is defined by the eigenvalues ν_k of M* (Eq. 10 and Section 3.3), yet these eigenvalues are never measured or estimated. Relevance in the long-ξ case is assigned from the observed loss gap and rank expansion, and those same observations are then cited as confirmation of the predicted phase transition. This is circular: the empirical signatures are consistent with a near-singular M*, but they do not identify the eigen-structure that the theory says governs relevance. An independent probe of M*, or at least of the stability Jacobians, is needed before the framework can be said to have predictive content in this regime.
- [§7, Table 6, Prediction 3] Prediction 3 states that τ_k should be monotonically increasing in |λ_k| for the TFM and that the MLP should be mode-blind. The reported long-ξ correlations are r_TFM = 0.435 and r_MLP = 0.553: the TFM does not show a stronger monotone trend than the MLP, and the MLP is not mode-blind by this measure. The text admits that the 'predicted monotone ordering of τ_k with |λ_k| is not cleanly recovered.' The dynamic-range contrast (5.4× vs 1.3×) is not equivalent to monotone selectivity, especially with non-monotone entries such as mode 10–11 (|λ|=0.137, τ=6.58). Appendix E explains why mixing weakens the test, but that means Prediction 3 is not confirmed at the level it was stated; 'regime-level contrast' is a post-hoc relaxation.
- [§3.5, Predictions 1–2] The paper explicitly states that Prediction 1 is 'not a strict consequence of the theory but a hypothesis' and that the Prediction 2 argument is 'suggestive rather than a strict consequence of the theory.' Given this, the abstract's and conclusion's language that the framework provides a 'predictive account' and that the predictions are 'confirmed' overstates the epistemic status. The empirical results are consistent with the hypotheses, but they do not validate the derivations because the key quantities (M*, its eigenvalues, and the smallness of δ) are not independently established.
- [§5, Eq. (17), Experiment 2] The layer-specific shift formula δ^(l) is introduced as 'motivated by (7) but is not implied by it.' Therefore the L0H0-dominance result (ΔCKA = 0.119 vs 0.027) is not a test of the derived shift formula. The paper acknowledges an alternative structural explanation (first layer sees highest-entropy input), which is good, but then in Section 8 still counts Prediction 4 as confirmed. The confirmation claim should be correspondingly qualified, or an experiment that actually tests the layer decomposition should be provided.
- [§4–§7, Tables 1, 3, 5, 6] All quantitative conclusions are based on a single training run per condition, with no error bars or seed variation. Given the controlled setting, this is fixable and necessary: the loss gap of 0.195 nats, the 4× head dominance, the 5.4× dynamic range, and the scaling exponents βρ all need variance estimates before they can support the strong claims. At present it is possible that some reported differences are within seed-to-seed variability.
minor comments (5)
- [§3.5] The mapping 'one iteration of F corresponds to one network layer' is an assumption and should be flagged as such where it is introduced; it is not directly verified, and Table 2's long-ξ drift makes it especially fragile.
- [Table 1] The negative loss gaps for MLP short and TFM short (≤0.004 nats) are described as within the noise floor, but the table reports them without uncertainty. A brief statement of the noise floor estimate would help.
- [§6.1] The disqualification of entropy and participation ratio as RG observables is well argued, but the effective-rank observable is still a statistic of A^(l,h), not of δ. The distinction between 'property of attention mechanism' and 'effect on residual stream' should be reiterated in the conclusion.
- [§7, Appendix E] Appendix E is honest about mode mixing, but the statement in Section 7 that injecting ϕ_k is 'the correct design for testing Prediction 3' should be reconciled with the fact that the resulting τ_k is explicitly not a single-eigenmode quantity. Either the prediction should be restated in terms of effective decay rates, or the test should be acknowledged as indirect.
- [References] Reference formatting is inconsistent (e.g., 'Fernando & Guitchounts' is cited after Wilson and Kogut; some entries mix author-year styles). This is minor but should be cleaned up.
Circularity Check
Partial circularity: long-ξ relevance is assigned from the observed phase transition and then reported as confirming the eigenvalue-based classification.
specific steps
-
self definitional
[Section 4.2 'RG interpretation' (cf. Section 3.3 classification and Section 8 Prediction 2 confirmation)]
"RG interpretation: The two regimes map directly onto the RG classification of Section 3.3: short-ξ attention behaves as an irrelevant perturbation (fixed-point shift small, MLP attractor preserved, same loss); long-ξ attention behaves as a relevant perturbation (large shift, qualitatively different attractor, loss gap of 0.212 nats)."
Section 3.3 defines 'relevant' and 'irrelevant' by the eigenvalues νk of M* (e.g., |1+νk| > 1 for relevant), but those eigenvalues are never measured. In the experimental sections, the labels are instead assigned from the very outcomes the predictions were meant to explain: short-ξ is called irrelevant because the shift is small and the MLP attractor is preserved, long-ξ is called relevant because the shift is large and the attractor differs. Section 8 then treats this labeling as confirmation of Prediction 2 ('consistent with the loss of perturbative control dictated by the near-singularity of M*'). The paper itself concedes that the driving mechanism (large a+b vs near-zero νk) 'cannot be determined from these measurements alone.' The classification is thus a redescription of the observe
-
self citation load bearing
[Section 3.1 and Section 3.4]
"As established in our previous work, the trained MLP residual stack converges to a fixed-point plateau... In (Haggi-Mani & Rish, 2026) we established that the network implements a discrete approximation to an RG flow."
The RG-flow interpretation and the central fixed-point premise are imported from the authors' own unpublished preprint (Haggi-Mani & Rish, 2026). Although Experiment 1's Table 2 independently measures near-zero MLP drift in both regimes, the identification of the forward pass with an RG flow is not re-derived or externally verified here; it functions as a self-cited interpretive frame for the paper's central claim. This is not the sole support for the fixed-point plateau, so it is not fully load-bearing, but it is a recurring self-citation that carries interpretive weight.
full rationale
The fixed-point shift formula δ = −M*^{-1}(a+b) is a genuine first-order expansion derived by linearizing the block map around x* (Appendix B), and the stability analysis of the TFM fixed point (Appendix C) is internally consistent. No fitted parameter is renamed as a prediction, and the numerical experiments are self-contained against a controlled Markov-chain benchmark. However, the paper's central relevance/irrelevance classification is partly circular: the operative meaning of 'relevant' in the experimental sections is the observed phase transition and large fixed-point shift, while the stated criterion is the unmeasured spectrum of M*. Section 8 treats the same observed transition as the confirming evidence for Prediction 2, and the paper acknowledges that the mechanism (large a+b vs near-zero νk) cannot be distinguished from the measurements. The long-ξ TFM also does not cleanly stabilize (Table 2: drift 0.014–0.060 through L6), so the fixed-point object x̃* is approximate in the very regime where the headline claim lives. These issues weaken the empirical test but do not make the shift-formula derivation itself circular, so the overall circularity is partial rather than total.
Axiom & Free-Parameter Ledger
free parameters (3)
- correlation-length control α =
10 (short-ξ) / 0.05 (long-ξ)
- decay lengths τk =
1.27–6.84 (long-ξ TFM), 0.60–1.39 (short-ξ TFM), 1.78–2.32 (MLP)
- scaling exponents βρ =
0.005–0.427 across layers/regimes
axioms (5)
- domain assumption The trained MLP residual stack converges to a fixed point x* with f(x*) = 0, and all eigenvalues of I + M lie inside the unit circle (−2 < μk < 0).
- ad hoc to paper One iteration of the block map F corresponds to one network layer (l counts layers in ϵl = (1+νk)^l ϵ0).
- ad hoc to paper The learned input projection W maps each Markov eigenvector ϕk into a dominant single eigenmode of M* such that the fitted decay length τk tracks |λk|.
- domain assumption For short-ξ, D[a] is small because token representations at x* are nearly identical across positions, giving M* ≈ D[b] ≈ M.
- domain assumption The new fixed point lies near x* (∥δ∥ ≪ 1) so attention output a = A(x*) is treated exactly while δ is treated to first order.
Cite this review
Pith. "Pith review of Relevant and Irrelevant: A Renormalization Group Analysis of Transformer Attention." pith.science (2026). https://pith.science/paper/YGQJK76Q
@misc{pith2026260715449,
author = {Pith},
title = {Pith review of: Relevant and Irrelevant: A Renormalization Group Analysis of Transformer Attention},
year = {2026},
howpublished = {\url{https://pith.science/paper/YGQJK76Q}},
note = {Machine review of arXiv:2607.15449}
}
read the original abstract
Using the language of Wilsonian renormalization group theory (RG), we treat the Transformer's attention mechanism as a perturbation of the trained MLP residual-stack fixed point and ask whether it constitutes a relevant, marginal, or irrelevant operator. We derive a fixed-point shift formula and obtain four testable predictions for the fixed-point geometry, effective rank profile, layer specificity, and perturbation decay spectrum. Testing these on synthetic Markov chain sequences with controlled correlation length, we find: (1) For large chains(long correlation), attention is strongly relevant: it closes a residual loss gap the MLP cannot bridge and drives a phase transition in representation space, with effective rank jumping above input dimensionality at layer 1 and stabilizing at a high-dimensional plateau. (2) For short chains(short correlation), attention is irrelevant: the Transformer converges to the same loss and fixed-point geometry as the MLP, though it contracts perturbations faster. (3) The transition is dominated by the first-layer head (L0H0), which accounts for more than 4 times the representational shift of any subsequent head, consistent with the prediction that the relevant operator acts before the MLP begins integrating out positional variation. (4) Perturbation decay experiments reveal a regime reversal: in the long correlation regime the Transformer selectively preserves slow Markov modes (5.4 times the dynamic range in decay length vs. 1.3 times for the MLP); in the short correlation regime it suppresses all modes faster than the MLP, with no spectral selectivity. Together, these results show that the relevance of attention is not a property of the architecture but of the spectral structure of the data-generating process, and that a first-order RG perturbation framework provides a predictive account of that difference.
Figures
Reference graph
Works this paper leans on
-
[9]
Association for Computational Linguistics. doi:10.18653/v1/P19-1452. Voita, E., Talbot, D., Moiseev, F., Sennrich, R., and Titov, I. Analyzing multi-head self- attention: Specialized heads do the heavy lifting, the rest can be pruned. InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 5797–5808, Florence, Italy,
-
[10]
Association for Computational Linguistics. doi:10.18653/v1/P19-1580. Wilson, K. G. Renormalization group and critical phenomena I.Physical Review B, 4(9): 3174–3183,
-
[13]
URL https://arxiv.org/abs/2605.14258. Makkuva, A. V., Bondaschi, M., Girish, A., Nagle, A., Jaggi, M., Kim, H., and Gastpar, M. Attention with Markov: A framework for principled analysis of transformers via Markov chains.arXiv preprint arXiv:2402.04161,
-
[15]
URL https://arxiv.org/abs/2510.25553. 27
-
[1971]
doi:10.1103/PhysRevB.4.3174. 26 Wilson, K. G. and Kogut, J. The renormalization group and the ε expansion.Physics Reports, 12(2):75–199,
-
[2013]
25 Bordelon, B., Atanasov, A., and Pehlevan, C
URLhttps://arxiv.org/abs/1301.3124. 25 Bordelon, B., Atanasov, A., and Pehlevan, C. A dynamical model of neural scaling laws. In Proceedings of the 41st International Conference on Machine Learning (ICML), volume 235, pp. 4345–4381. PMLR,
-
[2014]
URL https://arxiv.or g/abs/1410.3831. Roy, O. and Vetterli, M. The effective rank: A measure of effective dimensionality. In Proceedings of the 15th European Signal Processing Conference (EUSIPCO), pp. 606–610, Poznan, Poland,
-
[2015]
Kornblith, S., Norouzi, M., Lee, H., and Hinton, G
URL https://arxiv.org/abs/1412.6980. Kornblith, S., Norouzi, M., Lee, H., and Hinton, G. Similarity of neural network representations revisited. InProceedings of the 36th International Conference on Machine Learning (ICML), volume 97, pp. 3519–3529. PMLR,
-
[2019]
Association for Computational Linguistics. doi:10.18653/v1/W19-4828. Haggi-Mani, P. and Rish, I. Rank collapse, fixed points, and the renormalization group structure of MLP residual networks.Preprint,
-
[2024]
Clark, K., Khandelwal, U., Levy, O., and Manning, C
URL https://proceedings.mlr.press/v235/borde lon24a.html. Clark, K., Khandelwal, U., Levy, O., and Manning, C. D. What does BERT look at? An analysis of BERT’s attention. InProceedings of the 2019 ACL Workshop Black- boxNLP, pp. 276–286, Florence, Italy,
2019
- [2026]
This paper was first reviewed by deepseek-v4-flash on August 1, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.