Pith. sign in

REVIEW 3 major objections 4 minor 29 references

Clustered Attractor Manifolds and Dynamical Condensation in Self-Attention

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a positive overlap gap makes clustered token states stable attractors of self-attention dynamics, with a finite sharpness threshold for their nucleation from random initial conditions.

desk verdict Solid overlap-gap fixed-point analysis, but the headline finite-beta condensation transition from Gaussian initial conditions is not supported by the T=N numerics, which are consistent with the paper's own lifetime bound pushing the onset to zero. read the letter →

arxiv 2608.08922 v1 pith:Z7IUYV5Q submitted 2026-08-09 cond-mat.dis-nn cond-mat.stat-mechcs.LG

Qucheng Gao , Zuyi Yang , Xiao Chen This is my paper · ORCID
classification cond-mat.dis-nncond-mat.stat-mechcs.LG
keywords self-attentiondynamicsoverlapgapattentioncondensationrankcollapseclusteredattractormanifoldfragmentationrandom-energymodelparticipationratio
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper studies a minimal self-attention dynamics in which token representations set the attention matrix and the attention matrix updates the representations, in the joint limit where the number of tokens $N$ and the embedding dimension $d$ grow together. It claims that the overlap gap—the surplus similarity of a token to its own cluster over every other cluster—is the quantity that decides the attractor structure. A nonvanishing gap produces a $\sqrt{N}$ logit advantage, so inter-cluster attention is exponentially suppressed at fixed sharpness $\beta$, making clustered configurations stable fixed points across a whole manifold from a few macroscopic clusters to extensive microscopic fragmentation. Starting from unstructured Gaussian tokens, clusters form only above a finite $\beta$; below it, averaging erases diversity and the system rank-collapses, while above it, feedback amplifies fluctuations and condenses attention. The result matters because it locates a genuine dynamical transition at $\beta=O(1)$, whereas static random logits would need $\beta\sim\sqrt{\log N}$.

What carries the argument

The paper works with the self-masked softmax attention matrix $A_{ij}\propto\exp(\beta\sqrt{d}\,q_{ij})$ (diagonal excluded) and the normalized residual update $x_i(t+1)\propto(1-\gamma)x_i(t)+\gamma\sum_{j\neq i}A_{ij}x_j(t)$. The object that carries the argument is the overlap gap $\Delta_a=1-\max_{b\neq a}Q_{ab}$, the amount by which a token's overlap with members of its own cluster exceeds its largest overlap with any other cluster. When $\Delta_a$ is bounded away from zero, the total attention a source cluster sends to all other clusters is bounded by $N e^{-\beta\sqrt{N}\Delta_a}$, which vanishes for every fixed $\beta>0$ in the $d=N\to\infty$ limit. This exponential leakage bound simultaneously yields the fixed-point condition, the local stability of internal modes (contraction factor $1-\gamma n_a/(n_a-1)$), and the fragmentation lifetime $e^{\beta\sqrt{N}\Delta}$. A companion control parameter is the effective sharpness $\alpha=\beta\epsilon^2$ for a perturbed macroscopic cluster, which decides whether the perturbation heals or fragments into a narrow cone of microscopic descendants.

What would settle it

Simulate the model with $d=N$, $\gamma=0.3$, Gaussian initial conditions, and a fixed $\beta$ such as $0.8$, measuring the mean attention IPR $Y_A$ at time $T=N$ for increasing $N$. If $Y_A$ decays to zero with $N$, or if the apparent condensation onset $\beta_c(N)$ does not converge to a positive constant, the claimed finite-$\beta$ dynamical transition is false; a complementary check is whether the normalized overlap variance grows from $O(N^{-1})$ to $O(1)$ gaps rather than decaying.

Watch

Extended reading notes

Core claim

The central claim is that the overlap gap, not the average similarity, organizes the attractor phase diagram of minimal normalized self-attention. In the joint limit $d\sim N\to\infty$, a token whose overlap with its own cluster exceeds its overlap with every other cluster by a nonvanishing amount $\Delta$ gives that cluster a logit advantage $\beta\sqrt{N}\Delta$; because the competing set has only $O(N)$ targets and hence $O(\log N)$ entropy, the inter-cluster attention fraction decays like $N e^{-\beta\sqrt{N}\Delta}$, which vanishes for every fixed $\beta>0$. Consequently, every clustered configuration with a uniform positive gap is an exact fixed point of the limiting dynamics, and the manifold of such states spans diffuse macroscopic clusters with $Y_A=O(1/N)$ and condensed microscopic fragmentation with $Y_A=O(1)$, with narrow-cone fragments remaining low-rank in representation. These fixed points are normally attracting: internal deformations contract, while collective rotations are neutral. From Gaussian initial conditions the gap must be created by the dynamics, and the paper finds this happens only above a finite sharpness $\beta$; below it, attention averaging collapses the cloud to rank one, so the finite-$O(1)$ onset is a genuinely dynamical condensation transition rather than the static random-energy threshold $\beta\sim\sqrt{\log N}$.

Load-bearing premise

The load-bearing premise is that, from a random start, the feedback loop amplifies small accidental similarities into real clusters before averaging erases them; the paper supports this amplification by heuristic reasoning and finite-size simulations, not by an asymptotic proof in the thermodynamic limit.

Editorial extensions

If this is right

  • Clustered configurations with a uniform positive overlap gap are exact fixed points of the limiting dynamics at any fixed $\beta>0$, so self-attention alone can sustain a high-dimensional manifold of stable clustered states without external regularization.
  • In the thermodynamic limit taken first, extensive microscopic fragmentation is locally stable; at finite $N$ the same states coarsen on a time scale growing as $\exp(\beta\sqrt{N}\Delta)$, so the apparent stability boundary shifts toward $\beta=0$ as $N$ grows.
  • From Gaussian initial data the dynamics shows three regimes—diffuse rank collapse, condensed coexistence with one macroscopic cluster, and fragmented condensation—separated by transitions in the attention inverse participation ratio, the overlap variance, and the representation participation rank.
  • The finite-$\beta$ condensation onset contrasts with frozen random logits, which would require $\beta\sim\sqrt{\log N}$ according to the random-energy-model benchmark.
  • If the dimension is held fixed while $N\to\infty$, or if the attention kernel is linear or subquadratic, fixed-$\beta$ condensation disappears, so the exponential softmax together with $d\sim N$ scaling is the mechanism behind the transition.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A practical diagnostic follows for trained transformers: measure the gap between the mean intra-cluster and maximum inter-cluster overlap of token representations; heads with a finite gap should show exponentially suppressed attention leakage, and head specialization could be predicted from this gap.
  • The kernel-threshold result suggests a testable architectural prediction: replacing softmax with a quadratic nonlinearity should still produce condensation in the $d\sim N$ regime, while linear attention should not, separating the role of exponential selection from mere feedback.
  • The three dynamical regimes imply that depth acts as a control parameter: repeated layers at fixed $\beta$ below onset should drive rank collapse, while above onset they should drive further fragmentation or slow coarsening depending on the order of limits, which could be tested in deep attention stacks.
  • The coexistence regime of one macroscopic cluster plus many microscopic fragments resembles attention-sink phenomenology; testing whether the macroscopic cluster absorbs most inter-cluster attention could connect the model to observed training instabilities.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper studies a minimal normalized self-attention dynamics in which token representations determine the attention matrix and the attention matrix updates the representations, with token dimension d and number N taken together to infinity (d ~ N). The authors define normalized token overlaps q_ij and scaled logits z_ij = sqrt(d) q_ij, with softmax attention at inverse temperature beta. They show that, when tokens form clusters with a nonvanishing same-cluster overlap advantage, inter-cluster attention is exponentially suppressed, producing a manifold of clustered fixed points ranging from few macroscopic clusters to extensive microscopic fragmentation. They analyze the linear stability of these states, the noise-induced fragmentation of a macroscopic parent cluster characterized by effective sharpness alpha = beta epsilon^2, the lifetime of fragmented states under finite-N coarsening, and the order-of-limits distinction between t -> infinity first and N -> infinity first. The final part addresses dynamical accessibility from independent Gaussian initial conditions and claims a finite-beta attention-condensation transition, with three regimes: diffuse rank collapse, macroscopic-clustered condensation, and fragmented condensation.

Significance. If established, the paper would provide a clean statistical-mechanical picture of self-attention dynamics: the overlap gap as the organizing quantity, a normally attracting clustered manifold, exponentially long-lived fragmentation protected by sqrt(N) logit advantages, and a dynamical nucleation transition at O(1) beta from unstructured initial states. The fixed-point characterization, the leakage bound, the linear stability calculation, and the lifetime scaling (Eqs. 10, 23, 24 and Supplemental Sec. III) are internally consistent and are genuine contributions. The alpha-collapse of the fragmentation data and the explicit order-of-limits discussion are also valuable. However, the central dynamical-accessibility claim—that a finite beta_c survives in the joint limit—is not established by the evidence presented; the paper's own finite-size survival criterion in the Supplemental Material (Eq. 118) points to a possible drift of the apparent onset to zero. This gap is load-bearing because the finite-beta condensation transition is the headline result.

major comments (3)
  1. [Dynamical accessibility from Gaussian initial conditions; Fig. 3] The claim that "the condensation onset remains at finite beta as N -> infinity" is supported only by simulations at the single observation time T = N, with no beta_c(N) extrapolation, no scaling collapse, and no quantitative definition of the onset. This is exactly the regime where the paper's own survival criterion applies: Supplemental Eq. (118) gives beta_surv(N,T) ~ (log N + log T)/sqrt(N), which at T = N behaves as ~ 2 log N / sqrt(N) -> 0. The apparent onset seen in Fig. 3 could therefore be a finite-time artifact: for any fixed beta > 0, nucleation from O(1) initial logit fluctuations might occur on a time scale that grows with N, making beta_c(N) decay rather than converge to a positive constant. The time-window plateau shown in Supplemental Fig. 7 (T = N to 5N) does not resolve this, because the predicted coarsening time diverges faster than any power of N. The authors must provide either a scaling collapse of the onset with an N -> infinity extrapolation, or an asymptotic derivation of the nucleation time showing that it remains O(poly(N)) only for beta above a positive constant.
  2. [Dynamical accessibility from Gaussian initial conditions; Eqs. (26) and feedback-amplification paragraph] The mechanism invoked to justify the finite-beta onset is the statement that "feedback between attention and token geometry amplifies overlap fluctuations and generates finite overlap gaps." This is only a heuristic assertion; no linearized instability analysis of the diffuse Gaussian state, no mean-field nucleation calculation, and no bound on the growth rate of the overlap gap is provided. The initial logits are O(1), the same order as the static REM benchmark logits, so the claim that dynamical feedback reduces the threshold from beta_REM ~ sqrt(log N) to beta = O(1) requires a quantitative demonstration. Without such an analysis, the comparison to the static REM benchmark does not by itself support the dynamical transition.
  3. [Stability of extensive fragmentation; Supplemental Sec. III D, Eq. (118)] The manuscript itself states that the apparent small-beta stability boundary at finite size and finite time moves to zero in the thermodynamic limit, beta_surv(N,T) ~ (log N + log T)/sqrt(N). This is in direct tension with the main-text assertion that the Gaussian-initial-condition condensation onset remains finite as N -> infinity. The two statements could be consistent if the Gaussian nucleation process is qualitatively different from the survival of an already formed fragmented state, but the paper does not show this. The authors need to explain how the finite-time Gaussian-onset data avoid the drift predicted by Eq. (118), or provide numerical evidence that beta_c(N) converges to a positive value, in which case the central claim would be supported.
minor comments (4)
  1. [Abstract and Introduction] The abstract and introduction state the finite-beta condensation transition as an established result, but the body of the paper only provides finite-time numerical evidence; the wording should be softened until the scaling analysis is supplied.
  2. [Figure 1 and cluster diagnostics] The cluster statistics depend on a transitive overlap threshold q_th; while Supplemental Sec. V B shows robustness for 1 - q_th = 10^-4 and 10^-5, the main text should mention this threshold choice and its insensitivity explicitly in the caption or text.
  3. [Equation (24) and surrounding text] The lifetime formula tau_frag ~ (1/gamma N) exp(beta sqrt(N) Delta) is stated for bounded-size clusters, but the geometric factor g_a from the Supplemental derivation is omitted; a brief statement that g_a = O(1) for finite angular gaps would improve clarity.
  4. [References] The paper cites Refs. [13-20] for clustering and mean-field results, but the distinction between those results and the present overlap-gap mechanism could be made more explicit, especially regarding which results are new and which are refinements of prior work.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular reduction found: fixed-point existence, stability, and lifetime scaling follow directly from the model definitions; the finite-beta Gaussian-onset claim is an evidence-gap, not a circular argument.

full rationale

The paper's derivation chain is self-contained. Clustered fixed points are defined by Eq. (7); Eq. (8) gives inter-cluster attention; Eq. (10) shows leakage vanishes whenever the overlap gap stays bounded away from zero at fixed beta, a direct consequence of the logit scaling z=sqrt(d)q. Linear stability (Supplemental Eqs. 92-99) is computed from the update rule Eq. (4), yielding the contraction factor lambda_a=1-gamma n_a/(n_a-1), independent of any target result. The fragmentation lifetime Eq. (24) and the finite-size survival boundary Eq. (118) are derived from the leakage bound, not fitted. The alpha=beta*epsilon^2 collapse in Fig. 1 follows from Eq. (15) by definition of logits and noise variance, and is checked against simulations rather than used as an input. The REM benchmark is computed independently (Supplemental Sec. I) and used as a comparison, not as a premise. The central finite-beta onset for Gaussian initial conditions is asserted from Fig. 3 at T=N together with a heuristic feedback-amplification argument; the paper's own Eq. (118) concerning fragmented-state lifetime at finite N and T indicates the apparent onset could drift to zero, but that is an unsupported correctness claim, not a circular reduction: no equation in the paper assumes this onset in order to derive it. There is no fitted parameter renamed as a prediction, no load-bearing self-citation, and no ansatz smuggled in via citation, so the circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the model definitions plus standard concentration of measure for random unit vectors and a self-averaging assumption for the dynamics from random initial conditions. The self-averaging assumption is load-bearing for the finite-beta condensation onset and is not proven. The paper introduces no new physical entities; the overlap gap is a derived observable, not an invented degree of freedom.

free parameters (3)
  • gamma = 0.3
    Residual connection strength in Eq. (4), set to 0.3 throughout; quantitative curves depend on it, though the stability window is only 0<gamma<1.
  • qth = 1 - 10^-4
    Overlap threshold for defining clusters in numerics (Supplemental Sec. V.B); chosen with a robustness check against 1 - 10^-5.
  • observation time T = T = N
    Common observation time for all finite-size simulations; Gaussian-initiated observables plateau by then, but narrow-cone states in Fig. 9 still evolve.
assumptions (4)
  • standard math Concentration of measure for N random unit vectors in R^N: max_{a!=b} |c_a . c_b| = O(sqrt(log N / N)) with high probability.
    Used to establish the overlap gap for extensively fragmented states, Eq. (18)-(19) and Supplemental Eq. (107).
  • domain assumption The finite-N Gaussian initial condition self-averages: the dynamics of macroscopic observables is captured by a deterministic mean-field and fluctuations vanish as N -> infinity.
    Needed for the finite-beta condensation onset and for the O(log N) formation time; stated in Supplemental Sec. IV B, not proven for beta > 0.
  • domain assumption The relevant physical limit is N -> infinity before t -> infinity for stability of fragmented states.
    Invoked in 'Stability of extensive fragmentation', Eq. (25); if the opposite order is taken, finite-N coarsening restores consensus.
  • standard math The static random-energy-model benchmark describes frozen random logits: fixed beta gives diffuse attention and beta_REM ~ sqrt(log N).
    Used as the baseline in 'Dynamical accessibility' and Supplemental Sec. I; the paper's dynamic claim is the finite-beta improvement over this benchmark.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Clustered Attractor Manifolds and Dynamical Condensation in Self-Attention." pith.science (2026). https://pith.science/paper/Z7IUYV5Q

@misc{pith2026260808922,
  author       = {Pith},
  title        = {Pith review of: Clustered Attractor Manifolds and Dynamical Condensation in Self-Attention},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z7IUYV5Q}},
  note         = {Machine review of arXiv:2608.08922}
}
read the original abstract

Transformer layers generate state-dependent interaction networks: token representations determine the attention matrix, which in turn updates the representations. We study this feedback in a minimal normalized self-attention dynamics and identify the overlap gap as the central quantity governing its attractor structure in the thermodynamic limit. When tokens form internally aligned clusters and their similarity to members of the same cluster exceeds that to every other cluster by a nonvanishing amount, inter-cluster attention is exponentially suppressed as the dimension increases. This mechanism produces a high-dimensional manifold of clustered fixed points, ranging from a few macroscopic clusters to extensive microscopic fragmentation, and also controls their stability against perturbations. Starting from an unstructured Gaussian state, we find that clustered states nucleate from the diffuse background only above a finite threshold in attention sharpness, giving rise to a dynamical attention-condensation transition.

Figures

Figures reproduced from arXiv: 2608.08922 by the authors.

Figure 1
Figure 1. FIG. 1. Noise-induced fragmentation of a macroscopic cluster [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. FIG. 2. Scaling of the coarsening time for random pair [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. FIG. 3. Dynamical accessibility from normalized Gaussian initial conditions with [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (9 more)
Figure 1
Figure 1. Figure 1: FIG. 1. Formation times from Gaussian random initial states. For [PITH_FULL_IMAGE:figures/full_fig_p021_1.png]
Figure 2
Figure 2. Figure 2: FIG. 2. Logistic collapse of the polarization dynamics at [PITH_FULL_IMAGE:figures/full_fig_p022_2.png]
Figure 3
Figure 3. Figure 3: FIG. 3. Formation and sharpening of a macroscopic cluster for [PITH_FULL_IMAGE:figures/full_fig_p023_3.png]
Figure 4
Figure 4. Figure 4: FIG. 4. Snapshots of the dynamics under fixed three-dimensional random projections for [PITH_FULL_IMAGE:figures/full_fig_p024_4.png]
Figure 5
Figure 5. Figure 5: FIG. 5. Robustness of the cluster statistics with respect to the overlap threshold [PITH_FULL_IMAGE:figures/full_fig_p025_5.png]
Figure 6
Figure 6. Figure 6: FIG. 6. Attention, representation, and cluster observables for Gaussian initial conditions, measured at [PITH_FULL_IMAGE:figures/full_fig_p028_6.png]
Figure 7
Figure 7. Figure 7: FIG. 7. Observation-time dependence for Gaussian initial conditions at [PITH_FULL_IMAGE:figures/full_fig_p029_7.png]
Figure 8
Figure 8. Figure 8: FIG. 8. Attention, representation, and cluster observables for nearly rank-collapsed initial conditions with [PITH_FULL_IMAGE:figures/full_fig_p030_8.png]
Figure 9
Figure 9. Figure 9: FIG. 9. Observation-time dependence for nearly rank-collapsed initial conditions with [PITH_FULL_IMAGE:figures/full_fig_p031_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 20 canonical work pages

  1. [1]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, Attention is all you need, inAdvances in Neural Information Processing Systems, Vol. 30 (2017)

    work page 2017

  2. [2]

    Devlin, M.-W

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding, inProceedings of the 2019 Conference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Language Technologies(2019) pp. 4171–4186

    work page 2019

  3. [3]

    T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Ka- plan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sas- try, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. ...

    work page 2020

  4. [4]

    Dosovitskiy, L

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, An image is worth 16x16 words: Transformers for image recog- nition at scale, inInternational Conference on Learning Representations(2021)

    work page 2021

  5. [5]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, Learning transferable visual models from natural language supervision, inProceedings of the 38th International Conference on Machine Learn- ing, Proceedings of Machine Learning Research, Vol. 139 (PMLR, 2021) pp. 8748–8763

    work page 2021

  6. [6]

    Jumper, R

    J. Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool, R. Bates, A. ˇZ ´ ıdek, A. Potapenko, A. Bridgland, C. Meyer, S. A. A. Kohl, A. J. Ballard, A. Cowie, B. Romera-Paredes, S. Nikolov, R. Jain, J. Adler, T. Back, S. Petersen, D. Reiman, E. Clancy, M. Zielinski, M. Steinegger, M. Pacholska, T. Berghammer, S. Bodenste...

    work page 2021

  7. [7]

    Y. Lu, Z. Li, D. He, Z. Sun, B. Dong, T. Qin, L. Wang, and T.-Y. Liu, Understanding and improving transformer from a multi-particle dynamic system point of view, arXiv preprint arXiv:1906.02762 (2019)

    arXiv 2019

  8. [8]

    S. Bai, J. Z. Kolter, and V. Koltun, Deep equilibrium models, inAdvances in Neural Information Processing Systems, Vol. 32 (2019)

    work page 2019

Show all 29 references
  1. [9]

    Dong, J.-B

    Y. Dong, J.-B. Cordonnier, and A. Loukas, Attention is not all you need: Pure attention loses rank doubly exponentially with depth, inProceedings of the 38th In- ternational Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 139 (PMLR, 2021) pp. 2793–2803

    2021

  2. [10]

    L. Noci, S. Anagnostidis, L. Biggio, A. Orvieto, S. P. Singh, and A. Lucchi, Signal propagation in transformers: Theoretical perspectives and the role of rank collapse, inAdvances in Neural Information Processing Systems, Vol. 35 (2022) pp. 27198–27211

    2022

  3. [11]

    S. Zhai, T. Likhomanenko, E. Littwin, D. Busbridge, J. Ramapuram, Y. Zhang, J. Gu, and J. M. Susskind, 6 Stabilizing transformer training by preventing attention entropy collapse, inProceedings of the 40th International Conference on Machine Learning, Proceedings of Machine Le...

    2023

  4. [12]

    G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis, Ef- ficient streaming language models with attention sinks, inInternational Conference on Learning Representations (2024)

    2024

  5. [14]

    Geshkovski, C

    B. Geshkovski, C. Letrouit, Y. Polyanskiy, and P. Rigollet, A mathematical perspective on transformers, Bulletin of the American Mathematical Society62, 427 (2025), arXiv:2312.10794 [cs.LG]

    2025 arXiv

  6. [15]

    Burger, S

    M. Burger, S. Kabri, Y. Korolev, T. Roith, and L. Weigand, Analysis of mean-field models arising from self-attention dynamics in transformer architectures with layer normalization, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineer- ing Scie...

    2025 arXiv

  7. [16]

    Rigollet, The mean-field dynamics of transformers, in Proceedings of the International Congress of Mathemati- cians 2026—Volume 7: Invited Lectures (Sections 15–20), edited by S

    P. Rigollet, The mean-field dynamics of transformers, in Proceedings of the International Congress of Mathemati- cians 2026—Volume 7: Invited Lectures (Sections 15–20), edited by S. Friedlander and Y. Tschinkel (Society for Industrial and Applied Mathematics, 2026) pp. 389–404

    2026

  8. [17]

    Altafini, Multistability of self-attention dynamics in transformers, arXiv preprint arXiv:2511.11553 (2025), arXiv:2511.11553 [cs.LG]

    C. Altafini, Multistability of self-attention dynamics in transformers, arXiv preprint arXiv:2511.11553 (2025), arXiv:2511.11553 [cs.LG]

    2025

  9. [18]

    Karagodin, Y

    N. Karagodin, Y. Polyanskiy, and P. Rigollet, Clustering in causal attention masking, inAdvances in Neural Infor- mation Processing Systems, Vol. 37 (Curran Associates, Inc., 2024) arXiv:2402.11180 [cs.LG]

    2024 arXiv

  10. [19]

    Karagodin, S

    N. Karagodin, S. Ge, Y. Polyanskiy, and P. Rigollet, Nor- malization in attention dynamics, inAdvances in Neural Information Processing Systems, Vol. 38 (Curran Asso- ciates, Inc., 2025) arXiv:2505.15290 [cs.LG]

    2025 arXiv

  11. [20]

    Bruno, F

    G. Bruno, F. Pasqualotto, and A. Agazzi, A multiscale analysis of mean-field transformers in the moderate inter- action regime, inAdvances in Neural Information Pro- cessing Systems, Vol. 38 (Curran Associates, Inc., 2025) arXiv:2509.25040 [math.PR]

    2025

  12. [21]

    Giorlandino and S

    A. Giorlandino and S. Goldt, Two failure modes of deep transformers and how to avoid them: A unified the- ory of signal propagation at initialisation, inInterna- tional Conference on Learning Representations(2026) arXiv:2505.24333 [cs.LG]

    2026 arXiv

  13. [23]

    See Supplemental Material for the static REM benchmark, finite-dimensional scaling and non-softmax attention ker- nels, clustered fixed points and their stability, the dy- namical formation of macroscopic clusters, and additional numerical results

  14. [26]

    Katharopoulos, A

    A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret, Transformers are RNNs: Fast autoregressive transformers with linear attention, inProceedings of the 37th Interna- tional Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 119 (PMLR, 2020) pp. 5156–5165

    2020

  15. [28]

    Derrida, Random-energy model: An exactly solvable model of disordered systems, Physical Review B24, 2613 (1981)

    B. Derrida, Random-energy model: An exactly solvable model of disordered systems, Physical Review B24, 2613 (1981)

    1981

  16. [29]

    Geshkovski, C

    B. Geshkovski, C. Letrouit, Y. Polyanskiy, and P. Rigollet, The emergence of clusters in self-attention dynamics, inAdvances in Neural Information Processing Systems, Vol. 36 (Curran Associates, Inc., 2023) pp. 57026–57037, arXiv:2305.05465 [cs.LG]

    2023 arXiv

  17. [30]

    Geshkovski, H

    B. Geshkovski, H. Koubbi, Y. Polyanskiy, and P. Rigollet, Dynamic metastability in the self-attention model, arXiv preprint arXiv:2410.06833 (2024)

    2024 arXiv

  18. [31]

    Bruno, F

    G. Bruno, F. Pasqualotto, and A. Agazzi, Emergence of meta-stable clustering in mean-field transformer models, inThe Thirteenth International Conference on Learning Representations(2025)

    2025

  19. [32]

    Katharopoulos, A

    A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret, Transformers are RNNs: Fast autoregressive transformers with linear attention, inProceedings of the 37th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 119 (PMLR, 2020) pp. 5156–5165

    2020

  20. [33]

    Choromanski, V

    K. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlos, P. Hawkins, J. Davis, A. Mohiuddin, L. Kaiser, D. Belanger, L. Colwell, and A. Weller, Rethinking attention with performers, inInternational Conference on Learning Representations(2021) arXiv:2009.14794 [cs.LG]

    2021 arXiv

  21. [34]

    Rigollet, The mean-field dynamics of transformers, inProceedings of the International Congress of Mathematicians 2026—Volume 7: Invited Lectures (Sections 15–20), edited by S

    P. Rigollet, The mean-field dynamics of transformers, inProceedings of the International Congress of Mathematicians 2026—Volume 7: Invited Lectures (Sections 15–20), edited by S. Friedlander and Y. Tschinkel (Society for Industrial and Applied Mathematics, 2026) pp. 389–404. 2...

    2026

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.