arxiv: 2604.26085 · v1 · submitted 2026-04-28 · 🧮 math.DS · math.CA

Recognition: unknown

Spectral Selection in Symmetric Self-Attention Dynamics

Christian Kuehn, Jaeyoung Yoon

Pith reviewed 2026-05-07 14:29 UTC · model grok-4.3

classification 🧮 math.DS math.CA MSC 37N99

keywords self-attention dynamicsspectral selectiongradient flow on sphereeigenvalue dominancesign-split polarizationinteracting particle systemasymptotic mode selection

0 comments

The pith

Symmetric self-attention flows on the sphere select either full alignment to a dominant positive eigenvector or sign-split polarization to the most negative one.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper analyzes an idealized self-attention update as a system of particles moving on the unit sphere. When the weight matrices obey the symmetry condition that makes Q transpose K equal to a symmetric matrix V, the update becomes a gradient flow that can be rewritten exactly in the eigenbasis of V. This rewriting exposes a clean spectral selection rule: the particles converge either to the eigenvector of the largest positive eigenvalue when that eigenvalue strictly dominates in size, or they split evenly into opposite signs along the eigenvector of the most negative eigenvalue when V is negative definite. The analysis supplies both local stability conditions for these pure-mode states and global convergence statements in each regime.

Core claim

Under the symmetry Q^T K = V = V^T the self-attention particle system on the sphere admits an exact gradient-flow representation that diagonalizes in the eigenbasis of V. This structure produces two distinct long-term behaviors: homogeneous alignment to the unique dominant positive eigendirection whenever one positive eigenvalue exceeds all others in modulus, and sign-split polarization to the eigenvector belonging to the eigenvalue of largest negative magnitude whenever V is negative definite. Local stability criteria for the corresponding pure-mode equilibria are derived, and global selection of these modes is proved in both regimes.

What carries the argument

Exact reformulation of the flow in the eigenbasis of the symmetric matrix V, which decouples the modes and makes spectral dominance control the selection between alignment and polarization.

Load-bearing premise

The weight matrices satisfy the symmetry condition that Q transpose K equals a symmetric matrix V.

What would settle it

A direct numerical integration of the particle ODE under the imposed symmetry Q^T K = V = V^T with a matrix having one strictly dominant positive eigenvalue, in which the particles fail to converge to the corresponding eigenvector.

Figures

Figures reproduced from arXiv: 2604.26085 by Christian Kuehn, Jaeyoung Yoon.

**Figure 1.** Figure 1: Snapshots of the particle system in the positive-dominant regime at times view at source ↗

**Figure 2.** Figure 2: Time evolution of ρmin(t) := mini,j∈[n] ⟨xi(t), xj (t)⟩ for a fixed matrix V with dominant positive eigenvalue λ1. The values ρmin = 1 and −1 correspond to homogeneous alignment and to the presence of at least one antipodal pair, respectively. Each curve corresponds to one trial, and the same collection of initial data is used across the three panels. The particle number is n = 80, and each panel contains… view at source ↗

**Figure 3.** Figure 3: Numerical illustration of the two-particle negative-definite regime. (a) shows the view at source ↗

**Figure 4.** Figure 4: Monte Carlo observables for the negative-definite regime with view at source ↗

**Figure 5.** Figure 5: Representative β-dependent stability boundary for sign-split pure states, shown for λp = 1. The plotted curve is the upper bound λpσ(cβ, r) = σ(e 2β , r). Different line styles correspond to different population ratios r = n+/n−. The empty circles indicate the threshold points β = 1 2 | ln r|, where the admissible stability regime begins. For r = 1, the threshold point at β = 0 is interpreted as a limiting… view at source ↗

read the original abstract

We study self-attention dynamics on the unit sphere as an interacting particle system arising from an idealized Transformer-type update. Under a symmetry assumption on weight matrices given by $Q^\top K=V=V^\top$, the flow admits a gradient-flow structure and an exact reformulation in the eigenbasis of $V$, revealing a spectral mode-selection mechanism. We show that the dynamics exhibits two distinct asymptotic scenarios: homogeneous alignment toward the dominant eigendirection when one positive eigenvalue strictly dominates all others in modulus, and sign-split polarization toward the most negative eigendirection when $V$ is negative definite. In particular, we obtain local stability criteria for pure-mode equilibria and global selection results in both regimes. These results provide a rigorous finite-particle description of how the spectrum of the weight matrices organizes asymptotic patterns in a symmetric self-attention flow, and highlight how the symmetric setting renders the dynamics amenable to mathematical analysis.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The paper gives an exact eigenbasis reduction for symmetric self-attention flows that cleanly separates two asymptotic regimes: dominant-positive alignment or negative-definite sign-split polarization.

read the letter

The central new piece is the exact change of variables to the eigenbasis of the symmetric matrix V = Q^T K. This turns the particle system on the sphere into decoupled modal dynamics and yields two explicit long-term pictures: homogeneous alignment to the strictly dominant positive eigenvalue, or sign-split polarization to the most negative eigenvalue when V is negative definite. They also supply local stability criteria for the pure-mode equilibria and global selection statements in each case. The symmetry assumption directly produces the gradient-flow structure, and the reduction looks internally consistent with no obvious cross terms left behind. The finite-particle treatment is handled without mean-field limits, which is a plus for rigor. The main limitation is that the symmetry Q^T K = V = V^T is quite restrictive and may not survive in generic trained models, so the results function more as an idealized case study than a general explanation. Global selection also appears to rest on the strict dominance or negative-definiteness hypotheses; it would help to see how sensitive the conclusions are to small perturbations of those conditions. This work is aimed at dynamical-systems people who study attention or interacting particle models. A reader who wants precise organizing principles for asymptotic patterns in symmetric flows will find it useful. The derivations are clear enough that the paper deserves referee time to check the proofs and the scope of the global results.

Referee Report

0 major / 3 minor

Summary. The manuscript analyzes self-attention dynamics on the unit sphere as an interacting particle system derived from an idealized Transformer update. Under the symmetry assumption Q^⊤K = V = V^⊤, the flow is shown to admit a gradient-flow structure with an exact reformulation in the eigenbasis of V. This yields two distinct asymptotic regimes: homogeneous alignment toward the dominant positive eigendirection when one positive eigenvalue strictly dominates in modulus, and sign-split polarization toward the most negative eigendirection when V is negative definite, together with local stability criteria for pure-mode equilibria and global selection results in both regimes.

Significance. If the derivations hold, the work supplies a rigorous finite-particle description of how the spectrum of the weight matrices organizes long-term patterns in a symmetric self-attention flow. The gradient-flow structure and exact eigenbasis change of variables are genuine strengths that separate modes cleanly and make the local/global analysis tractable; these features distinguish the contribution from purely numerical or heuristic studies of attention dynamics.

minor comments (3)

§2: the precise statement of the symmetry assumption Q^⊤K = V = V^⊤ and the resulting gradient structure should be isolated as a numbered proposition or lemma to make the subsequent eigenbasis reduction easier to reference.
§4.1: the local stability criteria for pure-mode equilibria are stated in terms of eigenvalue dominance, but the linearization matrix is not displayed explicitly; adding the Jacobian at the equilibrium would clarify the spectral gap argument.
The global selection theorems would benefit from a short remark on the measure of the exceptional initial conditions (if any) that fail to converge to the predicted attractor.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for the positive assessment of our manuscript on spectral selection in symmetric self-attention dynamics and for recommending minor revision. The referee's summary accurately reflects the gradient-flow structure, eigenbasis reformulation, and the two distinct asymptotic regimes we analyze. Since no specific major comments were raised in the report, we have no point-by-point rebuttals to provide at this time.

Circularity Check

0 steps flagged

No significant circularity in derivation chain

full rationale

The paper's central results follow from the symmetry assumption Q^T K = V = V^T, which directly induces a gradient-flow structure and allows an exact change of variables to the eigenbasis of V. The asymptotic scenarios (homogeneous alignment for dominant positive eigenvalue and sign-split polarization for negative definite V) are then derived from the reduced dynamics on the sphere, with local stability criteria and global selection results obtained without any fitted parameters, self-referential predictions, or load-bearing self-citations. The derivation is self-contained under the stated hypotheses, as the mode-selection mechanism emerges mathematically from the reformulated system rather than being presupposed.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claims rest on the single symmetry assumption that enables the gradient-flow structure and eigenbasis decoupling. No free parameters, additional axioms, or invented entities are mentioned.

axioms (1)

domain assumption Symmetry assumption Q^⊤K = V = V^⊤
Invoked to obtain gradient-flow structure and exact eigenbasis reformulation of the flow.

pith-pipeline@v0.9.0 · 5446 in / 1185 out tokens · 55540 ms · 2026-05-07T14:29:11.358469+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

30 extracted references · 8 canonical work pages · 2 internal anchors

[1]

Acebr´ on, L

J. Acebr´ on, L. Bonilla, C. Vicente, F. Ritort, and R. Spigler. The Kuramoto model: A simple paradigm for synchronization phenomena.Rev. Mod. Phys., 77(1):137–185, 2005

2005
[2]

Bruno, F

G. Bruno, F. Pasqualotto, and A. Agazzi. Emergence of meta-stable clustering in mean-field transformer models.arXiv preprint arXiv:2410.23228, 2024

work page arXiv 2024
[3]

Bruno, F

G. Bruno, F. Pasqualotto, and A. Agazzi. A multiscale analysis of mean-field trans- formers in the moderate interaction regime.Advances in Neural Information Processing Systems, 2025

2025
[4]

Burger, S

M. Burger, S. Kabri, Y. Korolev, T. Roith, and L. Weigand. Analysis of mean-field models arising from self-attention dynamics in transformer architectures with layer nor- malization.Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 383(2298):20240233, 2025

2025
[5]

A Unified Perspective on the Dynamics of Deep Transformers.arXiv preprint arXiv:2501.18322, 2025

V. Castin, P. Ablin, J. A. Carrillo, and G. Peyr´ e. A unified perspective on the dynamics of deep transformers.arXiv preprint arXiv:2501.18322, 2025. 31

work page arXiv 2025
[6]

Chemnitz, M

D. Chemnitz, M. Engel, C. Kuehn, and S. Kuntz. A dynamical systems perspective on the analysis of neural networks.arXiv:2507.05164, 2025

work page arXiv 2025
[7]

R. T. Q. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud. Neural ordinary differential equations. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett, editors,Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018

2018
[8]

S. Chen, Z. Lin, Y. Polyanskiy, and P. Rigollet. Quantitative clustering in mean-field transformer models.arXiv preprint arXiv:2504.14697, 2025

work page internal anchor Pith review arXiv 2025
[9]

M. A. Cohen and S. Grossberg. Absolute stability of global pattern formation and par- allel memory storage by competitive neural networks.IEEE Transactions on Systems, Man, and Cybernetics, SMC-13(5):815–826, 1983

1983
[10]

W. E. A proposal on machine learning via dynamical systems.Communications in Mathematics and Statistics, 5:1–11, 2017

2017
[11]

Dynamic metastability in the self-attention model.arXiv preprint arXiv:2410.06833, 2024

B. Geshkovski, H. Koubbi, Y. Polyanskiy, and P. Rigollet. Dynamic metastability in the self-attention model.arXiv preprint arXiv:2410.06833, 2024

work page arXiv 2024
[12]

Geshkovski, C

B. Geshkovski, C. Letrouit, Y. Polyanskiy, and P. Rigollet. The emergence of clusters in self-attention dynamics. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors,Advances in Neural Information Processing Systems, volume 36. Curran Associates, Inc., 2023

2023
[13]

Geshkovski, C

B. Geshkovski, C. Letrouit, Y. Polyanskiy, and P. Rigollet. A mathematical perspective on transformers.Bulletin of the American Mathematical Society, 62:427–479, 2025

2025
[14]

Haber and L

E. Haber and L. Ruthotto. Stable architectures for deep neural networks.Inverse Problems, 34(1):014004, 2018

2018
[15]

Hofbauer and K

J. Hofbauer and K. Sigmund.Evolutionary Games and Population Dynamics. Cam- bridge University Press, 1998

1998
[16]

J. J. Hopfield. Neural networks and physical systems with emergent collective compu- tational abilities.Proceedings of the National Academy of Sciences, 79(8):2554–2558, 1982

1982
[17]

Karagodin, S

N. Karagodin, S. Ge, Y. Polyanskiy, and P. Rigollet. Normalization in attention dy- namics. InAdvances in Neural Information Processing Systems, 2025

2025
[18]

Kuehn and S.-V

C. Kuehn and S.-V. Kuntz. Embedding capabilities of neural odes.arXiv preprint arXiv:2308.01213, 2023

work page arXiv 2023
[19]

Kuehn and S.-V

C. Kuehn and S.-V. Kuntz. Analysis of the geometric structure of neural networks and neural odes via morse functions.Advances in Computational Mathematics, 52(1), 2026

2026
[20]

Kuramoto.Chemical Oscillations, Waves, and Turbulence

Y. Kuramoto.Chemical Oscillations, Waves, and Turbulence. Dover Publications, 1984. 32

1984
[21]

T. Lin, Y. Wang, X. Liu, and X. Qiu. A survey of transformers.AI Open, 3:111–132, 2022

2022
[22]

Pikovsky, M

A. Pikovsky, M. Rosenblum, and J. Kurths.Synchronization. Cambridge University Press, 2001

2001
[23]

The Mean-Field Dynamics of Transformers.arXiv preprint arXiv:2512.01868, 2025

P. Rigollet. The mean-field dynamics of transformers.arXiv preprint arXiv:2512.01868, 2025

work page arXiv 2025
[24]

Ruthotto and E

L. Ruthotto and E. Haber. Deep neural networks motivated by partial differential equations.Journal of Mathematical Imaging and Vision, 62(3):352–364, 2020

2020
[25]

W. H. Sandholm.Population games and evolutionary dynamics. MIT Press, 2010

2010
[26]

E. D. Sontag. A learning result for continuous-time recurrent neural networks.Systems & Control Letters, 34(3):151–158, 1998

1998
[27]

Strogatz

S. Strogatz. From Kuramoto to Crawford: exploring the onset of synchronization in populations of coupled oscillators.Physica D, 143:1–20, 2000

2000
[28]

LLaMA: Open and Efficient Foundation Language Models

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi` ere, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

work page internal anchor Pith review arXiv 2023
[29]

Vaswani, N

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors,Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017

2017
[30]

Zhang and R

B. Zhang and R. Sennrich. Root mean square layer normalization. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alch´ e-Buc, E. Fox, and R. Garnett, editors,Ad- vances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. A Proofs for Section 5 This appendix contains the technical proofs omitted from Section 5. A.1 Lineari...

2019