Gated attention enables non-flat and positively curved geometries in the Fisher-Rao manifold of representations that ungated attention cannot achieve.
Are transformers universal approximators of sequence-to-sequence functions? arXiv preprint arXiv:1912.10077
8 Pith papers cite this work. Polarity classification is still indexing.
8
Pith papers citing it
representative citing papers
Third-order co-skewness in fMRI is destroyed by BFM pretraining, causing poor cognition prediction; a co-skewness-preserving linear FC exceeds BFMs and raw FC.
Kerr-soliton attention realizes transformer attention in physical hardware via Kerr solitons in a resonator, with analytic training and experimental inference showing high-fidelity agreement between hardware and model.
Attention pooling produces a free-multiplicative-convolution bulk spectrum and two phase transitions for signal recovery; optimal weights are the top eigenvector of the positional correlation matrix R.
Lipschitz continuous transformations F of probability measures w.r.t. Wasserstein distance admit continuous transport maps f(·,μ) such that F(μ) = f(·,μ)_# μ.
One of the Q, K or V weights in transformer self-attention is redundant and replaceable by the identity matrix under mild assumptions, reducing parameters by 25 percent with no loss in small-model performance.
In a cellular automata rule-inference task designed to block memorization, neural models achieve high next-step accuracy but accuracy falls sharply with longer reasoning chains; depth, recurrence, memory, and test-time compute extend the reachable depth but do not remove the bound.
Residual networks admit progressive approximation trajectories with monotonically decreasing error, enabling useful predictions from any depth after a single training run via the LPA principle.
citing papers explorer
-
How Does Attention Help? Insights from Random Matrices on Signal Recovery from Sequence Models
Attention pooling produces a free-multiplicative-convolution bulk spectrum and two phase transitions for signal recovery; optimal weights are the top eigenvector of the positional correlation matrix R.