Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

Is In-Context Universality Enough? MLPs are Also Universal In-Context

T0 review · 3 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper proves that multilayer perceptrons match transformers in in-context universality, so approximation power does not explain the transformer's edge.

desk verdict The core claim is likely true but the proof has a concrete gap in Lemma 9 that must be fixed before the theorem is sound. read the letter →

arxiv 2502.03327 v1 pith:LPKFDEF2 submitted 2025-02-05 stat.ML cs.LGcs.NAcs.NEmath.NAmath.PR

Anastasis Kratsios , Takashi Furuya This is my paper · ORCID
classification stat.MLcs.LGcs.NAcs.NEmath.NAmath.PR MSC 68T0741A3049Q22
keywords in-contextlearninguniversalapproximationmultilayerperceptronstransformerspermutation-invariantcontexts1-WassersteindistanceReLUnetworksoptimaltransport
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether the transformer's ability to learn in context — adapting to a task from a prompt of examples without retraining — gives it an approximation-theoretic advantage over the classical multilayer perceptron (MLP). Recent results showed that transformers can approximate any continuous function of a context (a probability measure over tokens) and a query, uniformly on compact sets. The paper proves that MLPs with trainable activation functions are universal in this same permutation-invariant in-context sense: for any uniformly continuous context-to-prediction map and any tolerance, a ReLU-MLP approximates it to within the tolerance on a set of contexts of probability at least $\gtrsim 1-(\delta^q-\delta_*^q)$, with explicit depth and width bounds. The conclusion is negative for the transformer: in-context universality does not distinguish transformers from MLPs, so the transformer's empirical success must be explained by inductive bias, optimization, or training dynamics rather than by expressive power. The proof also yields a quantitative in-context universality theorem for multi-head transformers as a corollary.

What carries the argument

The central object is the space of permutation-invariant contexts (PICs): probability measures $\mu = \sum_{n=1}^N w_n \delta_{x_n}$ on $\mathbb{R}^d$ whose weights live on a discretized simplex (multiples of $1/C$, so that no token exceeds the context-window capacity), realized as row-permutation equivalence classes of $N \times (d+1)$ matrices and metrized by the 1-Wasserstein distance $W_1$. The argument is carried by four mechanisms. First, a non-Euclidean version of the classical 'trifling regions' decomposition cuts the compact context space $K$ into retracted Voronoi cells, leaving only annuli of total measure at most $CK(\delta^q-\delta_*^q)$ where approximation may fail. Second, an optimal piecewise-constant approximator assigns each cell the value of $f$ at its landmark, achieving uniform error at most $\omega(\delta)$. Third, exact MLP implementations of the ingredients — the 1-Wasserstein distance via the linear program over a transport polytope, squared $\ell^2$ norms, bump functions, coordinatewise multiplication, and componentwise minima — let a ReLU MLP reproduce the 0/1 cell indicators on the approximation region. Fourth, the 'transformerification' procedure converts the resulting MLP into a multi-head transformer with the same depth and width and at most twice the trainable parameters, so the MLP approximation bound transfers verbatim to the transformer.

What would settle it

A concrete check is to compute the detector mismatch in Lemma 9. In $\mathbb{R}^2$, with the context equal to the landmark measure and $x - x_k = (\sqrt{\delta_*/2}, \sqrt{\delta_*/2})$, one has $W_1(\mu,\mu_k) + \|x-x_k\|_2^2 = \delta_*$ while $W_1(\mu,\mu_k) + \|x-x_k\|_1 = \sqrt{2\delta_*} > \delta_*$; the defining cell test fails where the implemented detector fires, so the claimed 0/1 identity (14) is not a consequence of the ingredients as written, and the recursion's 'off' value requires $\delta < 1$. Measuring the $P$-mass of the sliver $\{W_1(\mu,\mu_k)+\|x-x_k\|_1 > \delta_* \ge W_1(\mu,\mu_k)+\|x-x_k\|_2^2\}$ inside the approximation region $K \setminus K^{\delta_*}$ settles the question: positive mass means the piecewise-constant approximator can assign a cell more than total probability mass 1, breaking the main bound; zero mass (or containment in the trifling annulus) would indicate a repairable gap.

Watch

Extended reading notes

Core claim

The central claim is Theorem 4. Fix a finite context window $C$, a token budget $N$, and a compact set $K$ of permutation-invariant contexts paired with queries, equipped with a probability measure $P$ that is Ahlfors $q$-regular (assigning mass comparable to $r^q$ to every ball of radius $r$). For any uniformly continuous target $f$ from $K$ into probability measures over outputs, with modulus of continuity $\omega$, and any tolerance $\delta \le \mathrm{diam}(K)$, there is a ReLU MLP $\hat f$ such that $W_1(f(\mu,x),\hat f(\mu,x)) \le \omega(\delta)$ on a subset of $K$ of $P$-measure $\gtrsim 1-(\delta^q-\delta_*^q)$, with depth $O(dN^{2C}/\omega^{-1}(\varepsilon)^{2q})$ and width $O(dN^{2C}/\omega^{-1}(\varepsilon)^q)$ when the ambient dimension satisfies $N^N \le d$. The construction produces a piecewise-constant approximator on a 'retracted Voronoi' decomposition of $K$ whose exposed 'trifling region' has small measure, implements the cell indicators with an MLP that exactly computes the 1-Wasserstein distance between weighted empirical measures, and assembles the whole via standard network operations. Because the same MLP can be rewritten as a multi-head transformer with identical depth and width and at most double the trainable parameters, the paper obtains a quantitative in-context universality statement for transformers as a corollary, and its overall message is that in-context universality is not a transformer privilege.

Load-bearing premise

The load-bearing premise is that the MLP can detect, with exact 0/1 outputs, whether a context-and-query pair lies in a given cell of the decomposition; but the cells are defined using one distance (Wasserstein-1 plus $\ell^1$ on the query), while the implemented detectors use another (Wasserstein-1 plus squared $\ell^2$), the proof never reconciles the two, and the recursion additionally needs the tolerance $\delta$ to be smaller than 1.

Editorial extensions

If this is right

  • If Theorem 4 is right, in-context universality gives transformers no approximation-theoretic edge over MLPs in the permutation-invariant setting, so the transformer's practical success must be attributed to inductive bias, optimization, or training dynamics.
  • The exact 1-Wasserstein implementation (Proposition 20 and Lemma 21) means ReLU MLPs can serve as built-in Wasserstein-1 solvers on weighted empirical measures with $N$ support points, a reusable primitive for neural optimal transport.
  • Corollary 5 transfers the MLP guarantee to multi-head transformers with $N$ attention heads per block and matching rate bounds, a quantitative sharpening of the earlier transformer universality result.
  • The depth and width bounds $O(dN^{2C}/\omega^{-1}(\varepsilon)^{2q})$ and $O(dN^{2C}/\omega^{-1}(\varepsilon)^q)$ make the approximation cost explicit in the token number $N$, the context window $C$, and the metric dimension $q$ of the context space.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's equivalence is stated for permutation-invariant contexts and for worst-case uniformly continuous targets; nothing in the theorems covers order-sensitive tasks or finite-sample behavior, so the natural reading is that approximation capacity is not where the transformer's advantage lives — not that MLPs will match transformers in practice.
  • The constructive widths are enormous (they involve factorial terms before the high-dimensional simplification $N^N \le d$), so the result is an existence statement about expressivity rather than a recipe for training; a useful next step would be matching lower bounds or sparse constructions showing when such sizes are necessary.
  • An empirical test of the paper's spirit is available: on permutation-invariant in-context tasks, transformer and MLP approximations of matched parameter count should approach the same limiting error as capacity grows; if instead they saturate at different levels, that gap would point to optimization or inductive-bias effects, exactly the factors the paper names as the real differentiators.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper addresses whether in-context universality explains the empirical advantage of transformers over MLPs. It proves that MLPs with the trainable activation function in (8) are universal in-context approximators for permutation-invariant contexts (PICs): Theorem 4 states that, on a compact set K of PICs and queries equipped with an Ahlfors q-regular measure, any uniformly continuous target f into a Wasserstein space of probability measures can be approximated by an MLP up to a modulus-of-continuity error with high probability outside a small ``trifling'' region, with explicit depth/width bounds. Corollary 5 converts this MLP into a multi-head transformer via a ``transformerification'' result. The proof constructs retracted Voronoi cells, a piecewise-constant approximator, exact MLP implementations of the 1-Wasserstein distance for the restricted context spaces, and pseudo-indicator MLPs for the cells.

Significance. If the main theorem is correct, the paper delivers a conceptually important negative answer: in-context universality does not separate transformers from MLPs, so the transformer's success must be explained by inductive bias, training dynamics, or other properties. The paper has several notable strengths: Lemma 21 gives an exact MLP implementation of the 1-Wasserstein distance on P_{C,N}, the approximation rates are quantified through metric doubling dimension and Ahlfors regularity rather than left asymptotic, and Proposition 11 provides a clean depth/width-preserving conversion from MLPs to multi-head transformers. The construction is substantive and goes beyond a trivial reduction to classical MLP universality. However, the central constructive step, Lemma 9, has specific technical gaps that affect Theorem 4 as stated; these appear repairable, but they are load-bearing rather than cosmetic.

major comments (3)
  1. [4.3, Lemma 9, Eqs. (21)-(24)] Equations (21)-(24) do not implement the indicator of the cells defined in (11). The cells C^{\delta*}_k are defined in Section 2.1.1 from balls B((\mu_k,x_k),r) = {(\nu,z): W1(\mu_k,\nu)+\|z-x_k\|_1<r}, but the proof applies the pseudo-indicator \phi_{\delta*,\delta} to W1(\mu,\mu_k)+\|x-x_k\|_2^2. For example, with d=1, \mu=\mu_1, \delta*=0.5, and |x-x_1|=0.6, the point is outside B((\mu_1,x_1),\delta*) because 0.6>0.5, yet \phi(0.36)=1 because 0.36<0.5. The proof must either use the \ell_1 norm, which Lemma 18 already implements, or redefine the metric and cells with \|\cdot\|_2^2 throughout. Since Lemma 10 constructs \hat f_\delta as the sum of \nu_k \tilde{\Phi}_k and relies on \tilde{\Phi}_k being the cell indicators, this mismatch breaks the constructive proof of Theorem 4 as written.
  2. [4.3, Lemma 9, recursive step] The recursive factor \phi_{\delta*,\delta}(1-\tilde{\Phi}_j(\mu,x)) equals zero only when |1|>\delta, i.e. when \delta<1. For \delta\geq 1, which the theorem permits because \delta\leq diam(K), the product does not necessarily vanish outside earlier cells and \tilde{\Phi}_k no longer equals the indicator of C^{\delta*}_k. The statement of Lemma 9 and Theorem 4 need an explicit \delta<1 condition, or the pseudo-indicator needs a second breakpoint above 1, or the domain must be rescaled so that diam(K)<1. Without this, the recursion in Lemma 9 does not produce the claimed partition of unity.
  3. [4.4, Lemma 10, proof of part (iii)] The proof of Lemma 10(iii) bounds W1(f_\delta(\mu,x),\hat f_\delta(\mu,x)) by \|\nu_k\|_{KR}, but the Kantorovich-Rubinstein norm on probability measures is never defined in the manuscript. Since the argument also uses the identity that the sets C^\delta_k \setminus C^{\delta*}_k partition the trifling region K_{\delta*}, this step depends on the correct cell indicators from Lemma 9. The norm should be defined (or replaced by a bound in terms of the Wasserstein diameter of f(K)), and the partitioning claim should be stated and proved explicitly, especially because Lemma 9 is currently the source of the metric and \delta<1 defects.
minor comments (5)
  1. [Theorem 4] The theorem statement contains a duplicated phrase: it first fixes a uniformly continuous f and then says ``For each \omega-uniformly continuous contextual mapping f''; one of these should be removed. In addition, the width/depth estimates are written with \varepsilon, but \varepsilon is not defined in the statement; the rates should use the stated approximation error \delta or introduce \varepsilon consistently.
  2. [Corollary 5] The displayed complexity bounds refer to ``the depth and width of \hat f'' but the object being constructed is the transformer \hat T; this is a typographical slip that should be corrected.
  3. [Proposition 2] The text says the map \Phi is a ``homomorphism''; in context this should be ``homeomorphism'' or ``isometry'' depending on the intended statement.
  4. [Appendix C.1, Lemma 6 proof] In the chain of inequalities, line (19) writes ``K C\delta^p - c\delta*^p'' with exponent p, but the rest of the proof uses q; the exponent should be q throughout.
  5. [Appendix C.2, Lemma 7 proof] The proof of Lemma 7(i) is only a one-sentence assertion and does not actually derive the separation bound \delta-\delta*. The bound does follow from the recursive deletion of earlier \delta-balls, but the proof should spell out the triangle inequality argument.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the MLP in-context universality theorem is a self-contained constructive proof; the Lemma 9 metric mismatch is a correctness bug, not a circular step.

full rationale

The derivation is a constructive approximation-theory proof. Theorem 4 is obtained by (i) a packing/covering decomposition of K into retracted Voronoi cells (Lemmas 6–7), (ii) a piecewise-constant oracle approximator fδ (Lemma 8), (iii) exact implementation of W1 and the cell indicators by ReLU/ReQU MLPs (Lemma 9, via Lemmas 18 and 21), and (iv) summation over cells (Lemma 10). None of these steps fits a parameter to data or defines the target function in terms of the network; the network is explicitly constructed from f and the partition. The self-citations (e.g., Furuya and Kratsios 2024 for exact polynomial/ReQU multiplication, and Furuya et al. 2024 for the PIC formalism) supply independent lemmas that do not assume the MLP-universality conclusion, so they are not load-bearing in a circular way. The transformerification procedure (Proposition 11) is also a constructive reduction from MLPs to transformers, not a premise. Separately, the proof has a genuine correctness risk in Lemma 9: equation (21) asserts that the bump applied to W1(µ,µ1)+||x−x1||_2^2 equals the indicator of B((µ1,x1),δ*), but that ball is defined using the ℓ1 term W1(µ,ν)+||z−x||_1. This is an internal metric mismatch, and the recursion also silently requires δ<1 for φ_{δ*,δ}(1)=0; these are correctness issues, not circularity, and they do not make the theorem equivalent to its inputs.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

The construction rests on standard approximation theory and on technical assumptions about the context space and activation function. No free parameters are fitted; the main burden is the exact Wasserstein implementation and the partition-of-unity proof. The axioms listed are the load-bearing premises beyond standard mathematics.

assumptions (6)
  • domain assumption PIC weights are restricted to the discrete simplex Δ_{C,N}, meaning all weights are multiples of 1/C and sum to 1.
    Definition 1 and Eq. (2); excludes contexts with arbitrary real frequencies, which the authors call mathematical artifacts. The Wasserstein implementation enumerates all weight pairs in this finite set.
  • domain assumption K is compact and carries an Ahlfors q-regular probability measure P.
    Setting 1 and Definition 3; needed for the high-probability and tail estimates via control of the trifling region in Lemma 6.
  • ad hoc to paper Activation function σθ in (8) can implement ReLU, identity, and ReQU by choosing θ.
    Appendix A; the theorem is called ReLU MLP but the proof uses ReQU multiplication (Lemma 14), so the model class is MLPs with trainable activation functions.
  • standard math Optimal transport plans between discrete measures can be chosen as extreme points of the transport polytope, with at most N^{N−1} extreme points.
    Proposition 20 uses Peyré et al. and Brualdi to enumerate extreme points; this underpins exact W1 implementation.
  • ad hoc to paper The token dimension d satisfies d ≥ N^N so that factorial and hyper-exponential terms in the W1 subnetwork are absorbed.
    Section 4.4 uses this to simplify the depth and width bounds; without it, the stated rates do not follow from Lemma 10.
  • ad hoc to paper The pseudo-indicator recursion assumes δ* < δ < 1 so that ϕ_{δ*,δ}(1)=0.
    Lemma 9's recursion uses ϕ_{δ*,δ}(1−Φ_j); for the product to act as an exact complement, 1 must lie outside the bump's support. This condition is not stated in the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Is In-Context Universality Enough? MLPs are Also Universal In-Context." pith.science (2026). https://pith.science/paper/LPKFDEF2

@misc{pith2026250203327,
  author       = {Pith},
  title        = {Pith review of: Is In-Context Universality Enough? MLPs are Also Universal In-Context},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LPKFDEF2}},
  note         = {Machine review of arXiv:2502.03327}
}
abstract

The success of transformers is often linked to their ability to perform in-context learning. Recent work shows that transformers are universal in context, capable of approximating any real-valued continuous function of a context (a probability measure over $\mathcal{X}\subseteq \mathbb{R}^d$) and a query $x\in \mathcal{X}$. This raises the question: Does in-context universality explain their advantage over classical models? We answer this in the negative by proving that MLPs with trainable activation functions are also universal in-context. This suggests the transformer's success is likely due to other factors like inductive bias or training stability.

Figures

Figures reproduced from arXiv: 2502.03327 by the authors.

Figure 1
Figure 1. As Mea￾sure in P7,6(R 2 ) [st]The cat likes the dog. [end] [st] the cat likes dog [end] [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 5
Figure 5. The activation function in (8). σθ(x) def. = ( α1x p if x ≥ 0 α2x if x < 0 . (8) Trainable Activation Functions As most modern deep learning implementations, such as trans￾formers using trainable Swish Ramachandran et al. (2017) or GLU activation variants Shazeer (2020), KANs Liu et al. (2024); Ismailov (2024) activated by trainable B-splines and SiLUs, or neural snowflakes which implement trainable fractal activati… view at source ↗
Figure 6
Figure 6. Our Regular Decomposition of the PIC Space K : The Retracted Voronoi cells (non￾redish coloured regions) C δ⋆ 1 , . . . , Cδ⋆ K , for δ⋆ > 0, whose union makes up the “large” approximation region K \ Kδ⋆ . The reddish region symbolizes our “small” trifling region whereon a uniform approximation may fail. Our approximators will uniformly approximate the given target PIC-function f : K → PM(Y) on the approximation reg… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. How Does the Pretraining Distribution Shape In-Context Learning? A Fundamental Trade-Off

    cs.LG 2025-10 conditional novelty 6.0 of 10

    Heavy-tailed pretraining distributions improve in-context task selection under distribution shift but worsen ICL generalization, especially in low-data regimes.

  2. Beyond Universal Approximation Theorems: Algorithmic Uniform Approximation by Neural Networks Trained with Noisy Data

    stat.ML 2025-08 reject novelty 6.0 of 10

    An explicit randomized training pipeline is claimed to yield uniform approximators from noisy data with minimax-optimal trainable parameters, but key sample-complexity claims are algebraically reversed and the proof s...

Reference graph

Works this paper leans on

77 extracted references · 47 canonical work pages · cited by 2 Pith papers

  1. [1]

    Designing universal causal deep learning models: The geometric (hyper) transformer

    Beatrice Acciaio, Anastasis Kratsios, and Gudmund Pammer. Designing universal causal deep learning models: The geometric (hyper) transformer. Mathematical Finance, 34 0 (2): 0 671--735, 2024

    work page 2024

  2. [2]

    What learning algorithm is in-context learning

    Ekin Aky \"u rek, D Schuurmans, J Andreas, T Ma, and D Zhou. What learning algorithm is in-context learning. Investigations with linear models. arXiv, 2211, 2022

    work page 2022

  3. [3]

    Linear extension operators between spaces of lipschitz maps and optimal transport

    Luigi Ambrosio and Daniele Puglisi. Linear extension operators between spaces of lipschitz maps and optimal transport. Journal f \"u r die reine und angewandte Mathematik (Crelles Journal) , 2020 0 (764): 0 1--21, 2020

    work page 2020

  4. [4]

    Neural machine translation by jointly learning to align and translate

    Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014

    arXiv 2014

  5. [5]

    Transformers as statisticians: Provable in-context learning with in-context algorithm selection

    Yu Bai, Fan Chen, Huan Wang, Caiming Xiong, and Song Mei. Transformers as statisticians: Provable in-context learning with in-context algorithm selection. Advances in neural information processing systems, 36, 2024

    2024

  6. [6]

    Simultaneous approximation of a smooth function and its derivatives by deep neural networks with piecewise-polynomial activations

    Denis Belomestny, Alexey Naumov, Nikita Puchkin, and Sergey Samsonov. Simultaneous approximation of a smooth function and its derivatives by deep neural networks with piecewise-polynomial activations. Neural Networks, 161: 0 242--253, 2023

    work page 2023

  7. [7]

    Introduction to linear optimization, volume 6

    Dimitris Bertsimas and John N Tsitsiklis. Introduction to linear optimization, volume 6. Athena Scientific Belmont, MA, 1997

    work page 1997

  8. [8]

    Optimal approximation with sparsely connected deep neural networks

    Helmut Bolcskei, Philipp Grohs, Gitta Kutyniok, and Philipp Petersen. Optimal approximation with sparsely connected deep neural networks. SIAM Journal on Mathematics of Data Science, 1 0 (1): 0 8--45, 2019

    work page 2019

Show all 77 references
  1. [9]

    Neural spacetimes for dag representation learning

    Haitz S \'a ez de Oc \'a riz Borde, Anastasis Kratsios, Marc T Law, Xiaowen Dong, and Michael Bronstein. Neural spacetimes for dag representation learning. arXiv preprint arXiv:2408.13885, 2024 a

    2024 arXiv

  2. [10]

    Scalable message passing neural networks: No need for attention in large graph representation learning

    Haitz S \'a ez de Oc \'a riz Borde, Artem Lukoianov, Anastasis Kratsios, Michael Bronstein, and Xiaowen Dong. Scalable message passing neural networks: No need for attention in large graph representation learning. arXiv preprint arXiv:2411.00835, 2024 b

    2024

  3. [11]

    Bridson and Andr\'e Haefliger

    Martin R. Bridson and Andr\'e Haefliger. Metric spaces of non-positive curvature, volume 319 of Grundlehren der mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences]. Springer-Verlag, Berlin, 1999. ISBN 3-540-64324-9. doi:10.1007/978-3-662-12494-9. UR...

    1999 doi

  4. [12]

    Richard A. Brualdi. Combinatorial matrix classes, volume 108 of Encyclopedia of Mathematics and its Applications. Cambridge University Press, Cambridge, 2006. ISBN 978-0-521-86565-4; 0-521-86565-4. doi:10.1017/CBO9780511721182. URL https://doi.org/10.1017/CBO9780511721182

    2006 doi

  5. [13]

    How smooth is attention? In ICML 2024, 2024

    Val \'e rie Castin, Pierre Ablin, and Gabriel Peyr \'e . How smooth is attention? In ICML 2024, 2024

    2024

  6. [14]

    Training dynamics of multi-head softmax attention for in-context learning: Emergence, convergence, and optimality

    Siyu Chen, Heejune Sheen, Tianhao Wang, and Zhuoran Yang. Training dynamics of multi-head softmax attention for in-context learning: Emergence, convergence, and optimality. arXiv preprint arXiv:2402.19442, 2024

    2024 arXiv

  7. [15]

    Efficient approximation of high-dimensional functions with neural networks

    Patrick Cheridito, Arnulf Jentzen, and Florian Rossmannek. Efficient approximation of high-dimensional functions with neural networks. IEEE Transactions on Neural Networks and Learning Systems, 2021 a

    2021

  8. [16]

    Efficient approximation of high-dimensional functions with neural networks

    Patrick Cheridito, Arnulf Jentzen, and Florian Rossmannek. Efficient approximation of high-dimensional functions with neural networks. IEEE Transactions on Neural Networks and Learning Systems, 33 0 (7): 0 3079--3093, 2021 b

    2021

  9. [17]

    Tighter bounds on the expressivity of transformer encoders

    David Chiang, Peter Cholak, and Anand Pillay. Tighter bounds on the expressivity of transformer encoders. In International Conference on Machine Learning, pages 5544--5562. PMLR, 2023

    2023

  10. [18]

    Conditional positional encodings for vision transformers

    Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, and Chunhua Shen. Conditional positional encodings for vision transformers. In The Eleventh International Conference on Learning Representations, 2022

    2022

  11. [19]

    Global universal approximation of functional input maps on weighted spaces

    Christa Cuchiero, Philipp Schmocker, and Josef Teichmann. Global universal approximation of functional input maps on weighted spaces. arXiv preprint arXiv:2306.03303, 2023

    2023

  12. [20]

    The density theorem and hausdorff inequality for packing measure in general metric spaces

    Colleen D Cutler. The density theorem and hausdorff inequality for packing measure in general metric spaces. Illinois journal of mathematics, 39 0 (4): 0 676--694, 1995

    1995

  13. [21]

    Neural snowflakes: Universal latent graph inference via trainable latent geometries

    Haitz S \'a ez de Oc \'a riz Borde and Anastasis Kratsios. Neural snowflakes: Universal latent graph inference via trainable latent geometries. In The Twelfth International Conference on Learning Representations, 2023

    2023

  14. [22]

    Finite sample analysis and bounds of generalization error of gradient descent in in-context linear regression

    Karthik Duraisamy. Finite sample analysis and bounds of generalization error of gradient descent in in-context linear regression. arXiv preprint arXiv:2405.02462, 2024

    2024 arXiv

  15. [23]

    Attention enables zero approximation error

    Zhiying Fang, Yidong Ouyang, Ding-Xuan Zhou, and Guang Cheng. Attention enables zero approximation error. arXiv preprint arXiv:2202.12166, 2022

    2022 arXiv

  16. [24]

    Simultaneously solving fbsdes with neural operators of logarithmic depth, constant width, and sub-linear rank

    Takashi Furuya and Anastasis Kratsios. Simultaneously solving fbsdes with neural operators of logarithmic depth, constant width, and sub-linear rank. arXiv preprint arXiv:2410.14788, 2024

    2024

  17. [25]

    Globally injective and bijective neural operators

    Takashi Furuya, Michael Puthawala, Matti Lassas, and Maarten V de Hoop. Globally injective and bijective neural operators. arXiv preprint arXiv:2306.03982, 2023

    2023 arXiv

  18. [26]

    Transformers are universal in-context learners

    Takashi Furuya, Maarten V de Hoop, and Gabriel Peyr \'e . Transformers are universal in-context learners. arXiv preprint arXiv:2408.01367, 2024

    2024 arXiv

  19. [27]

    What can transformers learn in-context? a case study of simple function classes

    Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems, 35: 0 30583--30598, 2022

    2022

  20. [28]

    Robust barycenter estimation using semi-unbalanced neural optimal transport

    Milena Gazdieva, Jaemoo Choi, Alexander Kolesov, Jaewoong Choi, Petr Mokrov, and Alexander Korotin. Robust barycenter estimation using semi-unbalanced neural optimal transport. arXiv preprint arXiv:2410.03974, 2024

    2024 arXiv

  21. [29]

    A survey on lipschitz-free banach spaces

    Gilles Godefroy. A survey on lipschitz-free banach spaces. Commentationes Mathematicae, 55 0 (2), 2015

    2015

  22. [30]

    Can a transformer represent a kalman filter? In 6th Annual Learning for Dynamics & Control Conference, pages 1502--1512

    Gautam Goel and Peter Bartlett. Can a transformer represent a kalman filter? In 6th Annual Learning for Dynamics & Control Conference, pages 1502--1512. PMLR, 2024

    2024

  23. [31]

    Multilayer feedforward networks are universal approximators

    Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural networks, 2 0 (5): 0 359--366, 1989

    1989

  24. [32]

    Addressing common misinterpretations of kart and uat in neural network literature

    Vugar Ismailov. Addressing common misinterpretations of kart and uat in neural network literature. arXiv preprint arXiv:2408.16389, 2024

    2024

  25. [33]

    Neural tangent kernel: Convergence and generalization in neural networks

    Arthur Jacot, Franck Gabriel, and Cl \'e ment Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018

    2018

  26. [34]

    arvenp\"a\

    Esa J\"arvenp\"a\"a, Maarit J\"arvenp\"a\"a, Antti K\"aenm\"aki, Tapio Rajala, Sari Rogovin, and Ville Suomala. Packing dimension and A hlfors regularity of porous sets in metric spaces. Math. Z., 266 0 (1): 0 83--105, 2010. ISSN 0025-5874,1432-1823. doi:10.1007/s00209-009-055...

    2010 doi

  27. [35]

    Universal approximation with deep narrow networks

    Patrick Kidger and Terry Lyons. Universal approximation with deep narrow networks. In Conference on learning theory, pages 2306--2327. PMLR, 2020

    2020

  28. [36]

    Transformers provably solve parity efficiently with chain of thought

    Juno Kim and Taiji Suzuki. Transformers provably solve parity efficiently with chain of thought. In NeurIPS 2024 Workshop on Mathematics of Modern Machine Learning, 2024 a

    2024

  29. [37]

    Transformers learn nonlinear features in context

    Juno Kim and Taiji Suzuki. Transformers learn nonlinear features in context. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2024 b

    2024

  30. [38]

    Transformers are minimax optimal nonparametric in-context learners

    Juno Kim, Tai Nakamaki, and Taiji Suzuki. Transformers are minimax optimal nonparametric in-context learners. In ICML 2024 Workshop on In-Context Learning, 2024. URL https://openreview.net/forum?id=WjrKBQTWKp

    2024

  31. [39]

    Wasserstein-2 generative networks

    Alexander Korotin, Vage Egiazarian, Arip Asadulaev, Alexander Safin, and Evgeny Burnaev. Wasserstein-2 generative networks. arXiv preprint arXiv:1909.13082, 2019

    1909 arXiv

  32. [40]

    Neural optimal transport

    Alexander Korotin, Daniil Selikhanovych, and Evgeny Burnaev. Neural optimal transport. In The Eleventh International Conference on Learning Representations, 2022

    2022

  33. [41]

    Universal approximation theorems for differentiable geometric deep learning

    Anastasis Kratsios and L \'e onie Papon. Universal approximation theorems for differentiable geometric deep learning. Journal of Machine Learning Research, 23 0 (196): 0 1--73, 2022

    2022

  34. [42]

    Universal approximation under constraints is possible with transformers

    Anastasis Kratsios, Behnoosh Zamanlooy, Tianlin Liu, and Ivan Dokmani \'c . Universal approximation under constraints is possible with transformers. In International Conference on Learning Representations, 2021

    2021

  35. [43]

    An approximation theory for metric space-valued functions with a view towards deep learning

    Anastasis Kratsios, Chong Liu, Matti Lassas, Maarten V de Hoop, and Ivan Dokmani \'c . An approximation theory for metric space-valued functions with a view towards deep learning. arXiv preprint arXiv:2304.12231, 2023

    2023 arXiv

  36. [44]

    Learnable fourier features for multi-dimensional spatial positional encoding

    Yang Li, Si Si, Gang Li, Cho-Jui Hsieh, and Samy Bengio. Learnable fourier features for multi-dimensional spatial positional encoding. Advances in Neural Information Processing Systems, 34: 0 15816--15829, 2021

    2021

  37. [45]

    Transformers as algorithms: Generalization and stability in in-context learning

    Yingcong Li, Muhammed Emrullah Ildiz, Dimitris Papailiopoulos, and Samet Oymak. Transformers as algorithms: Generalization and stability in in-context learning. In International Conference on Machine Learning, pages 19565--19594. PMLR, 2023

    2023

  38. [46]

    Reality only happens once: Single-path generalization bounds for transformers

    Yannick Limmer, Anastasis Kratsios, Xuwei Yang, Raeid Saqur, and Blanka Horvath. Reality only happens once: Single-path generalization bounds for transformers. arXiv preprint arXiv:2405.16563, 2024

    2024 arXiv

  39. [47]

    Kan: Kolmogorov-arnold networks

    Ziming Liu, Yixuan Wang, Sachin Vaidya, Fabian Ruehle, James Halverson, Marin Solja c i \'c , Thomas Y Hou, and Max Tegmark. Kan: Kolmogorov-arnold networks. arXiv preprint arXiv:2404.19756, 2024

    2024 arXiv

  40. [48]

    Asymptotic theory of in-context learning by linear attention

    Yue M Lu, Mary I Letey, Jacob A Zavatone-Veth, Anindita Maiti, and Cengiz Pehlevan. Asymptotic theory of in-context learning by linear attention. arXiv preprint arXiv:2405.11751, 2024

    2024

  41. [49]

    Your transformer may not be as powerful as you expect

    Shengjie Luo, Shanda Li, Shuxin Zheng, Tie-Yan Liu, Liwei Wang, and Di He. Your transformer may not be as powerful as you expect. Advances in Neural Information Processing Systems, 35: 0 4301--4315, 2022

    2022

  42. [50]

    Every complete doubling metric space carries a doubling measure

    Jouni Luukkainen and Eero Saksman. Every complete doubling metric space carries a doubling measure. Proc. Amer. Math. Soc., 126 0 (2): 0 531--534, 1998. ISSN 0002-9939,1088-6826. doi:10.1090/S0002-9939-98-04201-4. URL https://doi.org/10.1090/S0002-9939-98-04201-4

    1998 doi

  43. [51]

    The expresssive power of transformers with chain of thought

    William Merrill and Ashish Sabharwal. The expresssive power of transformers with chain of thought. arXiv preprint arXiv:2310.07923, 2023

    2023 arXiv

  44. [52]

    Length independent pac-bayes bounds for simple rnns

    Volodimir Mitarchuk, Clara Lacroce, R \'e mi Eyraud, R \'e mi Emonet, Amaury Habrard, and Guillaume Rabusseau. Length independent pac-bayes bounds for simple rnns. In International Conference on Artificial Intelligence and Statistics, pages 3547--3555. PMLR, 2024

    2024

  45. [53]

    In-context learning and induction heads

    Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11895, 2022

    2022 arXiv

  46. [54]

    Equivalence of approximation by convolutional neural networks and fully-connected networks

    Philipp Petersen and Felix Voigtlaender. Equivalence of approximation by convolutional neural networks and fully-connected networks. Proceedings of the American Mathematical Society, 148 0 (4): 0 1567--1581, 2020

    2020

  47. [55]

    Mathematical theory of deep learning

    Philipp Petersen and Jakob Zech. Mathematical theory of deep learning. arXiv preprint arXiv:2407.18384, 2024

    2024

  48. [56]

    Universal in-context approximation by prompting fully recurrent models

    Aleksandar Petrov, Tom A Lamb, Alasdair Paren, Philip HS Torr, and Adel Bibi. Universal in-context approximation by prompting fully recurrent models. Neural Information Processing Systems (NeurIPS) 2025, 2025

    2025

  49. [57]

    Computational optimal transport: With applications to data science

    Gabriel Peyr \'e , Marco Cuturi, et al. Computational optimal transport: With applications to data science. Foundations and Trends in Machine Learning , 11 0 (5-6): 0 355--607, 2019 a

    2019

  50. [58]

    Computational optimal transport: With applications to data science

    Gabriel Peyr \'e , Marco Cuturi, et al. Computational optimal transport: With applications to data science. Foundations and Trends in Machine Learning, 11 0 (5-6): 0 355--607, 2019 b

    2019

  51. [59]

    Searching for activation functions

    Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. arXiv preprint arXiv:1710.05941, 2017

    2017 arXiv

  52. [60]

    The mechanistic basis of data dependence and abrupt learning in an in-context classification task

    Gautam Reddy. The mechanistic basis of data dependence and abrupt learning in an in-context classification task. In The Twelfth International Conference on Learning Representations, 2023

    2023

  53. [61]

    Singular value perturbation and deep network optimization

    Rudolf H Riedi, Randall Balestriero, and Richard G Baraniuk. Singular value perturbation and deep network optimization. Constructive Approximation, 57 0 (2): 0 807--852, 2023

    2023

  54. [62]

    Glu variants improve transformer

    Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020

    2002 arXiv

  55. [63]

    Nonparametric estimation of non-crossing quantile regression process with deep requ neural networks

    Guohao Shen, Yuling Jiao, Yuanyuan Lin, Joel L Horowitz, and Jian Huang. Nonparametric estimation of non-crossing quantile regression process with deep requ neural networks. Journal of Machine Learning Research, 25 0 (88): 0 1--75, 2024

    2024

  56. [64]

    Optimal approximation rate of R e LU networks in terms of width and depth

    Zuowei Shen, Haizhao Yang, and Shijun Zhang. Optimal approximation rate of R e LU networks in terms of width and depth. J. Math. Pures Appl. (9), 157: 0 101--135, 2022. ISSN 0021-7824,1776-3371. doi:10.1016/j.matpur.2021.07.009. URL https://doi.org/10.1016/j.matpur.2021.07.009

    2022 doi

  57. [65]

    Expressivity of spiking neural networks

    Manjot Singh, Adalbert Fono, and Gitta Kutyniok. Expressivity of spiking neural networks. arXiv preprint arXiv:2308.08218, 2023

    2023 arXiv

  58. [66]

    Training dynamics of multi-head softmax attention for in-context learning: Emergence, convergence, and optimality

    Chen Siyu, Sheen Heejune, Wang Tianhao, and Yang Zhuoran. Training dynamics of multi-head softmax attention for in-context learning: Emergence, convergence, and optimality. In The Thirty Seventh Annual Conference on Learning Theory, pages 4573--4573. PMLR, 2024

    2024

  59. [67]

    What formal languages can transformers express? a survey

    Lena Strobl, William Merrill, Gail Weiss, David Chiang, and Dana Angluin. What formal languages can transformers express? a survey. Transactions of the Association for Computational Linguistics, 12: 0 543--561, 2024

    2024

  60. [68]

    Adaptivity of deep relu network for learning in besov and mixed smooth besov spaces: optimal rate and curse of dimensionality

    Taiji Suzuki. Adaptivity of deep relu network for learning in besov and mixed smooth besov spaces: optimal rate and curse of dimensionality. In International Conference on Learning Representations, 2018

    2018

  61. [69]

    A. W. van der Vaart and Jon A. Wellner. Weak convergence and empirical processes---with applications to statistics. Springer Series in Statistics. Springer, Cham, second edition, 2003. ISBN 978-3-031-29038-1; 978-3-031-29040-4. doi:10.1007/978-3-031-29040-4. URL https://doi.or...

    2003 doi

  62. [70]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  63. [71]

    Optimal transport, volume 338 of Grundlehren der mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences]

    C\'edric Villani. Optimal transport, volume 338 of Grundlehren der mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences]. Springer-Verlag, Berlin, 2009. ISBN 978-3-540-71049-3. doi:10.1007/978-3-540-71050-9. URL https://doi.org/10.1007/978-3-540-71050...

    2009 doi

  64. [72]

    Distance-based classification with lipschitz functions

    Ulrike von Luxburg and Olivier Bousquet. Distance-based classification with lipschitz functions. J. Mach. Learn. Res., 5 0 (Jun): 0 669--695, 2004

    2004

  65. [73]

    Transformers learn in-context by gradient descent

    Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, Jo \ a o Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference on Machine Learning, pages 35151--35174. PMLR, 2023

    2023

  66. [74]

    Lipschitz algebras

    Nik Weaver. Lipschitz algebras. World Scientific, 2018

    2018

  67. [75]

    Optimal approximation of continuous functions by very deep relu networks

    Dmitry Yarotsky. Optimal approximation of continuous functions by very deep relu networks. In Sébastien Bubeck, Vianney Perchet, and Philippe Rigollet, editors, Proceedings of the 31st Conference On Learning Theory, volume 75 of Proceedings of Machine Learning Research, pages ...

    2018

  68. [76]

    Trained transformers learn linear models in-context

    Ruiqi Zhang, Spencer Frei, and Peter L Bartlett. Trained transformers learn linear models in-context. Journal of Machine Learning Research, 25 0 (49): 0 1--55, 2024 a

    2024

  69. [77]

    In-context learning of a linear transformer block: benefits of the mlp component and one-step gd initialization

    Ruiqi Zhang, Jingfeng Wu, and Peter L Bartlett. In-context learning of a linear transformer block: benefits of the mlp component and one-step gd initialization. arXiv preprint arXiv:2402.14951, 2024 b

    2024 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.