Pith. sign in

REVIEW 3 major objections 4 minor 21 references

Training-Free Universal Approximation by Prompting Random Transformers

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A single-layer, never-trained random transformer can approximate any Hölder function on a compact manifold when steered by a soft prompt, at minimax-optimal rates.

desk verdict Clean prompt-based universal approximation with random weights, but the advertised minimax rate leans on two imported kernel-regression lemmas that need a closer look. read the letter →

arxiv 2608.09558 v1 pith:SDSP2RBI submitted 2026-08-10 cs.LG cs.NAmath.NAmath.STstat.TH

Alexander Hsu , Rongjie Lai This is my paper · ORCID
classification cs.LGcs.NAmath.NAmath.STstat.TH MSC 68T0762G0860B2041A25
keywords softpromptingrandomtransformersuniversalapproximationNadaraya-WatsonestimatorkernelregressionHölderfunctionsonmanifoldsminimax-optimalratestraining-free
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that pretraining is unnecessary for function regression with transformers: a single-layer softmax attention network whose weights are random and never trained can approximate any Hölder function on a compact manifold, provided it is given a soft prompt built for that function. The prompt is constructed explicitly, not learned: each prompt token solves a small linear system that aligns the attention logits with Gaussian kernel exponents, making the frozen network reproduce the Nadaraya-Watson kernel estimator in its forward pass. Because the network inherits kernel-regression guarantees, the approximation error reaches the minimax-optimal rate $\tilde{O}(n^{-2\alpha/(2\alpha+d_X)})$ in the sample count $n$, where $\alpha$ is the Hölder exponent and $d_X$ is the intrinsic manifold dimension. The construction needs only a rank condition on the random weights, which holds almost surely under Gaussian initialization, so the result applies to essentially every draw of the weights.

What carries the argument

The machinery is the explicit soft prompt as the minimum-norm solution of a linear system that matches attention logits to Gaussian kernel exponents. For each support point $x_i$ with label $y_i$, the token $p_i$ solves $\widetilde{W}p_i = v_i$, where $\widetilde{W}$ stacks the first $d_{\rm in}+1$ rows of the query-key product $W = W_Q W_K^\top/\sqrt{d_h}$ with the readout vector $w_V^\top$, and $v_i$ contains $x_i/h^2$, $-\|x_i\|^2/(2h^2)+\beta$, and $y_i$; in the fully random architecture the analogous system uses the random embedding and decoding maps. A logit shift $\beta$ added to the bias coordinate suppresses the query's self-attention term exponentially in $\beta$, and a rank condition on $\widetilde{W}$ (full row rank) guarantees solvability. This construction turns the attention layer into the Nadaraya-Watson kernel estimator, so the approximation theory of kernel regression on manifolds transfers to prompted transformers.

What would settle it

Run the constructed prompt on a sphere with $d_X=2$, $\alpha=1$, bandwidth $h=n^{-1/4}$ and the prescribed logit shift, measuring MSE over many test queries; if the error does not decay like $n^{-1/2}$ up to log factors as $n$ and $d_h$ grow, the claimed minimax-optimal rate is contradicted.

Watch

Extended reading notes

Core claim

The central claim is that a frozen, single-head softmax attention transformer with random Gaussian weights is a universal approximator for Hölder functions on a compact manifold, steered solely by a soft prompt. For any target function (given through noisy samples) the paper builds a query-independent prompt whose tokens are the minimum-norm solutions of the linear systems $\widetilde{W} p_i = v_i$ and $\widetilde{W}_{\mathrm{aff}} p_i = v'_i$, forcing the attention output to match the Nadaraya-Watson kernel estimator up to an exponentially small self-attention term suppressed by a logit shift $\beta$. This yields universal approximation theorems with rate $\tilde{O}(n^{-2\alpha/(2\alpha+d_X)})$ — the minimax-optimal rate for nonparametric regression on a $d_X$-dimensional manifold — for deterministic attention parameters (Theorem 2), random attention parameters (Theorem 4), and fully random transformers with random affine embedding and decoding (Theorem 6). The prompt construction is explicit and depends on the realized weights, and the paper also tracks the price of prompting: token norms grow like $O(n^{2/(2\alpha+d_X)})$ in the deterministic case, improving to $O(n^{2/(2\alpha+d_X)}/\sqrt{d_h}+1)$ with random attention and $O(n^{2/(2\alpha+d_X)}/d_h+\sqrt{d_h})$ for fully random transformers.

Load-bearing premise

The pure kernel-regression bounds on manifolds (the variance and bias lemmas imported from the companion paper) are the load-bearing premise; if they are not reproducible as stated, the minimax rate does not follow from this paper's argument.

Editorial extensions

If this is right

  • A fixed, never-trained random transformer can perform nonparametric regression in-context: the task-specific information lives entirely in the soft prompt, not in the weights.
  • Approximation error decays at the minimax-optimal rate for Hölder functions, so prompting alone is not a weaker substitute for training in this setting — it matches the best possible statistical rate up to logarithms.
  • The construction is explicit and computable via the Moore-Penrose pseudoinverse, so no gradient-based prompt optimization is needed to realize the guarantee.
  • Prompt length $n$ doubles as the number of support points, so longer prompts give both more data and more steering capacity at the same time.
  • Fully random embeddings and decoding spread the prompt constraints across the hidden dimension, changing the prompt-norm cost: larger $d_h$ reduces token norms in the random-attention case but eventually increases them in the fully random case.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension: the same logit-shift construction should transfer to multi-head attention by assigning one kernel per head, which the paper does not prove.
  • The paper's explicit prompt solves regression with one token per support point; a natural follow-up is to ask whether a single prompt of length $m \ll n$ can encode a coarser estimator, since the paper's bounds do not rule this out.
  • The experiments (a trained prompt beating the constructed one) suggest the construction is sufficient but not optimal; if that gap persists at larger $n$, the true cost of prompting may be below the paper's worst-case token norms.
  • If the mechanism is robust to finite precision, it offers a purely inference-time way to adapt frozen foundation models to new regression tasks without touching weights, which is the practical reading of 'pretraining is optional'.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper studies soft prompting of a single-layer softmax-attention transformer with frozen, randomly initialized weights. For any Hölder function on a compact manifold, the authors construct a query-independent soft prompt, of length equal to the number of support points, whose tokens are minimum-norm solutions of linear systems such as fW p_i = v_i in Eq. (3.5) or fWaff p_i = v'_i in Eq. (4.3). This makes the attention output match the Nadaraya-Watson kernel estimator up to a query self-attention term that is exponentially suppressed by a logit shift β. The construction requires only full row rank of fW, which is shown to hold almost surely for Gaussian attention weights; a further extension covers random affine embeddings and decoders. Building on standard kernel-regression bounds, Theorems 2, 4, and 6 state minimax-optimal squared-error rates O~(n^{-2α/(2α+d_X)}) and quantify prompt-token norms. Numerical experiments on the sphere S^2 illustrate the scaling of the construction and compare it with trained baselines.

Significance. If the stated results hold, the paper makes a striking conceptual contribution: a fixed random single-layer attention network can be steered to approximate a broad function class purely through an explicitly constructed prompt, without any training of the network weights. The core prompt construction is transparent and largely self-contained: the logit-matching equations, the pseudoinverse solution, and the logit-shift suppression are all checked in the text, and the random-matrix rank arguments are standard and sound. The paper also gives concrete prompt-norm tradeoffs in terms of n, d_h, and the Hölder/geometry parameters, and it provides experiments that support the predicted n^{-2} and β-scaling of the kernel-approximation error. The main caveat is that the advertised minimax rate is not derived in this manuscript: it is inherited from Lemmas 2 and 3, which are stated without proof and whose precise hypotheses are not fully specified.

major comments (3)
  1. [Section 3.2, Lemmas 2 and 3] The headline rates in Theorems 2, 4, and 6 rest entirely on the finite-sample variance bound (Lemma 2) and the manifold bias bound (Lemma 3), but neither lemma is proved in this manuscript and their statements are incomplete. Lemma 3 is stated for h > 0 with no restriction relative to the reach τ_X, and the O(h^α log(h^{-1})) bound is asserted uniformly over X without stating the required condition; Lemma 2 similarly does not state whether h ≤ τ_X/2 or whether the high-probability event and the constant are uniform in x. Since the proof of Theorem 2 invokes these lemmas with h = n^{-1/(2α+d_X)} and δ = 4h^2, the minimax claim cannot be verified from the text. The authors should include full proofs or quote the exact theorems from Shen et al. [2026] with all hypotheses, and confirm that the chosen h and δ satisfy them.
  2. [Section 3.2, proof of Theorem 2, Eq. (3.10)] The transition from Lemma 2's high-probability pointwise statement to the expectation bound (3.10) needs an explicit uniformity statement. As written, Lemma 2 is stated for a fixed x, while the theorem needs a bound on E_{x_i,ε_i,x} of the squared finite-sample gap; the proof uses the bounded-difference inequality with δ = 4h^2 but does not specify whether the O(·) constant in Lemma 2 is uniform over x ∈ X. If the constant depends on x through local reach or other geometric quantities, the expectation step is not justified as written. Please clarify or strengthen Lemma 2 accordingly.
  3. [Section 1 and Theorem 2] The paper presents the minimax-optimal rate as a main contribution, but the transformer-specific part of the proof establishes only the O(n^{-2}) prompt-construction error; the minimax rate itself comes from kernel-regression bounds imported from prior work. This division should be stated clearly in the introduction and in the statements of Theorems 2, 4, and 6, so that readers can distinguish the genuinely new prompt-construction mechanism from the inherited kernel-estimator guarantees.
minor comments (4)
  1. [Section 3.2, Lemma 3 statement] The phrase 'with at most exponential dependence on d_X' is too vague for a theorem statement; at minimum, state whether the bound is uniform in x and whether it requires h ≤ c τ_X for a constant c.
  2. [Section 5, Fig. 2a] For h = 0.2, the text attributes the initial steeper drop to insufficient local support; a brief explanation of why this does not contradict the predicted n^{-2} bound would help the reader interpret the transient.
  3. [Section 5, Fig. 4] The baselines are trained with Adam while the construction is not; the comparison is informative, but the reader should be reminded that the failure of the baselines in Architecture B may be due to initialization scale rather than fundamental capacity.
  4. [Theorem 4 and Theorem 6] The statements of Theorems 4 and 6 combine an almost-sure rate statement with a high-probability token-norm statement; the proof should explicitly separate the probability space of the initialization from the probability space of the samples so that the constants in the O(·) notation are unambiguous.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity; the prompt construction is self-contained, and the self-cited kernel-regression lemmas are external support rather than a restatement of the target result.

full rationale

The proof chain is not circular. Theorem 1 constructs the soft prompt explicitly as p_i^* = fW^† v_i (Eq. 3.5), matching the attention logits to Gaussian kernel exponents; the approximation error is then an explicit algebraic residual (Eq. 3.7) that vanishes with the logit shift and prompt length. Theorems 3 and 5 replace the deterministic rank condition by almost-sure rank properties of Gaussian matrices, proved from the standard Lemma 4. The minimax rates in Theorems 2, 4, and 6 come from a four-term decomposition in which term I is the self-contained Theorem 1 bound and terms II-IV are Lemmas 1-3. Lemma 1 is proved in the paper. Lemmas 2 and 3 are stated without proof and attributed to Shen et al. [2026], a work sharing two authors with the present paper; this is a genuine verifiability and provenance gap and a self-citation. It is not circular, however: those lemmas concern bias and variance of the Nadaraya-Watson estimator for Holder functions on manifolds, with assumptions that do not include the target transformer-prompting result, so they are external benchmarks rather than restatements of the conclusion. No fitted parameter is relabeled as a prediction; the prompt is constructed from the realized weights and training labels, and query independence is proved, not assumed. The modest self-citation justifies a nonzero low score, but the central derivation does not reduce to its inputs.

Assumptions & free parameters 2 free parameters · 7 assumptions · 0 invented entities

No new physical or mathematical entities are introduced. The only extra machinery is the soft prompt itself, which is a standard continuous prefix. The price of the construction is tracked through h, β, and prompt token norms rather than through hidden degrees of freedom.

free parameters (2)
  • kernel bandwidth h = h = n^{-1/(2α+dX)} in Theorems 2, 4, and 6
    Chosen in the proof to balance kernel bias and variance; not estimated from data. In practice α and dX would need to be known or tuned, so the rate is an oracle-rate statement.
  • logit shift β = β = (3R^2/2) n^{2/(2α+dX)} in Theorem 2, plus log+ C_G in Theorems 4 and 6
    Chosen to cancel the exponential bandwidth factor and suppress query self-attention; affects prompt token norm. It is a construction parameter, not a data fit.
assumptions (7)
  • domain assumption Assumption 1: X is a compact dX-dimensional Riemannian manifold with positive reach τX, isometrically embedded in R^din, with ||x|| ≤ R.
    Used throughout to bound kernel exponents, denominator, and small-ball probabilities; limits the result to nicely embedded compact manifolds.
  • domain assumption Assumption 2: f is α-Hölder with 0 < α ≤ 1 and constant L.
    Defines the function class of the universal approximation theorem and the exponent in the rate.
  • domain assumption Assumption 3: labels have independent, mean-zero noise bounded by S_noise.
    Needed for the noise term and for bounding the NW estimator by S = S_f + S_noise.
  • domain assumption Assumption 4: support points and query are iid uniform on X.
    Required for the minimax regression rate; Theorem 1 itself is deterministic in the support points.
  • ad hoc to paper Condition 1 or its random analogue: fW and fWaff have full row rank.
    This is the solvability condition for the linear systems that define the prompt. The paper proves it holds almost surely for Gaussian weights, but for fixed deterministic weights it is an extra condition.
  • standard math Lemmas 2 and 3: kernel estimator variance and bias bounds on manifolds, deferred to Shen et al. [2026].
    Stated without proof; used to turn NW approximation into minimax-optimal universal approximation rates. The cited authors overlap with the present authors, so independent verification from this text is not possible.
  • domain assumption Assumptions 5 and 6: Gaussian random initialization of attention, embedding, and decoding parameters.
    Defines the random transformer regime and enables the almost-sure rank and singular-value bounds.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Training-Free Universal Approximation by Prompting Random Transformers." pith.science (2026). https://pith.science/paper/SDSP2RBI

@misc{pith2026260809558,
  author       = {Pith},
  title        = {Pith review of: Training-Free Universal Approximation by Prompting Random Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SDSP2RBI}},
  note         = {Machine review of arXiv:2608.09558}
}
read the original abstract

How expressive is prompting a transformer? Answering this question is important for separating the roles of prompting, architecture, and pretraining in transformer models, and for determining whether task-specific behavior must be stored in model weights or can instead be induced at inference time through the prompt. We show, in an approximation-theoretic sense, that pretraining is optional: a single-layer softmax attention network with random, untrained weights can approximate any H\"older function on a compact manifold when steered by an appropriate soft prompt. Guided by the connection between softmax attention and kernel methods, we construct explicit soft prompts (a prompt per target function, independent of the query) as solutions to linear systems matching attention logits to Gaussian kernel exponents, under which the frozen transformer emulates the classical Nadaraya-Watson kernel estimator. The construction requires only a mild rank condition on the weights, which we show holds almost surely under Gaussian initialization. The prompted network inherits the theoretical guarantees of kernel regression, leading to universal approximation theorems with minimax-optimal rates that depend on the intrinsic dimension. We further quantify the cost of prompting, exposing a tradeoff between the norm of the constructed soft prompt tokens, prompt length, and hidden dimension. Numerical experiments corroborate the constructions and predicted rates.

Figures

Figures reproduced from arXiv: 2608.09558 by the authors.

Figure 1
Figure 1. Semi-log plot of normalized minimum singular value versus [PITH_FULL_IMAGE:figures/full_fig_p023_1.png] view at source ↗
Figure 2
Figure 2. Scaling of supremum squared error between construction and oracle NW estimator across different [PITH_FULL_IMAGE:figures/full_fig_p024_2.png] view at source ↗
Figure 3
Figure 3. Max token norm versus β across various bandwidths h. Fixed dh = 256 and n = 256. Solid lines show medians and shaded regions show IQR. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Log-log plots of MSE versus n for both architectures and various baselines, with fixed dh = 256 throughout. h and β are set according to the theory: h = n −1/4 and β = 3√ n/2 + log+(CG) (or β = 3 √ n/2 + log+(CG,aff) for Architecture B). MSE is computed over 1000 unifo…
Figure 5
Figure 5. Figure 5: Log-log plots of maximum prompt token norm of constructed soft prompt versus [PITH_FULL_IMAGE:figures/full_fig_p026_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 10 canonical work pages

  1. [2026]

    Understanding In-Context Learning for Nonlinear Regression with Transformers: Attention as Featurizer

    URLhttps://arxiv.org/abs/ 2605.05176. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. InInternational Conference on Learning Representations,

    work page Pith review arXiv

  2. [9]

    Aobo Kong, Shiwan Zhao, Hao Chen, Qicheng Li, Yong Qin, Ruiqi Sun, Xin Zhou, Enzhi Wang, and Xiaohang Dong

    URLhttps://openreview.net/forum?id=nJnky5K944. Aobo Kong, Shiwan Zhao, Hao Chen, Qicheng Li, Yong Qin, Ruiqi Sun, Xin Zhou, Enzhi Wang, and Xiaohang Dong. Better zero-shot reasoning with role-play prompting. In Kevin Duh, Helena Gomez, and Steven Bethard, editors,Proceedings of the 2024 Conference of the North American Chapter of the Association for Compu...

    work page 2024

  3. [10]

    URLhttps://aclanthology.org/2024.naacl-long.228/

    18653/v1/2024.naacl-long.228. URLhttps://aclanthology.org/2024.naacl-long.228/. Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,

    work page 2024

  4. [11]

    Wolfgang Maass, Thomas Natschläger, and Henry Markram

    URLhttps://arxiv.org/abs/2310.14201. Wolfgang Maass, Thomas Natschläger, and Henry Markram. Real-time computing without stable states: A new framework for neural computation based on perturbations.Neural Computation, 14(11):2531–2560,

    arXiv

  5. [12]

    Memory Limitations of Prompt Tuning in Transformers

    URLhttps://arxiv.org/abs/2509.00421. Elizbar A. Nadaraya. On estimating regression.Theory of Probability and Its Applications, 9(1):141–142,

    work page Pith review arXiv

  6. [13]

    Partha Niyogi, Stephen Smale, and Shmuel Weinberger

    URLhttps: //arxiv.org/abs/2503.20561. Partha Niyogi, Stephen Smale, and Shmuel Weinberger. Finding the homology of submanifolds with high confidence from random samples.Discrete & Computational Geometry, 39(1):419–441, March

    arXiv

  7. [15]

    neurips.cc/paper_files/paper/2008/file/0efe32849d230d7f53049ddc4a4b0c60-Paper.pdf

    URLhttps://proceedings. neurips.cc/paper_files/paper/2008/file/0efe32849d230d7f53049ddc4a4b0c60-Paper.pdf. Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Mondal, and Aman Chadha. A systematic survey of prompt engineering in large language models: Techniques and applications,

    work page 2008

  8. [16]

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettle- moyer, Nicola Cancedda, and Thomas Scialom

    URLhttps://arxiv.org/abs/2402.07927. Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettle- moyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. InThirty-seventh Conference on Neural Information Processing Systems,

    arXiv

Show all 21 references
  1. [17]

    Sheng Shen, Alexei Baevski, Ari S

    URL https://openreview.net/forum?id=Yacmpz84TH. Sheng Shen, Alexei Baevski, Ari S. Morcos, Kurt Keutzer, Michael Auli, and Douwe Kiela. Reservoir transformers. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International ...

    2021

  2. [18]

    Taylor Shin, Yasaman Razeghi, Robert L

    URLhttps://openreview.net/forum?id=WbRULwqsIy. Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. AutoPrompt: Elic- iting knowledge from language models with automatically generated prompts. InProceedings of the 2020 Conference on Empirical Metho...

    2020

  3. [19]

    URLhttps://arxiv.org/abs/2412. 13212. Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada, Louis-Philippe Morency, and Ruslan Salakhutdinov. Transformer dissection: An unified understanding for transformer’s attention via the lens of kernel. In Proceedings of the 2019 Conference ...

    2019

  4. [21]

    Ziqian Zhong and Jacob Andreas

    doi: 10.1007/s11704-026-60308-3. Ziqian Zhong and Jacob Andreas. Algorithmic capabilities of random transformers. InAdvances in Neural Information Processing Systems,

    doi

  5. [1993]

    Aman Bhargava, Cameron Witkowski, Shi-Zhuo Looi, and Matt Thomson

    doi: 10.1109/18.256500. Aman Bhargava, Cameron Witkowski, Shi-Zhuo Looi, and Matt Thomson. What’s the magic word? a control theory of llm prompting,

    doi

  6. [2001]

    URLhttps://www.sciencedirect.com/ science/article/pii/S1874584901800103

    doi: https://doi.org/10.1016/S1874-5849(01)80010-3. URLhttps://www.sciencedirect.com/ science/article/pii/S1874584901800103. Yihe Dong, Lorenzo Noci, Mikhail Khodak, and Mufan Li. Is random attention sufficient for sequence modeling? disentangling trainable components in the t...

    doi

  7. [2007]

    URLhttp://www.jstor.org/stable/20461468

    ISSN 07492170. URLhttp://www.jstor.org/stable/20461468. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, et al. Language models are few-shot learners. InAdvances in Neural Information Processing Systems,

  8. [2008]

    doi: 10.1007/s00454-008-9053-2

    ISSN 1432-0444. doi: 10.1007/s00454-008-9053-2. URLhttps://doi.org/10.1007/s00454-008-9053-2. Samet Oymak, Ankit Singh Rawat, Mahdi Soltanolkotabi, and Christos Thrampoulidis. On the role of attention in prompt-tuning. InInternational Conference on Machine Learning,

    doi

  9. [2015]

    Kurt Hornik

    doi: 10.1109/ICCV.2015.123. Kurt Hornik. Approximation capabilities of multilayer feedforward networks.Neural Networks, 4(2):251–257,

    2015 doi

  10. [2017]

    doi: https://doi.org/10.1016/j.neunet.2017.07.002

    ISSN 0893-6080. doi: https://doi.org/10.1016/j.neunet.2017.07.002. URLhttps://www. sciencedirect.com/science/article/pii/S0893608017301545. Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank J. Reddi, and Sanjiv Kumar. Are trans- formers universal approximators of s...

    2017 doi

  11. [2023]

    URLhttps://arxiv.org/abs/2310.04444. Peter J. Bickel and Bo Li. Local polynomial regression on unknown manifolds.Lecture Notes-Monograph Series, 54:177–186,

    arXiv

  12. [2024]

    URLhttps://proceedings.neurips.cc/paper_files/paper/2024/file/ 7f64034009f4a5fa417a57e1a987c5cd-Paper-Conference.pdf

    doi: 10.52202/079017-2202. URLhttps://proceedings.neurips.cc/paper_files/paper/2024/file/ 7f64034009f4a5fa417a57e1a987c5cd-Paper-Conference.pdf. Tokio Kajitsuka and Issei Sato. Are transformers with one layer self-attention using low-rank weight matrices universal approximator...

    2024 doi

  13. [2025]

    Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant

    URLhttps://arxiv.org/abs/ 2506.01115. Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. InAdvances in Neural Information Processing Systems,

    arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.