Understanding Two-Layer Neural Networks with Smooth Activation Functions

Changcun Huang

REVIEW 3 major objections 3 minor 30 references

Understanding Two-Layer Neural Networks with Smooth Activation Functions

T0 review · 3 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Two-layer sigmoid networks can be read as smooth spline constructions, with hidden units marking the knots and output weights carrying the polynomial coefficients.

desk verdict The spline interpretation is attractive and the question is right, but Lemma 3's divergent-coefficient step doesn't transfer from a shrinking interval to a fixed one, so the main approximation theorems are unsupported as written. read the letter →

arxiv 2507.14177 v2 pith:ZZL2Y2QI submitted 2025-07-11 cs.LG cs.AIcs.NAmath.NA

Changcun Huang This is my paper · ORCID

classification cs.LGcs.AIcs.NAmath.NA MSC 41A1541A3041A6368T07

keywords two-layerneuralnetworkssmoothactivationfunctionsuniversalapproximationsplinesbackpropagationexplanationgeneralizedsigmoidalunitszero-errorpointssmooth-continuityrestriction

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the 'black box' of a two-layer network with smooth sigmoidal or tanh hidden units is, at its core, a smooth-spline mechanism. The author proves that any sufficiently smooth function on the unit cube can be approximated to arbitrary accuracy by such a network, by first approximating the function with a multivariate smooth spline and then realizing each polynomial piece with hidden units whose weights and biases mark the spline's knots. In the resulting picture, the meaning of trained parameters is concrete: a weight vector and bias place a knot hyperplane, and the corresponding output weight sets the coefficient of the truncated-power term that joins two adjacent polynomial pieces. The paper further claims that backpropagation-trained solutions exhibit exactly this structure and verifies the claim on one- and two-dimensional examples. If correct, the result turns a heuristic fitting method into a deterministic, interpretable construction.

What carries the argument

The load-bearing mechanism is the truncated-power realization of a generalized sigmoidal unit (Lemma 3 in the univariate case, Lemma 11 in the multivariate case). By scaling the weight by $\rho > 1$ and adjusting the bias, the unit's activation becomes negligible on the 'zero part' before a knot while on the next region it behaves like $c(w^T x+b)^m_+$, with $c$ growing to infinity; this supplies the recurrence $s_\nu = s_{\nu-1} + c_\nu (w^T x+b)^m_+$ that connects spline pieces. Around it stand the generalized Wronskian matrix, which fixes the coefficients of the Taylor polynomial produced by global units, and the smooth-continuity restriction (Theorem 15), which propagates realized boundary polynomials into interior regions.

What would settle it

Train a two-layer sigmoidal network on $f(x)=x^3+3$ over $[0,1]$ until the $L^2$ error is below $10^{-4}$, compute each unit's zero-error point by truncation, and fit the best smooth spline of order 3 with those knots to the network output; if the best such spline cannot reproduce the output to within the training error, the Lemma 3 mechanism is not what training found.

Watch

Extended reading notes

Core claim

The central result, Theorem 18, states that for any $f \in C^m([0,1]^n)$, a two-layer network with generalized sigmoidal units can approximate $f$ with arbitrary precision by implementing a multivariate smooth spline $S(x) \in S^m_n(H,R)$; the number of hidden units required is at least $(\zeta^{1/n}-1)^n + \binom{n+m}{m}$, where $\zeta$ is the number of polynomial pieces in the spline. The construction has four parts: Taylor-series local approximation via a generalized Wronskian matrix; a strict partial order of knot hyperplanes that lets pieces be added one at a time; spline implementation in which a single unit's scaled, shifted activation mimics the truncated power $c(w^T x+b)^m_+$ on one side of a knot while vanishing on the other; and the smooth-continuity restriction, which says that in dimension $n \ge 2$, once the boundary polynomials around a region are realized, the interior polynomial is automatically produced. The author interprets this as the explanation of training solutions: hidden units are either global units building the initial polynomial or local units carrying spline knots, with output weights carrying the spline coefficients.

Load-bearing premise

Everything rests on Lemma 3: a single sigmoidal unit can be pushed so that its value on the zero side of a knot tends to zero while the coefficient of the truncated power on the other side tends to infinity, and the limiting identities derived from approximate equality on subintervals are valid.

Editorial extensions

If this is right

Any $f \in C^m([0,1]^n)$ is approximable by a two-layer sigmoidal network with the explicit unit count of Theorem 18.
Trained hidden units are interpretable: local units' weights and biases mark spline knots, and output weights are spline coefficients.
The backpropagation solution can in principle be constructed deterministically rather than found by gradient descent.
Negative units can be added freely at knots (two-sided bases) without changing the approximation, matching common experimental patterns.
For tanh units the same theory holds with one additional hidden unit, which experiments show is used to produce the constant term.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the mechanism is right, the same spline-and-knot reading should hold layer by layer in deep sigmoidal networks, with each layer implementing a hierarchical spline composition.
The zero-part scaling predicts a quantitative signature: in a trained network, each local unit's effective support should become more one-sided as training error decreases; this could be measured by truncation tests on held-out functions.
The theory suggests that the activation function's precise shape matters little beyond being smooth, monotone on the left tail, and tending to zero, so other saturating functions with similar tails should also produce spline-like solutions.
The smooth-continuity restriction implies a kind of boundary-value problem for network outputs, so some interior polynomial coefficients may be forced by boundary data; checking this in training could identify which parts of the solution are genuinely free.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Dear colleague,

The short version: the paper's core constructive claim — that a single sigmoidal unit can be squeezed into a truncated power on a fixed interval — does not go through as written. The stress-test note is right. In Lemma 3 the coefficient c'_{k-1} diverges on the shrinking interval (x'_k, x_k]; the proof then relabels the knot and asserts the same monomial on the fixed interval (x_k, x_{k+1}]. But the translated sigmoid either tends to 0 or to 1 on that fixed interval, depending on where the interval sits relative to the inflection point; it does not tend to (x-x_k)^m_+. Theorem 5, and everything that leans on it (Theorems 6, 7, 13, 16, 17, 18), collapses. The earlier coefficient-cancellation step (equations 3.30–3.34) is also hand-wavy: approximate equality on an interval doesn't imply the polynomials are coefficient-wise close, especially when the limit involves growing weights.

What's genuinely good: the paper asks a real question — what do trained two-layer smooth networks actually compute? The proposed answer, that trained units act as spline knots and polynomial pieces, is concrete and testable. Section 2's local Taylor/Wronskian construction is clearly presented and honestly acknowledges prior work (Xu & Cao 2005). The multivariate generalized-Wronskian ordering in Lemma 6 is a neat combinatorial device. The smooth-continuity restriction and boundary-determination analogue to ReLU networks is a plausible idea worth exploring, and the experimental plots do show that trained units often look like local/global units in the paper's sense.

But the load-bearing Lemma 3 is not a minor gap; it is the hinge of the entire universal-approximation mechanism. The multivariate Theorem 17 also depends on unproved boundary-determination results from Huang 2024, and the experimental verification uses manually tuned thresholds gamma1–gamma4 with no error analysis or baselines. Those are real but secondary concerns; the Lemma 3 problem is primary.

Who is this for? Researchers working on interpretability of sigmoidal networks or constructive approximation. They would find the framework suggestive, but they should not rely on the theorems as stated. With a repaired Lemma 3 (if one can be found) the paper could be a solid contribution; as it stands, the central claims are unsupported.

Recommendation: send it to peer review anyway. A serious referee could either find a fix for Lemma 3 or force a much weaker, honest statement (e.g., approximation on intervals whose length shrinks with the scale parameter). Desk-rejecting would bury a genuinely interesting research program over what may be a repairable error.

Referee Report

3 major / 3 minor

Summary. The paper proposes a constructive mechanism for two-layer networks with smooth activations, claiming that trained hidden units correspond to knots and polynomial pieces of a smooth spline. Section 2 builds local Taylor-series approximations using generalized Wronskian matrices; Section 3 introduces a 'zero-part error' lemma and uses it to implement univariate splines, culminating in a universal-approximation theorem for C^m functions; Sections 5-6 extend the construction to multivariate input via multivariate Wronskians and a smooth-continuity restriction; Section 7 gives qualitative experimental illustrations. The central claims are Theorem 18 (multivariate universal approximation) and the purported explanation of backpropagation-trained solutions.

Significance. If the construction were valid, it would give a deterministic, spline-based interpretation of two-layer sigmoidal networks, connecting local and global units to spline knots and providing a concrete mechanism for universal approximation. The local Wronskian-based construction is a plausible and interesting contribution, and the paper makes falsifiable experimental predictions about zero-error points and constant-producing tanh units. However, the global spline implementation rests on Lemma 3, whose proof has a load-bearing gap; without that lemma, Theorems 5-7 and 16-18 are not established. The experimental verification is suggestive but heuristic, relying on ad-hoc thresholds and not a quantitative validation of the theorems.

major comments (3)

[Section 3.3, Lemma 3, eqs. (3.42)-(3.45)] The proof obtains c'_k-1 -> +infinity from matching phi_t(x'_k)=epsilon' and phi_t(x_k)=epsilon on the interval (x'_k, x_k], whose length h=x_k-x'_k tends to 0 by eq. (3.41). It then deletes the knot x'_k, sets c_k=c'_k-1, and claims (3.19) on the fixed interval (x_k, x_{k+1}]. This transfer is not justified: a monomial approximation on a shrinking interval does not extend to a neighboring fixed interval. For the logistic sigmoid, the translated unit satisfies phi_a(x_k)=epsilon', but for every fixed x>x_k one has phi_a(x)->1 as rho->infinity; with output weight lambda=d_k/c_k set as in Theorem 5, lambda phi_a(x)->0 pointwise on (x_k, x_{k+1}], not d_k(x-x_k)^m. Hence the L2 error in Theorem 5 does not tend to 0 for the fixed spline S(x)=d_k(x-x_k)^m_+ unless the mesh interval also shrinks, which Theorem 5 does not assume. Since Theorem 6, and through it Theorems 7, 13, 16, 17, and 18, attach local units to fixed knots using Theorem 5, the central spline-implementation and universal-approximation claims collapse if this gap cannot be repaired.
[Section 3.3, eqs. (3.30)-(3.34)] The proof infers c_kappa approx 0 from s_{kappa+1}(x) approx 0 on intervals and from 'regarding' two polynomials as equal. Approximate equality on infinitely many points gives only a small L2 or pointwise difference; it does not imply exact identity of polynomial coefficients. In particular, eq. (3.31) cannot justify the conclusion s_1(x) approx -c_1(x-x_1)^m on [0,x_1] and hence c_1 approx 0. Even if eqs. (3.31)-(3.33) hold in the limit, the argument controls no rates and therefore cannot yield the monotone divergence of c_k claimed in part II of Lemma 3. This is the same load-bearing step as the previous major comment.
[Sections 6 and 7, Theorems 15-18] Theorem 17 is proved only by analogy with Huang (2024)'s lemma 10, and Theorem 15's proof invokes a 'boundary-determination principle' from the same unpublished reference. Since Lemma 11 reduces the multivariate zero-part construction to the univariate Lemma 3, the multivariate construction inherits the error identified above. Moreover, the manuscript does not state or prove the required boundary-determination principle for smooth splines; it only refers to a proof in Huang (2024). Consequently Theorem 18 is not independently verifiable from the material provided in this paper. The explanatory claims about training solutions in Section 7 are also not derived from the theorems: the local/global classification uses thresholds gamma_1,...,gamma_4 and heuristic criteria with no sensitivity analysis or quantitative error control connecting gradient-descent solutions to the constructed spline solutions.

minor comments (3)

[Section 3.3, Lemma 3] There are typos such as 'cure' for 'curve' in the text describing Figure 1a, and 'logistical' for 'logistic' in Section 7. The notation l0 and l+ is used in Sections 6-7 without a formal definition in this paper; it should be defined locally rather than only by reference to Huang (2024).
[Section 2.3, Lemma 2] The sentence 'then equation 2.24 becomes' appears to refer to equation 2.18, and the displayed determinant in eq. (2.22) has an extra parenthesis. These are minor presentation issues, but they make the construction harder to follow.
[Section 7.1] The thresholds gamma_1,...,gamma_4 are free parameters of the experimental classification. The paper does not report sensitivity to these thresholds, and the discrete criteria in eqs. (7.3)-(7.14) are not shown to converge to the theoretical definitions in Definitions 5 and 15. A brief discussion of this gap would strengthen the experimental section.

Circularity Check

3 steps flagged · score 5.0 of 10

Multivariate universal-approximation proof imports load-bearing definitions and a uniqueness principle from the same author's prior Huang (2024) preprint; univariate spline construction is not circular.

uniqueness imported from authors [Section 6.5, proof of Lemma 14 (Smooth-continuity restriction)]
"According to the 'boundary-determination principle' of splines in S1_n(H,R) (see the proof of Huang (2024)'s lemma 6), qi2(x) and qi4(x) for ŝi(x) can completely and uniquely determine qi3(x) due to the property of continuous piecewise linear functions, and so is the unique determination of q3(x) through q2(x) and q4(x) for Ŝ(x)."

The uniqueness that makes Theorem 15's 'simultaneously produced' conclusion go through is not derived in this paper; it is imported from the same author's prior arXiv preprint (Huang 2024, Lemma 6). This uniqueness is load-bearing because Theorem 15 is then used in Theorem 17 (and hence Theorem 18) to fill in un-implemented regions from boundary pieces. Since Huang (2024) is neither machine-checked nor independently reproduced in this manuscript, the step is a self-citation standing in for a proof.
self citation load bearing [Section 6.7, proof of Theorem 18]
"The proof is formally the same as that of Huang (2024)'s theorem 10. The preceding results of section 6 provided all the prerequisites to prove this theorem."

The paper's central claim (universal approximation for arbitrary input dimension) is explicitly handed off to Huang (2024), Theorem 10, from the same author. Theorem 17's proof similarly says 'the proof is analogous to Huang (2024)'s lemma 10.' Thus the final step of the claimed derivation chain—the n-dimensional implementation of splines over a standard partition—is not proved here but inherited from a same-author result whose assumptions and proof are not reproduced. This makes the headline universal-approximation result depend on a self-citation chain rather than on a self-contained proof in this manuscript.

1 more flagged steps

ansatz smuggled in via citation [Section 6.7, opening sentence defining the setting of Theorems 17-18]
"The concepts of 'standard partition' and 'universal global hyperplane' were defined in Huang (2024)."

Theorem 17, which feeds directly into Theorem 18, only applies to a 'standard partition' of U=[0,1]^n. That partition concept is adopted from the same author's prior work and is not defined or justified in this paper. As a result, the scope of the main universal-approximation construction is set by an imported ansatz from Huang (2024), whose definitions are not independently established here.

full rationale

The univariate part of the paper (Theorems 1-7) is not circular: Taylor's theorem, Wronskian nonsingularity, and spline recurrences are used to construct approximations, and the conclusion is not assumed in the premises. The same holds for the local multivariate construction of Theorems 9-10. However, the global multivariate architecture imports load-bearing structure from the author's own Huang (2024) preprint: the boundary-determination uniqueness principle (Lemma 14), the standard-partition framework (Theorem 17), and the proof skeleton of the central Theorem 18 are all taken from that same-author work and are not independently established in this manuscript. Because patterns 3 and 4 apply to the chain leading to the headline universal-approximation claim, the paper cannot be scored at 0-2. The score is 5: the central claim retains independent mathematical content in the univariate and spline-implementation steps, but one of the load-bearing multivariate uniqueness steps reduces to a self-citation. I do not classify the Lemma 3 shrinking-interval issue as circularity, as it is a mathematical correctness gap rather than an input-output identity.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical entities. Its conceptual constructs (local unit, zero-error point, smooth-continuity restriction) are definitions used to explain training solutions, but they rely on the hand-chosen thresholds and on external results from Huang (2024). The free parameters are the analysis thresholds, not the network parameters; the network weights and biases are constructed, not fitted.

free parameters (1)

classification thresholds gamma1, gamma2, gamma3, gamma4 = 0.01, 0.05, 0.001, 0.05 (varies per example)
Chosen by hand per experiment in Section 7 to classify units as local/global/inactive and constant-producing; no rule given, and values change across Figure 6 examples, allowing classification to adjust to theoretical expectations.

assumptions (5)

standard math Taylor's theorem with Lagrange remainder applies to smooth activation functions and target functions.
Used in Theorems 1 and 2 to control local approximation error.
domain assumption Generalized sigmoidal activation functions (monotone on (-inf,0], limit 0 at -inf) suffice; no upper-limit condition.
Definition 3; weaker than Cybenko's definition but used throughout.
ad hoc to paper Lemma 3's scaling construction: for any epsilon, there exist rho, gamma such that a scaled sigmoid approximates a truncated power with zero-part error arbitrarily small and coefficient c_k diverging to infinity.
This is the central constructive step; the proof is sketchy and contains an unflagged polynomial-identity argument (Section 3.3, equations 3.30-3.34).
ad hoc to paper Boundary-determination principle for continuous piecewise linear functions (from Huang 2024, lemma 6).
Used in Lemma 14 and Theorem 15; the result is cited from the author's prior preprint without proof in this paper.
ad hoc to paper Standard partition and universal global hyperplane constructions of Huang (2024) extend to smooth activations.
Theorem 17's proof is 'analogous to Huang (2024)'s lemma 10'; no details given here.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Understanding Two-Layer Neural Networks with Smooth Activation Functions." pith.science (2026). https://pith.science/paper/ZZL2Y2QI

@misc{pith2026250714177,
  author       = {Pith},
  title        = {Pith review of: Understanding Two-Layer Neural Networks with Smooth Activation Functions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZZL2Y2QI}},
  note         = {Machine review of arXiv:2507.14177}
}

read the original abstract

This paper aims to understand the training solution, which is obtained by the back-propagation algorithm, of two-layer neural networks whose hidden layer is composed of the units with smooth activation functions, including the usual sigmoid type most commonly used before the advent of ReLUs. The mechanism contains four main principles: construction of Taylor series expansions, strict partial order of knots, smooth-spline implementation and smooth-continuity restriction. The universal approximation for arbitrary input dimensionality is proved and the explanation of training solutions is given. Through the principles proposed, the mystery of ``black box'' of the solution space is largely revealed. The new proofs employed also enrich approximation theory.

Figures

Figures reproduced from arXiv: 2507.14177 by the authors.

**Figure 1.** Reduction of zero-part errors. Remark. The main purpose of constructing a spline from a piecewise linear approximation to the derivatives of f(x) is to make the knots of smooth splines evident, providing convenience for neural-network implementation. 3.3 Zero-Part Error of Units To activation function σ(x) of a generalized sigmoidal unit U, we call the interval (−∞, x0) with x0 < 0 the zero part of U, if the contr… view at source ↗

**Figure 2.** Principle of global approximation. 3.5 One-Sided Solutions To approximate univariate functions, the term “one sided” means that all wi’s of activation functions φi(x) = σ(wix + bi) are greater than (or less than) zero; otherwise, it is the “two-sided” case. Lemma 4. Let s(x) ∈ Sm 1 (∆) be a spline with Pk(x) the polynomial on the kth subinterval derived from ∆. Denote by φi(x) = σ(wix + bi) for i = 1, 2, . . . , ζ … view at source ↗

**Figure 3.** A two-sided solution. Lemma 5. Under theorem 6, to a local unit uk whose zero-error point is knot xk, suppose that a negative unit u ′ k is added whose activation function is φ ′ k (x) = σ(w ′ kx + b ′ k ) with w ′ k < 0 and whose zero-error point is also xk. Given any output weight λ ′ k of u ′ k , theorem 6 still holds by readjusting the remaining parameters of network N. Proof [PITH_FULL_IMAGE:figures/full_fig_p… view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Multiple expressions of a polynomial piece. [PITH_FULL_IMAGE:figures/full_fig_p042_4.png]

**Figure 5.** Figure 5: A standard partition. Proposition 8. Denote by H = {lij : 1 ≤ i ≤ n, 1 ≤ j ≤ Mi − 1} a set of n − 1- dimensional hyperplanes of R n , with the equation of lij being xi = j/Mi . The set H forms a standard partition of U = [0, 1]n and n strict partial orders including Pi…

**Figure 6.** Figure 6: Solution explanation for one-dimensional input. [PITH_FULL_IMAGE:figures/full_fig_p053_6.png]

**Figure 7.** Figure 7: Solution explanation for two-dimensional input. [PITH_FULL_IMAGE:figures/full_fig_p055_7.png]

**Figure 8.** Figure 8: Units for constant term predicted by corollary 1. [PITH_FULL_IMAGE:figures/full_fig_p056_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 28 canonical work pages

[1]

M., Lopez-de-Teruel, P

Almira, J. M., Lopez-de-Teruel, P. E., Romero-L\' o pez, D. J., & Voigtlaender, F. (2021). Negative results for approximation using single layer and multilayer feedforward neural networks. Journal of mathematical analysis and applications, 494(1), 124584

work page 2021
[2]

A., & Bonaiuto, J

Arbib, M. A., & Bonaiuto, J. J. (Eds.). (2016). From neuron to cognition via computational neuroscience. MIT Press, 73--127

work page 2016
[3]

Bishop, C. M. (1995). Neural networks for pattern recognition. Oxford university press, 126--128

work page 1995
[4]

Brualdi, R. A. (2004). Introductory combinatorics (4th ed.). Pearson Prentice Hall, 71--75

work page 2004
[5]

Cantarini, M., & Costarelli, D. (2025). Simultaneous approximation by neural network operators with applications to Voronovskaja formulas. Mathematische Nachrichten, 298(3), 871--885

work page 2025
[6]

Chen, T., Chen, H., & Liu, R. W. (1992). A constructive proof and an extension of Cybenko's approximation theorem. In Computing Science and Statistics: Statistics of Many Parameters: Curves, Images, Spatial Models, NY: Springer New York, 163--168

work page 1992
[7]

Chui, C. K. (1992). An introduction to wavelets. Academic press, 81--85

work page 1992
[8]

K., & Wang, R

Chui, C. K., & Wang, R. H. (1983). On smooth multivariate spline functions. Mathematics of computation, 41(163), 131--142

work page 1983

Show all 30 references

[9]

Costarelli, D., & Spigler, R. (2013). Constructive approximation by superposition of sigmoidal functions. Anal. Theory Appl, 29(2), 169--196

2013
[10]

Cybenko, G. (1989). Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems, 2, 303--314

1989
[11]

De Boor, C. (2001). A Practical Guide to Splines (Revised Edition). Springer-Verlag New York, 291--311

2001
[12]

J., & Ismailov, V

Guliyev, N. J., & Ismailov, V. E. (2018). On the approximation by single hidden layer feedforward neural networks with fixed weights. Neural Networks, 98, 296--304

2018
[13]

Haykin, S. (2009). Neural networks and learning machines (3rd ed.). Pearson Prentice Hall, 10--15

2009
[14]

Hecht-Nielsen, R. (1989). Theory of the backpropagation neural network. In Proceedings of the International Joint Conference on Neural Networks, Piscataway, NJ: IEEE, 593--605

1989
[15]

E., & Salakhutdinov, R

Hinton, G. E., & Salakhutdinov, R. R. (2006). Reducing the dimensionality of data with neural networks. Science, 313(5786), 504--507

2006
[16]

Hornik, K. (1993). Some new results on neural network approximation. Neural Networks, 6(8), 1069--1072

1993
[17]

Hornik, K., Stinchcombe, M., & White, H. (1989). Multilayer feedforward networks are universal approximators. Neural networks, 2(5), 359--366

1989
[18]

Huang, C. (2024). On the principles of ReLU networks with one-hidden layer. arXiv:2411.06728

2024 arXiv
[19]

Ismailov, V. E. (2012). Approximation by neural networks with weights varying on a finite set of directions. Journal of Mathematical Analysis and Applications, 389(1), 72--83

2012
[20]

R., Koester, J

Kandel, E. R., Koester, J. D., Mack, S. H., & Siegelbaum, S. A. (Eds.). (2021). Principles of neural science (6th ed.). New York: McGraw-hill, 241--272

2021
[21]

Karlin, S., & Studden, W. J. (1966). Tchebycheff systems: With applications in analysis and statistics. JOHN WILEY & SONS, 1--36

1966
[22]

Y., Pinkus, A., & Schocken, S

Leshno, M., Lin, V. Y., Pinkus, A., & Schocken, S. (1993). Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural Networks, 6(6), 861--867

1993
[23]

Nair, V., & Hinton, G. (2010). Rectified linear units improve restricted boltzmann machines. Proceedings of the 27th international conference on machine learning, New York: ACM, 807--814

2010
[24]

Pinkus, A. (1999). Approximation theory of the MLP model in neural networks. Acta numerica, 8, 143--195

1999
[25]

Powell, M. J. D. (1981). Approximation theory and methods. Cambridge university press, 241--253

1981
[26]

E., Hinton, G

Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323, 533--536

1986
[27]

Schumaker, L. (2007). Spline functions: basic theory (3rd. ed.). Cambridge university press, 363--461

2007
[28]

& White, H

Stinchcombe, M. & White, H. (1989) Universal approximation using feedforward networks with non-sigmoid hidden layer activation functions. In International Joint Conference on Neural Networks, IEEE, 613--617

1989
[29]

Walker, R. J. (1950). Algebraic curves. Princeton: Princeton University Press

1950
[30]

B., & Cao, F

Xu, Z. B., & Cao, F. L. (2005). Simultaneous L^p -approximation order for neural networks. Neural Networks, 18(7), 914--923

2005

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

M., Lopez-de-Teruel, P

Almira, J. M., Lopez-de-Teruel, P. E., Romero-L\' o pez, D. J., & Voigtlaender, F. (2021). Negative results for approximation using single layer and multilayer feedforward neural networks. Journal of mathematical analysis and applications, 494(1), 124584

work page 2021

[2] [2]

A., & Bonaiuto, J

Arbib, M. A., & Bonaiuto, J. J. (Eds.). (2016). From neuron to cognition via computational neuroscience. MIT Press, 73--127

work page 2016

[3] [3]

Bishop, C. M. (1995). Neural networks for pattern recognition. Oxford university press, 126--128

work page 1995

[4] [4]

Brualdi, R. A. (2004). Introductory combinatorics (4th ed.). Pearson Prentice Hall, 71--75

work page 2004

[5] [5]

Cantarini, M., & Costarelli, D. (2025). Simultaneous approximation by neural network operators with applications to Voronovskaja formulas. Mathematische Nachrichten, 298(3), 871--885

work page 2025

[6] [6]

Chen, T., Chen, H., & Liu, R. W. (1992). A constructive proof and an extension of Cybenko's approximation theorem. In Computing Science and Statistics: Statistics of Many Parameters: Curves, Images, Spatial Models, NY: Springer New York, 163--168

work page 1992

[7] [7]

Chui, C. K. (1992). An introduction to wavelets. Academic press, 81--85

work page 1992

[8] [8]

K., & Wang, R

Chui, C. K., & Wang, R. H. (1983). On smooth multivariate spline functions. Mathematics of computation, 41(163), 131--142

work page 1983