REVIEW 3 major objections 4 minor 30 references
Deep Symmetric Autoencoders from the Eckart-Young-Schmidt Perspective
T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Symmetric autoencoders get rigorous SVD-based error bounds, and a new EYS initialization built from iterated SVD consistently beats standard initializations in experiments.
desk verdict Symmetric autoencoder bounds are real, EYS init is useful, but the finite-sample gap needs work before the practical claim is fully supported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The Eckart-Young-Schmidt theorem—the Hilbert-space statement that truncated SVD gives the best low-rank approximation, with squared error equal to the tail sum of squared singular values—is the anchor. The load-bearing mechanism is Lemma 4.1, a per-layer Pythagorean decomposition: for an orthogonal layer, the squared distance between the current encoder output and the decoder's reconstruction splits exactly into the squared residual of the orthogonal projection onto the layer's column space plus the squared distance between the next encoder output and the next decoder reconstruction, inflated by Lipschitz factors of ρ and $ρ^{{-1}}$. Iterating this identity turns a deep reconstruction error into a shallow, layerwise sum, letting the SVD control each term.
What would settle it
Take a random vector u with a known covariance eigengap, fix a bilipschitz activation, and compare Algorithm 1 run with S=200 to the population greedy construction computed exactly. If the first-layer empirical subspace deviates substantially from the top eigenvectors, or if a trained SOAE's reconstruction error exceeds the upper bound in Eq. (15), the paper's finite-sample transfer or its Theorem 4.1 bound is falsified.
Extended reading notes
Core claim
The paper establishes that the reconstruction error of a symmetric orthogonal autoencoder with bilipschitz activation ρ is squeezed between two sums over layers: each term is the mean squared error of projecting the k-th encoder output onto the span of the next orthogonal weight matrix, weighted by powers of Lip(ρ)^{-2} or Lip($ρ^{{-1}}$)^2. Iterating this decomposition and optimizing each projection greedily by SVD gives an upper bound on the optimal symmetric autoencoder's error involving only eigenvalue tails of successive covariance matrices. This justifies the EYS initialization as a principled, data-driven starting point rather than an ad hoc heuristic. The paper also formalizes three nested architecture classes (symmetric, biorthogonal, orthogonal), shows biorthogonal networks act as representation-consistent nonlinear projectors, and reports experiments where the constrained SBAE class often beats the unconstrained SAE class, suggesting constraints can regularize the loss landscape.
Load-bearing premise
The error guarantees require the activation to be bilipschitz, and the practical algorithm replaces population covariance operators by SVDs of just 200 training snapshots; if a chosen activation is not bilipschitz or the small-sample SVD projections are poor, the theoretical backing for the EYS initialization no longer holds.
Editorial extensions
If this is right
- The EYS initialization gives every symmetric autoencoder—SAE, SBAE, or SOAE—the same starting point: a network whose initial reconstruction error is explicitly bounded by eigenvalue tails, instead of a random guess.
- Depth and activation sharpness degrade worst-case reconstruction exponentially: the upper bound carries Lip(ρ^{-1})^{2k} per layer, so a low-sharpness activation is important for deeper nets.
- The optimal errors of the three nested classes are sandwiched between the same POD eigenvalue tail and the greedy EYS upper bound, so the cost of imposing orthogonality or biorthogonality is quantifiable.
- In the numerical study, EYS initialization speeds convergence and lowers final test error across all architectures and datasets, and constrained biorthogonal autoencoders can outperform unconstrained ones.
- The same EYS starting point also improves classical (non-symmetric) autoencoders in the experiments, although the theory developed here no longer applies directly to that case.
Reading between the lines
- An implication the paper leaves implicit is that the practical guarantee depends on finite-sample stability: Algorithm 1 estimates population covariances from roughly 200 training snapshots, so a natural next step is perturbation bounds on the greedy subspace sequence as S varies.
- The layerwise eigenvalue-tail form suggests a principled architecture search: choose each width n_{k+1} as the number of eigenvalues of Σ[E_k^*(u)] needed to reach a target tail, which would automate skeleton selection currently done heuristically.
- The same greedy projection construction could potentially be carried over to convolutional autoencoders by replacing SVD with separable or low-rank filter approximations, but the per-layer Pythagorean argument would need substantial reworking.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper develops a mathematical framework for deep symmetric autoencoders, in which the encoder and decoder share a common activation ρ and the architecture is built from paired encoders/decoders. Three nested classes are studied: unconstrained symmetric autoencoders (SAE), biorthogonal symmetric autoencoders (SBAE), and symmetric orthogonal autoencoders (SOAE). The main theoretical contribution is Theorem 4.1, which bounds the expected reconstruction error of any SOAE from above and below by sums of per-layer linear projection errors, with weights that are powers of Lip(ρ) and Lip(ρ^{-1}). Corollary 4.1 then derives a population-level upper bound in terms of eigenvalue tails of covariance matrices of iteratively transformed random variables. Based on this, the authors propose the 'EYS initialization' (Algorithm 1), which greedily applies empirical SVD to the current latent variables. Numerical experiments on three parameterized PDE datasets compare EYS with He initialization and with the constrained initialization of Otto et al., and report favorable accuracy and training dynamics.
Significance. If the bounds are correct, the paper gives a rigorous connection between deep symmetric autoencoders and classical Eckart-Young-Schmidt theory, offering a principled, data-driven initialization procedure with no tunable parameters. The theoretical derivation in Section 4.1 is elegant and appears sound under the stated bilipschitz assumption: Lemma 4.1 decomposes the reconstruction error into per-layer projection terms, and Theorem 4.1 follows by iteration. The proposed EYS initialization is a natural byproduct and is tested against a genuinely different baseline (He initialization). The code is publicly released, which is a concrete strength. The main weakness is that the practical justification of EYS relies on an empirical, finite-sample surrogate of the population-level construction, and the experiments do not provide repeated trials or error bars, so the strength of the empirical transfer claim is not fully evidenced.
major comments (3)
- [§4.1, Corollary 4.1] The definition of E*_{k+1} is dimensionally inconsistent. The text states E*_{k+1}(v) := ρ(V*_{k+1} E*_k(v) + b*_k), but V*_{k+1} is an n_k × n_{k+1} matrix, E*_k(v) ∈ R^{n_k}, and b*_k ∈ R^{n_k}. Thus V*_{k+1} E*_k(v) is an n_{k+1}-dimensional vector while b*_k is n_k-dimensional, so the sum is undefined. The correct definition should be E*_{k+1}(v) := ρ(V*_{k+1}^T (E*_k(v) − b*_k)), which matches Theorem 4.1 and Algorithm 1. As written, Eq. (16) and the surrounding derivation do not parse, and this needs to be fixed before the Corollary can be evaluated.
- [§4.2, Algorithm 1] The theoretical justification for EYS is given at the population level in Corollary 4.1, but Algorithm 1 replaces the true law P by the empirical measure of S = 200 snapshots, and no finite-sample analysis is provided. This gap is load-bearing for the practical claim: in the ROD case n0 = 4347 while S = 200, so the first-layer sample covariance has rank at most 199, the top n1 = 15 empirical eigenvectors are not guaranteed to approximate the population subspace, and the empirical tail sums used to motivate the bound are downward biased. The paper would need either a finite-sample stability bound, a convergence study in S, or at least carefully repeated trials with error bars before the claim that EYS is a principled starting point is fully supported by the experiments.
- [§5.3, Table 3 and Figure 6] The experiments are reported as single runs without error bars or multiple seeds, and Figure 5 compares EYS against the best of 100 He initializations. This makes claims such as 'SBAE often outperforms SAE' and 'EYS consistently outperforms its standard counterpart' difficult to assess, since the observed gaps might be within run-to-run variability. For a paper whose contribution includes a new initialization strategy, reporting one trajectory per configuration is insufficient; at least a few seeds per experiment, or error bars on the final MSE, are needed to support the empirical conclusions.
minor comments (4)
- [§1, Introduction] There is a typo in the first sentence of Section 1: 'we introduce the some notation' should be 'we introduce some notation'.
- [§3.3, Remark 3.1] The surjectivity of the SOAE parametrization via π_orth and of the SBAE parametrization via Eq. (8) is asserted, but no proof is given. Since surjectivity is important for the claim that the parametrization does not restrict the hypothesis class, a short argument or a reference would improve the presentation.
- [§5.2, Table 2] The sentence 'We use the criteria proposed in [11] to set the latent dimension' is vague; the relevant criteria from [11] should be stated explicitly or a specific section/page should be cited.
- [Appendix B] The claim that the generalized He initialization 'coincides exactly with the initialization proposed in [13]' for LeakyReLU appears numerically incorrect. For α = 1 (ReLU), the formula in the appendix gives σ = 1/n, whereas the standard He initialization uses σ = √(2/n). The harmonic-mean step leading to the closed-form σ is also not shown, so the derivation is hard to verify.
Circularity Check
No circularity: the error bounds and the EYS initialization are derived from the external Eckart-Young-Schmidt/POD machinery rather than from the quantities they purport to predict.
full rationale
The central derivation is self-contained and does not reduce to its inputs. Lemma 4.1 uses only the bilipschitz property of rho and the Pythagorean orthogonal decomposition against span(V_k) to sandwich each layer's reconstruction error between the linear projection error at that layer and the same projection error scaled by Lip(rho^-1)^2; no constant is fitted and no target MSE enters the proof. Theorem 4.1 iterates this lemma, and Corollary 4.1 obtains its upper bound by explicitly constructing a candidate SOAE whose layer-k projection is the POD projector onto the top n_{k+1} eigenvectors of Sigma[E*_k(u)]; the bound is then exactly the classical EYS tail sum from Theorem 2.2. This is a genuine constructive bound, not a restatement of the infimum. The EYS initialization (Algorithm 1) is the stated empirical version of that population-level construction, so it is designed to match Corollary 4.1 rather than fitted to reported results; its experimental validation is against the external He initialization and the Otto et al. benchmark. The only self-citation, [11], is used solely to choose latent dimensions in the experiments and is not load-bearing for either the theory or the main comparison. The finite-sample gap, in which Algorithm 1 replaces the population law by 200 snapshots without a concentration or stability analysis, weakens the practical justification, but it is a robustness limitation rather than a circular reduction of the prediction to its input.
Assumptions & free parameters
assumptions (4)
- domain assumption The activation ρ is bilipschitz with 0 < η ≤ |ρ(x)-ρ(y)|/|x-y| ≤ L for all x≠y
- standard math Eckart-Young-Schmidt theorem for compact operators on Hilbert spaces
- ad hoc to paper The parametrization of biorthogonal matrices in Eq. (8) is surjective
- domain assumption The empirical distribution on the training set approximates the true distribution of u
Cite this review
Pith. "Pith review of Deep Symmetric Autoencoders from the Eckart-Young-Schmidt Perspective." pith.science (2026). https://pith.science/paper/QNFY6TYS
@misc{pith2026250611641,
author = {Pith},
title = {Pith review of: Deep Symmetric Autoencoders from the Eckart-Young-Schmidt Perspective},
year = {2026},
howpublished = {\url{https://pith.science/paper/QNFY6TYS}},
note = {Machine review of arXiv:2506.11641}
}
read the original abstract
Deep autoencoders have become a fundamental tool in various machine learning applications, ranging from dimensionality reduction and reduced order modeling of partial differential equations to anomaly detection and neural machine translation. Despite their empirical success, a solid theoretical foundation for their expressiveness remains elusive, particularly when compared to classical projection-based techniques. In this work, we aim to take a step forward in this direction by presenting a comprehensive analysis of what we refer to as symmetric autoencoders, a broad class of deep learning architectures ubiquitous in the literature. Specifically, we introduce a formal distinction between different classes of symmetric architectures, analyzing their strengths and limitations from a mathematical perspective. For instance, we show that the reconstruction error of symmetric autoencoders with orthonormality constraints can be understood by leveraging the well-renowned Eckart-Young-Schmidt (EYS) theorem. As a byproduct of our analysis, we end up developing the EYS initialization strategy for symmetric autoencoders, which is based on an iterated application of the Singular Value Decomposition (SVD). To validate our findings, we conduct a series of numerical experiments where we benchmark our proposal against conventional deep autoencoders, discussing the importance of model design and initialization.
Figures
Figures from the paper (7 more)
Reference graph
Works this paper leans on
-
[1]
Najmeh Abiri, Bj¨ orn Linse, Patrik Ed´ en, and Mattias Ohlsson. Establishing strong imputation per- formance of a denoising autoencoder in a wide range of missing data problems. Neurocomputing, 365:137–146, 2019
work page 2019
-
[2]
The gap between theory and practice in function approximation with deep neural networks
Ben Adcock and Nick Dexter. The gap between theory and practice in function approximation with deep neural networks. SIAM Journal on Mathematics of Data Science , 3(2):624–655, 2021
2021
-
[3]
Johannes Ball´ e, Valero Laparra, and Eero P. Simoncelli. End-to-end optimized image compression. In International Conference on Learning Representations , 2017
work page 2017
-
[4]
An introduction to the proper orthogonal decomposition
Anindya Chatterjee. An introduction to the proper orthogonal decomposition. Current Science , 78(7):808–817, 2000
work page 2000
-
[5]
Autoencoder-based network anomaly detection
Zhaomin Chen, Chai Kiat Yeo, Bu Sung Lee, and Chiew Tong Lau. Autoencoder-based network anomaly detection. In 2018 Wireless telecommunications symposium (WTS) , pages 1–5. IEEE, 2018
work page 2018
-
[6]
On the properties of neural machine translation: Encoder–decoder approaches
Kyunghyun Cho, Bart van Merri¨ enboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder–decoder approaches. In Dekai Wu, Marine Carpuat, Xavier Car- reras, and Eva Maria Vecchi, editors, Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation , pages 103–111, Doha, Q...
work page 2014
-
[7]
A course in functional analysis , volume 96
John B Conway. A course in functional analysis , volume 96. Springer, New York, 2019
work page 2019
-
[8]
Convergence rates for learning linear operators from noisy data
Maarten V de Hoop, Nikola B Kovachki, Nicholas H Nelsen, and Andrew M Stuart. Convergence rates for learning linear operators from noisy data. SIAM/ASA Journal on Uncertainty Quantification , 11(2):480–513, 2023. 20
work page 2023
Show all 30 references
-
[9]
Real analysis and probability
Richard M Dudley. Real analysis and probability . Chapman and Hall/CRC, 2018
2018
-
[10]
The approximation of one matrix by another of lower rank.Psychometrika, 1(3):211–218, Sep 1936
Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank.Psychometrika, 1(3):211–218, Sep 1936
1936
-
[11]
A deep learning approach to reduced order mod- elling of parameter dependent partial differential equations
Nicola Franco, Andrea Manzoni, and Paolo Zunino. A deep learning approach to reduced order mod- elling of parameter dependent partial differential equations. Mathematics of Computation, 92(340):483– 524, 2023
2023
-
[12]
A comprehensive deep learning-based approach to reduced order modeling of nonlinear time-dependent parametrized pdes
Stefania Fresca, Luca Dede’, and Andrea Manzoni. A comprehensive deep learning-based approach to reduced order modeling of nonlinear time-dependent parametrized pdes. Journal of Scientific Comput- ing, 87:1–36, 2021
2021
-
[13]
Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, 2015
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, 2015
2015
-
[14]
G. E. Hinton and R. R. Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 313(5786):504–507, 2006
2006
-
[15]
Autoencoders, minimum description length and helmholtz free energy
Geoffrey E Hinton and Richard Zemel. Autoencoders, minimum description length and helmholtz free energy. In J. Cowan, G. Tesauro, and J. Alspector, editors, Advances in Neural Information Processing Systems, volume 6. Morgan-Kaufmann, 1993
1993
-
[16]
M.A. Kramer. Autoassociative neural networks. Computers & Chemical Engineering , 16(4):313–328,
-
[17]
Nonlinear principal component analysis using autoassociative neural networks
Mark A Kramer. Nonlinear principal component analysis using autoassociative neural networks. AIChE journal, 37(2):233–243, 1991
1991
-
[18]
Error estimates for deeponets: A deep learning framework in infinite dimensions
Samuel Lanthaler, Siddhartha Mishra, and George E Karniadakis. Error estimates for deeponets: A deep learning framework in infinite dimensions. Transactions of Mathematics and Its Applications , 6(1):tnac001, 2022
2022
-
[19]
M. Loeve. Probability Theory II. F.W.Gehring P.r.Halmos and C.c.Moore. Springer, 1978
1978
-
[20]
M¨ ucke, Sander M
Nikolaj T. M¨ ucke, Sander M. Boht´ e, and Cornelis W. Oosterlee. Reduced order modeling for parameter- ized time-dependent pdes using spatially and memory aware deep learning. Journal of Computational Science, 53:101408, 2021
2021
-
[21]
Nguyen, Raymond K
Thanh V. Nguyen, Raymond K. W. Wong, and Chinmay Hegde. Benefits of jointly training autoen- coders: An improved neural tangent kernel analysis, 2020
2020
-
[22]
Otto, Gregory R
Samuel E. Otto, Gregory R. Macchio, and Clarence W. Rowley. Learning nonlinear projections for reduced-order modeling of dynamical systems using constrained autoencoders. Chaos: An Interdisci- plinary Journal of Nonlinear Science , 33(11):113130, 11 2023
2023
-
[23]
Deep learning for reduced order modelling and efficient temporal evolution of fluid simulations
Pranshu Pant, Ruchit Doshi, Pranav Bahl, and Amir Barati Farimani. Deep learning for reduced order modelling and efficient temporal evolution of fluid simulations. Physics of Fluids , 33(10):107101, 10 2021
2021
-
[24]
Pytorch: An imperative style, high-performance deep learning library
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu ...
2019
-
[25]
A graph convolutional autoencoder approach to model order reduction for parametrized pdes
Federico Pichi, Beatriz Moya, and Jan S Hesthaven. A graph convolutional autoencoder approach to model order reduction for parametrized pdes. Journal of Computational Physics , 501:112762, 2024
2024
-
[26]
Reduced basis methods for partial differential equations: an introduction , volume 92
Alfio Quarteroni, Andrea Manzoni, and Federico Negri. Reduced basis methods for partial differential equations: an introduction , volume 92. Springer, Cham, 2015
2015
-
[27]
Methods of modern mathematical physics
Michael Reed and Barry Simon. Methods of modern mathematical physics. vol. 1. Functional analysis . Academic San Diego, San Diego, 1980
1980
-
[28]
Zur theorie der linearen und nichtlinearen integralgleichungen
Erhard Schmidt. Zur theorie der linearen und nichtlinearen integralgleichungen. Mathematische An- nalen, 63(4):433–476, Dec 1907
1907
-
[29]
Invertible autoencoder for domain adaptation
Yunfei Teng and Anna Choromanska. Invertible autoencoder for domain adaptation. Computation, 7(2), 2019. 21 Appendices A. Technical proofs Proof of Theorem 2.2. Recall that for a generic random vector z ∈ Rm one has E∥z − d∥2 ≥ E∥z − E[z]∥2 = Tr(Σ[z]), for all deterministic ve...
2019
-
[1992]
Neutral network applications in chemical engineering
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.