Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Understanding Learning Invariance in Deep Linear Networks

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read For deep linear networks with MSE, hard-wiring, data augmentation, and regularization share the same attainable solution.

desk verdict Genuine extension of prior work on invariance in deep linear networks, but the regularization-path continuity theorem is false as stated and the critical-point identity proof is incomplete. read the letter →

arxiv 2506.13714 v1 pith:XHAVVRZW submitted 2025-06-16 stat.ML cs.LGmath.STstat.TH

Hao Duan , Guido Montúfar This is my paper · ORCID
classification stat.MLcs.LGmath.STstat.TH MSC 68T0715A18
keywords invarianceequivariancedataaugmentationregularizationdeeplinearnetworkslosslandscapecriticalpointslow-rankapproximation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that, for mean-squared-error regression with deep linear networks, three standard ways of imposing a symmetry on the learned model — hard-wiring the invariant architecture, training on a data-augmented orbit, and adding a penalty that pushes toward invariance — lead to the same achievable solution. It characterizes the global optimum of each problem and proves that the set of critical points for hard-wiring and data augmentation is identical: every non-global critical point is a saddle, and the global optimum is the unique local minimum. Regularization admits a larger critical set, but the extra points are saddles too, and the regularized solution varies continuously and converges to the hard-wired optimum as the penalty grows. If the results hold, engineers can choose among these methods based on compute and data cost rather than on what the model can ultimately learn.

What carries the argument

The function space is the determinantal variety of rank-at-most-$r$ linear maps, which is non-convex when $r<\min\{d_0,d_L\}$. Each method produces a target matrix to be approximated by a rank-$r$ truncation: the orthogonal projection $Z_{\mathrm{inv}}=Z(I-\tilde G\tilde G^+)$ for hard-wiring, the orbit-averaged $Z_{\mathrm{da}}$ for augmentation, and the rescaled $Z_{\mathrm{reg}}(\lambda)=ZB(\lambda)^{-1}$ for regularization, with $B(\lambda)^2=I+n\lambda\tilde G\tilde G^T$. The Eckart–Young truncation of these targets, followed by a change of variables back to the original coordinates, yields the optima; the critical-point classification follows from the normal-space description of the determinantal variety.

What would settle it

Construct a target matrix $Z$ and an invariance projector so that the $r$-th and $(r+1)$-th singular values of $Z_{\mathrm{reg}}(\lambda)$ cross at some $\lambda^*$, then evaluate the rank-$r$ truncation on both sides of $\lambda^*$; a discontinuous jump in $\widehat W_{\mathrm{reg}}(\lambda)$ would refute Theorem 3.6 as stated, while a continuous limit would support it.

Watch

Extended reading notes

Core claim

Under MSE loss and a unitary representation of a finite cyclic group, the paper proves that data augmentation and hard-wiring have the same global optima and the same critical points in function space (Theorems 3.6, 3.9, Proposition 3.10). In all three formulations every non-global critical point is a saddle; the only local minimum is the global one. The regularized problem has more critical points — $\binom{m}{r}$ instead of $\binom{d}{r}$ — but they remain saddles, and the regularization path continuously connects the unregularized optimum to the constrained invariant optimum.

Load-bearing premise

For the regularization-path continuity theorem, the paper assumes $Z_{\mathrm{reg}}(\lambda)=ZB(\lambda)^{-1}$ has full rank for every $\lambda\ge 0$, but the proof additionally requires its singular values to fall into separated groups that never merge; full rank alone does not prevent $\sigma_r(\lambda)$ and $\sigma_{r+1}(\lambda)$ from crossing, where the best rank-$r$ approximation could jump.

Editorial extensions

If this is right

  • For any unitary representation, a model trained on all group-orbit copies of the data converges to a map that is exactly invariant and identical to the hard-wired optimum.
  • Because all non-global critical points are saddles and the global optimum is the unique local minimum, gradient-based training on any of the three objectives is not trapped by spurious local minima.
  • Increasing the regularization strength $\lambda$ moves the solution along a continuous path that ends at the hard-wired invariant solution.
  • The extra critical points introduced by regularization are harmless in the sense that none of them is a local minimum.
  • In this linear setting, the choice among hard-wiring, augmentation, and regularization changes computational cost, not the attainable function.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not claim it, but its rank-truncation argument suggests a practical priority: if the symmetry group is known and a hard-wired architecture is feasible, augmentation adds data and parameter cost without improving the optimum.
  • A testable nonlinear analogue follows: train two-layer networks under MSE with a bottleneck and compare late-training losses of augmented versus hard-wired models; the paper's appendix observations suggest the gap widens as capacity grows.
  • The continuity theorem is only proven under a full-rank assumption, so a synthetic check where $\sigma_r(\lambda)$ and $\sigma_{r+1}(\lambda)$ cross would reveal whether the regularization path can jump; the paper leaves this case open.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies mean squared error regression with deep linear networks under cyclic group invariance, comparing three ways of imposing invariance: hard-wired constraints, data augmentation, and explicit regularization. The main claimed results are that hard-wiring and data augmentation have identical global optima (Theorems 3.3, 3.8, 3.9), that the regularization path is continuous and converges to the hard-wired solution (Theorem 3.6), and that the critical points of the hard-wired and data-augmented problems coincide and consist of saddles plus the global optimum, while regularization introduces additional saddles (Proposition 3.10). The paper also includes MNIST experiments and an extension to shallow nonlinear networks via the neural tangent kernel. The writing is clear and the paper addresses a timely question, but several load-bearing technical claims require repair.

Significance. If the stated results hold, the paper gives a valuable theoretical explanation of when data augmentation, regularization, and hard-wired invariance are interchangeable in a non-convex rank-constrained linear model. The exact characterizations of global optima and critical points in a determinantal variety are useful, and the treatment is essentially parameter-free, deriving closed-form solutions from Eckart-Young theory and known critical-point results for low-rank approximation. The experiments provide supporting evidence for the linear setting and an honest discussion of limitations. However, the correctness of the central claims currently depends on unproved or false technical assumptions, so the significance can only be assessed after the issues below are resolved.

major comments (3)
  1. [Theorem 3.6 and Appendix A.5] Theorem 3.6 is false as stated. The proof invokes Lemma A.4 (Dieci et al.), whose hypothesis requires the singular values to split into disjoint groups that never merge, but the theorem only assumes that Z_reg(λ) = Z B(λ)^{-1} has full rank for all λ ≥ 0. Full rank does not prevent σ_r(λ) and σ_{r+1}(λ) from crossing at the truncation boundary, and at such a crossing the best rank-r approximation is not unique and a single-valued continuous regularization path need not exist. Concretely, take d0 = dL = 3, X = I_3, Y = Z with columns (0,2,0), (1,0,0), (0,0,ε) for 0 < ε < 1, and eG = diag(2,0,0). Then r = 1, nullity(eG) = 2, so the stated setting applies. With B(λ)^{-1} = diag(a,1,1) and a = 1/sqrt(1+12λ), the singular values of Z_reg are 2a, 1, ε. At λ0 = 1/4 they satisfy σ_1 = σ_2 = 1; for λ < λ0 the unique global optimum is 2a^2 e2 e1^T, and for λ > λ0 it is e1 e2^T. The one-sided limits at λ0 are distinct, so no continuous single-valued selection of global optima exists. The theorem can likely be repaired by adding a non-crossing assumption for the singular values, but as written the statement and its proof do not establish the advertised continuity.
  2. [Proposition 3.10 and Appendix A.8] The claim that the critical points of the constrained problem and the data-augmented problem are identical is not proved. The proof says the statement follows directly from Proposition A.10 and the fact that Z_inv and Z_da both have rank d, but the argument only counts C(d,r) critical points in each case. Equal cardinality does not imply equal sets. The two descriptions, U_inv Σ_inv_I V_inv^T P^{-1} and U_da Σ_da_I V_da^T Q^{-1}, are not shown to coincide for arbitrary subsets I; Theorem 3.9 establishes equality of the global optima, i.e., only for I = [r]. To support the set-identity claim, the authors need either to prove that the two objective functions agree up to an additive constant on the invariant subspace, or to provide a direct identification of the critical sets. Otherwise the statement should be weakened to an equality of cardinalities or to a statement about correspondence under the coordinate change used in the proofs.
  3. [Appendix A.7, Eq. (50)] The proof of Theorem 3.9 contains an algebraic error in the simplification of the averaged matrix. The step (Λ_g A Λ_g^†)_{ij} = (diag(Λ_g) diag(Λ_g)^†)_{ij} A_{ij} is not valid off the diagonal: in fact (Λ_g A Λ_g^†)_{ij} = λ_i A_{ij} \bar{λ}_j. Averaging over the group gives 1 whenever λ_i = λ_j, even when i ≠ j, so the averaged matrix is block diagonal with respect to the eigenspaces of ρ_X(g), not merely the 1-eigenspace. The final identity in Eq. (47) may still be salvageable because the subsequent multiplication by G on the left retains only the trivial block, but as written the proof needs a corrected derivation of the block structure. This is a load-bearing point because Theorem 3.9 is the central equivalence result between data augmentation and hard-wiring.
minor comments (5)
  1. [Remark 3.7] Remark 3.7 states that the full-rank assumption in Theorem 3.6 is mild and holds with probability 1, but the discussion after the theorem should acknowledge that full rank alone does not imply the non-crossing condition actually needed for continuity; the remark should be revised accordingly.
  2. [Theorem 3.3 proof, Appendix A.3] In Case 1 of the proof of Theorem 3.3, the sentence 'We assume Z has rank d' introduces an assumption that is not stated in the theorem; the case should be justified or the theorem statement amended.
  3. [Appendix A.12] There is a typo in the first paragraph of Appendix A.12: 'Besdies' should be 'Besides'.
  4. [Notation, Proposition 3.1] The symbol G is used both for the group and for the matrix I_{d0} - ρ_X(g); although the context distinguishes them, a different symbol for the matrix would improve readability.
  5. [After Proposition 3.10] The sentence beginning 'We observe that fully-connected linear networks have no spurious local minima... suggesting that also for our three optimization problems there are no spurious local minima' should be phrased as a conjecture or supported by a precise reference, since the preceding proposition concerns function-space critical points and the parameter-space statement is only heuristic.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the main results reduce to external Eckart-Young and Trager et al. theorems plus independent algebraic identities, and the noted Theorem 3.6 issue is a correctness gap rather than a circular step.

full rationale

The paper's derivation chain is self-contained against external benchmarks. Theorem 3.3 solves the hard-wired problem by projecting the target to Z_inv = Z(I - eG eG+) and applying the Eckart-Young theorem; Proposition 3.5 rewrites the regularized loss as a Frobenius low-rank approximation of Z_reg(lambda) = Z B(lambda)^{-1}; Proposition 3.8 does the same for the augmented target Z_da. These reductions are algebraic rewritings, not definitions that presuppose the conclusions. Theorem 3.9, claiming cW_da = cW_inv, is proved via the nontrivial algebraic identity in Equation 47 (Appendix A.7), and the definitions of Z_da and Z_inv are independent, so the equivalence is not true by construction. Proposition 3.10 imports the critical-point characterization from Trager et al. (2020, Theorem 28), an external source with no author overlap with the present paper, and applies it to the three target matrices; this is independent support, not a self-citation chain. Self-citations such as Brechet et al. (2023), Karhadkar et al. (2024), and Kohn et al. (2024a) appear only in Related Work and are not load-bearing. No fitted parameter is relabeled as a prediction; the theoretical claims are parameter-free. The only notable weakness is in Theorem 3.6: the proof invokes Lemma A.4 (Dieci et al.) to obtain smooth SVD factors, but Lemma A.4 requires singular values to lie in disjoint groups that never merge, whereas Remark 3.7 assumes only that Z_reg(lambda) is full rank. A singular-value crossing at the truncation boundary can make the best rank-r approximation discontinuous, so the stated continuity may fail without an additional non-crossing assumption. This is a mathematical correctness and rigor concern, not a circularity, because the claimed result would still be a substantive independent statement if the missing hypothesis were added. Accordingly, no circular step is identified and the circularity score is 0.

Assumptions & free parameters 0 free parameters · 9 assumptions · 0 invented entities

No fitted parameters are introduced; the analysis imports standard matrix approximation results and imposes generic-position assumptions (full-rank data, distinct singular values, unitary representation). No new entities or mediators are postulated.

assumptions (9)
  • domain assumption XX^T has full rank d0 so the positive definite square root P=(XX^T)^{1/2} exists
    Used in Section 3.1 to reparametrize the loss as a low-rank approximation in fW=WP.
  • domain assumption rank(Z_inv) > r for the constrained problem
    Theorem 3.3 and Proposition 3.10 require the projected target to have rank exceeding the rank bound; otherwise the rank constraint is vacuous or the SVD truncation is not well-defined.
  • domain assumption Z_reg(λ)=Z B(λ)^{-1} is full rank for all λ≥0
    Assumed in Theorem 3.6 and Remark 3.7 to make the regularized low-rank approximation well-defined and to support an SVD continuity argument.
  • domain assumption All non-zero singular values of Z_inv, Z_da, Z_reg(λ) are pairwise distinct
    Proposition 3.10 relies on this to count critical points as C(d,r) and C(m,r) without multiplicity issues.
  • domain assumption The representation ρX is unitary
    Theorem 3.9 and Proposition 3.8 use unitarity to make the group-averaging operator Hermitian and to commute Q^{-1} with ρX.
  • domain assumption The group G is finite cyclic with a given representation ρX
    The main theorems are proven for finite cyclic groups; Proposition 3.1 relies on W(I-ρ(g))=0 characterizing invariance.
  • standard math Eckart-Young-Mirsky theorem for Frobenius-norm low-rank approximation
    Used throughout to identify global optima as SVD truncations of the relevant target matrices.
  • domain assumption Trager et al. characterization of critical points on determinantal varieties
    Proposition A.10 imports the result that critical points of squared distance to a determinantal variety are SVD truncations of the target; this underpins Proposition 3.10.
  • standard math Dieci et al. SVD continuity lemma (Lemma A.4)
    Invoked in the proof of Theorem 3.6 to obtain smooth SVD factors, but the paper does not verify the lemma's disjoint-singular-value-group hypothesis.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Understanding Learning Invariance in Deep Linear Networks." pith.science (2026). https://pith.science/paper/XHAVVRZW

@misc{pith2026250613714,
  author       = {Pith},
  title        = {Pith review of: Understanding Learning Invariance in Deep Linear Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XHAVVRZW}},
  note         = {Machine review of arXiv:2506.13714}
}
read the original abstract

Equivariant and invariant machine learning models exploit symmetries and structural patterns in data to improve sample efficiency. While empirical studies suggest that data-driven methods such as regularization and data augmentation can perform comparably to explicitly invariant models, theoretical insights remain scarce. In this paper, we provide a theoretical comparison of three approaches for achieving invariance: data augmentation, regularization, and hard-wiring. We focus on mean squared error regression with deep linear networks, which parametrize rank-bounded linear maps and can be hard-wired to be invariant to specific group actions. We show that the critical points of the optimization problems for hard-wiring and data augmentation are identical, consisting solely of saddles and the global optimum. By contrast, regularization introduces additional critical points, though they remain saddles except for the global optimum. Moreover, we demonstrate that the regularization path is continuous and converges to the hard-wired solution.

Figures

Figures reproduced from arXiv: 2506.13714 by the authors.

Figure 2
Figure 2. Training curves for data augmentation (DA), regulariza￾tion (λ), and constrained model under mean squared error loss [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 4
Figure 4. ∥W − W⊥∥ 2 F /∥W∥ 2 F . critical points in both data augmentation and constrained models are identical, while regularization generally intro￾duces more critical points (Proposition 3.10). Though our theoretical results are for linear networks, since we consider a non-convex function space, it is natural to conjecture that some of the conclusions might carry over to other mod￾els with non-convex function space. Empir… view at source ↗
Figure 3
Figure 3. ∥W⊥∥F , where W⊥ is the non-invariant part of W. Interestingly, we can see that for both data augmentation and regularization, ∥Wc⊥∥F displays a “double descent”. Our conjecture is that the loss may also be decomposed into two parts, one controlling the error from invariance, and the other one controlling the error from the target. There￾fore, the gradient of the weights during training can be decomposed into two di… view at source ↗
Figures from the paper (5 more)
Figure 5
Figure 5. Figure 5: The spectrum of target matrices in the MNIST dataset. A.8. Proof of Proposition 3.10 Proposition A.10. Suppose the target matrix Z ∈ R dL×d0 has rank m > d > r. The critical points of ℓZ restricted to the function space Mr are all matrices of the form UΣIV T where I ∈ …
Figure 7
Figure 7. Figure 7: Training curves for data augmentation (DA), regulariza￾tion (λ), and constrained model under cross-entropy loss. In [PITH_FULL_IMAGE:figures/full_fig_p027_7.png]
Figure 8
Figure 8. Figure 8: ∥W⊥∥F where W⊥ is the non-invariant part of W under cross-entropy loss. 0 500 1000 1500 2000 2500 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 DA λ = 1e-2 λ = 1e-1 λ = 1 Epochs Proportion [PITH_FULL_IMAGE:figures/full_fig_p028_8.png]
Figure 10
Figure 10. Figure 10: Training curves for a two-layer NN with different nonlinear activation functions via data augmentation and hard-wiring on MNIST. N = 5000 Gaussian N = 5000 MNIST N = 1000 Gaussian N = 1000 MNIST Leaky ReLU ReLU Swish Tanh Leaky ReLU ReLU Swish Tanh 0 10 20 30 0 10 20 …
Figure 11
Figure 11. Figure 11: Median of the measure ϵinv(f) of discrepancy from invariance for 2-layer neural networks with different activation functions, trained on MNIST with data augmentation. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_11.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Cascading Through the Hierarchy: Regularizer-Induced Feature Detection as Phase Transitions in Deep Linear Neural Networks

    cond-mat.stat-mech 2026-08 conditional novelty 6.0 of 10

    For deep linear networks satisfying an alignment condition, the authors derive exact critical regularization strengths β_c = η_j at which each singular direction of the data covariance becomes learnable, via Landau-ty...

Reference graph

Works this paper leans on

38 extracted references · 26 canonical work pages · cited by 1 Pith paper

  1. [1]

    GρX (h) =G for allh∈G

    work page

  2. [2]

    That is to say,G is a projection operator fromX to the subspace allG-fixed points

    G is idempotent, i.e.,G 2 =G. That is to say,G is a projection operator fromX to the subspace allG-fixed points

    work page

  3. [3]

    If ρX is unitary, i.e.,ρX (h)†ρX (h) = Id for allh∈G , thenG is Hermitian. Proof

    work page

  4. [5]

    Gerken, J

    URL https://openreview.net/forum? id=3aQs3MCSexD. Gerken, J. E. and Kessel, P. Emergent equivariance in deep ensembles, 2024. URL https://arxiv.org/abs/ 2403.03103. Gideoni, Y . Implicitly learned invariance and equiv- ariance in linear regression, 2023. URL https:// openreview.net/pdf?id=ZnxYNriPlg. Gillis, N. and Shitov, Y . Low-rank matrix approximatio...

    arXiv 2024

  5. [7]

    URL https://link.springer.com/10.1007/ 978-3-662-07931-7

    doi: 10.1007/978-3-662-07931-7. URL https://link.springer.com/10.1007/ 978-3-662-07931-7 . Br´echet, P., Papagiannouli, K., An, J., and Mont ´ufar, G. Critical points and convergence analysis of generative deep linear networks trained with Bures-Wasserstein loss. In Proceedings of the 40th International Confer- ence on Machine Learning, volume 202 of Proc...

    work page doi:10.1007/978-3-662-07931-7

  6. [8]

    URL https:// doi.org/10.1007/978-1-4757-2189-8_9

    doi: 10.1007/978-1-4757-2189-8 9. URL https:// doi.org/10.1007/978-1-4757-2189-8_9 . Horn, R. A. and Johnson, C. R. Matrix analysis. Cambridge University Press, second edition, corrected reprint edition,

    doi:10.1007/978-1-4757-2189-8

  7. [11]

    cc/paper_files/paper/2018/file/ 5a4be1fa34e62bb8a6ec6b91d2462f5a-Paper

    URL https://proceedings.neurips. cc/paper_files/paper/2018/file/ 5a4be1fa34e62bb8a6ec6b91d2462f5a-Paper. pdf. Karhadkar, K., Murray, M., Tseran, H., and Mont ´ufar, G. Mildly overparameterized ReLU networks have a favor- able loss landscape. Transactions on Machine Learning Research, 2024. URL https://openreview.net/ forum?id=10WARaIwFn. Kawaguchi, K. Dee...

    arXiv 2018

  8. [12]

    eP11 eP12 eP† 12 eP22 #2

    Id0−d (58) By definition, we know that eP−2eXeX† = Id0. Therefore, " eP11 eP12 eP† 12 eP22 #2" eX1:deX† 1:d eX1:deX† (d+1):d0 eX(d+1):d0 eX† 1:d eX(d+1):d0 eX† (d+1):d0 # = Id0, (59) " eP 2 11 + eP12eP† 12 eP11eP12 + eP12eP22 eP† 12eP11 + eP22eP† 12 eP 2 22 + eP† 12eP12 #" eX1:deX† 1:d eX1:deX† (d+1):d0 eX(d+1):d0 eX† 1:d eX(d+1):d0 eX† (d+1):d0 # = Id0. ...

    work page

Show all 38 references
  1. [13]

    cc/paper_files/paper/2018/file/ a41b3bb3e6b050b6c9067c67f663b915-Paper

    URL https://proceedings.neurips. cc/paper_files/paper/2018/file/ a41b3bb3e6b050b6c9067c67f663b915-Paper. pdf. Li, Z., Wang, R., Yu, D., Du, S. S., Hu, W., Salakhutdi- nov, R., and Arora, S. Enhanced convolutional neural tangent kernels, 2019. URL https://arxiv.org/ abs/1911.00...

    2018 arXiv

  2. [16]

    cc/paper_files/paper/2019/file/ bdbca288fee7f92f2bfa9f7012727740-Paper

    URL https://proceedings.neurips. cc/paper_files/paper/2019/file/ bdbca288fee7f92f2bfa9f7012727740-Paper. pdf. Poston, T., Lee, C.-N., Choie, Y ., and Kwon, Y . Local minima and back propagation. In IJCNN-91-Seattle In- ternational Joint Conference on Neural Networks , vol- ume...

    2019

  3. [17]

    Ruben, G

    URL https://proceedings.mlr.press/ v202/puny23a.html. Ruben, G. and Zamir, S. Lower rank approximation of matrices by least squares with any choice of weights. Technometrics, 21(4):489–498, 1979. URL http:// www.jstor.org/stable/1268288. 11 Towards a Theoretical Understanding ...

    1979

  4. [18]

    Soltanolkotabi, M., Javanmard, A., and Lee, J

    URL https://openreview.net/forum? id=MG0mYskXN2. Soltanolkotabi, M., Javanmard, A., and Lee, J. D. The- oretical insights into the optimization landscape of over-parameterized shallow neural networks. IEEE Transactions on Information Theory , 65(2):742–769,

  5. [19]

    Song, Z., Woodruff, D

    URL https://ieeexplore.ieee.org/ document/8409482. Song, Z., Woodruff, D. P., and Zhong, P. Low Rank Approx- imation with Entrywise L1-Norm Error. In Proceedings of the 49th Annual ACM SIGACT Symposium on Theory of Computing, STOC 2017, pp. 688–701, New York, NY , USA, 2017. A...

    2017

  6. [20]

    URL https://epubs.siam.org/doi/abs/10

    doi: 10.1137/1.9780692196380. URL https://epubs.siam.org/doi/abs/10. 1137/1.9780692196380. Tahmasebi, B. and Jegelka, S. The exact sample complexity gain from invariances for kernel regression. In Thirty- seventh Conference on Neural Information Processing Systems, 2023. URL h...

    doi

  7. [23]

    cc/paper_files/paper/2017/file/ f22e4747da1aa27e363d86d40ff442fe-Paper

    URL https://proceedings.neurips. cc/paper_files/paper/2017/file/ f22e4747da1aa27e363d86d40ff442fe-Paper. pdf. Zhang, Z. and Zhao, K. Low-rank matrix approximation with manifold regularization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(7):1717– 1729, 20...

    2017 doi

  8. [24]

    Zitnick, C

    URL https://openreview.net/forum? id=SysEexbRb. Zitnick, C. L., Chanussot, L., Das, A., Goyal, S., Heras- Domingo, J., Ho, C., Hu, W., Lavril, T., Palizhati, A., Riviere, M., Shuaibi, M., Sriram, A., Tran, K., Wood, B., Yoon, J., Parikh, D., and Ulissi, Z. An introduc- tion to...

    2021 arXiv

  9. [28]

    We have GρX (h) = Z G ρX (g)dλ(g)ρX (h) = Z G ρX (gh)dλ(g) = Z G ρX (gh)dλ(gh) =G

    Here, we need to use the fact that the Haar measure is left-invariant, i.e.,λ(gA) =λ(A) for allg∈G andA∈A . We have GρX (h) = Z G ρX (g)dλ(g)ρX (h) = Z G ρX (gh)dλ(g) = Z G ρX (gh)dλ(gh) =G. (36)

  10. [29]

    (37) 17 Towards a Theoretical Understanding of Learning Invariance in Deep Linear Networks via Loss Landscapes

    To show that G is idempotent, we have G 2 = Z G ρX (g)dλ(g) Z G ρX (h)dλ(h) = Z G Z G ρX (g)ρX (h)dλ(g)dλ(h) = Z G Z G ρX (gh)dλ(g)dλ(h) = Z G Z G ρX (gh)dλ(gh)dλ(h) = Z G Gdλ(h) =G. (37) 17 Towards a Theoretical Understanding of Learning Invariance in Deep Linear Networks via...

  11. [30]

    eX1:deX† 1:d −1 0d,d0−d 0d0−d,d 0d0−d,d0−d # = eP−1 Id0− eP−1(Λg− Id0) eP−1(Λg− Id0) + eP−1. (54) We can see that eP 2 eP−2−

    To see the last property, we have G † = Z G ρX (g)†dλ(g) = Z G ρX (g)−1dλ(g) = Z G ρX (g)dλ(g) =G. (38) Lemma A.7. Given a finite groupG with ordern and a representationρ ofG on vector spaceV over field C, then for every g∈G , there exists a basisPg in which the matrix ofρ(g) ...

    1936

  12. [32]

    The theorem is proved

    (61) We have shown that the LHS equals the RHS in Equation 47. The theorem is proved. 22 Towards a Theoretical Understanding of Learning Invariance in Deep Linear Networks via Loss Landscapes 2 4 6 8 5 10 15 20 Z_inv Z_da Z_reg(lambda = 0.1) Index Singular Value Figure 5. The ...

  13. [33]

    They are all in the form of U inv Σinv I V invT P−1, whereI∈ [d]r

    (Constrained Space) The number of critical points in the optimization problem (4) is d r . They are all in the form of U inv Σinv I V invT P−1, whereI∈ [d]r. The unique global minimum isU inv Σinv [r] V invT P−1, which is also the unique local minimum

  14. [34]

    They are all in the form of U da Σda I V daT Q−1, whereI∈ [d]r

    (Data Augmentation) The number of critical points in the optimization problem (9) is d r . They are all in the form of U da Σda I V daT Q−1, whereI∈ [d]r. These critical points are the same as the critical points in the constrained function space. The unique global minimum isU...

  15. [35]

    They are all in the form of U reg Σreg I V regT B(λ)−1P−1, whereI∈ [m]r

    (Regularization) The number of critical points in the optimization problem (7) is m r . They are all in the form of U reg Σreg I V regT B(λ)−1P−1, whereI∈ [m]r. The unique global minimum is U reg Σreg [r] V regT B(λ)−1P−1, which is also the unique local minimum. Proof. This fo...

    2018

  16. [36]

    This suggests that it is more difficult to learn invariance from the data when the model has more parameters

    Effect of the size of the model: Compared to the case without a bottleneck middle layer,ϵinv(f,x ) is significantly smaller when there is a bottleneck. This suggests that it is more difficult to learn invariance from the data when the model has more parameters

  17. [37]

    For underdetermined linear models, i.e., when the number of data points exceeds the input dimension, Proposition 3.10 shows that all critical points are invariant

    Effect of the amount of training data: We notice thatϵinv(f,x ) is smaller when there are more training data. For underdetermined linear models, i.e., when the number of data points exceeds the input dimension, Proposition 3.10 shows that all critical points are invariant. How...

  18. [38]

    This suggests that the invariance learned from the data is fairly robust

    Robustness under distribution shift:Though the model is trained on MNIST,ϵinv(f,x ) does not increase significantly even when the model is tested on a completely different dataset. This suggests that the invariance learned from the data is fairly robust. 28 Towards a Theoretic...

    2000

  19. [1987]

    URL https://www.sciencedirect.com/ science/article/pii/0024379587901145. Gori, M. and Tesi, A. On the problem of local min- ima in backpropagation. IEEE Transactions on Pat- tern Analysis and Machine Intelligence , 14(1):76–86,

  20. [1992]

    Gunasekar, S., Lee, J

    URL https://ieeexplore.ieee.org/ document/107014. Gunasekar, S., Lee, J. D., Soudry, D., and Srebro, N. Implicit bias of gradient descent on linear convolutional networks. In Advances in Neural Information Pro- cessing Systems , volume 31. Curran Associates, Inc.,

  21. [2017]

    ISBN 978-0-521-54823-6 978-0-521-83940-2. Hu, E. J., Shen, Y ., Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models, 2021. URL https://arxiv. org/abs/2106.09685. Igashov, I., St¨ark, H., Vignac, C., Schneuing,...

    2021 arXiv

  22. [2018]

    cc/paper_files/paper/2018/file/ 0e98aeeb54acf612b9eb4e48a269814c-Paper

    URL https://proceedings.neurips. cc/paper_files/paper/2018/file/ 0e98aeeb54acf612b9eb4e48a269814c-Paper. pdf. Harris, J. Determinantal Varieties, pp. 98–113. Springer New York, New York, NY , 1992. ISBN 978-1-4757-2189-

    2018

  23. [2019]

    Mei, S., Misiakiewicz, T., and Montanari, A

    URL https://openreview.net/forum? id=Syx72jC9tm. Mei, S., Misiakiewicz, T., and Montanari, A. Learning with invariances in random features and kernel mod- els. In Proceedings of Thirty Fourth Conference on Learning Theory, volume 134 of Proceedings of Machine Learning Research...

  24. [2020]

    Xu, Z., Min, H., Tarmoun, S., Mallada, E., and Vidal, R

    URL https://openreview.net/forum? id=rkgOlCVYvB. Xu, Z., Min, H., Tarmoun, S., Mallada, E., and Vidal, R. Linear convergence of gradient descent for finite width over-parametrized linear networks with general initializa- tion. In Proceedings of The 26th International Conferenc...

  25. [2021]

    Mirsky, L

    URL https://proceedings.mlr.press/ v134/mei21a.html. Mirsky, L. Symmetric Gauge Functions and Unitary Invari- ant Norms. The Quarterly Journal of Mathematics, 11 (1):50–59, 1960. Moskalev, A., Sepliarskaia, A., Bekkers, E. J., and Smeul- ders, A. On genuine invariance learning...

    1960 arXiv

  26. [2023]

    URL https://proceedings.mlr.press/ v202/brechet23a.html. Chen, Z. and Zhu, W. On the implicit bias of linear equiv- ariant steerable networks. In Advances in Neural In- formation Processing Systems , volume 36, pp. 6132–

  27. [2024]

    URL https://link.springer.com/10.1007/ s10107-024-02058-3

    doi: 10.1007/s10107-024-02058-3. URL https://link.springer.com/10.1007/ s10107-024-02058-3 . Li, H., Xu, Z., Taylor, G., Studer, C., and Gold- stein, T. Visualizing the loss landscape of neural nets. In Advances in Neural Information Process- ing Systems , volume 31. Curran As...

    doi

  28. [2284]

    URL https://proceedings

    PMLR, 2023. URL https://proceedings. mlr.press/v206/xu23c.html. Zaheer, M., Kottur, S., Ravanbakhsh, S., Poczos, B., Salakhutdinov, R. R., and Smola, A. J. Deep sets. In Advances in Neural Information Process- ing Systems , volume 30. Curran Associates, Inc.,

    2023

  29. [5454]

    URL https: //doi.org/10.1007/s00009-005-0038-6

    doi: 10.1007/s00009-005-0038-6. URL https: //doi.org/10.1007/s00009-005-0038-6 . Dutta, A. and Li, X. On a problem of weighted low-rank approximation of matrices. SIAM Journal on Matrix Analysis and Applications , 38(2):530–553, 2017. doi: 10.1137/15M1043145. URL https://doi.o...

    arXiv

  30. [6155]

    URL https: //openreview.net/forum?id=DnVjDRLwVu

    Curran Associates, Inc., 2023. URL https: //openreview.net/forum?id=DnVjDRLwVu. Cohen, T. and Welling, M. Group equivariant convo- lutional networks. In Balcan, M. F. and Weinberger, K. Q. (eds.), Proceedings of The 33rd International Con- ference on Machine Learning , volume ...

    2023

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.