REVIEW 5 major objections 5 minor 2 cited by
Parameter Symmetry Potentially Unifies Deep Learning Theory
T0 review · 5 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read Parameter symmetry could unify deep learning theory.
desk verdict A plausible, clearly written position paper that is worth refereeing, but its empirical support rests on thresholded correlations rather than causal tests, and its key theoretical props are mostly the authors' own informal or in-preparation results. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the parameter symmetry group and the projection matrix $P_G$ onto its invariant subspace. A parameter $\theta$ is in a $G$-symmetric state when $P_G \theta = \theta$; the symmetry-breaking distance $\Delta_G = \|\theta - P_G\theta\|_2^2$ measures how far training has moved the parameters from that state, and the degree of symmetry counts how many pairwise neuron distances exceed a chosen threshold. This gives an operational definition of symmetry breaking and restoration, and it connects to a theorem: at a $G$-symmetric state the effective parameter dimension drops by $\mathrm{rank}(P_G)$, which in the lazy-training regime is equivalent to masking the neural tangent kernel features. The same projection yields a decomposition of the model near a symmetric state into a quadratic kernel model plus a smaller residual model, which is why feature learning is tied to symmetry.
What would settle it
A decisive check is to train the same teacher-student networks with all parameter symmetries removed and ask whether the sharp loss plateaus, complexity jumps, and collapsed representations still appear with unchanged timing; if they do, the claim that symmetry breaking drives the three hierarchies is falsified. A second check is to vary the threshold $\Delta_G^{\mathrm{th}}$ across a wide range and show that the reported coincidence between symmetry-breaking events and loss jumps disappears at alternative reasonable thresholds.
Extended reading notes
Core claim
The central claim is a unifying hypothesis: the hierarchies of learning dynamics, model complexity, and representation formation are primarily determined by parameter symmetry breaking and restoration. Concretely, the paper argues that the loss landscape of a network is organized into symmetry classes, that gradient-based training moves between these classes, and that each transition changes the effective number of parameters and therefore the model's capacity. It further argues that invariant, hierarchical, and universal representations require parameter symmetry: removing permutation symmetry eliminates neural collapse, while the double rotation symmetry of deep linear networks provably forces universally aligned representations across different models. All three hierarchies are presented as corollaries of the same symmetry mechanism rather than as separate phenomena.
Load-bearing premise
The load-bearing premise is that the hand-chosen threshold $\Delta_G^{\mathrm{th}}$ (0.05–0.2 in most experiments, 1 in one figure) faithfully tracks whether a true group-theoretic symmetry is broken, and that the measured symmetry transitions actually cause the loss jumps and representation changes rather than merely tracking a third factor such as learning-rate or gradient-noise dynamics.
Editorial extensions
If this is right
- If symmetry dynamics drive learning, then sharp loss drops during training should coincide with symmetry-breaking events, and removing symmetries should eliminate the corresponding plateaus.
- The space-quantization conjecture implies that under weight decay a layer contains at most a regularizer-dependent number of non-identical neurons, making effective model complexity and generalization essentially width-independent.
- Neural collapse should disappear when permutation symmetry is removed and reappear when it is restored.
- The double-rotation symmetry of deep linear networks forces universal alignment of representations across arbitrarily different models, giving a proof of a Platonic-representation-style statement in that setting.
- Practitioners should be able to engineer desired hierarchies by deliberately introducing or removing symmetries in models, loss functions, and data.
Reading between the lines
- Editorial inference: If the symmetry-to-symmetry picture transfers from deep linear to nonlinear networks, representation alignment between models may become predictable from the symmetry groups of their architectures, offering a design rule for cross-model compatibility.
- Editorial inference: The framework suggests a clean experiment: add a single scalar multiplier to a layer's weights (creating one new symmetry) and vary only the regularization strength; if hierarchy-like loss plateaus appear and disappear with that symmetry, the causal story is directly tested.
- Editorial inference: The threshold-based symmetry metric is a practical proxy; replacing it with a scale-invariant or topologically defined order parameter would make the theory robust to the choice of $\Delta_G^{\mathrm{th}}$.
- Editorial inference: The space-quantization argument could be extended from neuron weights to attention heads and vocabulary embeddings, predicting that representation rank saturates with width in transformers under weight decay, consistent with the ViT observations in the paper.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This position paper argues that parameter symmetry breaking and restoration is the unifying mechanism behind three empirically observed hierarchies in deep learning: the temporal hierarchy of learning dynamics (e.g., saddle-to-saddle jumps), the complexity hierarchy (e.g., width-independent generalization), and the representational hierarchy (e.g., neural collapse and representation alignment). The paper formalizes parameter symmetries with Definition 1 and Definition 2, defines the symmetry-breaking distance Delta_G in Eq. (3), and presents experimental evidence from MLPs, transformers, ViTs, and ResNets in which symmetry-breaking events are aligned with loss jumps, complexity changes, and representation-rank changes. It also appeals to several theoretical results, including Theorem 1 from the authors' prior work, Theorem 2 from a companion preprint, and Conjecture 1 from an in-preparation manuscript. The stated position is that these three hierarchies are primarily determined by symmetry dynamics, with implications for deliberately engineering symmetries to control learning. The paper itself acknowledges in Section 7 that validating or falsifying its three hypotheses remains the most important next step.
Significance. If the central position is correct, it offers a genuinely unifying research program: distinct phenomena currently explained by separate theories could be understood as different manifestations of a single symmetry-breaking/restoration dynamic, and it would provide a design principle for introducing or removing symmetries in models. The paper is valuable as a synthesis: it connects extensive prior literature on symmetry-induced saddles, stochastic collapse, neural collapse, and representation alignment, and it states three concrete, individually testable hypotheses. Its strengths include explicit operational definitions of symmetry-breaking distance, reproducible experimental protocols in Section A, and honest acknowledgment of alternative views in Section 7. However, the current empirical support is largely correlational and depends on hand-chosen thresholds in the symmetry-breaking measure, and the main theoretical props are informal or unpublished self-references.
major comments (5)
- [Section 3, Eq. (3), Figure 2] The operational definition of a symmetry-breaking event is threshold-dependent, and no threshold-stability analysis is reported. The text sets Delta_G_th between 0.05 and 0.2 in most experiments but uses 1.0 in Figure 4, and for the double-rotation symmetry in Figure 3 the measured quantity is the eigenvalue-difference proxy in Eq. (13) rather than Eq. (3). The alignment between symmetry-breaking events and loss jumps or rank changes could therefore be an artifact of threshold choice. The authors should report the detected event times, or the coincidence statistics, over a sweep of Delta_G_th, or use a scale-invariant statistic such as a relative increase in Delta_G. This is load-bearing because all three hypotheses are evaluated with this same measurement style.
- [Sections 3-5, central Position statement] The experiments demonstrate temporal coincidence between Delta_G growth and loss drops or representation-rank changes, but coincidence along the same optimization trajectory does not establish that symmetry breaking 'primarily determines' these hierarchies. A common third factor, such as effective learning rate, gradient scale, or stochastic noise, could drive both the symmetry measure and the observed leaps. Figure 5 provides an intervention for the representation hypothesis (removing permutation symmetry suppresses neural collapse), but no analogous intervention is provided for the dynamics and complexity hypotheses. The authors should either soften the 'primarily determined' claim to a correlational one or add intervention experiments, e.g., maintaining the parameters on the symmetric subspace during training or removing/applying individual symmetries and measuring whether the loss jumps and complexity adaptation disappear.
- [Section 4 and Appendix B.1, Theorem 1/Theorem 3] The main text claims that at a G-symmetric state the effective model dimension decreases by exactly rank(PG) 'throughout training.' The formal statement in Appendix B.1 shows that the reduced-parameter representation f' exists for all GD/SGD iterations, but the NTK feature-masking statement is proved only in the lazy-training limit (Eqs. (21)-(25) with kappa < 1/(2 lambda_max(A))). The text conflates the exact reduction of parameter count with the stronger kernel-regime claim. This should be flagged explicitly, since Section 4 uses the NTK statement to argue that symmetric states are low-capacity states from which gradient methods cannot escape.
- [Section 4, Conjecture 1 and Appendix B.2, Theorem 4] The space-quantization result supporting the Occam's-razor argument relies on the unverified regularity condition in Eq. (26): ||nabla_theta ell_0(...)|| <= K||theta||^q with K = K0 m^{-alpha}. This assumption does the real work in the proof, as Eq. (37) makes clear; without independent evidence that K decays with the number of active neurons for realistic permutable losses, the bound in Eq. (27) is conditional. The authors should prove or at least numerically verify this scaling condition for concrete losses such as MSE with ReLU or attention layers, or otherwise state the result as conditional.
- [Sections 4, 5, Appendix B; references [73], [87], [88]] Several load-bearing theoretical items are informal self-references or unpublished: Theorem 1 generalizes the authors' ICLR 2025 paper [88], Theorem 2 is taken from the authors' companion preprint [87], and Conjecture 1 is attributed to an in-preparation paper by two of the authors, cited as 'Removed for anonymity.' For a journal submission, these should be replaced with peer-reviewed versions, precise statements included in the paper, or proofs supplied in the appendix. Reference [73] in particular cannot be verified in its current form and should not be treated as a citable source.
minor comments (5)
- [Section 6, Eq. (7)] Equation (7) and the surrounding text contain severe Unicode artifacts (e.g., '⌟⟨⟨⟪rl⟫l⟩⟩⟪...') that make the displayed formula unreadable; this must be fixed before publication.
- [Appendix A.1, Eq. (12)-(14)] The notation Ndos is defined as the number of small-Delta_G pairs, but then Ndosb = h - Ndos is called the degree of symmetry breaking; the relationship of these counts to the number of broken generators discussed in Section 3 should be stated explicitly, since h is the number of neurons only for the sorted-neighbor procedure.
- [Appendix B.1] Minor typos: 'intialized' should be 'initialized', 'iteraiton' should be 'iteration', and 'kernalized' should be 'kernelized'.
- [Figure 2 caption] The caption does not state the threshold Delta_G_th used for the black dotted lines; the reader must infer it from Section A.1. Please report the exact threshold in the caption or in the main text.
- [Section 6 and Eq. (4)] The Taylor expansion in Eq. (4) is stated for small Delta_G but the validity condition on Delta_G relative to the Hessian is not discussed; a sentence clarifying the assumed smoothness and the order of the remainder would improve rigor.
Circularity Check
The Complexity and Representation hypotheses rest heavily on the authors' own companion results, including an anonymous in-preparation conjecture, and the empirical 'match' between symmetry-breaking counts and rank/loss is partly a re-thresholding of the same weight geometry; the paper retains independent experiments, so the circularity is real but partial.
-
self citation load bearing
[Section 4 ('Parameter Symmetry as an Occam's Razor'), Conjecture 1; references [73], [87], [88]]
"Conjecture 1. (Space Quantization Conjecture [73], Informal) In every layer with permutation symmetry, for two nonidentical neurons i and j, ∆ij > O(κβ) after training, where β > 0 and κ is the regularization strength. ... Thus, when regularized, there are, at most, a finite number of nonidentical neurons within a layer, however wide it is. This implies that the actual complexity of the trained network must decrease as a function polynomial in the regularization strength."
The claim that regularized networks have width-independent complexity is the paper's own Complexity Hypothesis, and it is derived from Conjecture 1, which is attributed to an in-preparation paper by two of the authors ('[73] Yizhou Xu and Liu Ziyin. Removed for anonymity. In Preparation.'). The special-case proof in Section B.2 assumes an unverified scaling K = K0 m^{-alpha} and does not establish the full conjecture. The central complexity conclusion is therefore supported by the authors' own unverified conjecture rather than by an independently checkable external result. The same self-citation pattern appears in the Representation Hypothesis, whose formal support is Theorem 2 imported from the authors' companion preprint [87].
-
other
[Section 5, Figure 4 caption; Appendix A.1 and A.4]
"This figure repeats the experiment in Ref. [47] and shows the rank and degree of symmetry breaking, which can be seen as the simplest metrics of the representation complexity, of different layers in a 5-layer FCN trained on CIF AR-10. ... As Figure 4 shows, these changes in the representation ranks match well the symmetry-breaking level of the layer, a strong evidence that symmetry may drive the formation of hierarchical representations."
The paper itself calls both quantities 'simplest metrics of the representation complexity.' Symmetry breaking is operationalized (Section A.1/A.4) as the number of pairwise neuron distances exceeding a hand-chosen threshold, and representation rank is a thresholded count of the same layer's effective dimensions. Because both are monotone summaries of the same weight geometry, their layer-wise correspondence is partly built into the definitions. The alignment is presented as causal evidence that symmetry drives representation hierarchy, but the explanatory variable is a re-thresholding of the quantity being explained, so it cannot independently confirm the Representation Hypothesis.
full rationale
This is a position paper, not a derivation of hard predictions, so much of it is legitimately a synthesis of prior results. The original experiments (Figures 2, 3, 4, 5, 10) give some independent content: for example, removing permutation symmetry changes neural collapse in Figure 5, an external manipulation rather than a definitional identity. However, the formal backbone for the Complexity and Representation hypotheses is unusually dependent on the authors' own prior work: Theorem 1 [88], Theorem 2 [87], Conjecture 1 [73], and the symmetry-removal method [88] all come from the same group, and [73] is even labeled 'Removed for anonymity.' The proof of Theorem 3 in Appendix B.1 and the special case of Conjecture 1 in B.2 are self-contained and reduce the citation dependence, but the full Space Quantization Conjecture remains an unverified self-citation that carries the width-independence claim. Separately, the empirical alignment between 'degree of symmetry breaking' and rank/loss jumps is weakened by the operational definition: thresholded pairwise distances and thresholded rank are two views of the same weight geometry, so a match is not an independent test of causation. These are real circularity concerns, but they are partial; the paper does not reduce entirely to its inputs, and a score of 5 reflects moderate circularity rather than a fully forced derivation.
Assumptions & free parameters
free parameters (2)
- Delta_G_th (symmetry-breaking distance threshold) =
0.05-0.2 in most experiments; 1.0 in Figure 4
- alpha and K0 (scaling exponent and constant in Theorem 4) =
unspecified
assumptions (4)
- domain assumption Parameter symmetries are the primary origins of the saddle points that shape learning dynamics (Refs [41, 83]).
- domain assumption Theorem 2 (informal) from Ref [87] on universal alignment of deep linear networks holds as stated.
- domain assumption Inequality (26): the gradient of ell_0 along swap directions is bounded by K times the distance to the q-th power, with K = K0 m^{-alpha}.
- domain assumption The three hierarchy phenomena (complexity jumps, simplicity bias, neural collapse, and related observations) are robust universal behaviors of trained networks.
Cite this review
Pith. "Pith review of Parameter Symmetry Potentially Unifies Deep Learning Theory." pith.science (2026). https://pith.science/paper/MH2EIJU4
@misc{pith2026250205300,
author = {Pith},
title = {Pith review of: Parameter Symmetry Potentially Unifies Deep Learning Theory},
year = {2026},
howpublished = {\url{https://pith.science/paper/MH2EIJU4}},
note = {Machine review of arXiv:2502.05300}
}
read the original abstract
The dynamics of learning in modern large AI systems is hierarchical, often characterized by abrupt, qualitative shifts akin to phase transitions observed in physical systems. While these phenomena hold promise for uncovering the mechanisms behind neural networks and language models, existing theories remain fragmented, addressing specific cases. In this position paper, we advocate for the crucial role of the research direction of parameter symmetries in unifying these fragmented theories. This position is founded on a centralizing hypothesis for this direction: parameter symmetry breaking and restoration are the unifying mechanisms underlying the hierarchical learning behavior of AI models. We synthesize prior observations and theories to argue that this direction of research could lead to a unified understanding of three distinct hierarchies in neural networks: learning dynamics, model complexity, and representation formation. By connecting these hierarchies, our position paper elevates symmetry -- a cornerstone of theoretical physics -- to become a potential fundamental principle in modern AI.
Figures
Figures from the paper (13 more)
Forward citations
Cited by 2 Pith papers
-
Observable- and Positional-Encoding-Dependent Symmetry Readout from Neural Network Weights
Symmetries readable from PE-equipped neural-field weights obey G_obs^exact ⊆ G_lift^exact(φ) ∩ G_true, so PE design structurally gates which groups Gram-type weight observables can detect.
-
PAC--Bayes Bounds on Quotient Parameter Spaces: Geometry-induced Implicit-Bias Priors
Identifying parameters that define the same predictor reduces the PAC-Bayes KL complexity term, and a geometry-tilted 'implicit-bias' prior can tighten the certificate when it is closer to the learned posterior.
Reference graph
Works this paper leans on
-
[88]
Remove symmetries to control model expressivity
Liu Ziyin, Yizhou Xu, and Isaac Chuang. Remove symmetries to control model expressivity. Interna- tional Conference on Learning Representations , 2025. A Experiments A.1 Measurement of ∆G Permutation Symmetry For the permutation symmetry in fully connected layers, we have described the ∆G for these pairwise symmetries. However, for a layer of width ψ, the...
work page 2025
-
[87]
Neural thermodynamics i: Entropic forces in deep and universal representation learning
Liu Ziyin, Yizhou Xu, and Isaac Chuang. Neural thermodynamics i: Entropic forces in deep and universal representation learning. arXiv preprint arXiv:2505.12387 , 2025
- [73]
-
[1]
Sgd learning on neural networks: leap complexity and saddle-to-saddle dynamics
Emmanuel Abbe, Enric Boix Adsera, and Theodor Misiakiewicz. Sgd learning on neural networks: leap complexity and saddle-to-saddle dynamics. In The Thirty Sixth Annual Conference on Learning Theory, pages 2552–2623. PMLR, 2023
2023
-
[2]
Understanding intermediate layers using linear classifier probes
Guillaume Alain. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644, 2016
arXiv 2016
-
[3]
Negative eigenvalues of the hessian in deep neural networks
Guillaume Alain, Nicolas Le Roux, and Pierre-Antoine Manzagol. Negative eigenvalues of the hessian in deep neural networks. arXiv preprint arXiv:1902.02366 , 2019
arXiv 1902
-
[4]
More is different: Broken symmetry and the nature of the hierarchical structure of science
Philip W Anderson. More is different: Broken symmetry and the nature of the hierarchical structure of science. Science, 177(4047):393–396, 1972
1972
-
[5]
The prevalence of neural collapse in neural multivariate regression
George Andriopoulos, Zixuan Dong, Li Guo, Zifan Zhao, and Keith Ross. The prevalence of neural collapse in neural multivariate regression. arXiv preprint arXiv:2409.04180 , 2024
arXiv 2024
Show all 91 references
-
[6]
Thermal forces from a micro- scopic perspective
Pietro Anzini, Gaia Maria Colombo, Zeno Filiberti, and Alberto Parola. Thermal forces from a micro- scopic perspective. Physical Review Letters, 123(2):028002, 2019. 10
2019
-
[7]
Neural networks as kernel learners: The silent alignment effect
Alexander Atanasov, Blake Bordelon, and Cengiz Pehlevan. Neural networks as kernel learners: The silent alignment effect. arXiv preprint arXiv:2111.00034 , 2021
2021 arXiv
-
[8]
Revisiting model stitching to compare neural representations
Yamini Bansal, Preetum Nakkiran, and Boaz Barak. Revisiting model stitching to compare neural representations. Advances in neural information processing systems , 34:225–236, 2021
2021
-
[9]
Nearly-tight vc-dimension and pseudodimension bounds for piecewise linear neural networks
Peter L Bartlett, Nick Harvey, Christopher Liaw, and Abbas Mehrabian. Nearly-tight vc-dimension and pseudodimension bounds for piecewise linear neural networks. Journal of Machine Learning Research , 20(63):1–17, 2019
2019
-
[10]
Representation learning: A review and new perspectives
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence , 35(8):1798–1828, 2013
2013
-
[11]
Geometric deep learning: Grids, groups, graphs, geodesics, and gauges
Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veliˇ ckovi´ c. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13478 , 2021
2021 arXiv
-
[12]
Stochastic collapse: How gradient noise attracts sgd dynamics towards simpler subnetworks
Feng Chen, Daniel Kunin, Atsushi Yamamura, and Surya Ganguli. Stochastic collapse: How gradient noise attracts sgd dynamics towards simpler subnetworks. arXiv preprint arXiv:2306.04251 , 2023
2023 arXiv
-
[13]
On lazy training in differentiable programming
Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. arXiv preprint arXiv:1812.07956 , 2018
2018 arXiv
-
[14]
Gradient descent on neural networks typically occurs at the edge of stability
Jeremy M Cohen, Simran Kaur, Yuanzhi Li, J Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. arXiv preprint arXiv:2103.00065 , 2021
2021 arXiv
-
[15]
A kernel theory of modern data augmentation
Tri Dao, Albert Gu, Alexander Ratner, Virginia Smith, Chris De Sa, and Christopher R´ e. A kernel theory of modern data augmentation. InInternational conference on machine learning, pages 1528–1537. PMLR, 2019
2019
-
[16]
L. Dinh, R. Pascanu, S. Bengio, and Y. Bengio. Sharp Minima Can Generalize For Deep Nets. ArXiv e-prints, March 2017
2017
-
[17]
An image is worth 16x16 words: Transformers for image recognition at scale
Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020
2010 arXiv
-
[18]
Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced
Simon S Du, Wei Hu, and Jason D Lee. Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced. Advances in neural information processing systems , 31, 2018
2018
-
[19]
Why molecules move along a temperature gradient
Stefan Duhr and Dieter Braun. Why molecules move along a temperature gradient. Proceedings of the National Academy of Sciences , 103(52):19678–19682, 2006
2006
-
[20]
The role of permutation invariance in linear mode connectivity of neural networks
Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. arXiv preprint arXiv:2110.06296 , 2021
2021 arXiv
-
[21]
The representation theory of finite groups
Walter Feit. The representation theory of finite groups . Elsevier, 1982
1982
-
[22]
Spontaneous symmetry breaking in neural networks
Ricky Fok, Aijun An, and Xiaogang Wang. Spontaneous symmetry breaking in neural networks. arXiv preprint arXiv:1710.06096, 2017
2017 arXiv
-
[23]
A regularity condition of the information matrix of a multilayer perceptron network
Kenji Fukumizu. A regularity condition of the information matrix of a multilayer perceptron network. Neural networks, 9(5):871–879, 1996
1996
-
[24]
Local minima and plateaus in hierarchical structures of multilayer perceptrons
Kenji Fukumizu and Shun-ichi Amari. Local minima and plateaus in hierarchical structures of multilayer perceptrons. Neural networks, 13(3):317–327, 2000
2000
-
[25]
On the role of neural collapse in transfer learning
Tomer Galanti, Andr´ as Gy¨ orgy, and Marcus Hutter. On the role of neural collapse in transfer learning. arXiv preprint arXiv:2112.15121 , 2021
2021 arXiv
-
[26]
Norm-based generalization bounds for compositionally sparse neural networks
Tomer Galanti, Mengjia Xu, Liane Galanti, and Tomaso Poggio. Norm-based generalization bounds for compositionally sparse neural networks. arXiv preprint arXiv:2301.12033 , 2023. 11
2023 arXiv
-
[27]
Stochastic training is not necessary for generalization
Jonas Geiping, Micah Goldblum, Phillip E Pope, Michael Moeller, and Tom Goldstein. Stochastic training is not necessary for generalization. arXiv preprint arXiv:2109.14119 , 2021
2021 arXiv
-
[28]
The implicit bias of depth: How incremental learning drives generalization
Daniel Gissin, Shai Shalev-Shwartz, and Amit Daniely. The implicit bias of depth: How incremental learning drives generalization. arXiv preprint arXiv:1909.12051 , 2019
1909 arXiv
-
[29]
On the symmetries of deep learn- ing models and their internal representations
Charles Godfrey, Davis Brown, Tegan Emerson, and Henry Kvinge. On the symmetries of deep learn- ing models and their internal representations. Advances in Neural Information Processing Systems , 35:11893–11905, 2022
2022
-
[30]
The platonic representation hypoth- esis
Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypoth- esis. arXiv preprint arXiv:2405.07987 , 2024
2024 arXiv
-
[31]
Batch normalization: Accelerating deep network training by reduc- ing internal covariate shift
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reduc- ing internal covariate shift. arXiv preprint arXiv:1502.03167 , 2015
2015 arXiv
-
[32]
Neural tangent kernel: Convergence and general- ization in neural networks
Arthur Jacot, Franck Gabriel, and Cl´ ement Hongler. Neural tangent kernel: Convergence and general- ization in neural networks. arXiv preprint arXiv:1806.07572 , 2018
2018 arXiv
-
[33]
Saddle-to-saddle dynamics in deep linear networks: Small initialization training, symmetry, and sparsity
Arthur Jacot, Fran¸ cois Ged, Berfin S ¸im¸ sek, Cl´ ement Hongler, and Franck Gabriel. Saddle-to-saddle dynamics in deep linear networks: Small initialization training, symmetry, and sparsity. arXiv preprint arXiv:2106.15933, 2021
2021 arXiv
-
[34]
Sgd on neural networks learns functions of increasing complexity
Dimitris Kalimeris, Gal Kaplun, Preetum Nakkiran, Benjamin Edelman, Tristan Yang, Boaz Barak, and Haofeng Zhang. Sgd on neural networks learns functions of increasing complexity. Advances in neural information processing systems , 32, 2019
2019
-
[35]
Weight decay induces low-rank attention layers
Seijin Kobayashi, Yassir Akram, and Johannes Von Oswald. Weight decay induces low-rank attention layers. arXiv preprint arXiv:2410.23819 , 2024
2024 arXiv
-
[36]
Similarity of neural network representations revisited
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning , pages 3519–3529. PMLR, 2019
2019
-
[37]
Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning
Daniel Kunin, Allan Ravent´ os, Cl´ ementine Domin´ e, Feng Chen, David Klindt, Andrew Saxe, and Surya Ganguli. Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning. Advances in Neural Information Processing Systems , 37:81157–81...
2024
-
[38]
Rethinking the limiting dynamics of sgd: modified loss, phase space oscillations, and anomalous diffusion
Daniel Kunin, Javier Sagastuy-Brena, Lauren Gillespie, Eshed Margalit, Hidenori Tanaka, Surya Gan- guli, and Daniel LK Yamins. Rethinking the limiting dynamics of sgd: modified loss, phase space oscillations, and anomalous diffusion. 2021
2021
-
[39]
Statistical Physics: Volume 5 , volume 5
Lev Davidovich Landau and Evgenii Mikhailovich Lifshitz. Statistical Physics: Volume 5 , volume 5. Elsevier, 2013
2013
-
[40]
Neural network renormalization group
Shuo-Hui Li and Lei Wang. Neural network renormalization group. Physical review letters , 121(26):260601, 2018
2018
-
[41]
Symmetry, saddle points, and global geometry of nonconvex matrix factorization
Xingguo Li, Zhaoran Wang, Junwei Lu, Raman Arora, Jarvis Haupt, Han Liu, and Tuo Zhao. Symmetry, saddle points, and global geometry of nonconvex matrix factorization. arXiv preprint arXiv:1612.09296, 1:5–1, 2016
2016 arXiv
-
[42]
Reconciling modern deep learning with traditional op- timization analyses: The intrinsic learning rate
Zhiyuan Li, Kaifeng Lyu, and Sanjeev Arora. Reconciling modern deep learning with traditional op- timization analyses: The intrinsic learning rate. Advances in Neural Information Processing Systems , 33:14544–14555, 2020
2020
-
[43]
What happens after sgd reaches zero loss?–a mathe- matical framework
Zhiyuan Li, Tianhao Wang, and Sanjeev Arora. What happens after sgd reaches zero loss?–a mathe- matical framework. In International Conference on Learning Representations , 2021
2021
-
[44]
The empirical impact of neural parameter symmetries, or lack thereof
Derek Lim, Theo Moe Putterman, Robin Walters, Haggai Maron, and Stefanie Jegelka. The empirical impact of neural parameter symmetries, or lack thereof. arXiv preprint arXiv:2405.20231 , 2024. 12
2024 arXiv
-
[45]
Abide by the law and follow the flow: Conserva- tion laws for gradient flows
Sibylle Marcotte, R´ emi Gribonval, and Gabriel Peyr´ e. Abide by the law and follow the flow: Conserva- tion laws for gradient flows. 2023
2023
-
[46]
Invariant and equivariant graph networks
Haggai Maron, Heli Ben-Hamu, Nadav Shamir, and Yaron Lipman. Invariant and equivariant graph networks. arXiv preprint arXiv:1812.09902 , 2018
2018 arXiv
-
[47]
The tunnel effect: Building data representations in deep neural networks
Wojciech Masarczyk, Mateusz Ostaszewski, Ehsan Imani, Razvan Pascanu, Piotr Mi lo´ s, and Tomasz Trzcinski. The tunnel effect: Building data representations in deep neural networks. Advances in Neural Information Processing Systems, 36, 2024
2024
-
[48]
Charge symmetry, quarks and mesons
Gerald A Miller, Bernard Marie Karel Nefkens, and Ivo ˇSlaus. Charge symmetry, quarks and mesons. Physics Reports, 194(1-2):1–116, 1990
1990
-
[49]
Deep neural networks have an inbuilt occam’s razor
Chris Mingard, Henry Rees, Guillermo Valle-P´ erez, and Ard A Louis. Deep neural networks have an inbuilt occam’s razor. Nature Communications, 16(1):220, 2025
2025
-
[50]
Towards understanding the role of over-parametrization in generalization of neural networks
Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. Towards understanding the role of over-parametrization in generalization of neural networks. arXiv preprint arXiv:1805.12076, 2018
2018 arXiv
-
[51]
In search of the real inductive bias: On the role of implicit regularization in deep learning
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. In search of the real inductive bias: On the role of implicit regularization in deep learning. arXiv preprint arXiv:1412.6614 , 2014
2014 arXiv
-
[52]
On connected sublevel sets in deep learning
Quynh Nguyen. On connected sublevel sets in deep learning. In International conference on machine learning, pages 4790–4799. PMLR, 2019
2019
-
[53]
Neural networks should be wide enough to learn disconnected decision regions
Quynh Nguyen, Mahesh Chandra Mukkamala, and Matthias Hein. Neural networks should be wide enough to learn disconnected decision regions. In International conference on machine learning , pages 3740–3749. PMLR, 2018
2018
-
[54]
Prevalence of neural collapse during the terminal phase of deep learning training
Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40):24652–24663, 2020
2020
-
[55]
An introduction to quantum field theory
Michael E Peskin. An introduction to quantum field theory . CRC press, 2018
2018
-
[56]
On generalization bounds for neural networks with low rank layers
Andrea Pinto, Akshay Rangamani, and Tomaso Poggio. On generalization bounds for neural networks with low rank layers. arXiv preprint arXiv:2411.13733 , 2024
2024 arXiv
-
[57]
Cosmology and broken discrete symmetry
John Preskill, Sandip P Trivedi, Frank Wilczek, and Mark B Wise. Cosmology and broken discrete symmetry. Nuclear Physics B , 363(1):207–220, 1991
1991
-
[58]
Neural collapse in deep homogeneous classifiers and the role of weight decay
Akshay Rangamani and Andrzej Banburski-Fahey. Neural collapse in deep homogeneous classifiers and the role of weight decay. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 4243–4247. IEEE, 2022
2022
-
[59]
Feature learning in deep classifiers through intermediate neural collapse
Akshay Rangamani, Marius Lindegaard, Tomer Galanti, and Tomaso A Poggio. Feature learning in deep classifiers through intermediate neural collapse. In International Conference on Machine Learning, pages 28729–28745. PMLR, 2023
2023
-
[60]
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120 , 2013
2013 arXiv
-
[61]
On the stepwise nature of self-supervised learning
James B Simon, Maksis Knutins, Liu Ziyin, Daniel Geisz, Abraham J Fetterman, and Joshua Albrecht. On the stepwise nature of self-supervised learning. arXiv preprint arXiv:2303.15438 , 2023
2023 arXiv
-
[62]
Geometry of the loss landscape in overparameterized neural networks: Symmetries and invariances
Berfin Simsek, Fran¸ cois Ged, Arthur Jacot, Francesco Spadaro, Cl´ ement Hongler, Wulfram Gerstner, and Johanni Brea. Geometry of the loss landscape in overparameterized neural networks: Symmetries and invariances. In International Conference on Machine Learning , pages 9722–...
2021
-
[63]
On the origin of implicit regular- ization in stochastic gradient descent
Samuel L Smith, Benoit Dherin, David GT Barrett, and Soham De. On the origin of implicit regular- ization in stochastic gradient descent. arXiv preprint arXiv:2101.12176 , 2021. 13
2021 arXiv
-
[64]
Noether’s learning dynamics: Role of symmetry breaking in neural networks
Hidenori Tanaka and Daniel Kunin. Noether’s learning dynamics: Role of symmetry breaking in neural networks. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 25646–25660....
2021
-
[65]
Optimizing mode connectivity via neuron alignment.Advances in Neural Information Processing Systems, 33:15300– 15311, 2020
Norman Tatro, Pin-Yu Chen, Payel Das, Igor Melnyk, Prasanna Sattigeri, and Rongjie Lai. Optimizing mode connectivity via neuron alignment.Advances in Neural Information Processing Systems, 33:15300– 15311, 2020
2020
-
[66]
Equivalences between sparse models and neural networks
Ryan J Tibshirani. Equivalences between sparse models and neural networks. Working Notes. URL https://www. stat. cmu. edu/˜ ryantibs/papers/sparsitynn. pdf , 2021
2021
-
[67]
The information bottleneck method
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000
2000 arXiv
-
[68]
Deep learning and the information bottleneck principle
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 ieee information theory workshop (itw) , pages 1–5. IEEE, 2015
2015
-
[69]
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017
2017
-
[70]
Asymptotic equivalence of bayes cross validation and widely applicable information criterion in singular learning theory
Sumio Watanabe and Manfred Opper. Asymptotic equivalence of bayes cross validation and widely applicable information criterion in singular learning theory. Journal of machine learning research , 11(12), 2010
2010
-
[71]
Linguistic collapse: Neural collapse in (large) language models
Robert Wu and Vardan Papyan. Linguistic collapse: Neural collapse in (large) language models. arXiv preprint arXiv:2405.17767, 2024
2024 arXiv
-
[72]
The janus effects of sgd vs gd: high noise and low rank
Mengjia Xu, Tomer Galanti, Akshay Rangamani, Lorenzo Rosasco, and Tomaso Poggio. The janus effects of sgd vs gd: high noise and low rank. 2023
2023
-
[74]
When does feature learning happen? perspective from an analytically solvable model
Yizhou Xu and Liu Ziyin. When does feature learning happen? perspective from an analytically solvable model. arXiv preprint arXiv:2401.07085 , 2024
2024 arXiv
-
[75]
Performance-optimized hierarchical models predict neural responses in higher visual cortex
Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan A Solomon, Darren Seibert, and James J DiCarlo. Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences , 111(23):8619–8624, 2014
2014
-
[76]
Feature learning in infinite-width neural networks
Greg Yang and Edward J Hu. Feature learning in infinite-width neural networks. arXiv preprint arXiv:2011.14522, 2020
2011 arXiv
-
[77]
Visualizing and understanding convolutional networks
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceed- ings, Part I 13 , pages 818–833. Springer, 2014
2014
-
[78]
Understanding deep learning requires rethinking generalization
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. 2017
2017
-
[79]
Symmetry teleportation for accelerated opti- mization
Bo Zhao, Nima Dehmamy, Robin Walters, and Rose Yu. Symmetry teleportation for accelerated opti- mization. Advances in Neural Information Processing Systems , 35:16679–16690, 2022
2022
-
[80]
Understanding mode connectivity via param- eter space symmetry
Bo Zhao, Nima Dehmamy, Robin Walters, and Rose Yu. Understanding mode connectivity via param- eter space symmetry. In UniReps: the First Workshop on Unifying Representations in Neural Models , 2023
2023
-
[81]
Improving convergence and generalization using parameter symmetries
Bo Zhao, Robert M Gower, Robin Walters, and Rose Yu. Improving convergence and generalization using parameter symmetries. arXiv preprint arXiv:2305.13404 , 2023. 14
2023 arXiv
-
[82]
Quadratic models for understanding neural network dynamics
Libin Zhu, Chaoyue Liu, Adityanarayanan Radhakrishnan, and Mikhail Belkin. Quadratic models for understanding neural network dynamics. arXiv preprint arXiv:2205.11787 , 2022
2022 arXiv
-
[83]
Symmetry induces structure and constraint of learning
Liu Ziyin. Symmetry induces structure and constraint of learning. In Forty-first International Confer- ence on Machine Learning , 2024
2024
-
[84]
Formation of representations in neural networks
Liu Ziyin, Isaac Chuang, Tomer Galanti, and Tomaso Poggio. Formation of representations in neural networks. International Conference on Learning Representations , 2025
2025
-
[85]
The probabilistic stability of stochastic gradient descent, 2023
Liu Ziyin, Botao Li, Tomer Galanti, and Masahito Ueda. The probabilistic stability of stochastic gradient descent, 2023
2023
-
[86]
Parameter symmetry and noise equilibrium of stochastic gradient descent
Liu Ziyin, Mingze Wang, Hongchao Li, and Lei Wu. Parameter symmetry and noise equilibrium of stochastic gradient descent. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024
2024
-
[89]
For all time steps t under GD or SGD, there exists a model f′(x, θ′) and sequence of parameters θ′ t such that for all x, f′(x, θ′ t)= f(x, θt), (18) where dim(θ′)= dim(PG)
-
[90]
Here, A ∶= PG ∑x∇θf(x, θ0)T∇θf(x, θ0)PG and A+ denotes the Moore–Penrose inverse of A
The kernalized model, g(x, θ)= limλ→0(λ−1f(x, λθ+ θ0)− f(x, θ0)), converges to θ∗ = A+ ∑ x ∇θf(x, θ0)T y(x) (19) under GD for a sufficiently small learning rate. Here, A ∶= PG ∑x∇θf(x, θ0)T∇θf(x, θ0)PG and A+ denotes the Moore–Penrose inverse of A. The second part of the theor...
-
[91]
(21) Therefore, g(x, θ) simplifies to a kernel model g(x, θ)=∇θ0 f(x, θ0)PGθ
By (20), close to any symmetric point θ0 (any θ0 for which PGθ0 = θ0), for all x, we have f(x, θ)− f(x, θ0)=∇θf(x, θ0)PG∆+ O(∥∆∥)2. (21) Therefore, g(x, θ) simplifies to a kernel model g(x, θ)=∇θ0 f(x, θ0)PGθ. (22) Let us consider the squared lossℓ(θ)= ∑x∣∣y(x)−g(x, θ)∣∣2 and ...
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.