Data Augmentation and Regularization for Learning Group Equivariance

Axel Flinth; Oskar Nordenfors

REVIEW 2 major objections 5 minor 18 references

Data Augmentation and Regularization for Learning Group Equivariance

T0 review · 2 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Training on augmented data with a penalty that shrinks the non-equivariant part of the weights makes the equivariant subspace an exponentially attracting set for gradient flow, provided the regularization strength is large enough.

desk verdict A modest but real extension: regularization can make the equivariant subspace an attractor, and the proof gap in the manuscript is repairable. read the letter →

arxiv 2502.06547 v1 pith:OW6R2KYX submitted 2025-02-10 stat.ML cs.LGmath.OC

Oskar Nordenfors , Axel Flinth This is my paper · ORCID

classification stat.MLcs.LGmath.OC MSC 68T0720C3537C10

keywords groupequivariancedataaugmentationregularizationgradientflowattractorequivariantsubspaceneuralnetworksrepresentationsofcompactgroups

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper aims to prove that a neural network can become group-equivariant through training rather than by design. The authors show that if ordinary, non-constrained layers are trained on data augmented by a compact group's symmetries, and a penalty is added that shrinks the part of the weights lying outside the equivariant subspace, then the equivariant subspace becomes an exponentially attracting set for the training dynamics. The result matters because it offers a theoretical justification for a widely used practical trick: with enough regularization, augmented training should yield approximately equivariant models without hard-coding the symmetry into the architecture.

What carries the argument

The central object is the regularized augmented gradient flow $\dot A = -\Pi_L \nabla R_{\mathrm{aug}}(A) - \gamma \Pi_{E^\perp} A$, where $E$ is the subspace of equivariant architectures, $L$ is the admissible architecture subspace, and $\Pi_{E^\perp}$ projects onto the orthogonal complement of $E$. The mechanism that carries the proof is the decoupling of this flow into motion along $E$ and motion perpendicular to $E$: Fact A identifies $\Pi_L \nabla R_{\mathrm{aug}}(X)$ with $\Pi_E \nabla R(X)$ on $E$, and Fact B keeps the curvature term inside $TE^\perp$, yielding $\dot Y = -\Pi_L (R_{\mathrm{aug}})''(X)Y - \gamma Y + O(\|Y\|^2)$.

What would settle it

Take a network and an invariant loss for which the augmented risk at points at distance $r$ from $E$ grows without bound as the point moves along $E$, start the regularized augmented gradient flow from those points, and check whether one finite $\gamma$ gives exponential convergence for all of them; if the required $\gamma$ depends on position along $E$, the theorem's 'for any $r$' statement fails.

Watch

Extended reading notes

Core claim

The central claim is Theorem 3: under an invariant loss, a commuting projection condition, a lower bound on the second derivative of the augmented risk along the orthogonal complement of the equivariant subspace, and a uniform bound on the third derivative, for every $r > 0$ one can choose the regularization parameter $\gamma$ large enough that the projected gradient flow $\dot A = -\Pi_L \nabla R_{\mathrm{aug}}(A) - \gamma \Pi_{E^\perp} A$, started within distance $r$ of $E$, converges to $E$ exponentially fast. The proof splits the dynamics into motion along $E$ and motion perpendicular to $E$, shows that the perpendicular component obeys $\dot Y = -\Pi_L (R_{\mathrm{aug}})''(X)Y - \gamma Y + O(\|Y\|^2)$, and uses Grönwall's inequality to get exponential decay when $\gamma$ exceeds a threshold set by the curvature bound and an a priori bound on $\|Y\|$.

Load-bearing premise

The proof needs a uniform upper bound on the augmented risk at the starting point across the whole allowed neighborhood of the equivariant subspace; if that risk is unbounded along the subspace, no single regularization strength chosen from the radius alone may force convergence.

Editorial extensions

If this is right

If Theorem 3 is correct, equivariant models can be obtained without a priori weight constraints: training on augmented data plus a computable penalty drives the network into the equivariant subspace, so equivariance emerges from the training procedure.
Because the convergence is exponential, small deviations from equivariance introduced by stochastic or early-stopped training are damped rather than amplified, provided $\gamma$ is large enough.
The comparison with plain weight decay shows augmentation gives a strict advantage: the cross term $\langle \Pi_L \nabla R(X), Y \rangle$ vanishes only in the augmented case, so the required $\gamma$ can be smaller.
Starting near a strict local minimum of the equivariant-mode training, a large $\gamma$ makes the regularized augmented flow converge to that minimum, giving a local stability guarantee for the trained model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

An immediate extension, not studied here, is to anneal $\gamma$ during training: starting with large $\gamma$ to pull the network to $E$, then lowering it to escape poor equivariant minima; the local-stability analysis suggests this could work but is not proven.
The result is stated for gradient flow, but the experiment uses SGD with random augmentations; whether the attractor property survives finite step sizes and stochastic gradients is an implicit conjecture that the experiment only partially supports.
The theorem's quantitative bound hides a dependence on initial risk; an explicit bound on that risk would turn the result into a practical recipe for choosing $\gamma$.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Look, this is a short paper that does two things: it summarizes the authors' prior framework for data augmentation, and it adds one new theorem—if you add weight decay on the non-equivariant part of the weights, the equivariant subspace E becomes an attractor for augmented gradient flow. That's a genuine extension over their earlier work, which only had weak stability. The proof is a Lyapunov/Gronwall argument, and it's mostly clean. The assumptions are stated, and the paper doesn't oversell: it explicitly notes the experiment uses SGD with random augmentation while the theory is gradient flow with perfect augmentation.

The main soft spot is the quantifier step in Theorem 3. As written, the proof claims that because α depends only on ||Y0||, you can choose γ for all starts within distance r. That's not quite right—α also depends on R_aug(A0), which can grow as you move along E. However, this is fixable. The stress-test note shows you can use a forward-invariance argument: the energy estimate gives d/dt ||Y||² ≤ −2(σ+γ)||Y||² + O(||Y||³), so for ||Y0|| < r you get exponential decay with a single γ depending on r, M, and σ, without any bound on the initial risk. So the claim stands, but the manuscript's proof as written has a gap that needs correcting.

The experiment is small—MNIST, 30 runs, distance to E only. It's not a strong test of the theorem, and the SGD/random augmentation mismatch means it's suggestive rather than confirmatory. That's acknowledged, and I don't think it's a fatal flaw, but a referee should ask for either a better experiment or at least a clear statement that the experiment is only a heuristic check.

The citation pattern is fine. The paper leans on the authors' own [10], but that's an independent published result with proofs, so it's not circular.

Bottom line: the core claim is plausible and the proof gap is repairable. This deserves a serious referee, but the referee should ask for the revised proof and a more careful discussion of the experiment's limitations. If you work on augmentation or equivariance theory, this is worth a read, and I'd probably cite the attractor result if I were writing in this area.

Referee Report

2 major / 5 minor

Summary. The paper studies whether data augmentation plus a regularization penalty on the non-equivariant part of the weights can make the equivariant subspace E an attractor for training dynamics. Building on the authors' prior framework [10], it states Theorem 3: under an invariant loss, the compatibility condition ΠLΠG = ΠGΠL, a lower curvature bound on TE⊥, and a uniform third-derivative bound, for every r > 0 there exists γ large enough such that the regularized augmented gradient flow A_dot = -ΠL∇R_aug(A) - γΠ_{E⊥}A, started within distance r of E, converges to E exponentially fast. The paper also presents a small numerical experiment on rotated MNIST with CNNs, comparing augmented and non-augmented SGD training under various γ, and reports that augmented training stays closer to E for moderate and large γ.

Significance. If Theorem 3 is correct, the paper provides a useful theoretical justification for a common practical recipe: rather than restricting weights a priori to equivariant architectures, one can train with augmented data and a simple regularization penalty and still obtain approximately equivariant networks. The result is a natural and worthwhile extension of the authors' earlier work, and the paper is clearly written. Credit is due for making the experimental code publicly available and for honestly summarizing the prior framework. The main caveat is that the numerical experiments use SGD with random augmentation, which is not covered by the gradient-flow theorem, so the empirical confirmation is indirect. Overall the contribution is incremental but sound in its central idea, provided the proof gap in Theorem 3 is repaired.

major comments (2)

[Section III, Theorem 3 proof, Eq. (3)] The proof of Theorem 3 contains a quantification gap. The inequality in Eq. (3) gives d/dt(½||Y||²) ≤ (C√α - σ - γ)||Y||², where α = 2R_aug(A0)/γ + ||Y0||². The text then states "Since α only depends on ||Y0||" and concludes that γ can be chosen depending only on r. This is not correct as written: α depends on R_aug(A0), which is not uniformly bounded over the r-neighborhood of E. Consequently, the written argument does not establish that a single γ works for all starts with dist(A0,E) < r. This is a load-bearing issue for the theorem's statement. The gap is repairable: using the uniform third-derivative bound, the Y-dynamics satisfy d/dt(½||Y||²) ≤ (M||Y|| - σ - γ)||Y||², and for γ > Mr - σ the sublevel set ||Y|| ≤ r is forward invariant, yielding exponential decay with a γ depending only on r, M, and σ. The manuscript should incorporate such a repair or explicitly state the additional uniform bound on initial risk.
[Section IV, Experiments] The numerical experiment uses SGD with random augmentations, whereas Theorem 3 concerns gradient flow with the full (Haar-averaged) augmented risk. The paper acknowledges this, saying the results "should also have an impact in the stochastic setting," but that is a conjecture, not a consequence of the theorem. As written, the experiment does not directly verify the theorem's assumptions or conclusions; it only provides suggestive evidence. The claim in the conclusion that the results "are born out in practice" is therefore stronger than what the experiment can support. The authors should either state this limitation clearly in the experimental section or prove a stochastic analogue; at minimum they should temper the concluding claim.

minor comments (5)

[Notation throughout] The notation for the orthogonal complement is inconsistent: the paper uses both TE⊥ and Π_{E⊥} and sometimes writes E⊥ without specifying the ambient space. A single consistent definition, e.g., Π_{E⊥} for the projection onto (TE)⊥, would improve readability.
[Figure 1 caption] The caption contains a typo: "If a too low value of γ is chosen" should be "If too low a value of γ is chosen."
[Proof of Theorem 3, big-O term] The Taylor expansion of ∇R_aug(A) around X has a remainder O(||Y||²) that is stated to be independent of X due to assumption (iii). This should be made explicit by stating the uniform bound in assumption (iii) over the whole affine space L, not just locally, or by noting that only a neighborhood of E matters due to the a priori bound.
[Remark 2] Remark 2 is a sketch and uses a linearization around (X*,0) without stating precise conditions under which the higher-order terms can be dominated. This is acceptable as a remark, but it should be labeled as a formal local analysis rather than a full theorem.
[Reference and spelling] The name "Grönwall" appears as "Gr¨onwall" in the proof of Theorem 3 due to a LaTeX encoding issue; this should be corrected to "Grönwall" for consistency with standard rendering.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Theorem 3 derives a genuine attractor property from separately proved prior lemmas; the proof contains a repairable gap but no input-output tautology.

full rationale

The paper's central new claim is Theorem 3, which states that regularized augmented gradient flow makes the equivariant subspace E an attractor. The proof invokes Fact A (Pi_L grad R_aug(A) = Pi_E grad R(A) for A in E) and Fact B (Pi_L (R_aug)''(A)Y in T E^perp) from the authors' prior paper [10]. These are parameter-free mathematical identities with proofs in a separate published work; they do not restate Theorem 3, are not fitted to the present experiment, and their assumptions (invariant loss, compatibility condition, affine subspace L) do not include the target attractor conclusion. The regularization term gamma/2 ||Pi_E^perp A||^2 is deliberately designed to penalize distance to E, but the theorem's content - that this penalty enforces exponential convergence under stated curvature and third-derivative bounds - does not reduce to the definition of the regularizer. No fitted parameter is relabeled as a prediction, and no known empirical pattern is merely renamed. A genuine issue in the written proof is that alpha = 2 R_aug(A0)/gamma + ||Y0||^2 is claimed to depend only on ||Y0||, whereas R_aug(A0) can grow along E; this is a proof gap, repairable via forward invariance using the uniform third-derivative bound, but it is not circularity. The experiment with SGD and random augmentations is an independent check, not an output of the theory. Overall, the derivation chain is self-contained apart from normal citation of prior lemmas, so the circularity score is 0.

Assumptions & free parameters 1 free parameters · 7 assumptions · 0 invented entities

The central theorem rests on prior structural assumptions from the authors' framework (loss invariance, compatibility condition, Facts A and B from [10]) plus three regularity conditions on R_aug. The only hand-chosen scalar is the regularization strength γ, which the theorem treats existentially and the experiment varies by hand. No new entities are introduced.

free parameters (1)

γ (regularization strength) = varied 1e-4, 1e-2, 1e0, 1e2 in the experiment; otherwise existential
The theorem does not fix γ; it only asserts existence for sufficiently large γ. In the experiment, γ is a hyperparameter chosen by hand, not fitted to data.

assumptions (7)

domain assumption Loss invariance: ℓ(ρL(g)x, ρL(g)x') = ℓ(x,x') for all g∈G
Assumption (a) in Section II-C; used to derive Fact A and the equivalence of augmented and equivariant dynamics on E.
domain assumption Compatibility: ΠLΠG = ΠGΠL
Assumption (b) in Section II-C; needed for the projections to commute so the dynamics decouple in the proof of Theorem 3.
domain assumption R_aug(A) ≥ 0 for every A ∈ L
Assumption (i) in Section III; gives the a priori bound ||Y||² ≤ α.
domain assumption There exists σ such that ⟨(R_aug)''(A)Y,Y⟩ ≥ σ||Y||² for A∈E, Y∈TE⊥
Assumption (ii) in Section III; controls the decay of the non-equivariant component Y.
domain assumption Third derivative of R_aug is uniformly bounded
Assumption (iii) in Section III; makes the Taylor remainder O(||Y||²) uniform in X.
domain assumption Gradient flow with exact Haar-measure augmentation
The theorem is proven for continuous gradient flow and perfect augmentation; the experiment uses SGD with random augmentations, so the theory does not directly cover the experiment.
standard math Facts A and B from Nordenfors, Ohlsson and Flinth [10]
Fact A: ΠL∇R_aug(A)=ΠE∇R(A) for A∈E. Fact B: ΠL(R_aug)''(A)Y∈TE⊥. Cited from [10, Prop. 3.14] and used without reproof in Theorem 3.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Data Augmentation and Regularization for Learning Group Equivariance." pith.science (2026). https://pith.science/paper/OW6R2KYX

@misc{pith2026250206547,
  author       = {Pith},
  title        = {Pith review of: Data Augmentation and Regularization for Learning Group Equivariance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OW6R2KYX}},
  note         = {Machine review of arXiv:2502.06547}
}

read the original abstract

In many machine learning tasks, known symmetries can be used as an inductive bias to improve model performance. In this paper, we consider learning group equivariance through training with data augmentation. We summarize results from a previous paper of our own, and extend the results to show that equivariance of the trained model can be achieved through training on augmented data in tandem with regularization.

Figures

Figures reproduced from arXiv: 2502.06547 by the authors.

**Figure 1.** Regardless of how far from E we start training we can select the regularization parameter γ large enough that the dynamics A˙ = −∇Sγ(A) converge to E exponentially fast. If a too low value of γ is chosen, the dynamics might not converge to E at all. (i) The risk is bounded below by zero. (ii) The second derivative of the augmented risk is bounded below in the following sense: There exists a (not necessarily positiv… view at source ↗

**Figure 2.** Projection errors for the two non-equivariant models for different [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 15 canonical work pages

[10]

Optimization Dynamics of Equivariant and Augmented Neural Networks

O. Nordenfors, F. Ohlsson, and A. Flinth, “Optimization dynamics of equivariant and augmented neural networks,” Transactions of Machine Learning Research , vol. 4, 2025. [Online]. Available: https://arxiv.org/abs/2303.13458

work page Pith review arXiv 2025
[1]

Highly accurate protein structure prediction with alphafold,

J. Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool, R. Bates, A. ˇZ´ıdek, A. Potapenko et al. , “Highly accurate protein structure prediction with alphafold,” nature, vol. 596, pp. 583–589, 2021

work page 2021
[2]

Group equivariant convolutional net- works,

T. S. Cohen and M. Welling, “Group equivariant convolutional net- works,” in Proceedings of the 33rd International Conference on Machine Learning, 2016, pp. 2990–2999

work page 2016
[3]

Steerable cnns,

——, “Steerable cnns,” in Proceedings of the 5th International Confer- ence on Learning Representations , 2017, pp. 689–703

work page 2017
[4]

A group-theoretic framework for data augmentation,

S. Chen, E. Dobriban, and J. H. Lee, “A group-theoretic framework for data augmentation,” The Journal of Machine Learning Research , vol. 21, no. 1, pp. 9885–9955, 2020

work page 2020
[5]

On the benefits of invariance in neural networks,

C. Lyle, M. van der Wilk, M. Kwiatkowska, Y . Gal, and B. Bloem-Reddy, “On the benefits of invariance in neural networks,” arXiv:2005.00178, 2020

arXiv 2005
[6]

Provably strict generalisation benefit for equivariant models,

B. Elesedy and S. Zaidi, “Provably strict generalisation benefit for equivariant models,” inProceedings of the 38th International Conference on Machine Learning . PMLR, 2021, pp. 2959–2969

work page 2021
[7]

Implicit bias of linear equivariant networks,

H. Lawrence, K. Georgiev, A. Dienes, and B. T. Kiani, “Implicit bias of linear equivariant networks,” in International Conference on Machine Learning. PMLR, 2022, pp. 12 096–12 125

work page 2022

Show all 18 references

[8]

On the implicit bias of linear equivariant steerable networks,

Z. Chen and W. Zhu, “On the implicit bias of linear equivariant steerable networks,” Advances in Neural Information Processing Systems , vol. 36, 2024

2024
[9]

Accurate structure prediction of biomolecular interactions with alphafold 3,

J. Abramson, J. Adler, J. Dunger, R. Evans, T. Green, A. Pritzel, O. Ronneberger, L. Willmore, A. J. Ballard, J. Bambricket al., “Accurate structure prediction of biomolecular interactions with alphafold 3,” Nature, vol. 630, pp. 493–500, 2024

2024
[11]

Emergent equivariance in deep ensembles,

J. E. Gerken and P. Kessel, “Emergent equivariance in deep ensembles,” in Proceedings of the 41st International Conference on Machine Learn- ing, vol. 235, 2024, pp. 15 438–15 465

2024
[12]

Symmetries in overparametrized neural networks: A mean-field view,

J. Maass and J. Fontbona, “Symmetries in overparametrized neural networks: A mean-field view,” 2024. [Online]. Available: https: //arxiv.org/abs/2405.19995

2024 arXiv
[13]

Ensembles provably learn equivariance through data augmentation,

O. Nordenfors and A. Flinth, “Ensembles provably learn equivariance through data augmentation,” 2024. [Online]. Available: https://arxiv. org/abs/2410.01452

2024
[14]

Representation theory,

W. Fulton and J. Harris, “Representation theory,” Graduate Texts in Mathematics, vol. 129, 2004

2004
[15]

Geometric deep learning: Grids, groups, graphs, geodesics, and gauges,

M. M. Bronstein, J. Bruna, T. Cohen, and P. Veliˇckovi´c, “Geometric deep learning: Grids, groups, graphs, geodesics, and gauges,” arXiv preprint arXiv:2104.13478, 2021

2021 arXiv
[16]

Invariant and equivariant graph networks,

H. Maron, H. Ben-Hamu, N. Shamir, and Y . Lipman, “Invariant and equivariant graph networks,” in Proceedings of the 7th International Conference on Learning Representations , 2019

2019
[17]

A general theory of equivari- ant CNNs on homogeneous spaces,

T. Cohen, M. Geiger, and M. Weiler, “A general theory of equivari- ant CNNs on homogeneous spaces,” Advances in Neural Information Processing Systems, vol. 32, 2019

2019
[18]

A practical method for con- structing equivariant multilayer perceptrons for arbitrary matrix groups,

M. Finzi, M. Welling, and A. G. Wilson, “A practical method for con- structing equivariant multilayer perceptrons for arbitrary matrix groups,” in Proceedings of the 38th International Conference on Machine Learn- ing. PMLR, 2021, pp. 3318–3328

2021

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [10]

Optimization Dynamics of Equivariant and Augmented Neural Networks

O. Nordenfors, F. Ohlsson, and A. Flinth, “Optimization dynamics of equivariant and augmented neural networks,” Transactions of Machine Learning Research , vol. 4, 2025. [Online]. Available: https://arxiv.org/abs/2303.13458

work page Pith review arXiv 2025

[2] [1]

Highly accurate protein structure prediction with alphafold,

J. Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool, R. Bates, A. ˇZ´ıdek, A. Potapenko et al. , “Highly accurate protein structure prediction with alphafold,” nature, vol. 596, pp. 583–589, 2021

work page 2021

[3] [2]

Group equivariant convolutional net- works,

T. S. Cohen and M. Welling, “Group equivariant convolutional net- works,” in Proceedings of the 33rd International Conference on Machine Learning, 2016, pp. 2990–2999

work page 2016

[4] [3]

Steerable cnns,

——, “Steerable cnns,” in Proceedings of the 5th International Confer- ence on Learning Representations , 2017, pp. 689–703

work page 2017

[5] [4]

A group-theoretic framework for data augmentation,

S. Chen, E. Dobriban, and J. H. Lee, “A group-theoretic framework for data augmentation,” The Journal of Machine Learning Research , vol. 21, no. 1, pp. 9885–9955, 2020

work page 2020

[6] [5]

On the benefits of invariance in neural networks,

C. Lyle, M. van der Wilk, M. Kwiatkowska, Y . Gal, and B. Bloem-Reddy, “On the benefits of invariance in neural networks,” arXiv:2005.00178, 2020

arXiv 2005

[7] [6]

Provably strict generalisation benefit for equivariant models,

B. Elesedy and S. Zaidi, “Provably strict generalisation benefit for equivariant models,” inProceedings of the 38th International Conference on Machine Learning . PMLR, 2021, pp. 2959–2969

work page 2021

[8] [7]

Implicit bias of linear equivariant networks,

H. Lawrence, K. Georgiev, A. Dienes, and B. T. Kiani, “Implicit bias of linear equivariant networks,” in International Conference on Machine Learning. PMLR, 2022, pp. 12 096–12 125

work page 2022