REVIEW 2 major objections 5 minor 18 references
Monge SAM: Robust Reparameterization-Invariant Sharpness-Aware Minimization Based on Loss Geometry
T0 review · 2 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read M-SAM replaces the Euclidean ball of sharpness-aware minimization with the Monge metric induced by the loss graph, yielding a closed-form perturbation between SAM and gradient descent and reducing attraction to saddles.
desk verdict The closed-form update is real and the conservative behavior is plausible, but the reparametrization-invariance claim fails against the paper's own equations. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Monge metric $G(\theta)=I+\nabla\ell(\theta)\nabla\ell(\theta)^\top$, the metric induced on the graph of the loss surface $M=\{(\theta,\ell(\theta))\}$ by the parameterization. Its inverse has the closed form $G^{-1}=I-\nabla\ell\nabla\ell^\top/(1+\|\nabla\ell\|^2)$ by the Sherman-Morrison formula. This gives M-SAM's perturbation in closed form and makes the effective perturbation radius $\tilde{\rho}_{\mathrm{M-SAM}} = \rho/(\|\nabla\ell\|\sqrt{1+\|\nabla\ell\|^2})$ an upper-bounded, gradient-norm-dependent version of SAM's radius. This radius is what connects the geometry to the dynamics: the stability analysis of the SAM gradient flow near equilibria uses $\tilde{\rho}$ in the criterion $\tilde{\rho}>-1/\lambda_i$, and the smaller M-SAM radius makes that criterion fail more often, reducing attraction to saddles.
What would settle it
Run M-SAM under the reparameterization that multiplies one layer's weights by $c$ and divides the next layer's by $c$; because the network function is unchanged, a truly invariant optimizer must produce identical updates in the original coordinates. The paper reports no such test, and the metric formula given implies the perturbations differ by a factor depending on $J_\psi$.
Extended reading notes
Core claim
The central claim is that a sharpness-aware minimizer can be made reparameterization-invariant by defining the adversarial search region with the metric induced by the loss graph rather than the Euclidean metric. The paper derives the worst-case perturbation $\delta_{\mathrm{M-SAM}} = \rho\,\nabla\ell(\theta)/(\|\nabla\ell(\theta)\|\sqrt{1+\|\nabla\ell(\theta)\|^2})$, which is SAM's perturbation multiplied by $1/\sqrt{1+\|\nabla\ell(\theta)\|^2}$. Because this factor vanishes as the gradient grows and approaches one near stationary points, M-SAM behaves like gradient descent on steep slopes and like SAM in flat regions; the paper calls this conservative behavior and uses perturbation theory of the SAM gradient flow to argue that it weakens the attraction to saddles and maxima. On a multi-modal alignment task, M-SAM fine-tuning of a CLIP model reaches a mutual-kNN alignment score of 0.446, compared with 0.351 for the pretrained model and 0.405 for SAM, while staying stable across learning rates that make SAM diverge.
Load-bearing premise
The invariance claim rests on treating $G(\theta)=I+\nabla\ell(\theta)\nabla\ell(\theta)^\top$ as the induced metric of the loss graph in every coordinate system; that formula is only the identity-chart expression, and under a general reparameterization $\phi=h(\theta)$ the induced metric is $J_\psi^\top J_\psi + \nabla_\phi L \nabla_\phi L^\top$ with $J_\psi=\mathrm{d}\theta/\mathrm{d}\phi$, so M-SAM's perturbation is not shown to be invariant.
Editorial extensions
If this is right
- M-SAM requires only one extra forward-backward pass over SAM, since the worst-case perturbation has a closed form.
- Because $\tilde{\rho}_{\mathrm{M-SAM}}\leq \tilde{\rho}_{\mathrm{SAM}}$, training with M-SAM is more tolerant of large learning rates and perturbation radii than SAM.
- The stability criterion $\tilde{\rho}>-1/\lambda_i$ is satisfied less often for M-SAM, so saddles and maxima attract M-SAM dynamics less than they attract SAM dynamics.
- When fine-tuning a CLIP model, M-SAM raises the mutual-kNN alignment score to 0.446, above the pretrained model's 0.351 and above SAM's 0.405 in the reported configuration.
Reading between the lines
- Editorial inference: the invariance claim can be tested directly by reparameterizing a trained network and comparing M-SAM updates; if it fails, the reported benefits may still hold but would need to be attributed to conservativeness rather than invariance.
- Editorial inference: the perturbation's dependence on $1/\sqrt{1+\|\nabla\ell\|^2}$ resembles adaptive gradient clipping, so M-SAM may be understood as SAM with a gradient-norm-aware step size; comparing it to clipped SAM on the same benchmarks would separate the two effects.
- Editorial inference: the Monge metric is the pullback of the Euclidean metric on the loss graph, so M-SAM might generalize to other graph-induced metrics, such as ones including second-order terms, at the same computational cost, though the paper does not explore this.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Monge SAM (M-SAM), an optimizer that replaces SAM's Euclidean perturbation ball with the ellipsoid defined by G(θ)=I+∇ℓ(θ)∇ℓ(θ)^T, called the Monge metric induced by the loss graph. The authors derive a closed-form adversarial perturbation, Eq. (9), of the form δ* = ρ∇ℓ/(‖∇ℓ‖√(1+‖∇ℓ‖²)), argue that M-SAM interpolates between SAM and gradient descent, analyze attraction to suboptimal equilibria through a stability ODE in Appendix A, and report experiments on fine-tuning a poorly generalizing ResNet-18 and on CLIP fine-tuning for multimodal representational alignment. The abstract, Section 3, and the conclusion all present reparameterization invariance as the core contribution.
Significance. If the central claim were correct, M-SAM would be a computationally cheap and broadly applicable geometry-aware replacement for SAM, avoiding the probabilistic restriction of Fisher SAM and the diagonalization needed in practice. The paper has genuine strengths: the perturbation formula is closed-form and follows correctly from the stated constrained optimization under Eq. (6), the computational cost is the same as SAM, and the CLIP alignment experiments (Table 1) show a substantial gain (0.446 vs. 0.405 for SAM). However, the advertised invariance is the paper's headline, and it is not supported by the equations: Eq. (6) is a coordinate-chart expression, not a tensor, so the algorithm changes under non-orthogonal reparameterizations. The secondary stability analysis in Appendix A also omits a term in the derivative of the perturbed point and incorrectly treats the perturbation as vanishing near equilibria. These issues are load-bearing for the paper's main theoretical claims.
major comments (2)
- [Section 3.1, Eq. (6); abstract; Section 3.2] The reparameterization-invariance claim is not correct for general smooth reparameterizations. For the graph embedding g(θ)=(θ,ℓ(θ)), the induced metric in a new chart θ=ψ(φ) is G_φ = J^T J + ∇_φℓ∇_φℓ^T with J=∂θ/∂φ, whereas the algorithm as stated uses I+∇_φℓ∇_φℓ^T in every chart. These two expressions coincide only when J^T J=I, i.e. for orthogonal reparameterizations. For a simple scaling or any nonlinear reparameterization, the perturbation (9) computed in φ-coordinates is not the pushforward of the perturbation computed in θ-coordinates. Thus the advertised invariance, which is repeated in the abstract and conclusion, is unsupported; moreover, the paper contains no experiment that tests invariance.
- [Appendix A, Eqs. (10)-(11)] The stability ODE derivation for z(θ)=θ+ρ̃(θ)∇ℓ(θ) omits the term ∇ℓ(θ)(∇ρ̃(θ))^T when computing ∂z/∂θ; the full derivative is I+ρ̃∇²ℓ(θ)+∇ℓ(θ)(∇ρ̃(θ))^T. In addition, near an equilibrium with ‖∇ℓ‖→0, the M-SAM effective radius behaves as ρ̃≈ρ/‖∇ℓ‖ and ‖δ*_M-SAM‖ tends to ρ, not 0, so the claim that 'δ*≈0' and hence z(θ*)=θ* is not justified. This weakens the theoretical conclusion that M-SAM is less attracted to saddle points than SAM.
minor comments (5)
- [Section 3.1] The dimension notation is inconsistent: the manifold is said to be embedded in RK+1 and in Rd+1, and the Jacobian Jg is described as a map to RK+1×K; please standardize.
- [Section 3.2] The norm notation alternates between ‖∇ℓ(θ)‖_2 and ‖∇ℓ(θ)‖; using a single symbol throughout would improve readability.
- [Conclusion] There is a typo in 'pertubation radii'; the word should be 'perturbation'.
- [Limitations and future work] The limitations paragraph acknowledges the Taylor-approximation and conservativeness issues, but it does not mention the coordinate-chart dependence of Eq. (6), which is the main obstacle to the invariance claim.
- [Figure 4] The caption says 'The square is used for reference' without identifying which square in the figure is meant; please clarify.
Circularity Check
No circularity: M-SAM's perturbation and saddle-attraction claims follow algebraically from the Monge-metric definition and an external stability criterion; the invariance claim is unproven but not circular.
full rationale
The derivation chain is self-contained: M-SAM's perturbation (Eq. 9) is obtained by solving the quadratically constrained linear program in Eq. (8) with the explicit Monge metric G = I + ∇ℓ∇ℓ^T from Eq. (6), so the 'between SAM and GD' behavior and the ρ̃ bound are algebraic consequences of the method's definition rather than fitted predictions. The saddle-attraction comparison uses the stability criterion ρ̃ > −1/λ_i attributed to Kim et al. (2023), an external cited result, and the comparison M-SAM vs SAM follows from ρ̃_M-SAM ≤ ρ̃_SAM, which is a direct inequality. No parameter is fitted to data and then reported as a prediction, and no load-bearing premise is justified only by a self-citation; the sole self-citation (Bergamin et al. 2024, related work on Riemannian Laplace approximations) is not used to support the central claims. The advertised reparametrization invariance is asserted from the induced-graph-metric construction rather than demonstrated, and the stated metric in Eq. (6) is in fact chart-dependent; however, an unsupported or incorrect claim is a correctness issue, not a circular reduction. Hence no circularity.
Assumptions & free parameters
free parameters (2)
- perturbation radius rho =
grid-selected: 0.005 to 0.04
- learning rate alpha =
grid-selected: 1e-5 to 1e-2
assumptions (4)
- ad hoc to paper The parameter manifold is represented in identity coordinates so J_g = [I_K, grad ell]^T.
- ad hoc to paper The effective perturbation radius rho_tilde is treated as locally constant when differentiating z(theta) = theta + rho_tilde grad in Appendix A.
- domain assumption The loss surface is smooth enough for the graph manifold and Taylor expansions.
- domain assumption The stability criterion from Kim et al. (2023): an equilibrium is stable under SAM dynamics if rho > -1/lambda_i for all Hessian eigenvalues lambda_i.
Cite this review
Pith. "Pith review of Monge SAM: Robust Reparameterization-Invariant Sharpness-Aware Minimization Based on Loss Geometry." pith.science (2026). https://pith.science/paper/ECQNCRNC
@misc{pith2026250208448,
author = {Pith},
title = {Pith review of: Monge SAM: Robust Reparameterization-Invariant Sharpness-Aware Minimization Based on Loss Geometry},
year = {2026},
howpublished = {\url{https://pith.science/paper/ECQNCRNC}},
note = {Machine review of arXiv:2502.08448}
}
read the original abstract
Recent studies on deep neural networks show that flat minima of the loss landscape correlate with improved generalization. Sharpness-aware minimization (SAM) efficiently finds flat regions by updating the parameters according to the gradient at an adversarial perturbation. The perturbation depends on the Euclidean metric, making SAM non-invariant under reparametrizations, which blurs sharpness and generalization. We propose Monge SAM (M-SAM), a reparametrization invariant version of SAM by considering a Riemannian metric in the parameter space induced naturally by the loss surface. Compared to previous approaches, M-SAM works under any modeling choice, relies only on mild assumptions while being as computationally efficient as SAM. We theoretically argue that M-SAM varies between SAM and gradient descent (GD), which increases robustness to hyperparameter selection and reduces attraction to suboptimal equilibria like saddle points. We demonstrate this behavior both theoretically and empirically on a multi-modal representation alignment task.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Ahn, K., Jadbabaie, A., and Sra, S. How to escape sharp minima. arXiv preprint arXiv:2305.15659,
-
[3]
CLIP performance after fine-tuning. We evaluate pre-trained CLIP on the COCO Captions dataset as well as models fine-tuned to the Wiki dataset. We consider SGD, Adam, SAM and M-SAM with varying (α, ρ)-settings where α is the learning rate and ρ is the perturbation radius. We present the loss and alignment scores at their optimal iterations, T ∗ loss and T...
work page 2000
-
[4]
Sharpness-aware minimization for efficiently improving generalization
Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412,
arXiv 2010
-
[6]
Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., and Wilson, A. G. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407,
-
[8]
Fantastic generalization measures and where to find them
Jiang, Y ., Neyshabur, B., Mobahi, H., Krishnan, D., and Bengio, S. Fantastic generalization measures and where to find them. arXiv preprint arXiv:1912.02178,
arXiv 1912
-
[13]
Xing, C., Arpit, D., Tsirigotis, C., and Bengio, Y . A walk with sgd. arXiv preprint arXiv:1802.08770,
-
[14]
Riemannian laplace approximation with the fisher metric
Yu, H., Hartmann, M., Williams, B., Girolami, M., and Klami, A. Riemannian laplace approximation with the fisher metric. arXiv preprint arXiv:2311.02766,
-
[15]
Zhu, Z., Wu, J., Yu, B., Wu, L., and Ma, J. The anisotropic noise in stochastic gradient descent: Its behavior of escap- ing from sharp minima and regularization effects. arXiv preprint arXiv:1803.00195,
Show all 18 references
-
[16]
10 Monge SAM: Robust Reparameterization-Invariant Sharpness-Aware Minimization Based on Loss Geometry A. Proof of ODE describing local SAM dynamics We extend the discrete SAM update to continuous time, hence establishing the SAM gradient flow: dθ dt = −f (z (θ)) where f (z (θ)...
2023
-
[17]
QT Q = I and Λ = diag (λ1,
In such settings the SAM perturbation disappears as z (θ) |θ=θ∗ = θ which results in Aρ (θ) |θ=θ∗ = −∇2ℓ (θ∗) · Id + ˜ρ∇2ℓ (θ∗) = −QΛQ⊤ · I + ˜ρQΛQ⊤ = −QΛQ⊤ + ˜ρQΛQ⊤QΛQ⊤ = −QΛQ⊤ + ˜ρQΛ2Q⊤ = −Q Λ + ˜ρΛ2 Q⊤ where the Hessian matrix, ∇2ℓ (θ∗), is real and symmetric and thus facto...
2023
-
[1997]
The platonic representation hypothesis
Huh, M., Cheung, B., Wang, T., and Isola, P. The platonic representation hypothesis. arXiv preprint arXiv:2405.07987,
-
[2016]
Stability anal- ysis of sharpness-aware minimization
Kim, H., Park, J., Choi, Y ., and Lee, J. Stability anal- ysis of sharpness-aware minimization. arXiv preprint arXiv:2301.06308,
-
[2017]
Dziugaite, G. K. and Roy, D. M. Computing nonvacuous generalization bounds for deep (stochastic) neural net- works with many more parameters than training data. arXiv preprint arXiv:1703.11008,
-
[2018]
Three factors influencing minima in sgd
Jastrzebski, S., Kenton, Z., Arpit, D., Ballas, N., Fischer, A., Bengio, Y ., and Storkey, A. Three factors influencing minima in sgd. arXiv preprint arXiv:1711.04623,
-
[2020]
Rethinking sharpness-aware minimization as varia- tional inference
Ujv´ary, S., Telek, Z., Kerekes, A., M´esz´aros, A., and Husz´ar, F. Rethinking sharpness-aware minimization as varia- tional inference. arXiv preprint arXiv:2210.10452,
-
[2021]
Lin, T.-Y ., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Doll´ar, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740–
2014
-
[2022]
S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang, P
Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang, P. T. P. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836,
-
[2024]
Entropy-sgd: Biasing gradient descent into wide val- leys
Chaudhari, P., Choromanska, A., Soatto, S., LeCun, Y ., Bal- dassi, C., Borgs, C., Chayes, J., Sagun, L., and Zecchina, R. Entropy-sgd: Biasing gradient descent into wide val- leys. Journal of Statistical Mechanics: Theory and Ex- periment, 2019(12):124018,
2019
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.