Pith. sign in

REVIEW 3 major objections 4 minor 2 cited by

An optimizer that lands on the orthogonal manifold every iteration—using just five matrix products—makes thousands of orthogonality constraints practical at deep-learning scale.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 23:07 UTC pith:ZTXGJRCR

load-bearing objection A practical, fast orthogonal-constraint optimizer that deserves a serious look; the exact-landing claim is overstated and the theorem's key assumption is unverified in practice, but the empirical package is strong. the 3 major comments →

arxiv 2602.14656 v2 pith:ZTXGJRCR submitted 2026-02-16 cs.LG math.DGmath.OC

An Embarrassingly Simple Way to Optimize Orthogonal Matrices at Scale

Adri\'an Javaloy , Antonio Vergari This is my paper
classification cs.LG math.DGmath.OC
keywords orthogonal optimizationStiefel manifoldLanding algorithmadaptive optimizersretraction-free optimizationVectorAdamCIFAR-10probabilistic circuits
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Orthogonality constraints improve stability and robustness, but existing optimizers are either slow (retraction-based) or let matrices wander off the manifold before eventually landing. This paper argues that the Landing update already contains the ingredients for an exact one-step return: take a tangent step to an intermediate point, then choose a step along the normal direction that puts the iterate back on the orthogonal matrices. The authors show that a fixed normal step of 1/2 achieves this whenever the base optimizer's output is uniformly bounded, giving squared distance to the manifold of order ξ^7, and that the step can be wrapped around adaptive optimizers such as VectorAdam. The result is POGO: five GPU-friendly matrix multiplications, no retractions, and empirically the same accuracy as unconstrained Adam on CIFAR-10 with 218,624 orthogonal 3×3 kernels trained in minutes. If right, this removes a key bottleneck and lets orthogonality constraints be used at deep-learning scale.

Core claim

The central discovery is that optimizing under orthogonality constraints can be decomposed into a tangent step followed by a normal step whose optimal size is both computable in closed form (the landing polynomial, a quartic) and, under a uniform gradient-norm bound, well approximated by the constant 1/2. Theorem 3.5 states that starting on the Stiefel manifold and iterating X_{t+1} = M_t + (1/2)(I - M_t M_t^T) M_t with M_t = X_t - η X_t S_t keeps every iterate within o(ξ^7) of the manifold, where ξ = ηL and L bounds the base optimizer's output. Because the update uses only matrix multiplications and because the skew-symmetric relative gradient is preserved by linear base optimizers, standar

What carries the argument

The landing polynomial P(λ) = ‖(M − λ∇N(M))(M − λ∇N(M))^T − I‖², a quartic in λ whose coefficients can be computed in O(p²n); the paper proves that its root gives the exact step to return to the manifold. The key simplification is the reformulated update M = X − ηXS (a step in the tangent space) followed by the normal step with λ: the bound in Proposition 3.2 shows ‖MM^T − I‖ ≤ η²‖S²‖, so that λ = 1/2 yields P(1/2) = o(ξ^7). The constant 1/2 is the first-order Taylor expansion of (MM^T)^{-1/2}, revealing POGO as an approximate polar retraction that uses only matrix products.

Load-bearing premise

The whole guarantee rests on a single uniform bound: the base optimizer's output G must satisfy ‖G‖ ≤ L across all iterations with ηL < 1; the paper gives no mechanism that provably enforces this, and the ablation shows that when the learning rate is too large POGO escapes the manifold.

What would settle it

Run the released code on the CIFAR-10 orthogonal-kernel benchmark (218,624 3×3 kernels) with VectorAdam, keeping ηL < 1; if POGO fails to stay within, say, 1e−4 manifold distance while matching Adam's final accuracy in under ~10 minutes on a modern GPU, the central practical claim is refuted. For the theory, the same run could record ξ_t = η‖G_t‖ at every iteration and check whether the observed ‖X_t X_t^T − I‖² is consistently below the o(ξ_t^7) rate on a small-scale quadratic.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • POGO can wrap modern adaptive optimizers (e.g., VectorAdam) and still keep every iterate close to the orthogonal manifold, closing the gap between constrained and unconstrained optimization in practice.
  • Because it needs only five matrix products and no QR/SVD retractions, POGO scales to problems with hundreds of thousands of orthogonal matrices (218,624 3×3 kernels) in minutes on a GPU, where retraction methods take hours.
  • The constant λ = 1/2 reduces hyperparameters: users only set the base optimizer and learning rate, and observe manifold distance ~ξ^7.
  • The complex-Stiefel extension means POGO applies to squared unitary probabilistic circuits, matching RGD performance in half the time.
  • POGO maintains feasibility throughout training, enabling early stopping and reliable constraint satisfaction without a post-hoc correction step.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the uniform bound ξ < 1 could be enforced algorithmically (e.g., via gradient clipping or a schedule), POGO's feasibility guarantee would become unconditional; the paper leaves that enforcement mechanism implicit.
  • The λ = 1/2 update is a low-order Taylor approximation of a polar retraction, so one could derive higher-order corrections (e.g., λ as a function of ‖MM^T − I‖) to trade one extra matrix product for an even tighter manifold bound.
  • The same two-step leap-and-land pattern could be adapted to other constraint sets with a quadratic normal potential, such as doubly stochastic matrices or low-rank subspaces, whenever the normal step has a closed-form root.
  • Because the base optimizer only needs to be linear, future momentum variants that remain linear in the gradient can be dropped into POGO; Adam's elementwise normalization should be avoided in favor of vector-wise normalization.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces POGO, an optimizer for problems with orthogonality constraints of the form X X^T = I. POGO builds on the Landing algorithm, but (i) decouples the tangent-space step from the normal correction and (ii) shows that a fixed correction strength λ = 1/2 keeps iterates provably close to the Stiefel manifold under a uniform bound on the base optimizer's output norm. The algorithm wraps any base optimizer (SGD, VAdam, etc.), uses only matrix products, and is evaluated on PCA, Procrustes, O-ViT, CNN, and squared unitary PC benchmarks, where it matches unconstrained Adam accuracy while being order-of-magnitude faster than retraction-based methods.

Significance. If the theoretical and empirical claims hold, POGO is a significant practical advance: it scales to hundreds of thousands of orthogonal constraints on a GPU, removes several hyperparameters from Landing, and achieves state-of-the-art downstream performance while staying on the manifold. The explicit bound in Prop. 3.3 is elegant and the empirical results are strong. However, several load-bearing assumptions and claims need clarification or verification before the results can be accepted as stated.

major comments (3)
  1. [§3.2, Lemma 3.1 and 'Choosing a step size'] The paper repeatedly states that POGO 'can compute exactly the optimal step size to land back on the manifold' and 'calculate the exact λ to stay within the manifold' (Fig. 3). This is not supported: P(λ) is a real quartic and for p>1 with unequal singular values it need not have a real root. The heuristic 'take the real part of the root with the least imaginary part' does not guarantee P(λ)=0 or even a local minimum of P(λ). The ablation in Fig. C.2 (left, η=0.025) shows iterates escaping the manifold even with root-based λ. Please revise the text to claim instead that POGO minimizes (or approximately minimizes) P over real λ, and remove the 'exact landing' wording unless a real-root existence result is proved. The default λ=1/2 variant is unaffected, but the §3.2 contribution as written is inaccurate.
  2. [§3.3, Assumption 1 and Theorem 3.5] Assumption 1 (∥G∥≤L, ξ=ηL<1) is the key condition in Prop. 3.3 and the induction in Thm. 3.5, but no mechanism in POGO or VAdam enforces it. The paper never reports the observed ∥G_t∥ or ξ for the headline VAdam experiments in §5.2–5.3. The ablation in Fig. C.2 shows that with SGD and η=0.025 the manifold distance grows, so the condition is not vacuous. The statements in §3.3 ('we can keep ∥G∥ under control') and App. C.6 ('gradient normalization ... helps us to adaptively control ∥G∥') are not proofs. Please either prove a uniform bound for VAdam or another base optimizer, or empirically report ∥G_t∥ and ξ for the runs used to support the 'maintains orthogonality at all times' claim. Without this, Thm. 3.5 cannot be invoked to explain the empirical feasibility of the main experiments.
  3. [§3.1, Definition 1 and Eq. (8)] The paper claims VAdam 'satisfies Def. 1' because it uses vector-wise normalization m/∥v∥_2. This is not a linear function of the input gradient: the denominator ∥v∥_2 is computed from squared past gradients, so the equivariance in Eq. (8) does not hold exactly. If VAdam is used as the base optimizer in §5, the theoretical rationale for replacing ∇f by G in the relative-gradient construction is weakened. Please either state the precise equivariance property that VAdam actually satisfies (e.g., rotation equivariance per vector) or clearly mark the use of VAdam as an empirical contribution. The λ=1/2 theory for SGD remains valid, but the current text overstates the theoretical support for adaptive base optimizers.
minor comments (4)
  1. [Throughout] Typos: 'hyperparemeters' (abstract), 'dowstream' (Section 6), and inconsistent use of 'o(ξ7/2)' vs 'o(ξ^{7/2})' in several places (e.g., Thm. 3.5). Consider using the explicit bound P(1/2) ≤ (3/4+ξ²/4)² ξ⁸ from Prop. A.7 instead of only writing o(ξ⁷) in the main text; the explicit bound is stronger and more informative.
  2. [Fig. 5 caption] The phrase 'finishes 5 minutes quicker than RSDM' could be replaced with 'is 5 minutes faster than RSDM' for clarity.
  3. [§3.2, root selection] The notation 'arg min_{i} min_{λ_i∈R} (λ_i − bar λ_i)²' is confusing. Clarify that you choose the root with the smallest imaginary part and then take its real component.
  4. [App. C.6 / Fig. C.3] The caption of Fig. C.3 appears to be copied from Fig. C.2 (it mentions 'Left plots solve the landing polynomial' but the figure shows combined curves). Please correct.

Circularity Check

0 steps flagged

No significant circularity: POGO's update and feasibility guarantee are derived in-paper from an explicit norm assumption; the only self-reference is the squared-PC benchmark from the authors' own group, which is not load-bearing.

full rationale

The derivation chain is internally consistent and self-contained. The POGO update (Eqs. 9-10) is defined directly from the Stiefel constraint; Lemma 3.1 gives the landing polynomial explicitly, Prop. 3.2 bounds the intermediate distance solely in terms of η and S, and Prop. 3.3/Thm. 3.5 derive the o(ξ^7) bound under the explicit Assumption 1 (∥G∥≤L, ξ=ηL<1). No fitted quantity is relabeled as a prediction: λ=1/2 is not fit to outcomes but bounded from the same polynomial, and the convergence discussion appeals to external RGD/polar-retraction theory (Boumal 2023) and to the acknowledged equivalence with SLPG (Liu et al. 2024), both of which are independent support. The only self-reference is the squared-unitary-PC evaluation from Loconte et al. (2025a), with overlapping authorship; it is an application benchmark, not a premise of any theorem, so it is not load-bearing. The unverified character of Assumption 1 for VAdam (the paper does not report observed ∥G_t∥ or ξ) is a correctness/robustness gap, not circularity: the theorem states a conditional guarantee and does not assume its own conclusion.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

The central algorithm introduces no fitted constants and no new physical entities. Its theoretical guarantees rest mainly on the standard Stiefel-manifold toolkit plus the bounded-gradient assumption (Assumption 1). The optional exact-landing variant relies on an unproven root-selection heuristic. The lambda=1/2 default is derived, not fitted.

axioms (4)
  • standard math Stiefel manifold geometry: tangent space {XS : S skew-symmetric}, relative gradient Skew(X^T G), polar retraction Taylor expansion
    Used throughout Sections 2-3; standard results from Edelman et al. 1998 and Absil et al. 2008.
  • domain assumption Assumption 1: ||G|| <= L for all iterations, and xi := eta*L < 1 (Prop 3.3, Thm 3.5)
    Load-bearing for the o(xi^7) feasibility bound; never verified in practice; VAdam is claimed to control ||G|| but no proof is given.
  • domain assumption Base optimizer linearity (Def. 1) is needed to claim adaptive optimizers preserve tangent-space semantics
    Used in Section 3.1 to justify plugging VAdam; VAdam's vector-wise normalization is not strictly linear in the current gradient, so this is an idealization.
  • ad hoc to paper The quartic landing polynomial has a root whose real part yields a good lambda for real matrices
    Section 3.2 root-selection rule (real part of least-imaginary root) is asserted without proof; P(lambda)=0 has no real root in general, so the exact-landing claim is unsupported.

pith-pipeline@v1.3.0-alltime-deepseek · 26220 in / 23361 out tokens · 199084 ms · 2026-08-02T23:07:47.931018+00:00 · methodology

0 comments
read the original abstract

Orthogonality constraints are ubiquitous in robust and probabilistic machine learning. Unfortunately, current optimizers are computationally expensive and do not scale to problems with hundreds or thousands of constraints. One notable exception is the Landing algorithm (Ablin et al., 2024) which, however comes at the expense of temporarily relaxing orthogonality. In this work, we revisit and improve on the ideas behind Landing, enabling the inclusion of modern adaptive optimizers while ensuring that orthogonal constraints are effectively met. Remarkably, these improvements come at little to no cost, and reduce the number of required hyperparemeters. Our algorithm POGO is fast and GPU-friendly, consisting of only 5 matrix products, and in practice maintains orthogonality at all times. On several challenging benchmarks, POGO greatly outperforms recent optimizers and shows it can optimize problems with thousands of orthogonal matrices in minutes while alternatives would take hours. As such, POGO sets a milestone to finally exploit orthogonality constraints in ML at scale. A PyTorch implementation of POGO is publicly available at https://github.com/adrianjav/pogo.

Figures

Figures reproduced from arXiv: 2602.14656 by Adri\'an Javaloy, Antonio Vergari.

Figure 1
Figure 1. Figure 1: POGO optimizes thousands of orthogonal matrices or￾ders of magnitude faster than retraction methods while achiev￾ing performance comparable to unconstrained optimizers, as shown in this CIFAR-10 (Krizhevsky et al., 2009) classification problem with a tailored CNN (Jordan, 2024) parameterized with orthogonal filters or kernels. While RSDM (Han et al., 2025) takes 17 hours to train on average, POGO trains in… view at source ↗
Figure 2
Figure 2. Figure 2: Illustration of the landing algorithm, adapted from (Ablin et al., 2024). Landing combines two orthogonal gradients at each it￾eration and adapts the learning rate η to keep X1 within ε-distance from the Stiefel manifold. However, all of these retractions are either numerically un￾stable (e.g., the Cayley map involves matrix inversions), or rely on iterative processes (e.g., QR or SVD) that also re￾quire G… view at source ↗
Figure 4
Figure 4. Figure 4: POGO reduces the optimality gap the fastest across all baselines while staying on the manifold. Results are averaged over 10 independent runs and the orthogonal matrices are of size 1500 × 2000 for PCA and 2000 × 2000 for Procrustes. In particular, Liu et al. (2024) propose a sequential linear￾ized proximal gradient method (SLPG) where at each itera￾tion it applies an approximate orthogonalization step. Wh… view at source ↗
Figure 6
Figure 6. Figure 6: Normalized distance and test-time accuracy on the CNN experiment from §5.2. We observe that in all instances POGO obtains similar accuracy to the unconstrained baseline while staying on the Stiefel manifold. 0 50 100 150 200 250 Time (minutes) 0.2 0.4 0.6 0.8 Validation accuracy Adam (unconst.) RGD RSDM Landing SLPG POGO (ours) [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Evolution of the accuracy for the CNN experiment with orthogonal kernels. POGO can learn at the same pace as the un￾constrained Adam baseline despite orthogonality constraints. Convolutional neural networks. To test the scalability of POGO (D2), we now consider a CNN tailored for CIFAR￾10 classification (Jordan, 2024). Let O×I ×k× k be the parameter dimensions of each convolutional layer, where O and I are… view at source ↗
Figure 8
Figure 8. Figure 8: Training curves for validation bits-per-dimension and manifold distance for the squared unitary PC experiment. POGO obtains state-of-the-art results for this class of models, while converging the fastest and not leaving the Stiefel manifold. Most remarkably, we observe that POGO consistently ob￾tains accuracy comparable to Adam with minimal extra overhead (D3). Also, while SLPG matches POGO in the orthogon… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. The Orthogonalized Read Is a Removable Training Scaffold for Recurrent Memory

    cs.LG 2026-07 accept novelty 7.0

    The orthogonalized read on noisy recall is a removable training scaffold: it multiplies escape hazard ~6.4x during plateau training, but a stock mLSTM achieves full accuracy once the read is annealed away.

  2. A second-order method landing on the Stiefel manifold via Newton$\unicode{x2013}$Schulz iteration

    math.OC 2026-05 unverdicted novelty 7.0

    A second-order method achieves local quadratic convergence on the Stiefel manifold without retractions by combining a modified Newton tangent step with Newton-Schulz normal steps for constraint satisfaction.

Reference graph

Works this paper leans on

54 extracted references · 7 linked inside Pith · cited by 2 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    and Peyr \'e , G

    Ablin, P. and Peyr \'e , G. Fast and accurate optimization on the orthogonal manifold without retraction. In Camps - Valls, G., Ruiz, F. J. R., and Valera, I. (eds.), International Conference on Artificial Intelligence and Statistics, AISTATS 2022, 28-30 March 2022, Virtual Event , volume 151 of Proceedings of Machine Learning Research, pp.\ 5636--5657. P...

    2022

  3. [3]

    Infeasible deterministic, stochastic, and variance-reduction algorithms for optimization under orthogonality constraints

    Ablin, P., Vary, S., Gao, B., and Absil, P.-A. Infeasible deterministic, stochastic, and variance-reduction algorithms for optimization under orthogonality constraints. Journal of Machine Learning Research, 25 0 (389): 0 1--38, 2024

    2024

  4. [4]

    Conjugate gradient algorithm for optimization under unitary matrix constraint

    Abrudan, T., Eriksson, J., and Koivunen, V. Conjugate gradient algorithm for optimization under unitary matrix constraint. Signal Processing, 89 0 (9): 0 1704--1714, 2009

    2009

  5. [5]

    E., Eriksson, J., and Koivunen, V

    Abrudan, T. E., Eriksson, J., and Koivunen, V. Steepest descent algorithms for optimization under unitary matrix constraint. IEEE Transactions on Signal Processing, 56 0 (3): 0 1134--1147, 2008

    2008

  6. [6]

    G., and Gallivan, K

    Absil, P.-A., Baker, C. G., and Gallivan, K. A. Trust-region methods on riemannian manifolds. Foundations of Computational Mathematics, 7 0 (3): 0 303--330, 2007

    2007

  7. [7]

    Optimization algorithms on matrix manifolds

    Absil, P.-A., Mahony, R., and Sepulchre, R. Optimization algorithms on matrix manifolds. Princeton University Press, 2008

    2008

  8. [8]

    The pad \'e method for computing the matrix exponential

    Arioli, M., Codenotti, B., and Fassino, C. The pad \'e method for computing the matrix exponential. Linear algebra and its applications, 240: 0 111--130, 1996

    1996

  9. [9]

    Unitary evolution recurrent neural networks

    Arjovsky, M., Shah, A., and Bengio, Y. Unitary evolution recurrent neural networks. In Balcan, M. and Weinberger, K. Q. (eds.), Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016 , volume 48 of JMLR Workshop and Conference Proceedings , pp.\ 1120--1128. JMLR.org, 2016. URL http://proce...

    2016

  10. [10]

    Can we gain more from orthogonality regularizations in training deep networks? In Bengio, S., Wallach, H

    Bansal, N., Chen, X., and Wang, Z. Can we gain more from orthogonality regularizations in training deep networks? In Bengio, S., Wallach, H. M., Larochelle, H., Grauman, K., Cesa - Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, Decemb...

    2018

  11. [11]

    Pattern Recognition and Machine Learning

    Bishop, C. Pattern Recognition and Machine Learning. Springer, 2006. URL https://www.microsoft.com/en-us/research/publication/pattern-recognition-machine-learning/

    2006

  12. [12]

    An introduction to optimization on smooth manifolds

    Boumal, N. An introduction to optimization on smooth manifolds. Cambridge University Press, 2023

    2023

  13. [13]

    and White, R

    Cai, L. and White, R. E. Reduction of model order based on proper orthogonal decomposition for lithium-ion battery simulations. Journal of the Electrochemical Society, 156 0 (3): 0 A154, 2008

    2008

  14. [14]

    and Laheld, B

    Cardoso, J.-F. and Laheld, B. H. Equivariant adaptive source separation. IEEE Transactions on signal processing, 44 0 (12): 0 3017--3030, 2002

    2002

  15. [15]

    Casado, M. L. Trivializations for gradient-based optimization on manifolds. In Wallach, H. M., Larochelle, H., Beygelzimer, A., d'Alch \'e - Buc, F., Fox, E. B., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Ca...

    2019

  16. [16]

    Casado, M. L. and Mart \'i nez - Rubio, D. Cheap orthogonal constraints in neural networks: A simple parametrization of the orthogonal and unitary group. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA , volume 97 of Proceedings of M...

    2019

  17. [17]

    Proximal gradient method for nonsmooth optimization over the stiefel manifold

    Chen, S., Ma, S., Man-Cho So, A., and Zhang, T. Proximal gradient method for nonsmooth optimization over the stiefel manifold. SIAM Journal on Optimization, 30 0 (1): 0 210--239, 2020

    2020

  18. [18]

    Chen, S., Deng, Z., Ma, S., and So, A. M.-C. Manifold proximal point algorithms for dual principal component pursuit and orthogonal dictionary learning. IEEE transactions on signal processing, 69: 0 4759--4773, 2021

    2021

  19. [19]

    Probabilistic circuits: A unifying framework for tractable probabilistic modeling

    Choi, Y., Vergari, A., and Van den Broeck, G. Probabilistic circuits: A unifying framework for tractable probabilistic modeling. Technical report, University of California, Los Angeles (UCLA), 2020

    2020

  20. [20]

    The singular value decomposition: Anatomy of optimizing an algorithm for extreme scale

    Dongarra, J., Gates, M., Haidar, A., Kurzak, J., Luszczek, P., Tomov, S., and Yamazaki, I. The singular value decomposition: Anatomy of optimizing an algorithm for extreme scale. SIAM Review, 60 0 (4): 0 808--865, 2018. doi:10.1137/17M1117732. URL https://doi.org/10.1137/17M1117732

    doi:10.1137/17m1117732 2018

  21. [21]

    A., and Smith, S

    Edelman, A., Arias, T. A., and Smith, S. T. The geometry of algorithms with orthogonality constraints. SIAM journal on Matrix Analysis and Applications, 20 0 (2): 0 303--353, 1998

    1998

  22. [22]

    O-vit: Orthogonal vision transformer

    Fei, Y., Liu, Y., Wei, X., and Chen, M. O-vit: Orthogonal vision transformer. ArXiv preprint, abs/2201.12133, 2022. URL https://arxiv.org/abs/2201.12133

    Pith/arXiv arXiv 2022

  23. [23]

    Francis, J. G. F. The qr transformation a unitary analogue to the lr transformation--part 1. The Computer Journal, 4 0 (3): 0 265--271, 1961. ISSN 0010-4620. doi:10.1093/comjnl/4.3.265. URL https://doi.org/10.1093/comjnl/4.3.265

    doi:10.1093/comjnl/4.3.265 1961

  24. [24]

    Parallelizable algorithms for optimization problems with orthogonality constraints

    Gao, B., Liu, X., and Yuan, Y.-x. Parallelizable algorithms for optimization problems with orthogonality constraints. SIAM Journal on Scientific Computing, 41 0 (3): 0 A1949--A1983, 2019

    2019

  25. [25]

    An orthogonalization-free parallelizable framework for all-electron calculations in density functional theory

    Gao, B., Hu, G., Kuang, Y., and Liu, X. An orthogonalization-free parallelizable framework for all-electron calculations in density functional theory. SIAM Journal on Scientific Computing, 44 0 (3): 0 B723--B745, 2022

    2022

  26. [26]

    Improving normalizing flows via better orthogonal parameterizations

    Goli\' n ski, A., Lezcano-Casado, M., and Rainforth, T. Improving normalizing flows via better orthogonal parameterizations. In ICML Workshop on Invertible Neural Networks, 2019. URL https://api.semanticscholar.org/CorpusID:271877803

    2019

  27. [27]

    Gower, J. C. and Dijksterhuis, G. B. Procrustes problems, volume 30. Oxford university press, 2004

    2004

  28. [28]

    Riemannian coordinate descent algorithms on matrix manifolds

    Han, A., Jawanpuria, P., and Mishra, B. Riemannian coordinate descent algorithms on matrix manifolds. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net, 2024. URL https://openreview.net/forum?id=bdKaQmrM81

    2024

  29. [29]

    Efficient optimization with orthogonality constraint: a randomized riemannian submanifold method

    Han, A., Poirion, P.-L., and Takeda, A. Efficient optimization with orthogonality constraint: a randomized riemannian submanifold method. ArXiv preprint, abs/2505.12378, 2025. URL https://arxiv.org/abs/2505.12378

    Pith/arXiv arXiv 2025

  30. [30]

    Henry, G., Tang, P. T. P., and Heinecke, A. Leveraging the bfloat16 artificial intelligence datatype for higher-precision computations. In 2019 IEEE 26th Symposium on Computer Arithmetic (ARITH), pp.\ 69--76. IEEE, 2019

    2019

  31. [31]

    Hyv \"a rinen, A., Hurri, J., and Hoyer, P. O. Independent component analysis. In Natural Image Statistics: A Probabilistic Approach to Early Computational Vision, pp.\ 151--175. Springer, 2001

    2001

  32. [32]

    and Valera, I

    Javaloy, A. and Valera, I. Rotograd: Gradient homogenization in multitask learning. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022. URL https://openreview.net/forum?id=T8wHz4rnuGL

    2022

  33. [33]

    94\ ArXiv preprint, abs/2404.00498, 2024

    Jordan, K. 94\ ArXiv preprint, abs/2404.00498, 2024. URL https://arxiv.org/abs/2404.00498

    Pith/arXiv arXiv 2024

  34. [34]

    Muon: An optimizer for hidden layers in neural networks, 2024

    Jordan, K., Jin, Y., Boza, V., You, J., Cesista, F., Newhouse, L., and Bernstein, J. Muon: An optimizer for hidden layers in neural networks, 2024. URL https://kellerjordan.github.io/posts/muon/

    2024

  35. [35]

    T., Balestriero, R., LeCun, Y., and Lloyd, S

    Kiani, B. T., Balestriero, R., LeCun, Y., and Lloyd, S. projunn: efficient method for training deep networks with unitary matrices. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orlea...

    2022

  36. [36]

    Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In Bengio, Y. and LeCun, Y. (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015. URL http://arxiv.org/abs/1412.6980

    Pith/arXiv arXiv 2015

  37. [37]

    Learning multiple layers of features from tiny images

    Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009

    2009

  38. [38]

    Kublanovskaya, V. N. On some algorithms for the solution of the complete eigenvalue problem. USSR Computational Mathematics and Mathematical Physics, 1 0 (3): 0 637--657, 1962

    1962

  39. [39]

    MNIST handwritten digit database

    LeCun, Y., Cortes, C., and Burges, C. MNIST handwritten digit database . ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist, 2, 2010

    2010

  40. [40]

    Orthogonal deep neural networks

    Li, S., Jia, K., Wen, Y., Liu, T., and Tao, D. Orthogonal deep neural networks. IEEE transactions on pattern analysis and machine intelligence, 43 0 (4): 0 1352--1368, 2019

    2019

  41. [41]

    Vectoradam for rotation equivariant geometry optimization

    Ling, S., Sharp, N., and Jacobson, A. Vectoradam for rotation equivariant geometry optimization. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December...

    2022

  42. [42]

    A penalty-free infeasible approach for a class of nonsmooth optimization problems over the stiefel manifold

    Liu, X., Xiao, N., and Yuan, Y.-x. A penalty-free infeasible approach for a class of nonsmooth optimization problems over the stiefel manifold. Journal of Scientific Computing, 99 0 (2): 0 30, 2024

    2024

  43. [43]

    M., Mengel, S., Trapp, M., Solin, A., Gillis, N., and Vergari, A

    Loconte, L., Sladek, A. M., Mengel, S., Trapp, M., Solin, A., Gillis, N., and Vergari, A. Subtractive mixture models via squaring: Representation and learning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024. URL https://openreview.net/forum?id=xIHi5nxu9P

    2024

  44. [44]

    How to square tensor networks and circuits without squaring them

    Loconte, L., Javaloy, A., and Vergari, A. How to square tensor networks and circuits without squaring them. ArXiv preprint, abs/2512.17090, 2025 a . URL https://arxiv.org/abs/2512.17090

    Pith/arXiv arXiv 2025

  45. [45]

    Sum of squares circuits

    Loconte, L., Mengel, S., and Vergari, A. Sum of squares circuits. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp.\ 19077--19085, 2025 b

    2025

  46. [46]

    and Loan, C

    Moler, C. and Loan, C. Nineteen dubious ways to compute the exponential of a matrix, twenty-five years later. SIAM Review, 45: 0 3--49, 2006. doi:10.1137/S00361445024180

    doi:10.1137/s00361445024180 2006

  47. [47]

    and Bourlard, H

    Morgan, N. and Bourlard, H. Generalization and parameter estimation in feedforward nets: Some experiments. Advances in neural information processing systems, 2, 1989

    1989

  48. [48]

    Tensor networks for complex quantum systems

    Or \'u s, R. Tensor networks for complex quantum systems. Nature Reviews Physics, 1 0 (9): 0 538--550, 2019

    2019

  49. [49]

    and Okatani, T

    Ozay, M. and Okatani, T. Optimization on submanifolds of convolution kernels in cnns. ArXiv preprint, abs/1610.07008, 2016. URL https://arxiv.org/abs/1610.07008

    Pith/arXiv arXiv 2016

  50. [50]

    Distributed retraction-free and communication-efficient optimization on the stiefel manifold

    Song, Y., Li, P., Gao, B., and Yuan, K. Distributed retraction-free and communication-efficient optimization on the stiefel manifold. ArXiv preprint, abs/2506.02879, 2025. URL https://arxiv.org/abs/2506.02879

    Pith/arXiv arXiv 2025

  51. [51]

    Richtungsfelder und Fernparallelismus in n-dimensionalen Mannigfaltigkeiten

    Stiefel, E. Richtungsfelder und Fernparallelismus in n-dimensionalen Mannigfaltigkeiten. PhD thesis, ETH Zurich, 1935

    1935

  52. [52]

    Optimization without retraction on the random generalized stiefel manifold

    Vary, S., Ablin, P., Gao, B., and Absil, P. Optimization without retraction on the random generalized stiefel manifold. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net, 2024. URL https://openreview.net/forum?id=QLtxj3erlJ

    2024

  53. [53]

    Vergari, A., Choi, Y., Liu, A., Teso, S., and den Broeck, G. V. A compositional atlas of tractable circuit operations for probabilistic inference. In Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, Neur...

    2021

  54. [54]

    Wang, J., Chen, Y., Chakraborty, R., and Yu, S. X. Orthogonal convolutional neural networks. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020 , pp.\ 11502--11512. IEEE , 2020. doi:10.1109/CVPR42600.2020.01152. URL https://doi.org/10.1109/CVPR42600.2020.01152

    arXiv 2020