Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

This paper argues that transformers can be reformulated to compute attention and optimize their parameters using only coordinate-independent relational quantities, removing the large redundancy caused by continuous symmetries in representat

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 21:47 UTC pith:MHSQ2ZQV

load-bearing objection A genuinely novel symmetry-reduction framework for transformers, correct on the parameter invariants, but the relational attention is an expressivity-limited prototype and the optimization claims are untested. the 3 major comments →

arxiv 2602.18948 v2 pith:MHSQ2ZQV submitted 2026-02-21 cs.LG cs.NEhep-thstat.ML

Toward Manifest Relationality in Transformers via Symmetry Reduction

J. Fran\c{c}ois , L. Ravera This is my paper
classification cs.LG cs.NEhep-thstat.ML MSC 68T07
keywords transformersattention mechanismssymmetry reductionrelational representationsGram matrixinvariant compositesquotient-space optimizationgauge-theory inspiration
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes a symmetry-reduction program for transformers: instead of breaking the continuous symmetries that make many parameter settings functionally identical, it rewrites representations, attention, and optimization in terms of invariant relational quantities. Token states are captured by their Gram matrix, which is unchanged by arbitrary orthogonal changes of basis in the hidden space; attention weights become a learned function of Gram entries. Attention-head parameters are likewise reduced to invariant composites (the query-key bilinear form and the value-output product), and training is moved to the quotient of parameter space by the symmetry group so updates never waste budget along redundant directions. The paper is explicitly conceptual—it fixes the symmetries for idealized blocks, flags that LayerNorm and biases break them in practice, and leaves empirical tests to future work. If its central assumption holds, the payoff is a principled geometric basis for leaner, more interpretable models whose internals reflect relations between tokens rather than arbitrary coordinates.

Core claim

On the paper's own terms, the central discovery is that the functionally meaningful content of a transformer layer can be expressed through manifestly invariant relatives: the Gram matrix G = XX⊤ of token states for representation, the composite G_QK = W_Q⊤ W_K for query-key scoring, and the composite G_VO = W_O W_V for the value-output sector. Attention weights are computed as s_ij = f(G_ij, G_ii, G_jj) with a learned scalar function, making weights strictly invariant under O(d) frame changes, while vector features are transported equivariantly and the Gram matrix itself propagates through layers as G⁺ = A G A⊤. In head space, the exact reparameterization symmetries O(d_h) and GL(d_h,R) lea

What carries the argument

The load-bearing objects are O(d)-invariant relational quantities. The Gram matrix of token states encodes pairwise relations and is invariant under global rotations of the hidden space; attention weights are defined as a learned function of its entries, so the scoring step contains no frame-dependent information. For parameters, the invariant composites G_QK and G_VO capture everything the forward pass uses, subject to rank constraints from the head dimension. A 'dressing' construction—choosing a canonical representative of each symmetry orbit via the singular value decomposition of the token matrix—lets vector-valued features be carried along without reintroducing frame dependence. These p

Load-bearing premise

The claim that attention scores computed from Gram-matrix entries alone (s_ij = f(G_ij, G_ii, G_jj)) can match the expressive power of standard dot-product attention; the paper calls this an idealized, illustrative prototype and offers no proof or experiment.

What would settle it

Take any fixed learned query-key pair (W_Q, W_K) and any two token matrices X and X' with the same Gram matrix but different scores x_i^T W_Q^T W_K x_j; since the Gram-based scoring family sees only Gram entries, it cannot distinguish X from X', so a task rewarded on that distinction would show a clear accuracy gap. Concretely, train a small transformer with standard attention on such a task and compare with the Gram-only version; a performance drop falsifies expressivity preservation.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Attention weights become independent of arbitrary basis choices in the hidden space, so the same relational content is preserved under any O(d) rotation of representations.
  • Training dynamics on the quotient space Θ/G have no component along symmetry orbits, so optimization budget is not spent on directions that leave the model function unchanged.
  • The functionally relevant parameters of attention heads reduce to invariant composites G_QK and G_VO; optimizing them directly requires enforcing rank ≤ d_h, either by low-rank parameterization or by projection after each step.
  • Since LayerNorm and biases break the idealized symmetries, the framework's guarantees apply exactly only to the linear attention submodule and approximately to standard blocks.
  • The framework provides a common language for representation geometry and optimization degeneracy, connecting transformer analysis to relational, gauge-style reasoning without inserting preferred coordinate directions.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural test is to enrich the invariant scoring family to s_ij = f(x_i^T A x_j, x_i^T x_i, x_j^T x_j) with a learned bilinear form A, which would preserve O(d)-invariance while recovering dot-product expressivity; the paper does not pursue this intermediate step.
  • Because G_QK and G_VO have rank at most d_h, direct optimization in these composites changes the model class unless the constraint is enforced; this suggests the rank constraint could itself serve as an architectural regularizer, an idea implicit but not developed in the paper.
  • The discrete head-permutation symmetry is treated as a residual relabeling freedom; a relational formulation could absorb it by representing heads as a set rather than an ordered list, potentially aligning with permutation-invariant ensemble ideas.
  • The framework predicts that symmetry-reduced training should show reduced sensitivity to initialization and flatter effective loss landscapes; these are testable predictions that should hold approximately even when LayerNorm breaks exact symmetry, provided the symmetry-breaking is small.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a symmetry-reduction framework for transformer architectures, inspired by the dressing field method (DFM) from gauge theory. It distinguishes two types of redundancy: (i) O(d) frame dependence of token representations in model space, and (ii) O(d_h) and GL(d_h,R) reparameterization symmetries inside attention heads. For the former, it introduces Gram-matrix-based relational attention (Eqs. 15-19) and an SVD-based dressing map (Eqs. 23-28). For the latter, it formulates quotient-space optimization using invariant composites G_QK = W_Q^T W_K and G_VO = W_O W_V (Eqs. 40-47), with projected gradient schemes sketched in Section V and Appendix A. The paper is explicitly conceptual: it states that empirical evaluation and engineering trade-offs are future work, and several constructions are labeled 'idealized prototypes'. The mathematical identities are elementary and mostly correct, but the central claim that this reformulation preserves expressivity while eliminating redundant degrees of freedom is not established for the Gram-only attention mechanism, and the dressing map is only well-defined on an open dense subset.

Significance. If the framework is made rigorous, it could provide a principled alternative to explicit symmetry breaking in transformer training, potentially reducing optimization degeneracies and offering interpretable relational variables. The identification of the exact reparameterization symmetries of linear attention submodules and the invariant composites G_QK and G_VO (Section V) is correct and useful, and the paper is careful to distinguish exact symmetries from architecture-dependent approximate ones. However, the contribution is currently a vision/proposal rather than a complete theory: the expressivity of Gram-only attention is unresolved, the SVD dressing has known degeneracy issues, and the optimization benefits are asserted without experiments or convergence analysis. As a conceptual framework, it is thought-provoking, but the central claims need to be either proven, substantially qualified, or re-scoped.

major comments (3)
  1. [Section IV, Eq. (16); cf. footnote [52]] The Gram-only attention score s_ij = f(G_ij, G_ii, G_jj) is not a reformulation of standard dot-product attention (Eq. 10). Standard scores are x_i^T W_Q^T W_K x_j, which depend on the learned bilinear form M = W_Q^T W_K. For a non-scalar M, two token matrices X and X' with the same Gram matrix (e.g., X' = X Q with Q ∈ O(d)) generally give different scores because x_i^T M x_j changes unless Q M Q^T = M. Hence the family in Eq. (16) is strictly smaller than the standard attention family and does not preserve model expressivity. Footnote [52] and the surrounding text call this an 'idealized, illustrative prototype', but the abstract and Section I state that attention mechanisms are 'reformulated' in invariant relational form 'while preserving model expressivity'. This is a load-bearing inconsistency. The authors should either prove a representational equivalence or universality result for
  2. [Section IV.A.b, Eqs. (23)-(28)] The dressing map u[X] = V_X^T from the SVD is not well-defined on the entire space R^{n×d}. For rank-deficient X or degenerate singular values, V_X is ambiguous up to an O(m) rotation within degenerate subspaces, and even with distinct singular values, a sign convention is required. The paper acknowledges this in Section IV.A.b, but then uses the dressed representative \hat{X} = U_X \Sigma_X as if it were strictly O(d)-invariant. The equivariance identity (27) holds only on the open dense set where a deterministic SVD convention is consistent. If the dressing map is to be used in optimization, as suggested by Eq. (38)-(39) and Appendix A, the non-smoothness and discontinuity at degenerate points can affect gradient-based methods. The paper should either show that optimization trajectories generically avoid these degeneracies, or explain how the residual symmetry is treated in the quotien
  3. [Section V.C and Appendix A, Eqs. (A5)-(A9)] The proposed invariant update on G_VO uses an ambient gradient step followed by projection onto the rank-≤d_h manifold. This is a heuristic projected-gradient scheme for a non-convex constraint, not a canonical quotient-gradient update. The paper correctly notes in Section V.B.c that the choice of inner product (and hence the transverse projection) is non-canonical, but the stronger claims that this 'eliminates motion along symmetry directions' and potentially 'improves optimization efficiency and stability' are not supported by analysis. In particular, the projected update (A9) is not shown to be a descent direction for the original loss, nor is it shown to converge to a stationary point of the constrained problem. Since the paper explicitly leaves empirical evaluation to future work, these optimization claims should be framed as open conjectures, and the conditions under which the proj
minor comments (5)
  1. [Abstract and Section I] The abstract and introduction would benefit from immediate qualification that the relational attention construction is a prototype, not a drop-in replacement for standard dot-product attention; as written, the phrase 'reformulate ... attention mechanisms ... preserving model expressivity' overstates what is demonstrated.
  2. [Eq. (16)] The function f is defined on (G_ij, G_ii, G_jj). Unless f is explicitly constrained to be symmetric in its first and third arguments, the score s_ij need not equal s_ji. This may be intentional, but the asymmetry should be noted, especially since standard dot-product attention is also asymmetric due to distinct query and key projections.
  3. [Eq. (18)] The choice to set values to the token states themselves (rather than learned value projections) is a significant architectural departure. It deserves a brief discussion of the expressivity implications, beyond the statement that it is a symmetry-manifest choice.
  4. [Section V.A] The statement that points of Θ/G correspond to distinct functions assumes that the group action captures all functional equivalences. In practice, there may be accidental symmetries not in G, or different orbits may implement the same function. The wording should be softened to 'distinct functions up to the specified symmetry'.
  5. [References] The paper leans heavily on the authors' own prior work on the dressing field method ([27]-[38]). While this is natural for the motivating framework, the ML reader would benefit from a more self-contained explanation of DFM or a shorter list of representative references.

Circularity Check

0 steps flagged

No significant circularity: the invariance identities (Gram matrix, SVD dressing, G_QK and G_VO composites, rank constraints) are derived in-text from elementary algebra; the 12 DFM self-citations are explicitly disclaimed as conceptual motivation, and Eq. 16's invariance-by-construction is honestly scoped as an idealized prototype (footnote [52]).

full rationale

The paper's derivation chain is self-contained elementary algebra. The O(d)-invariance of G=XX^T (Eq. 15), the equivariance of the SVD dressing field u[XU^T]=u[X]U^T (Eqs. 25-27), the invariance of G_QK=W_Q^T W_K under shared O(d_h) rotations (Eqs. 40-43), the invariance of G_VO=W_O W_V under GL(d_h) (Eqs. 44-47), and the rank constraints rank(G_QK)<=d_h and rank(G_VO)<=d_h (Eqs. 48, A6) are all proven directly in the text and do not depend on any citation. The 12 DFM self-citations [27]-[38] and the relational-terminology citation [42] are expressly disclaimed as load-bearing: Section II states 'We do not claim that transformer models possess gauge symmetries in the field-theoretic sense, nor that the Dressing Field Method (DFM) can be directly applied to ML architectures,' framing the review as 'methodological inspiration.' So the heavy self-citation is self-referential framing, not a load-bearing premise. The one result that is true 'by construction'—Eq. 16's O(d)-invariant Gram scoring—is honestly scoped: footnote [52] calls it 'an idealized, illustrative prototype rather than a drop-in replacement for standard dot-product attention,' and Section IV warns that generic learned feature maps 'would generally break O(d) unless explicitly constrained.' The Introduction's 'preserving model expressivity' phrase is thus qualified by the paper's own limitations; whether Eq. 16's function class covers standard dot-product attention is an expressivity/correctness question, not circularity, and the paper presents Eq. 16 as a construction, not as a derived theorem. The quotient-optimization claims are likewise framed as proposals ('we outline three practical schemes'), with 'Empirical evaluation... left for future work,' so there is no fitted-input-called-prediction. No uniqueness theorem is imported: the dressing choice is explicitly non-unique (Section IV A a 'Residual ambiguity'). The score of 1 reflects only the volume of self-referential framing and borrowed vocabulary; the central derivation is independent.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 0 invented entities

No free parameters are fitted in this paper; the framework introduces learned functions (f, G_QK, G_VO) but no numeric values. The axioms are the mathematical/domain assumptions the framework relies on, several of which the paper explicitly flags as idealizations. No new physical entities are introduced; 'dressing field' and 'relational invariants' are mathematical constructions adapted from physics.

axioms (5)
  • domain assumption Attention-head reparameterization symmetries: (W_Q,W_K)→(S W_Q, S W_K), S∈O(d_h) and (W_V,W_O)→(S^{-1}W_V, W_O S), S∈GL(d_h) are exact for the idealized linear submodule.
    Eqs. (13)-(14). The paper states in Section III.A.1 that this is exact only for the minimal block and approximate in practical transformers with LayerNorm/biases.
  • ad hoc to paper Gram matrix G = XX^T captures functionally relevant token relations, so attention can be scored from G alone without expressivity loss.
    Eqs. (15)-(16); footnote [52] calls this an idealized prototype, and the paper does not prove it preserves the expressivity of dot-product attention.
  • ad hoc to paper SVD dressing map u[X]=V_X^T is well-defined and usable in optimization (deterministic conventions fix signs; degeneracies are residual).
    Section IV.A.b; the paper acknowledges non-uniqueness on degenerate spectra and leaves the practical resolution open.
  • ad hoc to paper Symmetry-reduced quotient optimization improves or at least does not worsen training relative to unconstrained optimization.
    Section V; the benefit is motivated via flat directions and conserved quantities but not proven or empirically demonstrated.
  • domain assumption LayerNorm, biases, and MLP nonlinearities act as small symmetry-breaking perturbations, making approximate reduction meaningful.
    Section III.A.1.b; explicitly stated as an assumption.

pith-pipeline@v1.3.0-alltime-deepseek · 15091 in / 13157 out tokens · 114439 ms · 2026-08-02T21:47:20.526551+00:00 · methodology

0 comments
read the original abstract

Transformer models contain substantial internal redundancy arising from coordinate-dependent representations and continuous symmetries, in model space and in head space, respectively. While recent approaches address this by explicitly breaking symmetry, we propose a complementary framework based on symmetry reduction. We reformulate representations, attention mechanisms, and optimization dynamics in terms of invariant relational quantities, eliminating redundant degrees of freedom by construction. This perspective yields architectures that operate directly on relational structures, providing a principled geometric framework for reducing parameter redundancy and analyzing optimization.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. The Token Is a Group Element: On Lie-Algebra Attention over Matrix Lie Groups

    cs.LG 2026-06 unverdicted novelty 8.0

    Lie-Algebra Attention defines tokens as bare matrix Lie group elements and computes attention scores from the negative squared algebra norm of the relative pose log(g_i^{-1} g_j).

  2. Lecture Notes on Symmetry Reduction via the Dressing Field Method

    hep-th 2026-03 unverdicted novelty 1.0

    Lecture notes on the Dressing Field Method for symmetry reduction, presenting a framework for invariant observables with examples from Chern-Simons theory, electromagnetism, Higgs model, supersymmetry, and general relativity.

Reference graph

Works this paper leans on

69 extracted references · 36 linked inside Pith · cited by 2 Pith papers

  1. [1]

    Dressing

    “Dressing” internal representation frames by re- placing coordinate-dependent vectors with “rela- tional invariants”, hence reformulating attention mechanisms in “invariant relational form”

  2. [2]

    dressing

    Considering optimization dynamics in reduced, symmetry-free parameter spaces. Our immediate goal is not yet to replace existing archi- tectures, but to establish a conceptual and mathematical framework in which learning proceeds directly on mean- ingful relational d.o.f. rather than on arbitrary coordi- nate representations. Such a formulation may reduce ...

  3. [3]

    internal symmetry

    What symmetry actually holds (architecture-dependent) Before proceeding with our approach, to avoid over- claiming, let us fix here an explicit class of blocks and state the precise reparameterization symmetry. a. What is symmetric. In standard transformers, most nonlinear components (MLP with elementwise activation, LayerNorm, biases) are defined with re...

  4. [4]

    dressing

    Reduced variables as invariants: Learning onΘ red A practical approach to quotient optimization is to pa- rameterize the model usinginvariantsunder the group G. In the attention setting, an example of such invari- ant combinations arises from the observation that dot- product scores depend onW Q andW K only through the bilinear form GQK :=W ⊤ Q WK,(40) si...

  5. [5]

    Internal representation frames are dressed by re- placing coordinate-dependent vectors with rela- tional invariants, and attention mechanisms are rewritten in invariant relational form

  6. [6]

    Optimization dynamics are studied on reduced pa- rameter spaces, eliminating motion along redun- dant symmetry orbits. Together, these would yield a framework in which rep- resentations, attention, optimization, and structure are all formulated in manifestly relational (with invariant weights and equivariant vector carriers). Of course, both standard dot-...

  7. [7]

    Section III A 1)

    Invariant gradient and direct update onGV O Restricting attention to the symmetric submodule, the loss may be viewed as a function of the invariant, L=L(G V O),(A3) up to symmetry-breaking components such as Layer- Norm, biases, and nonlinear MLPs (cf. Section III A 1). One may therefore define the invariant gradient ∇GV OL ∈R d×d,(A4) and perform symmetr...

  8. [8]

    Two standard strategies may be used to address this

    Rank constraint and low-rank realizations SinceG V O factors through ad h-dimensional head space, it necessarily satisfies rank(GV O)≤d h.(A6) Consequently, unrestricted updates of the form (A5) may leave the representable set unless this constraint is en- forced (if one updatedG V O freely, it may become full- rank, hence no longer realizable by any (WO,...

  9. [9]

    For example, from a truncated SVDGV O= UΣV ⊤, with Σ∈R dh×dh , one may set WO :=UΣ 1/2, W V := Σ1/2V ⊤,(A10) so thatW OWV =G V O

    Choosing representatives for standard implementations If one wishes to implement the forward pass in the conventional factorized formW O,W V , a representative 11 factorization of the current invariantG V Omay be chosen at any stage. For example, from a truncated SVDGV O= UΣV ⊤, with Σ∈R dh×dh , one may set WO :=UΣ 1/2, W V := Σ1/2V ⊤,(A10) so thatW OWV =...

  10. [10]

    Devlin, M.-W

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, Bert: Pre-training of deep bidirectional transformers for lan- guage understanding,NAACL, arXiv:1810.04805 (2019)

    Pith/arXiv arXiv 2019

  11. [11]

    T. B. Brown, B. Mann, N. Ryder, M. Subbiah, and J. e. a. Kaplan, Language models are few-shot learners, NeurIPS, arXiv:2005.14165 (2020)

    Pith/arXiv arXiv 2005

  12. [12]

    Dosovitskiy, L

    A. Dosovitskiy, L. Beyer, and A. e. a. Kolesnikov, An im- age is worth 16x16 words: Transformers for image recog- nition at scale,ICLR, arXiv:2010.11929 (2021)

    Pith/arXiv arXiv 2010

  13. [13]

    Radford and J

    A. Radford and J. W. e. a. Kim, Learning transferable visual models from natural language supervision,ICML, arXiv:2103.00020 (2021)

    Pith/arXiv arXiv 2021

  14. [14]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, At- tention is all you need,Advances in Neural Information Processing Systems (NeurIPS), arXiv:1706.03762 (2017)

    Pith/arXiv arXiv 2017

  15. [15]

    Merrill, G

    W. Merrill, G. Weiss, Y. Goldberg, R. Schwartz, N. A. Smith, and E. Yahav, A formal hierarchy of rnns and transformers, arXiv:2004.08500 (2020)

    Pith/arXiv arXiv 2004

  16. [16]

    T. Lian, Y. Wang, X. Liu, and X. Qiu, A survey of trans- formers, arXiv:2106.04554 (2023)

    Pith/arXiv arXiv 2023

  17. [17]

    Elhage, N

    N. Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mann, A. Askell, Y. Bai, A. Chen, and T. e. a. Conerly, A mathematical frame- work for transformer circuits, https://transformer- circuits.pub/2021/framework/index.html (2023)

    2021

  18. [18]

    In transformers, equivariance means that if you apply a transformationgto the input (or to an internal rep- resentation) and there is a corresponding transforma- tionρ(g) on the output (or next-layer representation), then the layerF(hereFis simply the learned function implemented by the layer) commutes with that action: F(g·X) =ρ(g)·F(X). It is an archite...

  19. [19]

    T. S. Cohen and M. Welling, Group equivariant convo- lutional networks,International Conference on Machine Learning (ICML), arXiv:1602.07576 (2016)

    Pith/arXiv arXiv 2016

  20. [20]

    Kondor and S

    R. Kondor and S. Trivedi, On the generalization of equiv- ariance and convolution in neural networks to the action of compact groups,International Conference on Machine Learning (ICML), arXiv:1802.03690 (2018)

    Pith/arXiv arXiv 2018

  21. [21]

    Kunin, J

    D. Kunin, J. Sagastuy-Brena, S. Ganguli, D. L. K. Yamins, and H. Tanaka, Neural mechanics: Symmetry and broken conservation laws in deep learning dynamics, arXiv:2012.04728 (2020)

    Pith/arXiv arXiv 2012

  22. [22]

    Tanaka and D

    H. Tanaka and D. Kunin, Noether’s learning dynam- ics: Role of symmetry breaking in neural networks, arXiv:2105.02716 (2021)

    Pith/arXiv arXiv 2021

  23. [23]

    L. Dinh, R. Pascanu, S. Bengio, and Y. Bengio, Sharp minima can generalize for deep nets, arXiv:1703.04933 (2017)

    Pith/arXiv arXiv 2017

  24. [24]

    Zhang, Z

    B. Zhang, Z. Zheng, Z. Chen, and J. Li, Beyond the per- mutation symmetry of transformers: The role of rotation for model fusion, arXiv:2502.00264 (2025)

    Pith/arXiv arXiv 2025

  25. [25]

    Silverstein, D

    E. Silverstein, D. Kunin, and V. Shyam, Symmetry breaking in transformers for efficient and interpretable training, arXiv:2601.22257 (2026)

    arXiv 2026

  26. [26]

    M. F. da Silva, F. Dangel, and S. Oore, Hide & seek: Transformer symmetries obscure sharpness & riemannian geometry finds it, arXiv:2505.05409 (2025)

    Pith/arXiv arXiv 2025

  27. [27]

    B. Zhao, I. Ganev, R. Walters, R. Yu, and N. Dehmamy, Symmetries, flat minima and the conserved quantities of gradient flows,International Conference on Learning Representations (ICLR), arXiv:2210.17216 (2023), open- Review: 9ZpciCOunFb

    Pith/arXiv arXiv 2023

  28. [28]

    Santoro, D

    A. Santoro, D. Raposo, D. G. Barrett, M. Malinowski, R. Pascanu, P. Battaglia, and T. Lillicrap, A simple neu- ral network module for relational reasoning,NeurIPS, arXiv:1706.01427 (2017)

    Pith/arXiv arXiv 2017

  29. [29]

    Zaheer, S

    M. Zaheer, S. Kottur, S. Ravanbakhsh, B. Poc- zos, R. Salakhutdinov, and A. Smola, Deep sets, arXiv:1703.06114 (2017)

    Pith/arXiv arXiv 2017

  30. [30]

    P. W. Battaglia and J. B. e. a. Hamrick, Relational inductive biases, deep learning, and graph networks, arXiv:1806.01261 (2018)

    Pith/arXiv arXiv 2018

  31. [31]

    V. P. Dwivedi and X. Bresson, A generalization of trans- former networks to graphs, arXiv:2012.09699 (2020)

    Pith/arXiv arXiv 2012

  32. [32]

    C. Ying, T. Cai, S. Luo, S. Zheng, G. Ke, D. He, Y. Shen, and T.-Y. Liu, Do transformers really perform bad for graph representation?, inAdvances in Neural Informa- tion Processing Systems (NeurIPS)(2021)

    2021

  33. [33]

    Henneaux and C

    M. Henneaux and C. Teitelboim,Quantization of Gauge Systems(Princeton University Press, Princeton, NJ, 1992) paperback edition (original hardcover also 069108775X)

    1992

  34. [34]

    Guillemin and S

    V. Guillemin and S. Sternberg,Symplectic Techniques in Physics(Cambridge University Press, Cambridge, 1990) paperback reprint (original hardcover 1984)

    1990

  35. [35]

    Gieres, Covariant canonical formulations of classical field theories, SciPost Physics Lecture Notes 10.21468/SciPostPhysLectNotes.77 (2023), 12 arXiv:2109.07330

    F. Gieres, Covariant canonical formulations of classical field theories, SciPost Physics Lecture Notes 10.21468/SciPostPhysLectNotes.77 (2023), 12 arXiv:2109.07330

    Pith/arXiv arXiv 2023

  36. [36]

    Fran¸ cois, Artificial versus Substantial Gauge Symme- tries: A Criterion and an Application to the Electroweak Model, Phil

    J. Fran¸ cois, Artificial versus Substantial Gauge Symme- tries: A Criterion and an Application to the Electroweak Model, Phil. Sci.86, 472 (2019), arXiv:1801.00678 [physics.hist-ph]

    Pith/arXiv arXiv 2019

  37. [37]

    J. T. Francois Andr´ e, The dressing field method for dif- feomorphisms: a relational framework, J. Phys. A57, 305203 (2024), arXiv:2310.14472 [math-ph]

    Pith/arXiv arXiv 2024

  38. [38]

    J. T. Fran¸ cois and L. Ravera, Geometric Relational Framework for General-Relativistic Gauge Field Theo- ries, Fortsch. Phys.73, 2400149 (2025), arXiv:2407.04043 [gr-qc]

    Pith/arXiv arXiv 2025

  39. [39]

    Fran¸ cois and L

    J. Fran¸ cois and L. Ravera, Dressing fields for supersym- metry: the cases of the Rarita-Schwinger and gravitino fields, JHEP07, 041, arXiv:2405.04379 [hep-th]

    Pith/arXiv arXiv

  40. [40]

    Fran¸ cois and L

    J. Fran¸ cois and L. Ravera, Unconventional supersymme- try via the dressing field method, Phys. Rev. D111, 125022 (2025), arXiv:2412.01898 [hep-th]

    Pith/arXiv arXiv 2025

  41. [41]

    Fran¸ cois and L

    J. Fran¸ cois and L. Ravera, Reassessing the foundations of metric-affine gravity, Eur. Phys. J. C85, 902 (2025), arXiv:2505.05349 [gr-qc]

    Pith/arXiv arXiv 2025

  42. [42]

    Fran¸ cois and L

    J. Fran¸ cois and L. Ravera, Mechanics as a general- relativistic gauge field theory, and Relational Quantiza- tion, arXiv:2510.19845 [physics.gen-ph] (2025)

    arXiv 2025

  43. [43]

    Fran¸ cois and L

    J. Fran¸ cois and L. Ravera, Off-shell supersymmetry via manifest invariance, Phys. Lett. B868, 139633 (2025), arXiv:2504.06392 [hep-th]

    Pith/arXiv arXiv 2025

  44. [44]

    Fran¸ cois and L

    J. Fran¸ cois and L. Ravera, Raising galaxy rotation curves via dressing, Phys. Rev. D112, L081501 (2025), arXiv:2510.18549 [gr-qc]

    arXiv 2025

  45. [45]

    J. T. Fran¸ cois and L. Ravera, Relational Bundle Geomet- ric Formulation of Non-Relativistic Quantum Mechan- ics, Fortsch. Phys.73, e70040 (2025), arXiv:2501.02046 [quant-ph]

    Pith/arXiv arXiv 2025

  46. [46]

    Fran¸ cois and L

    J. Fran¸ cois and L. Ravera, Spacetime boundaries do not break diffeomorphism and gauge symmetries, Phys. Rev. D112, 125029 (2025), arXiv:2504.20945 [gr-qc]

    arXiv 2025

  47. [47]

    Berghofer, J

    P. Berghofer, J. Fran¸ cois, and L. Ravera, What Price Fiber Bundle Substantivalism? On How to Avoid Holes in Fibers, arXiv:2505.12876 [physics.hist-ph] (2025)

    Pith/arXiv arXiv 2025

  48. [48]

    Rovelli, What Is Observable in Classical and Quantum Gravity?, Class

    C. Rovelli, What Is Observable in Classical and Quantum Gravity?, Class. Quant. Grav.8, 297 (1991)

    1991

  49. [49]

    Rovelli, Partial observables, Phys

    C. Rovelli, Partial observables, Phys. Rev. D65, 124013 (2002), arXiv:gr-qc/0110035

    Pith/arXiv arXiv 2002

  50. [50]

    Rovelli, Why Gauge?, Found

    C. Rovelli, Why Gauge?, Found. Phys.44, 91 (2014), arXiv:1308.5599 [hep-th]

    Pith/arXiv arXiv 2014

  51. [51]

    Fran¸ cois and L

    J. Fran¸ cois and L. Ravera, On the Meaning of Local Sym- metries: Epistemic-ontological Dialectics, Found. Phys. 55, 38 (2025), arXiv:2404.17449 [physics.hist-ph]

    Pith/arXiv arXiv 2025

  52. [52]

    T. S. Cohen, M. Weiler, B. Kicanaoglu, and M. Welling, Gauge equivariant convolutional networks and the icosa- hedral cnn, arXiv:1902.04615 (2019)

    Pith/arXiv arXiv 1902

  53. [53]

    Theodosis, D

    E. Theodosis, D. E. Ba, and N. Dehmamy, In- corporating gauge-invariance in equivariant networks, OpenReview: YAINolpm8n (2024), openReview: https://openreview.net/forum?id=YAINolpm8n

    2024

  54. [54]

    Theodosis, D

    E. Theodosis, D. Ba, and N. Dehmamy, Constructing gauge-invariant neural networks for scientific applica- tions, ICML 2024 (2024)

    2024

  55. [55]

    Choi and C.-K

    Y. Choi and C.-K. Kim, Gauge-equivariant graph net- works via self-interference cancellation, arXiv:2511.16062 (2025)

    Pith/arXiv arXiv 2025

  56. [56]

    Huang, O

    L. Huang, O. Balabanov, H. Linander, M. Granath, D. Persson, and J. E. Gerken, Learning chern num- bers of multiband topological insulators with gauge equivariant neural networks,Advances in Neural In- formation Processing Systems (NeurIPS), NeurIPS 2025 Poster; arXiv:2502.15376 (2025), openReview: https://openreview.net/forum?id=6pjzFIyFBo

    Pith/arXiv arXiv 2025

  57. [57]

    Strunk and R

    A. Strunk and R. Assam, Gauge flow models, arXiv:2507.13414 (2025)

    arXiv 2025

  58. [58]

    Honda, A gauge-theory-based graph neu- ral network,Submitted to ICLR 2026, Open- Review: QxoyccprRp (2026), openReview: https://openreview.net/forum?id=QxoyccprRp

    H. Honda, A gauge-theory-based graph neu- ral network,Submitted to ICLR 2026, Open- Review: QxoyccprRp (2026), openReview: https://openreview.net/forum?id=QxoyccprRp

    2026

  59. [59]

    Then attention scores areq ⊤ i kj,which is just the dot product between query and key

    They take a token vectorx i ∈R d and produceq i = WQxi,k i =W K xi,whereq i, ki ∈R dh live in the head space. Then attention scores areq ⊤ i kj,which is just the dot product between query and key

  60. [60]

    , zn)∈R n is defined as softmax(z) i = exp(zi)Pn j=1 exp(zj ) ,i= 1,

    The ‘softmax’ function applied to a vector of raw scores (logits)z= (z 1, . . . , zn)∈R n is defined as softmax(z) i = exp(zi)Pn j=1 exp(zj ) ,i= 1, . . . , n. It maps the scores to a prob- ability distribution (non-negative entries summing to 1), with larger values receiving exponentially higher weight

  61. [61]

    This construction is intended as an idealized, illustrative prototype rather than a drop-in replacement for standard dot-product attention, meant to make the symmetry- reduced structure explicit

  62. [62]

    Let us remark that, whenR ij is taken to be an inner product (or a learned bilinear form) andfis scalar, the resulting attention weights are a form ofkernelized at- tention. Our contribution here is not to claim novelty of pairwise-kernel attention per se, but to place such con- structions in a symmetry-reduction program: (i) choose relational invariants ...

  63. [63]

    The SVD factorizesXinto orthogonal matricesU X ,V X and a diagonal matrix Σ X , enabling a canonical extrac- tion of the right singular vectorsV X

  64. [64]

    are eliminated by construction

    This symmetry reduction parallels gauge reduction in physical systems (this is an analogy, not a literal gauge symmetry): rather than introducing preferred directions, redundant d.o.f. are eliminated by construction

  65. [65]

    This is the ideal objective one would minimize knowing Dexactly

  66. [66]

    Writing (x, y)∼ Dmeans: ‘sample a random training example from the underlying population distribution’

  67. [67]

    ∆θ ∥ = P a ca δξa θ, for some basis{ξ a}ofgand coefficientsc a ∈R

    Here ∆θ ∥ ∈span{δ ξθ:ξ∈g}, i.e. ∆θ ∥ = P a ca δξa θ, for some basis{ξ a}ofgand coefficientsc a ∈R

  68. [68]

    Berghofer and J

    P. Berghofer and J. Fran¸ cois, Dressing vs. Fixing: On How to Extract and Interpret Gauge-Invariant Con- tent, Found. Phys.54, 72 (2024), arXiv:2404.18582 [physics.hist-ph]

    Pith/arXiv arXiv 2024

  69. [69]

    The SVD factorizesG V Ointo orthogonal matricesU,V and a diagonal matrix Σ of nonnegative singular values, providing the optimal rank-≤d h approximation in Frobe- nius norm (the truncated SVD gives the lowest possi- ble total squared error when compressingG V Oto rank- ≤d h)