Pith. sign in

REVIEW 4 major objections 5 minor 32 references

Self-Attention as a Parametric Endofunctor: A Categorical Framework for Transformer Architectures

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Pith's one-line read Stacking linear self-attention layers is exactly the free monad construction on a single-layer endofunctor.

desk verdict The central endofunctor/free monad theorem fails on a type error, but the parametric morphism construction for Q/K/V is sound. read the letter →

arxiv 2501.02931 v2 pith:IGVLV34G submitted 2025-01-06 cs.LG

classification cs.LG MSC 18C1518D0568T07
keywords self-attentiontransformerscategorytheoryendofunctorfreemonadparametricmorphismpositionalencodingsequivariance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper gives a category-theoretic account of the linear core of transformer self-attention. It argues that the query, key, and value maps form a single parametric morphism in the 2-category Para(Vect), and that this morphism induces an endofunctor on vector spaces whose repeated iteration is what happens when attention layers are stacked. The central proof asserts that the colimit of the chain $\mathrm{id} \to F \to F^2 \to \cdots$ is the free monad on $F$, meaning a stack of linear self-attention layers is the most general monad extending one layer. The same framework casts additive positional encodings as affine monoid actions, sinusoidal encodings as injective position labelings, token permutations as equivariances, and mechanistic-interpretability circuits as compositions of parametric morphisms. If correct, the result gives layered linear attention a universal algebraic description and connects interpretability heuristics to categorical structure.

What carries the argument

The central object is the 2-category Para(Vect), whose 1-morphisms are pairs $(P, f)$ with $f : P \otimes X \to Y$ linear, capturing parameterised linear maps without copying. From the parametric morphism $(\mathrm{AttP}, \mathrm{att})$ one extracts an endofunctor $F : \mathbf{Vect} \to \mathbf{Vect}$, and the load-bearing construction is the infinite ladder $\mathrm{id} \to F \to F^2 \to \cdots$ whose colimit in Vect is claimed to be the free monad $\mathrm{Free}(F)$. The paper also uses a monoid-action encoding for positional information and the group-action viewpoint of equivariance, all expressed as compositions of parametric morphisms.

What would settle it

Take $d = d_k = d_v = n = 1$ and $X = \mathbb{R}$. Then $F(X) = \mathbb{R} \otimes \mathbb{R} \otimes \mathbb{R} \simeq \mathbb{R}$, but the paper's formula defines $F(f) : \mathrm{AttP} \otimes X \to Y(X')$ by $\mathrm{att} \circ (\mathrm{id} \otimes f)$; for $f = \mathrm{id}$, this sends $(\theta, x)$ to $(\theta_1 x, \theta_2 x, \theta_3 x)$, which is not the identity map on $F(X)$. Exhibiting this domain mismatch and the absence of a natural map $\mathrm{id} \Rightarrow F$ on objects shows that the claimed endofunctor on Vect is not functorial, so the free-monad chain and its colimit cannot be formed.

Watch

Extended reading notes

Core claim

On the paper's own terms, the linear portion of self-attention can be packaged as one parametric 1-morphism $(\mathrm{AttP}, \mathrm{att})$ from the input vector space to a tensor product of query, key, and value spaces, with parameter space the direct sum of the three weight spaces. Restricting to Vect, this parametric morphism induces an endofunctor $F$, and stacking layers is the colimit of iterating $F$; Theorem 3.2 states that this colimit forms the free monad on $F$, so any monad extending $F$ factors uniquely through the stack. The paper also derives that strictly additive positional encodings are affine monoid actions, that sinusoidal encodings are not additive but can be initial objects among injective position-preserving maps when their vectors span, that the linear projections are permutation-equivariant, and that Elhage-style QK/OV circuits are compositions of parametric morphisms with weight sharing as 2-morphisms. The scope is deliberately the linear skeleton, with softmax, layer norm, and activations deferred to future categorical settings.

Load-bearing premise

The load-bearing premise is that the parametric 1-morphism $(\mathrm{AttP}, \mathrm{att})$ induces a genuine endofunctor $F$ on Vect together with a coherent chain of natural transformations $\mathrm{id} \Rightarrow F \Rightarrow F^2 \Rightarrow \cdots$, whose colimit exists and forms the free monad; the natural transformations are assumed without explicit construction, and the functor's action on morphisms is defined on $\mathrm{AttP} \otimes X$ rather than on $F(X)$.

Editorial extensions

If this is right

  • If Theorem 3.2 holds, every multi-layer linear self-attention stack has a universal property: any monad extending the single-layer endofunctor factors through it, making the stack the most general iteration of one attention layer.
  • Iterating the endofunctor $F$ precisely models layer stacking, so analysis of deep linear attention can reduce to studying one endofunctor and its iterates.
  • Additive positional encodings form genuine affine monoid actions, while sinusoidal encodings are faithful position labelings, suggesting that transformers need injectivity rather than additivity for position.
  • The linear parts of self-attention are permutation-equivariant, so symmetry breaking in actual transformers enters through nonlinear operations like softmax and layer norm.
  • Mechanistic-interpretability circuits, including composed virtual heads, correspond to compositions of parametric morphisms, giving those interpretive heuristics a category-theoretic reading.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the free-monad theorem is the most fragile part of the paper; even if the endofunctor construction fails, the weaker statement that Q/K/V form a parametric morphism may survive, and the circuits mapping is comparatively robust.
  • Editorial inference: a testable extension is to check whether softmax attention can be incorporated as a lax functor into a category of probability measures, preserving part of the monadic structure suggested here.
  • Editorial inference: the fixed codomain $Y(X)$ used in the endofunctor definition suggests the construction may be constant on objects; modifying it to make $F$ genuinely variable on objects could rescue functoriality.
  • Editorial inference: the parametric-morphism view suggests architectural design principles, such as using 2-morphisms for weight tying as a way to impose consistency constraints across layers.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript proposes a category-theoretic account of the linear parts of transformer self-attention. Its main claims are: (i) the query/key/value maps form a parametric 1-morphism (AttP, att) in the 2-category Para(Vect); (ii) this parametric morphism induces an endofunctor F on Vect whose iterates model stacked attention layers; (iii) stacking layers is exactly the free-monad construction on F; (iv) additive positional encodings are affine monoid actions while sinusoidal encodings have a universal property among injective position embeddings; (v) the linear parts of self-attention are permutation-equivariant; and (vi) mechanistic-interpretability circuits can be read as compositions of parametric morphisms. Proofs are in Appendices A and B. The parametric-morphism construction and the equivariance statement are essentially correct, but the endofunctor and free-monad claims rest on a type error in the definition of F.

Significance. If the free-monad result were valid, it would give a striking universal algebraic description of deep linear attention and would unify the QK/OV factorization with stackable layer structure. The parametric-morphism viewpoint is a reasonable dictionary: a linear map f: P⊗X → Y is naturally a 1-morphism in Para(Vect), and the Q/K/V triple does assemble into such a map. The equivariance theorem is correct, and the circuits discussion is a plausible high-level translation. However, the central advertised contribution—that iterative attention builds Free(F)—does not follow, because the alleged endofunctor F is not a well-defined functor on Vect. The paper therefore offers a suggestive reformulation of linear attention but not a proven universal construction. The significance of the paper as it stands is accordingly much lower than its abstract claims.

major comments (4)
  1. [Appendix A, proof of Theorem A.1] The definition of the induced endofunctor F is ill-typed. The object map is set to F(X) = Y(X) ≅ (R^{dk})^n ⊗ (R^{dk})^n ⊗ (R^{dv})^n, which is constant in X, while the arrow map is defined as F(f) = att ∘ (id_{AttP} ⊗ f), whose domain is AttP ⊗ X and codomain Y(X'). Thus F(f) is not a Vect morphism from F(X) to F(X'), because F(X) and AttP ⊗ X are different spaces. The displayed functoriality check F(g∘f) = F(g) ∘ F(f) is also ill-typed: the codomain of F(f) is Y(X'), whereas the domain of F(g) is AttP ⊗ X', and these spaces are not equal in general. Replacing the object map by F(X) = AttP ⊗ X would make the arrows type-check but would ignore att entirely and would not model self-attention; moreover, F^2(X) would be AttP ⊗ AttP ⊗ X, not a second attention layer. Consequently Theorem 3.1's final claim and the premise of Theorem 3.2 are unsupported.
  2. [Theorem 3.2 and Appendix B] The free-monad theorem assumes the existence of a coherent chain of natural transformations id ⇒ F ⇒ F^2 ⇒ ⋯, but no such natural transformations are constructed. The proof says each α_i: F^i ⇒ F^(i+1) arises from a 'usual composition embedding id ⇒ F plus coherence maps', yet η: id ⇒ F is never defined, and even for a genuine endofunctor F a natural map id ⇒ F is not automatically available. Additionally, the colimit of an ω-chain of iterates of an endofunctor is not, in general, the free monad on F; free monads require existence and a universal property, and the proof's assertion that 'the colimit construction provides' a monad structure and that µ can be defined by flattening F^n(T(X)) ≅ F^(n+m)(X) is not justified. With the constant object map of Appendix A, F^n(X) and F^(n+m)(X) are the same space, so the chain does not represent iterated self-attention at all.
  3. [Section 3.1 and Appendix A, composition statement] The claim that stacking multiple attention layers corresponds to composition in Para(Vect) is not established because the codomain of the first layer does not match the domain of a standard second attention layer. The constructed morphism (AttP, att) has domain X = (R^d)^n and codomain Y = (R^{dk})^n ⊗ (R^{dk})^n ⊗ (R^{dv})^n, the space of Q/K/V triples. A second self-attention layer, as implemented in transformers, takes token embeddings in (R^d)^n as input, not a triple of projected query/key/value spaces. The proof only checks composition with an arbitrary morphism AttP' : Y → Z; it does not show that this composition represents stacking self-attention layers. This is a separate gap from the endofunctor type error and further undermines the iterative-stacking claim.
  4. [Section 4.3 and Appendix C.3] The claim that sinusoidal encodings are universal among injective position-preserving maps is overbroad. The factorisation property requires the sinusoidal vectors {p_sin(m)} to generate (or span) the ambient space X_sin; the paper acknowledges this in Remark C.1 but still states the universality claim as a main result. For standard sinusoidal encodings with n positions in R^d, the n vectors generally do not span R^d when n is small relative to d, so the initial-object property fails in general. The theorem should be restricted to the case where the encoding is a basis or generating set, and even then the universal property is a statement about linear extension, not about the specific sinusoidal formula.
minor comments (5)
  1. [Abstract and Section 1] The abstract refers to 'an endofunctor whose iterated composition precisely models multi-layer attention', but the construction in Appendix A does not define such a functor; this phrasing should be corrected or qualified throughout, including in the introduction and Figure 1.
  2. [Definition 2.3] The definition of 2-morphisms in Para(Vect) would benefit from stating that the direction of ρ is the standard convention or explicitly explaining why ρ: Q → P is chosen; the current diagram is understandable but the convention is not motivated.
  3. [Theorem 5.1] Theorem 5.1 is correct but essentially restates that componentwise maps are permutation-equivariant; its proof is a direct verification. The presentation would be clearer if the theorem were phrased as a known elementary property rather than as a new categorical result.
  4. [Appendix D] Appendix D introduces GDL monads and M-algebra homomorphisms but does not connect them to the main Para(Vect) framework in a precise way; the claim that transformer layers can be viewed as T-algebra homomorphisms is asserted rather than proved.
  5. [Throughout] There are numerous typographical and formatting issues, including broken words such as 'learn ing' and 'specifically', inconsistent notation between ⊗ and ⊕ for the output space Y, and a reference list that is not consistently formatted. These should be cleaned up in any revision.

Circularity Check

2 steps flagged · score 7.0 of 10

The headline theorem that stacking self-attention equals Free(F) is definitional: 'free monad' is introduced as the colimit of iterates of F and 'stacking' as iterating F, so the claimed correspondence is established by naming, and the Para(Vect) composition claim likewise renames the generic composition rule.

  1. self definitional [Section 3.2, Theorem 3.2; Appendix B.1-B.2]
    "In category theory, collecting such iterates is precisely the free monad construction on F. Concretely, the free monad Free(F) is the colimit of all finite powers F n, with universal maps gluing them. Thus: Theorem 3.2 ... Then the colimit (in Vect) of the sequence id −→ F −→ F 2 −→ . . . forms the free monad on F. In other words, stacking self-attention layers is exactly building Free(F)."

    The paper itself fixes both sides of the claimed equivalence. Free(F) is defined in the same passage as the colimit of all finite powers of F, while stacking was previously defined as repeatedly composing F. The theorem therefore asserts that stacking equals Free(F) because both terms were assigned to the same colimit construction by definition. The Appendix B proof adds no attention-specific content: it declares the α_i to represent adding one more layer, defines η as the colimit cocone φ0, and defines μ by flattening F^a∘F^b=F^(a+b). These are the canonical colimit maps, so the universal property is imported from the colimit construction rather than derived from query/key/value structure.

  2. renaming known result [Section 3.1, composition in Para(Vect) and Theorem 3.1]
    "For self-attention, stacking multiple heads or layers amounts to this Para(Vect)-style composition of parametric morphisms, preserving the bilinear in parameters × data viewpoint. Furthermore, this 1-morphism (AttP, att) is stable under composition in Para(Vect). Concretely, stacking multiple attention layers corresponds to repeated composition of (AttP, att) with similar parametric morphisms, matching how self-attention is practically stacked in transformer architectures."

    Composition of parametric morphisms is closed by the definition of Para(Vect): any (P,f):X→Y and (Q,g):Y→Z compose to (Q⊗P,h). Thus the 'stability under composition' asserted in Theorem 3.1 holds for every parametric morphism, not because of anything specific to Q/K/V attention. The claim that this is how self-attention layers stack is a renaming of the generic composition rule as 'stacking layers'; the match with real transformer stacking is asserted interpretively, not derived from the attention equations.

full rationale

Score is 7 rather than 0-2 because the paper's central load-bearing claim, Theorem 3.2, is not an independent derivation. Section 3.2 defines the free monad as the colimit of all finite powers of F and simultaneously describes stacking as repeatedly composing F; the theorem then relabels that colimit as 'stacking self-attention layers', so the headline result reduces to the names assigned to the objects. Appendix B fills in the universal property by assuming the α_i maps that encode 'adding a layer' and by taking η and μ from the colimit itself, which is the same construction restated. Section 3.1's composition claim is likewise a generic property of Para(Vect) renamed as attention stacking. The equivariance and positional-encoding sections contain independent content, but they are not the central derivation and do not rescue the free-monad claim from being definitional. A separate well-definedness problem with F(f) in Appendix A is a correctness issue rather than a circularity issue; it only strengthens the impression that the free-monad reduction is stipulated rather than exhibited. There is no fitted-parameter prediction and no self-citation chain, so the score is below 8-10, but the central claimed result is forced by definition, justifying 7.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

The framework rests on the choice to ignore nonlinearities, on fixed sequence length, on the assertion that parametric morphisms induce endofunctors, and on conditional spanning assumptions for sinusoidal encodings. No free parameters are fitted to data; the mathematical objects are defined so that the stated conclusions follow by construction, which is why the circularity burden is high.

assumptions (6)
  • domain assumption Self-attention can be analyzed by restricting to linear components Q, K, V and ignoring softmax and layer norm.
    Stated in the Introduction and Section 7; the actual Attention(Q,K,V) includes softmax and nonlinearities, so the categorical claims apply only to a linear skeleton that is not the full mechanism.
  • ad hoc to paper A parametric 1-morphism in Para(Vect) with tensor product induces an endofunctor on the base category Vect.
    Used in Theorem 3.1 and Appendix A; this is not a standard consequence of the Para construction and is in fact inconsistent as written.
  • ad hoc to paper There is a natural transformation id => F and a coherent chain id -> F -> F^2 -> ... for the self-attention endofunctor.
    Used in Appendix B to build the colimit/free monad; such a natural transformation does not exist for a general endofunctor and none is constructed.
  • domain assumption Sinusoidal positional embeddings generate (span) the embedding space, or the relevant finite subset of positions is linearly independent.
    Needed for the claimed universal/initial property of sinusoidal encodings in Section 4.3 and Appendix C; the paper acknowledges this is conditional in Remark C.1.
  • domain assumption Fixed sequence length n.
    The constructions use X = (R^d)^n and Y = (R^dk)^n tensor (R^dk)^n tensor (R^dv)^n; variable-length sequences are deferred in the Discussion.
  • domain assumption Circuits from mechanistic interpretability can be represented as composition of linear parametric morphisms.
    This is an interpretive mapping in Section 6, not derived from the circuit literature; actual circuits include softmax and nonlinear path expansions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Self-Attention as a Parametric Endofunctor: A Categorical Framework for Transformer Architectures." pith.science (2026). https://pith.science/paper/IGVLV34G

@misc{pith2026250102931,
  author       = {Pith},
  title        = {Pith review of: Self-Attention as a Parametric Endofunctor: A Categorical Framework for Transformer Architectures},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IGVLV34G}},
  note         = {Machine review of arXiv:2501.02931}
}
abstract

Self-attention mechanisms have revolutionised deep learning architectures, yet their core mathematical structures remain incompletely understood. In this work, we develop a category-theoretic framework focusing on the linear components of self-attention. Specifically, we show that the query, key, and value maps naturally define a parametric 1-morphism in the 2-category $\mathbf{Para(Vect)}$. On the underlying 1-category $\mathbf{Vect}$, these maps induce an endofunctor whose iterated composition precisely models multi-layer attention. We further prove that stacking multiple self-attention layers corresponds to constructing the free monad on this endofunctor. For positional encodings, we demonstrate that strictly additive embeddings correspond to monoid actions in an affine sense, while standard sinusoidal encodings, though not additive, retain a universal property among injective (faithful) position-preserving maps. We also establish that the linear portions of self-attention exhibit natural equivariance to permutations of input tokens, and show how the "circuits" identified in mechanistic interpretability can be interpreted as compositions of parametric 1-morphisms. This categorical perspective unifies geometric, algebraic, and interpretability-based approaches to transformer analysis, making explicit the underlying structures of attention. We restrict to linear maps throughout, deferring the treatment of nonlinearities such as softmax and layer normalisation, which require more advanced categorical constructions. Our results build on and extend recent work on category-theoretic foundations for deep learning, offering deeper insights into the algebraic structure of attention mechanisms.

Figures

Figures reproduced from arXiv: 2501.02931 by the authors.

Figure 1
Figure 1. Conceptual overview of how the monoid M (left) provides positional encodings p : BM → Vect that are added to the embedding space Xn. The main self-attention block is formalised as a parametric endofunctor with learnable queries (Q), keys (K), and values (V ). Repeated application of this block yields the stacked self-attention layers, interpreted categorically as a free monad on F. Para(Vect). Restricting to the un… view at source ↗
Figure 2
Figure 2. A string diagram illustrating two parametric morp [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 19 canonical work pages

  1. [1]

    Tensorflow: Large-scale machine learning on heterogeneous distributed systems

    Mart \' n Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467, 2016

  2. [2]

    Metric Space Magnitude and Generalisation in Neural Networks

    Rayna Andreeva, Katharina Limbeck, Bastian Rieck, and Rik Sarkar. Metric space magnitude and generalisation in neural networks, 2023. URL https://arxiv.org/abs/2305.05611

  3. [3]

    Convenient categories of smooth spaces

    John Baez and Alexander Hoffnung. Convenient categories of smooth spaces. Transactions of the American Mathematical Society, 363 0 (11): 0 5789--5825, 2011

  4. [4]

    Automatic differentiation in machine learning: a survey

    Atilim Gunes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. Automatic differentiation in machine learning: a survey. Journal of machine learning research, 18 0 (153): 0 1--43, 2018

  5. [5]

    Cartesian differential categories

    Richard F Blute, J Robin B Cockett, and Robert AG Seely. Cartesian differential categories. Theory and Applications of Categories, 22 0 (23): 0 622--672, 2009

  6. [6]

    Geometric algebra transformers

    Johann Brehmer, Pim de Haan, S \"o nke Behrends, and Taco Cohen. Geometric algebra transformers. ArXiv, abs/2305.18415, 2023. URL https://api.semanticscholar.org/CorpusID:269738259

  7. [7]

    Geometric deep learning: Grids, groups, graphs, geodesics, and gauges

    Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veli c kovi \'c . Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13478, 2021

  8. [8]

    Towards foundations of categorical cybernetics

    Matteo Capucci, Bruno Gavranovi \'c , Jules Hedges, and Eigil Fjeldgren Rischel. Towards foundations of categorical cybernetics. arXiv preprint arXiv:2105.06332, 2021

Show all 32 references
  1. [9]

    A simple and effective positional encoding for transformers

    Pu-Chin Chen, Henry Tsai, Srinadh Bhojanapalli, Hyung Won Chung, Yin-Wen Chang, and Chun-Sung Ferng. A simple and effective positional encoding for transformers. arXiv preprint arXiv:2104.08698, 2021

  2. [10]

    Differential structure, tangent structure, and sdg

    J Robin B Cockett and Geoff SH Cruttwell. Differential structure, tangent structure, and sdg. Applied Categorical Structures, 22 0 (2): 0 331--417, 2014

  3. [11]

    Spherical cnns

    Taco S Cohen, Mario Geiger, Jonas K \"o hler, and Max Welling. Spherical cnns. arXiv preprint arXiv:1801.10130, 2018

  4. [12]

    Categorical foundations of gradient-based learning

    Geoffrey SH Cruttwell, Bruno Gavranovi \'c , Neil Ghani, Paul Wilson, and Fabio Zanasi. Categorical foundations of gradient-based learning. In European Symposium on Programming, pp.\ 1--28. Springer International Publishing Cham, 2022

  5. [13]

    Euclidean, projective, conformal: Choosing a geometric algebra for equivariant transformers

    Pim de Haan, Taco Cohen, and Johann Brehmer. Euclidean, projective, conformal: Choosing a geometric algebra for equivariant transformers. ArXiv, abs/2311.04744, 2023. URL https://api.semanticscholar.org/CorpusID:265050809

  6. [14]

    Differentials and distances in probabilistic coherence spaces

    Thomas Ehrhard. Differentials and distances in probabilistic coherence spaces. Logical Methods in Computer Science, 18, 2022

  7. [15]

    A mathematical framework for transformer circuits

    Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1 0 (1): 0 12, 2021

  8. [16]

    Symmetry-driven graph neural networks, 2021

    Francesco Farina and Emma Slade. Symmetry-driven graph neural networks, 2021. URL https://arxiv.org/abs/2105.14058

  9. [17]

    Backprop as functor: A compositional perspective on supervised learning

    Brendan Fong, David Spivak, and R \'e my Tuy \'e ras. Backprop as functor: A compositional perspective on supervised learning. In 2019 34th Annual ACM/IEEE Symposium on Logic in Computer Science (LICS), pp.\ 1--13. IEEE, 2019

  10. [18]

    Position: Categorical deep learning is an algebraic theory of all architectures

    Bruno Gavranovi \'c , Paul Lessard, Andrew Joseph Dudzik, Tamara von Glehn, Jo \ a o Guilherme Madeira Ara \'u jo, and Petar Veli c kovi \'c . Position: Categorical deep learning is an algebraic theory of all architectures. In Forty-first International Conference on Machine Learning

  11. [19]

    Marshall, Andrew S

    Wilson Gregory, Josu \'e Tonelli-Cueto, Nicholas F. Marshall, Andrew S. Lee, and Soledad Villar. Learning equivariant tensor functions with applications to sparse vector recovery. ArXiv, abs/2406.01552, 2024. URL https://api.semanticscholar.org/CorpusID:270217098

  12. [20]

    2-dimensional categories

    Niles Johnson and Donald Yau. 2-dimensional categories. Oxford University Press, USA, 2021

  13. [21]

    Categories enriched on two sides

    Max Kelly, Anna Labella, Vincent Schmitt, and Ross Street. Categories enriched on two sides. Journal of Pure and Applied Algebra, 168 0 (1): 0 53--98, 2002

  14. [22]

    A 2-categories companion

    Stephen Lack. A 2-categories companion. In Towards higher categories, pp.\ 105--191. Springer, 2009

  15. [23]

    Generalized enrichment for categories and multicategories

    Tom Leinster. Generalized enrichment for categories and multicategories. arXiv preprint math/9901139, 1999

  16. [24]

    Categories for the working mathematician, volume 5

    Saunders Mac Lane. Categories for the working mathematician, volume 5. Springer Science & Business Media, 2013

  17. [25]

    Pytorch: An imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019

  18. [26]

    A survey of graphical languages for monoidal categories

    Peter Selinger. A survey of graphical languages for monoidal categories. New structures for physics, pp.\ 289--355, 2011

  19. [27]

    The formal theory of monads

    Ross Street. The formal theory of monads. Journal of Pure and Applied Algebra, 2 0 (2): 0 149--168, 1972

  20. [28]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017

  21. [29]

    The topos of transformer networks

    Mattia Jacopo Villani and Peter McBurney. The topos of transformer networks. ArXiv, abs/2403.18415, 2024. URL https://api.semanticscholar.org/CorpusID:268723539

  22. [30]

    Equivariant and coordinate independent convolutional networks

    Maurice Weiler, Patrick Forr \'e , Erik Verlinde, and Max Welling. Equivariant and coordinate independent convolutional networks. A Gauge Field Theory of Neural Networks, 2023

  23. [31]

    Deep sets

    Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. Advances in neural information processing systems, 30, 2017

  24. [32]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.