Pith. sign in

REVIEW 3 major objections 5 minor 16 references

A two-layer network with holomorphic activation z^k represents the modular task ma+nb mod p exactly when m+n=k; every other linear target and all nonlinear targets like ab lie outside its fixed function class, so they cannot be memorised, l

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 03:49 UTC pith:MPFETBY2

load-bearing objection Solid algebraic core, but Theorem 3 as stated is false at N=k+1 and the proof defers to a missing appendix; the rest is worth engaging seriously. the 3 major comments →

arxiv 2607.13749 v1 pith:MPFETBY2 submitted 2026-07-15 cs.LG stat.ML

Algebraic Representability as the Limiting Regime of Grokking: An Exactly Solvable Model with Holomorphic Activations

Chon-Fai Kam , Xavier Cadet , Miloud Bessafi , Frederic Cadet This is my paper
classification cs.LG stat.ML MSC 68T0743A75
keywords grokkingrepresentabilitymodular arithmeticholomorphic activationFourier analysisfinite groupsspectral gapneural network expressivity
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Most grokking studies assume the network is large enough to memorise its training set, so capacity only controls timing. This paper constructs the opposite regime: a two-layer complex network with activation z^k whose expressible class is a fixed, (k+1)-dimensional subspace no matter how wide the hidden layer. Within that class, a modular linear task ma+nb is representable exactly when m+n=k; targets such as ab have broad Fourier support and lie outside for every degree. The key consequence is that unrepresentable targets cannot be memorised at all—the training loss is bounded below by a positive constant independent of width—so the familiar memorise-then-grok sequence is replaced by instant success or flat failure. The algebraic predictions match 584 of 585 training runs, and the same rank-based reasoning explains what happens when the input encoding is learned rather than fixed.

Core claim

At its core, the paper proves that the holomorphic monomial architecture acts as a hard Fourier bandpass filter. Because (W1j x1 + W2j x2)^k expands by the binomial theorem, every output is a linear combination of the k+1 characters ω^{la+(k−l)b}, l=0,...,k. With k<p these characters are distinct, so the expressible class F_k is exactly a (k+1)-dimensional subspace of the p²-dimensional function space, and a task is representable if and only if its Fourier support lies in S_k = {(l,k−l): 0≤l≤k}. For linear-phase targets f(a,b)=ma+nb mod p this reduces to the arithmetic condition m+n=k (Theorem 1). Nonlinear targets ab, a²+b, a+b², a²+b² have Fourier support on a line or the whole plane and a

What carries the argument

The central object is the expressible class F_k = span{ω^{la+(k−l)b}: l=0,...,k}, produced by the binomial expansion of each hidden unit under the activation σ(z)=z^k. F_k is a fixed (k+1)-dimensional subspace of characters on (Z_p)², and S_k is the degree-limited set of k+1 frequencies on the diagonal u+v≡k. The argument is carried by Fourier orthogonality: an inner-product test shows a target character lies in F_k exactly when its frequency pair is one of those k+1 points. Theorem 3 adds the spectral gap δ² = ‖ω^f − Proj_{F_k}ω^f‖², the squared distance from the target to the subspace, and proves this gap propagates to a random finite training set. Proposition 1 generalises the filter to a

Load-bearing premise

The no-memorisation theorem (Theorem 3) assumes that a concentration-style argument, deferred to a missing appendix, ensures the expressible subspace restricted to a random training set is well-conditioned; if that restriction were ill-conditioned, a non-representable target could in principle be fit on a finite sample, and the paper's central no-memorisation claim would fail.

What would settle it

Train the fixed roots-of-unity network on the task ab with p=97, any degree k≤6, and any hidden width for at least 50,000 steps over multiple seeds; any run whose training accuracy departs from chance (1/97) or whose training loss falls below the spectral floor δ² = 1 − (k+1)/p would falsify Theorem 3.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • For this architecture, a trained outcome can be predicted before training from the single arithmetic check m+n=k; no analysis of optimisation dynamics is needed.
  • The memorisation phase that defines grokking cannot exist for unrepresentable targets, so the capacity-based timing picture only applies after representability is satisfied.
  • Learned embeddings broaden representability to all rank-one targets but do not restore grokking: ab stays unsolvable at any degree in the tested range.
  • Compressing a standard piecewise-linear network through a bottleneck reproduces failure, memorisation-without-generalisation, then grokking with shrinking gap as capacity rises, placing the holomorphic case at one end of the same spectrum.
  • The training-loss lower bound gives a practical signature: a task whose training accuracy remains at chance regardless of width and runtime is outside the expressible class.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The Fourier-support criterion should transfer to other finite abelian groups and polynomial activations: any such network is a bandpass filter whose pass-band is the set of reachable frequency pairs, and the representability boundary is set by whether the target's support fits inside it. A concrete test is to repeat the grid on a composite modulus and check that the diagonal condition becomes a co
  • If the deferred concentration argument for Theorem 3 is supplied, the no-memorisation bound would be fully rigorous for finite training sets; a direct check is whether training loss on ab ever dips below the spectral floor δ² across many seeds and widths.
  • The learned-embedding delay that grows with k suggests an operational decomposition of grokking: representation-acquisition time versus the memorisation-generalisation gap. Measuring those two timescales separately in ordinary networks would test whether the acquisition phase is the one that the rank condition predicts.
  • The rank criterion suggests a fast pre-training filter for algorithmic tasks: the target matrix's rank bounds representability independent of the input encoding, allowing a cheap check before any training run.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies a two-layer complex-valued network with holomorphic activation σ(z)=z^k trained on modular arithmetic tasks encoded via roots of unity (z=ω^a,ω^b, ω=e^{2πi/p}). Its central theoretical results are: (i) Lemma 2, the output of the network, for any hidden width H, lies in the (k+1)-dimensional character subspace F_k = span{χ_{l,k-l}: l=0..k}; (ii) Lemma 3, every function in F_k is reachable when H≥k+1; (iii) Theorem 1, a linear-phase target ma+nb mod p is representable iff (m,n)≡(l,k-l) mod p, which for 0≤m+n≤k reduces to m+n=k; (iv) Theorem 2, the nonlinear targets ab, a^2+b, a+b^2, a^2+b^2 are not representable for any k<p; and (v) Theorem 3, a claimed width-independent lower bound on training loss for non-representable targets, implying that such targets cannot be memorized even on a finite training set. Experiments on p=97 with 585 runs report 99.8% agreement with the representability prediction, a clean binary INST/FAIL outcome, and a bottleneck ablation that connects the algebraic extreme to standard ReLU grokking behavior. A learned-embedding ablation tests the rank-based criterion of Proposition 1, leaving ab as the sole full-rank obstruction.

Significance. If fully established, the paper would provide a rare exactly solvable model of the capacity–grokking spectrum, with parameter-free representability predictions that are sharply falsifiable. The Fourier characterization (Theorems 1–2) is elementary, correct as far as I can verify, and cleanly confirmed by experiments. The paper also makes a valuable conceptual point: before the two-time-scale competition that produces grokking can occur, the network must be able to fit the training data at all. The bottleneck ablation and the learned-encoding experiment empirically connect this algebraic limit to standard architectures. However, the central no-memorisation claim depends entirely on Theorem 3, whose proof is incomplete and whose statement is false in an edge case admitted by the text. A corrected, complete version of Theorem 3 is necessary before the paper's headline conclusions can be accepted.

major comments (3)
  1. [§3.5, Theorem 3, Eq. (7)] The statement of Theorem 3 is not correct as written. It allows N≥k+1, and for N=k+1 the restriction map F_k→C^S is generically an isomorphism: the (k+1)×(k+1) evaluation matrix of the characters φ_l is invertible for most S, so every target vector, including a non-representable one, is fitted exactly, giving zero training loss. This contradicts the claimed positive lower bound (7). The theorem needs an explicit large-N condition (e.g., N ≥ c·k·log p) to avoid this edge case, and that condition must be stated in the theorem, not only in the experiments.
  2. [§3.5, proof of Theorem 3] The proof defers the crucial uniform concentration step to an appendix that is not present in the arXiv submission ('by a Schwartz–Zippel-type argument over Zp, detailed in the appendix'). This is not a cosmetic omission: Hoeffding–Serfling is applied to a fixed bounded function g, but the empirical minimizer over the (k+1)-dimensional linear subspace F_k can have arbitrarily large coefficients, so the summands |T_i−g(a_i,b_i)|^2 are not uniformly bounded over the class. A valid proof requires a genuine empirical-process or covering argument over a compact subset of F_k, or an explicit bound on the coefficients of the minimizer. Injectivity of the restriction map alone is insufficient.
  3. [Abstract, §1, §3.5, §6] The paper repeatedly states the unconditional conclusion that 'a non-representable target cannot be fit even on the training set.' This thesis is exactly the content of Theorem 3, and as long as Theorem 3 is unproven and false in the N=k+1 regime, the narrative overstates what is established. The experimental regime (N≈3763, p=97, k≤6) does avoid the edge case and is consistent with a qualified large-N version, but the main text should either prove the theorem with the required qualifications or clearly restrict the claim to the tested regime.
minor comments (5)
  1. [§4.4, Table 3] The bottleneck sweep reports median gaps over three seeds with no confidence intervals or seed-level spread. The claimed 'monotonic decrease' from 15000 to 5250 steps is not strongly supported by point estimates with n=3. Please include individual seed values or error bars, and soften the wording if they are not available.
  2. [§2, Eq. (2) and §4.1] The training set size is said to be 40% of 9409, but 0.4×9409 is not an integer. Please state the exact sampling scheme (e.g., |S|=3763 or 3764) and whether S is drawn without replacement.
  3. [§4.5, Figure 4] The legend includes a category 'GROK? (200–500)' with a question mark; define this intermediate category explicitly in the protocol of §4.1 and state how many runs fall into it for each initialisation.
  4. [§4.2, Table 1] For the single off-diagonal cell (k=6, a+5b), the text says a longer budget resolves it; please report the actual training/test accuracy and loss for that re-run so the reader can verify that it reaches INST.
  5. [§5, References] Several references to 2026 preprints (e.g., [13]) may not yet be available or verifiable. Please include full arXiv identifiers and, where possible, version numbers; otherwise the related-work discussion is hard to audit.

Circularity Check

0 steps flagged

No circularity: the representability criterion is a parameter-free Fourier consequence and the experiments check it against task grids fixed before training.

full rationale

The derivation chain is self-contained. Lemma 1 expands the forward pass by the binomial theorem; Lemma 2 identifies the output span with the (k+1)-dimensional character subspace F_k = span{χ_{ℓ,k−ℓ}}; Lemma 3 proves surjectivity via a Vandermonde construction; Theorem 1 is a direct Schur-orthogonality argument; Theorem 2 computes Fourier supports with Gauss sums; Theorem 3 attempts to transfer the spectral gap δ² to finite samples. None of these steps assumes the target conclusion, fits a parameter to the data it then 'predicts', or imports a load-bearing result from a self-citation. The experimental protocol fixes the task grid before the 585 runs, and the single off-diagonal cell is disclosed and re-tested with a longer budget rather than retroactively folded into the theorem. The only flagged deficiency is a proof-completeness gap, not circularity: the proof of Theorem 3 defers the key Schwartz–Zippel concentration step to an appendix absent from the arXiv v1, and the claimed injectivity of the restriction map at N=k+1 is not established in the text. That affects the soundness of the no-memorisation bound as a formal theorem, but it does not make the prediction equivalent to its inputs. The representability criterion is a closed-form, parameter-free Fourier-support condition, so the central claim has independent mathematical content. No self-citation is load-bearing. Verdict: no significant circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

The central proof rests on standard finite-group Fourier analysis and one unproved concentration lemma (Theorem 3 appendix). No fitted parameters are used; no new physical or mathematical entities are introduced.

axioms (4)
  • domain assumption The activation degree k is assumed < p so the characters chi_{l,k-l} for l=0..k are distinct (used in Lemma 2 and Theorems 1-2).
    Stated in Section 2; all experiments use p=97, k<=6, so the assumption is satisfied, but the theoretical claims are scoped to k<p.
  • standard math Schur orthogonality and the Fourier basis of (Z_p)^2 are used without proof (Lemmas 1-2, Theorem 1).
    Standard finite-group harmonic analysis; the paper invokes Schur's theorem.
  • standard math The quadratic Gauss-sum identity |sum_{x in Z_p} omega^{x^2}| = sqrt(p) for odd prime p is used in Theorem 2.
    Standard number-theoretic fact; stated in Section 3.4.
  • ad hoc to paper Theorem 3 assumes a uniform concentration bound over the (k+1)-dimensional family F_k (Hoeffding-Serfling plus a Schwartz-Zippel-type injectivity argument) that is stated but not proved in the submission.
    The proof text says 'detailed in the appendix', but no appendix is present; this is a load-bearing unproved lemma for the 'no memorisation' claim.

pith-pipeline@v1.3.0-alltime-deepseek · 16587 in / 17448 out tokens · 162172 ms · 2026-08-02T03:49:38.170898+00:00 · methodology

0 comments
read the original abstract

Neural networks trained on modular arithmetic exhibit grokking, a delayed transition from memorisation to generalisation known to depend on model capacity: too little and the network memorises slowly or not at all, too much and it generalises almost immediately. What happens at the extreme of this spectrum, when the architecture's expressible function class collapses to a finite-dimensional algebraic variety? We study two-layer networks with a holomorphic monomial activation sigma(z)=z^k, trained on modular tasks encoded via roots of unity. Here the network output, regardless of hidden width, is confined to a (k+1)-dimensional subspace of characters of (Z_p)^2, an O(k/p^2) slice of the full function space. We give a complete algebraic characterisation of this subspace: a task is representable if and only if its discrete Fourier support lies on the diagonal u+v = k (mod p), which for linear-phase targets reduces to the arithmetic criterion m+n=k. This is not merely a constraint on eventual generalisation but on memorisation itself: because the outputs are algebraically confined, a non-representable target cannot be fit even on the training set, and we prove a positive lower bound on the training loss, independent of width. Across 585 runs the algebraic prediction matches the observed outcome with 99.8% accuracy, with no memorisation regime and no grokking; outcomes split cleanly into instant success and outright failure. This binary behaviour is the limiting case of the capacity-grokking relationship: when the expressible class shrinks to a fixed algebraic object, the question of when a network will grok dissolves into whether it can represent the target at all. A bottleneck ablation connects this extreme to standard networks, tracing a continuous path from representational failure, through memorisation without generalisation, to grokking with a shrinking gap as capacity grows.

Figures

Figures reproduced from arXiv: 2607.13749 by Chon-Fai Kam, Frederic Cadet, Miloud Bessafi, Xavier Cadet.

Figure 1
Figure 1. Figure 1: Phase diagram for the holomorphic monomial network across activation degrees [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Outcome distribution for the holomorphic and [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Phase diagrams under three encodings, across degrees [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Train (red) and test (blue dashed) loss under a learned random-phase embedding, at [PITH_FULL_IMAGE:figures/full_fig_p013_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

16 extracted references · 13 linked inside Pith

  1. [1]

    A basin-selection perspective on grokking via singular learning theory.arXiv preprint arXiv:2603.01192, 2026

    Ben Cullen, Sergio Estan-Ruiz, Riya Danait, and Jiayi Li. A basin-selection perspective on grokking via singular learning theory.arXiv preprint arXiv:2603.01192, 2026

    Pith/arXiv arXiv 2026

  2. [2]

    Approximation by superpositions of a sigmoidal function.Mathematics of Control, Signals and Systems, 2(4):303–314, 1989

    George Cybenko. Approximation by superpositions of a sigmoidal function.Mathematics of Control, Signals and Systems, 2(4):303–314, 1989

    1989

  3. [3]

    Grokking modular polynomials.arXiv preprint arXiv:2406.03495, 2024

    Darshil Doshi, Tianyu He, Aritra Das, and Andrey Gromov. Grokking modular polynomials.arXiv preprint arXiv:2406.03495, 2024

    Pith/arXiv arXiv 2024

  4. [4]

    Grokking modular arithmetic.arXiv preprint arXiv:2301.02679, 2023

    Andrey Gromov. Grokking modular arithmetic.arXiv preprint arXiv:2301.02679, 2023

    Pith/arXiv arXiv 2023

  5. [5]

    Approximation capabilities of multilayer feedforward networks.Neural Networks, 4(2):251–257, 1991

    Kurt Hornik. Approximation capabilities of multilayer feedforward networks.Neural Networks, 4(2):251–257, 1991

    1991

  6. [6]

    Neural tangent kernel: Convergence and generalization in neural networks

    Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. InAdvances in Neural Information Processing Systems (NeurIPS), 2018. arXiv:1806.07572

    Pith/arXiv arXiv 2018

  7. [7]

    On the expressive power of deep polynomial neural networks

    Joe Kileel, Matthew Trager, and Joan Bruna. On the expressive power of deep polynomial neural networks. InAdvances in Neural Information Processing Systems (NeurIPS), volume 32, pages 10310–10319, 2019. arXiv:1905.12207

    Pith/arXiv arXiv 2019

  8. [8]

    Gershman, and Cengiz Pehlevan

    Tanishq Kumar, Blake Bordelon, Samuel J. Gershman, and Cengiz Pehlevan. Grokking as the transition from lazy to rich training dynamics. InInternational Conference on Learning Representations (ICLR), 2024. arXiv:2310.06110

    Pith/arXiv arXiv 2024

  9. [9]

    Michaud, Max Tegmark, and Mike Williams

    Ziming Liu, Ouail Kitouni, Niklas Nolte, Eric J. Michaud, Max Tegmark, and Mike Williams. Towards under- standing grokking: An effective theory of representation learning. InAdvances in Neural Information Processing Systems (NeurIPS), volume 35, pages 34651–34663, 2022. arXiv:2205.10343

    Pith/arXiv arXiv 2022

  10. [10]

    Progress measures for grokking via mechanistic interpretability

    Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. InInternational Conference on Learning Representations (ICLR),

  11. [11]

    Grokking: Generalization beyond overfitting on small algorithmic datasets.arXiv preprint arXiv:2201.02177, 2022

    Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets.arXiv preprint arXiv:2201.02177, 2022

    Pith/arXiv arXiv 2022

  12. [12]

    Droplets of good representations: Grokking as a first order phase transition in two layer networks.arXiv preprint arXiv:2310.03789, 2023

    Noa Rubin, Inbar Seroussi, and Zohar Ringel. Droplets of good representations: Grokking as a first order phase transition in two layer networks.arXiv preprint arXiv:2310.03789, 2023

    Pith/arXiv arXiv 2023

  13. [13]

    Model capacity determines grokking through competing memorisation and generalisation speeds.arXiv preprint arXiv:2605.09724, 2026

    Yiding Song and Hanming Ye. Model capacity determines grokking through competing memorisation and generalisation speeds.arXiv preprint arXiv:2605.09724, 2026

    Pith/arXiv arXiv 2026

  14. [14]

    Benefits of depth in neural networks

    Matus Telgarsky. Benefits of depth in neural networks. In29th Annual Conference on Learning Theory (COLT), volume 49 ofProceedings of Machine Learning Research, pages 1517–1539. PMLR, 2016. arXiv:1602.04485

    Pith/arXiv arXiv 2016

  15. [15]

    Chiheb Trabelsi, Olexa Bilaniuk, Ying Zhang, Dmitriy Serdyuk, Sandeep Subramanian, João Felipe Santos, Soroush Mehri, Negar Rostamzadeh, Yoshua Bengio, and Christopher J. Pal. Deep complex networks. In International Conference on Learning Representations (ICLR), 2018. arXiv:1705.09792

    Pith/arXiv arXiv 2018

  16. [16]

    Grokking phase transitions in learning local rules with gradient descent

    Bojan Žunkoviˇc and Enej Ilievski. Grokking phase transitions in learning local rules with gradient descent. Journal of Machine Learning Research, 25(199):1–52, 2024. arXiv:2210.15435. 15

    Pith/arXiv arXiv 2024