A Counterexample to Fourier Alignment in Single-Neuron Modular Addition

Gautam Neelakantan Memana

A Counterexample to Fourier Alignment in Single-Neuron Modular Addition

T0 review · 0 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read For every prime p at least 5, a single ReLU neuron can reach a dead state whose Fourier energy is split evenly among all nonzero frequency classes; the almost-sure single-frequency alignment conjecture fails.

desk verdict A clean, checkable counterexample to a well-posed open problem; the main caveat is the dead-region and ReLU'(0)=0 scope, which the paper states explicitly. read the letter →

arxiv 2608.04451 v2 pith:LZSCXM54 submitted 2026-08-05 math.OC cs.LGcs.NE

Gautam Neelakantan Memana This is my paper · ORCID

classification math.OCcs.LGcs.NE MSC 68T0749J52

keywords modularadditionFourieralignmentsingle-neuronnetworkReLUdyingClarketrajectorycross-entropylossgrokking

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper studies one ReLU neuron trained by exact full-table cross-entropy on addition in $\mathbb{Z}/p\mathbb{Z}$, with no weight decay, and asks whether an initially active neuron must converge to a direction supported on one nonzero real Fourier class. The answer is no. The paper constructs an open set of initial parameters whose selected Clarke trajectories (solutions of the nonsmooth gradient flow through the ReLU kink) reach a completely inactive state in finite time and freeze at a limit with Fourier energy $2/(p-1)$ in every nonzero class, which is less than 1 for every prime $p\geq 5$. Since Gaussian initialization gives every open set positive probability, the almost-sure single-frequency conclusion fails. An appendix strengthens this to every Clarke trajectory from an open set, to smooth dead-zone activations, and to fixed-step full-batch gradient descent.

What carries the argument

The central object is the single-active-gate reduction: initial weights are chosen so that exactly one table entry has positive preactivation while all other gates are negative. In this chamber the loss is controlled by the softmax probability of the correct class, and the dynamics are captured by the scalar gate $s$, the output gap $g$, and the conservation law $g^2 - p/(2(p-1))s^2 = \text{const}$. This invariant forces the correct-class probability to stay below $1/p$, so the active gate dies transversely in finite time. The frozen state is the centered point mass $q=e_0-(1/p)\mathbf{1}$, whose Fourier coefficients all have magnitude $1/\sqrt{p}$; this one vector carries the flat spectrum $2/(p-1)$. In the appendix the same reduction is combined with Clarke kink coefficients on cross-gate faces and a nodewise balance invariant to make the construction selection-independent and to extend it to smooth dead-zone activations and discrete gradient descent.

What would settle it

For $p=5$, take the symmetric initial condition (2.1), integrate the reduced ODEs (2.2)--(2.5) until $s=0$, and compute the combined centered Fourier fraction $F_\zeta$ of the frozen state; Theorem 1.5 predicts $F_1=F_2=1/2$. If a direct numerical integration with $\mathrm{ReLU}'(0)=0$ gives any $F_\zeta$ at or near $1$, the frozen-state calculation or the ODE reduction is wrong.

Watch

Extended reading notes

Core claim

Theorem 1.5 states that for every prime $p\geq 5$ there is a nonempty open set $U$ of initially active parameters and a measurable assignment of complete Clarke trajectories such that every selected trajectory starts in $U$, reaches a stationary dead-neuron state in finite time, and has normalized limiting direction satisfying $F_\zeta=2/(p-1)$ for every nonzero real Fourier class. Because $2/(p-1)<1$, the limiting direction is not pure in any single class. The proof starts from a symmetric initialization in which exactly one gate is active; the dynamics reduce to the gate $s=u(0)+v(0)$ and the output gap $g=w(j)-w(0)$, which preserve $g^2 - p/(2(p-1))s^2$. The gate $s$ hits zero transversely, and with the convention $\mathrm{ReLU}'(0)=0$ the parameter vector freezes; all three blocks are scalar multiples of the centered point mass $q=e_0-(1/p)\mathbf{1}$, whose Fourier transform has equal magnitude on every nonzero mode. The appendix proves a persistent sliding trajectory with the same flat limit, death of every Clarke trajectory from an open set, the same result for smooth dead-zone activations, and the same one-step death for fixed-step full-batch gradient descent.

Load-bearing premise

The construction depends on the activation having an exact dead region and on the convention $\mathrm{ReLU}'(0)=0$ at the kink; for everywhere-positive activations such as softplus, or for optimizers outside vanilla full-batch gradient descent, the paper does not claim the counterexample applies.

Editorial extensions

If this is right

MAIS-O60 is resolved negatively as stated: single-frequency purity is not an almost-sure consequence of training one ReLU neuron on modular addition under the Clarke-trajectory convention.
The flat terminal fraction $2/(p-1)$ is a sharp signature of the dead or memorized state; any trajectory reaching it is certified not to be monosemantic.
The failure is robust to small perturbations of the initial condition, so it is not a measure-zero artifact of exact symmetry.
Because the persistent branch has loss tending to $(p^2-1)/p^2 \log p$, the counterexample realizes the memorization side of the memorization-to-generalization dichotomy in the smallest possible model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension is to run full-batch training with $\mathrm{ReLU}'(0)=0$ from near-symmetric initial conditions for $p=5$; the predicted terminal fraction $1/2$ for each nonzero class should be observable in finite-precision numerics.
If an optimizer with momentum or adaptivity is used, the finite-time freeze proof no longer applies, so an empirical question is whether such optimizers can leave the dead region and restore single-frequency alignment.
The same one-active-gate mechanism likely transfers to other finite groups with a distinguished table entry, so a similar flat-spectrum failure may occur in the $S_3$ purity problem mentioned in the paper.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The paper does what it says: it gives a negative solution to MAIS-O60 for a single ReLU neuron on modular addition, and the construction is simpler than I expected. The invariant g^2 - p/(2(p-1)) s^2 is exact, the finite-time hit is transverse, and the frozen state has Fourier fraction 2/(p-1) in every nonzero class, so single-frequency purity fails. I checked the core ODE calculation by hand and it holds up. The appendix goes further: a persistent sliding trajectory with unique Clarke trajectory and flat spectrum, a dead-region set where every Clarke trajectory dies, smooth dead-zone approximations, and one-step death under full-batch GD. Those strengthen the main counterexample and remove most of the wiggle room around non-uniqueness.

What is genuinely new: prior emergence theorems need width at least 4(p-1) or analyze transformer internals; this shows the simplest model already has a robust failure mode. The two mechanisms, dying ReLU and one-example memorization, are familiar heuristics, but here they become exact statements with checkable invariants. The paper is honest about its boundary: the dead half-line of ReLU is essential, and softplus or Adam are not covered (Remark A.5). That is a scope limit, not a hidden gap.

Soft spots are minor. The main theorem uses a measurable assignment of Clarke trajectories and the ReLU'(0)=0 convention; if the open problem intended a canonical trajectory under automatic differentiation, the appendix mostly addresses it for the dead-region set (A.2) and gives uniqueness on the sliding set (A.1). The measurable patching outside the open set is gestured at rather than written out, but that is standard. The appendix is co-authored with an AI and labeled as such; I do not think that affects the math, though journals may want to confirm authorship policies.

Bottom line: this is a serious, carefully scoped counterexample, worth a referee. It will not change the world, since it is one neuron and no full task learning, but it resolves a well-posed open problem in the minimal model and provides a clean template for negative results in Fourier alignment. I would cite it if I worked on nonsmooth gradient flow.

Recommendation: send it to a math.OC or theory-of-NN venue and engage it seriously. The author should clarify the trajectory-selection convention relative to MAIS-O60 as posed; otherwise the math is solid.

Referee Report

0 major / 4 minor

Summary. This paper gives a negative solution to the open problem MAIS-O60, which asks whether a single ReLU neuron trained on modular addition must almost surely converge to a direction whose Fourier energy is supported on a single nonzero frequency class. The authors construct, for every prime p ≥ 5, an open set of initially active initial conditions for which the continuous-time Clarke gradient flow reaches a completely inactive (dead) neuron in finite time. At the frozen endpoint, the centered parameter vectors are all scalar multiples of the same centered point mass, so the normalized limiting direction has Fourier-energy fraction 2/(p−1) in every nonzero real frequency class. Since 2/(p−1) < 1, the limit is not single-frequency pure. The open set has positive Gaussian probability conditional on initial activity, giving a counterexample to the almost-sure claim under the paper's explicit measurable-Clarke-trajectory convention. An appendix (initially drafted by GPT-5.6 Sol and reviewed by the author) strengthens the result: a persistently active trajectory with the same flat Fourier spectrum that is independent of the Clarke selection; an open dead-region set where every Clarke trajectory dies under the ReLU'(0)=0 convention; smooth dead-zone activations; and one-step full-batch gradient descent. The scope is explicitly limited to activations with an exact dead half-line and to vanilla full-batch dynamics, as stated in Remark A.5.

Significance. The result is a clean and checkable counterexample in a simple setting. The main invariant (2.6) and the Fourier computation in Section 2.1 are exact and fully explicit, and the appendix provides substantial robustness, showing that the failure is not an artifact of a single kink selection or of the continuous-time formalism. The paper gives credit for these strengths: all key computations are reproducible, the open-set arguments are standard, and the honest scope limitations in Remark A.5 prevent overclaiming. If accepted, the paper decisively resolves a formalized open question and provides a useful warning against over-generalizing single-frequency Fourier alignment from empirical studies of modular arithmetic. The strengthened appendix results (Theorems A.1–A.4) are valuable in their own right, particularly the persistent sliding trajectory with a flat Fourier spectrum and the one-step death under full-batch gradient descent.

minor comments (4)

[Section 2.2, final paragraph] The assertion that the trajectories on U may be patched with any measurable assignment outside U presupposes the existence of a global Borel-measurable selection of complete Clarke trajectories for every initial condition. Please add a citation to a standard measurable-selection result for differential inclusions (e.g., Filippov's theorem) or observe that Theorem A.2 already provides an open set where every Clarke trajectory fails, so the global patching is not needed for the refutation.
[Abstract and Introduction] The scope limitation is stated precisely in Remark A.5, but the abstract and introduction could state more prominently that the counterexample uses the exact dead half-line of ReLU and the kink convention, and therefore does not apply to softplus or to non-vanilla optimizers; this would avoid any impression that the result refutes single-frequency alignment in broader settings.
[Lemma A.11, proof] The sentence 'The state remains in a compact set' is somewhat terse; it would help to spell out that x is bounded by the assumed integrability of A and that w is then bounded by the balance invariant (A.10), so that the lower bound a0 on A follows from continuity and the strict positivity of the gaps d_j.
[References] Reference [RW98] is cited in the text as 'R W98' with a space; the citation should be formatted consistently with the reference list and the other citations.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the counterexample is self-contained and does not reduce to its own inputs.

full rationale

The paper's central claim is a constructed ReLU dead-neuron counterexample to a posed open problem, not a fitted prediction. The invariant (2.6) is derived from the gradient-flow ODEs (2.2)-(2.5), the finite transverse hit is obtained from the explicit bound on s-dot, and the flat Fourier fraction F_zeta = 2/(p-1) is computed from the frozen or sliding limiting parameter vector, never imposed as a target. The open set U is obtained from continuous dependence and transversality, with positive Gaussian probability; no parameter is fitted to any subset of data. The appendix's strengthened claims are likewise proven from the Clarke calculus and semiconvexity estimates, and the citation [GPT26] is only a provenance note for the appendix draft, not a load-bearing mathematical input. References such as [DHL18], [MT06], and [RW98] provide standard external support. The acknowledged scope limitation in Remark A.5 (exact dead half-line and vanilla full-batch GD) is an explicit boundary of the result rather than a hidden circular step. No self-referential chain or definitional identification between input and output was found.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

All positive constants are existential construction choices satisfying open inequalities; none is fitted to data. The proof relies on standard nonsmooth calculus, finite-group Fourier analysis, and one imported balance invariant, all stated or proved. No new entities are introduced.

free parameters (6)

epsilon (main proof) = s(0) = epsilon > 0
Initial central-gate margin; any sufficiently small positive value works subject to B > epsilon/2 and A^2 > p/(2(p-1)) epsilon^2.
A (main proof) = A^2 > p/(2(p-1)) epsilon^2
Initial gap between wrong-class and correct logits; makes the invariant Gamma positive and forces finite-time death of the central gate.
B (main proof) = B > epsilon/2
Magnitude of inactive u and v coordinates; guarantees exactly one active gate at initialization.
X, gamma_star (U_slide, Appendix A.3) = X > (p-1)/(e gamma_star), gamma_star > 0
Open-set inequalities for the persistent sliding trajectory; existential construction choices, not fitted to data.
s0, M, gamma_in (U_dead, Appendix A.6) = gamma_in^2 > s0^2/2, M > s0/2
Open-set conditions for finite-time dead-neuron behavior; arbitrary feasible choices.
eta (Theorem A.4) = any eta > 0, with s0 < 2 eta g0 / p^2
Step size in fixed-step full-batch GD; theorem holds for all positive eta after choosing initial data depending on eta.

assumptions (5)

standard math Clarke generalized-gradient framework and its calculus for nonsmooth ReLU
Used to define the differential inclusion (1.6) and to handle the kink via Lemma A.6; textbook material from Clarke 1990.
standard math Semiconvexity implies hypomonotone subdifferentials and unique subgradient trajectories
Used in Proposition A.16 to prove uniqueness of the sliding trajectory; cited to Rockafellar-Wets and Marcellin-Thibault.
domain assumption Du-Hu-Lee balance invariant for one hidden unit
Lemma A.8 imports the invariant ||u||^2 + ||v||^2 - ||w||^2 = constant and proves it extends to Clarke selections; load-bearing for the projective asymptotics.
domain assumption Training is full-table cross-entropy without weight decay; ReLU'(0) convention is part of the dynamics
The problem definition (1.1) and the zero-at-kink convention in the dead and GD theorems are explicitly stated and scoped.
domain assumption Initialization is independent standard Gaussian
Converts an open set of initial conditions into positive probability, matching the almost-surely formulation of Problem 1.3.

how reviews work

0 comments

Cite this review

Pith. "Pith review of A Counterexample to Fourier Alignment in Single-Neuron Modular Addition." pith.science (2026). https://pith.science/paper/LZSCXM54

@misc{pith2026260804451,
  author       = {Pith},
  title        = {Pith review of: A Counterexample to Fourier Alignment in Single-Neuron Modular Addition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LZSCXM54}},
  note         = {Machine review of arXiv:2608.04451}
}

abstract

We give a negative solution to MAIS-O60. We first construct an example in which an initially active ReLU neuron becomes completely inactive in finite time and thereafter remains frozen at a limit whose Fourier energy is equally distributed among all nonzero real frequency classes. The counterexample holds on an open set of initial conditions and therefore occurs with positive probability under Gaussian initialization. An appendix prepared by GPT-5.6 Sol strengthens the counterexample by showing that the same failure can occur for every Clarke trajectory from an open set of initial conditions, under the convention $\mathrm{ReLU}'(0)=0$, for smooth dead-zone approximations of ReLU, and for fixed-step full-batch gradient descent. Thus, single-frequency alignment is not a general consequence of training a single neuron on modular addition.

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 14 canonical work pages

[1]

2026 , eprint =

On the Mechanism and Dynamics of Modular Addition: Fourier Features, Lottery Ticket, and Grokking , author =. 2026 , eprint =

work page 2026
[2]

The Eleventh International Conference on Learning Representations , year =

Progress Measures for Grokking via Mechanistic Interpretability , author =. The Eleventh International Conference on Learning Representations , year =. 2301.05217 , archivePrefix =

arXiv
[3]

2026 , howpublished =

Open Problem. 2026 , howpublished =

work page 2026
[4]

Which Irreducible Representations Does Training Select? , year =

work page
[5]

2026 , howpublished =

Neuron Purity and Representation Selection for. 2026 , howpublished =

work page 2026
[6]

2026 , howpublished =

Levine, Lionel , title =. 2026 , howpublished =

work page 2026
[7]

and Hu, Wei and Lee, Jason D

Du, Simon S. and Hu, Wei and Lee, Jason D. , title =. Advances in Neural Information Processing Systems , volume =. 2018 , eprint =

work page 2018
[8]

, title =

Clarke, Frank H. , title =. 1990 , doi =

1990

Show all 19 references

[9]

Tyrrell and Wets, Roger J.-B

Rockafellar, R. Tyrrell and Wets, Roger J.-B. , title =. 1998 , doi =

1998
[10]

Journal of Convex Analysis , volume =

Marcellin, Sylvie and Thibault, Lionel , title =. Journal of Convex Analysis , volume =. 2006 , url =

2006
[11]

Communications in Computational Physics , volume =

Lu, Lu and Shin, Yeonjong and Su, Yanhui and Karniadakis, George Em , title =. Communications in Computational Physics , volume =. 2020 , doi =. 1903.06733 , archivePrefix =

2020 arXiv
[12]

Journal of Machine Learning Research , volume =

Soudry, Daniel and Hoffer, Elad and Nacson, Mor Shpigel and Gunasekar, Suriya and Srebro, Nathan , title =. Journal of Machine Learning Research , volume =. 2018 , eprint =

2018
[13]

The Eighth International Conference on Learning Representations , year =

Lyu, Kaifeng and Li, Jian , title =. The Eighth International Conference on Learning Representations , year =. 1906.05890 , archivePrefix =

1906 arXiv
[14]

Advances in Neural Information Processing Systems , volume =

Ji, Ziwei and Telgarsky, Matus , title =. Advances in Neural Information Processing Systems , volume =. 2020 , eprint =

2020
[15]

Advances in Neural Information Processing Systems , volume =

Zhong, Ziqian and Liu, Ziming and Tegmark, Max and Andreas, Jacob , title =. Advances in Neural Information Processing Systems , volume =. 2023 , eprint =

2023
[16]

and Oncescu, Costin-Andrei and Zhao, Rosie and Kakade, Sham , title =

Morwani, Depen and Edelman, Benjamin L. and Oncescu, Costin-Andrei and Zhao, Rosie and Kakade, Sham , title =. The Twelfth International Conference on Learning Representations , year =. 2311.07568 , archivePrefix =

arXiv
[17]

Numerical Influence of

Bertoin, David and Bolte, J. Numerical Influence of. Advances in Neural Information Processing Systems , volume =. 2021 , eprint =

2021
[18]

The Outcome Law of One Rectifier Neuron , year =
[19]

2026 , month = aug, note =

Strengthened Counterexamples to Single-Frequency Alignment in. 2026 , month = aug, note =

2026

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [1]

2026 , eprint =

On the Mechanism and Dynamics of Modular Addition: Fourier Features, Lottery Ticket, and Grokking , author =. 2026 , eprint =

work page 2026

[2] [2]

The Eleventh International Conference on Learning Representations , year =

Progress Measures for Grokking via Mechanistic Interpretability , author =. The Eleventh International Conference on Learning Representations , year =. 2301.05217 , archivePrefix =

arXiv

[3] [3]

2026 , howpublished =

Open Problem. 2026 , howpublished =

work page 2026

[4] [4]

Which Irreducible Representations Does Training Select? , year =

work page

[5] [5]

2026 , howpublished =

Neuron Purity and Representation Selection for. 2026 , howpublished =

work page 2026

[6] [6]

2026 , howpublished =

Levine, Lionel , title =. 2026 , howpublished =

work page 2026

[7] [7]

and Hu, Wei and Lee, Jason D

Du, Simon S. and Hu, Wei and Lee, Jason D. , title =. Advances in Neural Information Processing Systems , volume =. 2018 , eprint =

work page 2018

[8] [8]

, title =

Clarke, Frank H. , title =. 1990 , doi =

1990