REVIEW 1 major objections 1 minor 2 cited by
Reviewed by Pith at T0; open to challenge.
T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →
T0 review · grok-4.3
Exploiting the non-differentiability of ReLU yields a complete classification of symmetries in shallow ReLU networks.
2026-05-10 13:43 UTC pith:ENI3YBMC
load-bearing objection The paper gives a symmetry classification for shallow ReLU networks by using non-differentiability, filling a gap left by analytic-activation methods, though completeness may not hold in degenerate cases. the 1 major comments →
A Complete Symmetry Classification of Shallow ReLU Networks
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
By exploiting the non-differentiability of the ReLU activation, a complete classification of all symmetries in shallow ReLU networks is obtained. This classification enumerates every distinct set of weights and biases that realize the same function, without imposing restrictions on network width or requiring analyticity of the activation.
What carries the argument
The non-differentiability of ReLU, which creates piecewise-linear regions whose boundaries are used to match and classify all equivalent parameter configurations.
Load-bearing premise
The non-differentiability of ReLU suffices by itself to produce an exhaustive list of all symmetries for shallow networks of arbitrary width.
What would settle it
A concrete shallow ReLU network whose set of symmetries cannot be fully recovered by the classification procedure would refute the central claim.
If this is right
- Every symmetry of any shallow ReLU network can be identified explicitly.
- The geometry of the neuromanifold becomes fully describable for the shallow ReLU case.
- Optimization trajectories can be analyzed with complete knowledge of all equivalent parameter points.
- No extra regularity conditions on weights or width are required for the classification to hold.
Where Pith is reading between the lines
- The same piecewise-linear matching technique could be adapted to classify symmetries in deeper ReLU architectures.
- Symmetry-aware regularization or initialization schemes could be derived from the explicit list of equivalences.
- The classification supplies a concrete testbed for studying how symmetries affect generalization and convergence rates.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper claims that by exploiting the non-differentiability of the ReLU activation function, it is possible to provide a complete classification of the symmetries in the parameter space of shallow ReLU networks, i.e., distinct parameters that realize the same realized function, in contrast to prior approaches requiring analytic activations.
Significance. If the classification is exhaustive for arbitrary shallow ReLU networks, the result would be significant for understanding the geometry of the neuromanifold in the practically important ReLU case, with potential implications for parameter identifiability and optimization dynamics.
major comments (1)
- [Abstract] Abstract: The assertion of a 'complete classification' for the shallow case does not address whether genericity assumptions on weights/biases are required. Degenerate configurations (multiple neurons with identical or linearly dependent weights) merge activation regions and can enlarge the stabilizer group with additional discrete symmetries not generated by permutations and sign flips; without an explicit treatment or proof that such cases are covered, the exhaustiveness of the classification is not established.
minor comments (1)
- The abstract is dense and would benefit from a concise statement of the classified symmetry group (e.g., generated by what operations under what conditions) to clarify the main result.
Simulated Author's Rebuttal
We thank the referee for their careful reading of the manuscript and for recognizing the potential significance of a complete symmetry classification for shallow ReLU networks. We address the major comment below and will revise the manuscript accordingly to strengthen the presentation.
read point-by-point responses
-
Referee: [Abstract] Abstract: The assertion of a 'complete classification' for the shallow case does not address whether genericity assumptions on weights/biases are required. Degenerate configurations (multiple neurons with identical or linearly dependent weights) merge activation regions and can enlarge the stabilizer group with additional discrete symmetries not generated by permutations and sign flips; without an explicit treatment or proof that such cases are covered, the exhaustiveness of the classification is not established.
Authors: We thank the referee for this observation. Our classification exploits the locations of non-differentiability of the realized function to recover the hyperplane arrangement and the associated symmetries; this geometric approach does not rely on differentiability or on any genericity assumption on the weights and biases. When neurons have identical or linearly dependent weights, the activation regions merge, but the only additional freedoms are permutations among the identical neurons (already accounted for in the symmetry group) and sign flips; any other discrete transformation would alter the piecewise-linear slopes or the locations of the kinks, which can be detected directly from the function. We nevertheless agree that an explicit treatment would improve clarity. In the revised manuscript we will add a dedicated subsection proving that no further discrete symmetries arise in degenerate configurations, thereby confirming that the classification remains exhaustive without genericity restrictions. revision: yes
Circularity Check
No circularity: classification follows from ReLU non-differentiability without reduction to inputs or self-citations
full rationale
The abstract and description present a direct mathematical exploitation of ReLU kinks to enumerate symmetries for shallow networks. No equations or steps are shown that define a quantity in terms of itself, rename a fitted parameter as a prediction, or rely on load-bearing self-citations for the completeness claim. The derivation chain is self-contained as an exhaustive case analysis on activation regions, with no evidence of the enumerated circular patterns.
Axiom & Free-Parameter Ledger
axioms (1)
- domain assumption Non-differentiability of ReLU suffices for a complete symmetry classification in the shallow case
read the original abstract
Parameter space is not function space for neural network architectures. This fact, investigated as early as the 1990s under terms such as ``reverse engineering," or ``parameter identifiability", has led to the natural question of parameter space symmetries\textemdash the study of distinct parameters in neural architectures which realize the same function. Indeed, the quotient space obtained by identifying parameters giving rise to the same function, called the \textit{neuromanifold}, has been shown in some cases to have rich geometric properties, impacting optimization dynamics. Thus far, techniques towards complete classifications have required the analyticity of the activation function, notably excising the important case of ReLU. Here, in contrast, we exploit the non-differentiability of the ReLU activation to provide a complete classification of the symmetries in the shallow case.
Figures
Forward citations
Cited by 2 Pith papers
-
Most ReLU Networks Admit Identifiable Parameters
For ReLU networks with input and hidden widths at least 2, most parameters are identifiable up to symmetry, so the functional dimension equals the parameter count minus the number of hidden neurons.
-
Most ReLU Networks Admit Identifiable Parameters
For ReLU networks with width at least two in input and hidden layers, an open set of parameters is identifiable, implying functional dimension equals parameter count minus hidden neurons.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.