On Space Folds of ReLU Neural Networks

Bernhard A.Moser; Bernhard Heinzl; Hamid Eghbalzadeh; Michal Lewandowski; Raphael Pisoni

REVIEW 3 major objections 4 minor 63 references

On Space Folds of ReLU Neural Networks

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proves that convexity of a union of activation regions is equivalent to convexity of its activation patterns in the Hamming cube only for first-hidden-layer hyperplanes, and introduces a range-ratio measure $\chi$ that…

desk verdict A clean new measure of ReLU space folding with a genuinely flawed proof of its main lemma and thin empirics; worth a real referee, not desk rejection. read the letter →

arxiv 2502.09954 v1 pith:PUBSE6QS submitted 2025-02-14 cs.LG cs.NE

Michal Lewandowski , Hamid Eghbalzadeh , Bernhard Heinzl , Raphael Pisoni , Bernhard A.Moser This is my paper · ORCID

classification cs.LGcs.NE MSC 68T0752A20

keywords spacefoldingReLUnetworksHammingactivationconvexityrangemeasureslinearregionspatternsCantorNet

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's goal is to turn the informal idea that deep ReLU networks fold the input space into a measurable quantity. Its central theoretical result, Lemma 1, states that for hyperplanes of the first hidden layer, a union of activation regions is convex in Euclidean space exactly when the corresponding set of activation patterns is convex in the Hamming cube. The authors then define a space folding measure $\chi(\Gamma)$ as one minus the ratio of the maximum Hamming distance reached along a straight input path to the total Hamming distance traveled; $\chi$ lies in $[0,1]$ and equals zero for a perfectly flat (unfolded) path. Experiments on the synthetic CantorNet benchmark and on MNIST show that $\chi$ increases with network depth in well-trained networks, and that wider networks show a larger fraction of paths exhibiting folding. The paper argues that $\chi$ can serve as a network feature describing how twisted the learned input representation is.

What carries the argument

The load-bearing object is the space folding measure $\chi(\Gamma)$, a ratio of two range metrics on the Hamming cube: $r_1(\Gamma) = \max_i d_H(\pi_i, \pi_1)$ is the maximum distance from the starting activation pattern, and $r_2(\Gamma) = \sum_i d_H(\pi_i, \pi_{i+1})$ is the total Hamming length of the path. The ratio $\phi = r_1/r_2$ equals $1$ for a convex (flat) path, so $\chi = 1 - \phi$ measures deviation from convexity. Lemma 1 supplies the convexity equivalence that makes $\chi$ meaningful: it guarantees that for first-layer hyperplanes, straight input segments map to shortest Hamming paths, so any $\chi > 0$ signals genuine folding; the paper notes this guarantee lapses for deeper layers, where the measure still quantifies path inefficiency but not necessarily convexity loss.

What would settle it

Train a single-hidden-layer ReLU network to high accuracy on any classification task and measure $\chi$ on straight paths between test points; Lemma 1 predicts $\chi = 0$ for all such paths, so any nonzero value would refute the convexity equivalence.

Watch

Extended reading notes

Core claim

The central claim is that ReLU networks fold the input space in a way that can be read off from activation patterns along straight lines. Lemma 1 establishes an equivalence: for a first hidden layer whose hyperplanes intersect the whole input space, a union of linear regions is convex in $\mathbb{R}^n$ if and only if the corresponding set of activation patterns is convex in the Hamming space $H^m$. Because this equivalence uses the fact that crossing a hyperplane flips exactly one activation bit, it fails for deeper layers, where neighboring regions can have Hamming distance larger than one. To quantify the resulting folding, the paper defines $\chi(\Gamma) = 1 - \max_i d_H(\pi_i, \pi_1) \big/ \sum_i d_H(\pi_i, \pi_{i+1})$ for a straight path $\Gamma$, interprets the numerator as a range measure and the denominator as total path length, and proves $0 \le \chi \le 1$. Empirically, the median of non-zero $\chi$ maxima across MNIST digit pairs correlates with depth (Pearson $r = 0.987$ for the best-performing seed), and the fraction of folded paths rises sharply in wider networks.

Load-bearing premise

The proof that straight paths correspond to shortest paths in the Hamming cube assumes each hyperplane crossing flips exactly one activation bit, which holds only for first-hidden-layer hyperplanes that cut across the entire input space; for deeper layers the paper itself shows neighboring regions can differ by more than one bit.

Editorial extensions

If this is right

Single-hidden-layer ReLU networks cannot exhibit space folding along straight input paths; any nonzero $\chi$ indicates depth is at play.
$\chi$ provides a bounded, algorithmically computable network feature with complexity $O(n(N+C))$ per sample pair that can be compared across architectures.
In well-trained MNIST models, aggregated $\chi$ grows with depth, while poorly trained models show lower folding, suggesting a link between folding and learned structure.
Wider networks do not necessarily fold more per path, but a much larger share of digit-pair paths exhibit folding, e.g., the ratio of folded paths grows from about $0.35$ for a $2\times 30$ network to about $0.97$ for a $2\times 300$ network.
The measure extends in principle to residual, normalization, and ReLU-based attention layers, since it operates on binarized activations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If Lemma 1's assumption fails for deep layers, then $\chi$ on full-network activations may count non-adjacent Hamming jumps as folding even when the underlying geometry is locally flat; a natural test is to compute $\chi$ using only first-layer activations and compare with the full-network value on the same paths.
The reported correlation between $\chi$ and generalization is suggestive but not causal; training the same architectures with shuffled labels would test whether high folding is a learned property or an architecture artifact.
Because $\chi$ is a per-path quantity, aggregating it as a median of maxima loses information about where along the path folding occurs; a finer analysis could localize folds to specific layer transitions and reveal which layers contribute most.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper introduces a simple, clean measure of space folding for ReLU networks: sample a straight line in input space, map it to activation patterns, and compute chi = 1 - (max distance from start)/(total path length). This is genuinely new as a quantitative handle on a phenomenon that has only been discussed qualitatively. The definition is straightforward, Lemma 2's bound (0 <= chi <= 1) is correct, and the CantorNet demo is a nice illustration. The related-work survey looks fair, including the self-citations for the range-measure inspiration and CantorNet; those are not load-bearing for the main result.

The central theoretical claim is Lemma 1, an equivalence between Euclidean convexity of a union of activation regions and Hamming convexity of the corresponding activation patterns. The proof is not valid as written. The contradiction step assumes A is not Hamming-convex, so some shortest path leaves A, but then notes that the straight line [P,Q] gives a shortest path inside A. That only shows one shortest path stays in A; non-convexity requires that some shortest path leaves A, and the existence of one shortest path inside A does not contradict that. The lemma may well be true — the "observable points" caveat in Definition 1 hints the authors saw the issue of non-realized sign patterns — but the proof needs real repair. The paper itself acknowledges the limitation to first-layer hyperplanes, and deep layers can have adjacent regions at Hamming distance >1. Yet the deep-network experiments and the interpretation of chi as "deviation from convexity" lean on this lemma. So the theoretical foundation for the most interesting application is currently unsupported.

The empirical section is thin: six MNIST architectures with fixed total hidden neurons, a post-hoc chosen seed (4 among the few that trained well), no confidence intervals on the reported Pearson correlation of 0.987, and a claim based on medians over digit pairs. It is suggestive but not convincing.

For a revision, I would ask for a corrected proof of Lemma 1, an honest statement of what chi means for deep layers without that lemma, all seeds reported, and some measure of uncertainty on the depth correlation. The measure itself is well defined and could become a useful descriptive tool. It does not resolve a big open problem and will not change practice on its own, but as a first quantitative fold measure it deserves referee time rather than a desk reject. I just would not accept it in its current form.

Referee Report

3 major / 4 minor

Summary. The paper proposes a quantitative measure of 'space folding' in ReLU networks by studying how straight line segments in the Euclidean input space map to paths in the Hamming space of activation patterns. The main theoretical claim (Lemma 1) is an equivalence between convexity of a union of activation regions and convexity of the corresponding set of activation patterns in Hamming space, for tessellations formed by hyperplanes that intersect the entire input space. The paper then introduces a space folding measure chi(Γ) = 1 - max_i d_H(π_1,π_i) / Σ_i d_H(π_i,π_{i+1}), proves bounds on it (Lemma 2), gives an algorithm for its computation, and reports experiments on CantorNet and MNIST suggesting that the measure increases with network depth.

Significance. If Lemma 1 were established, the convexity equivalence would be a clean structural statement about single-hidden-layer ReLU networks, and chi would be a novel, parameter-free probe of how deep layers deviate from that behavior. The measure itself is well-defined, Lemma 2's bounds are correct, the algorithm with its complexity analysis is useful, and the CantorNet example provides a helpful sanity check. However, the proof of Lemma 1 is not valid as written, and the deep-layer experiments rely on an assumption that the paper itself shows fails. The measure remains a promising descriptive statistic, but the paper's main theoretical and interpretational claims require substantial additional work.

major comments (3)

[Section 4, Lemma 1 (proof, first direction)] The contradiction argument is logically invalid. Non-convexity of A in the Hamming cube means that there exists at least one shortest path between two patterns in A that leaves A; it does not mean that every shortest path leaves A. The proof exhibits the path induced by the straight line segment [P,Q] and notes that it is a shortest path contained in A, which does not contradict the existence of a different shortest path that exits A. Definition 1 explicitly requires all shortest paths to remain in A. Additionally, the proof does not account for the 'observable points' qualification in Definition 1: a shortest path that leaves A only through non-observable binary vectors is not a violation of convexity, but the proof treats any exit from A as a contradiction. The lemma may be true, but this proof does not establish it; a correct proof would need to show that every observable pattern on every shortest path between two patterns in A occurs in some region contained in the convex union R.
[Section 5 (Eq. 4 discussion) and Section 6] The interpretation of chi as a deviation from convexity is invoked for deep networks despite the paper's own limitation note after Lemma 1. That note states that the single-bit-flip property holds only for first-hidden-layer hyperplanes that intersect the entire input space, and Example 2 shows that for deeper layers adjacent regions can have Hamming distance greater than one. In that setting, the denominator r2(Γ) in Eq. (3) accumulates multiple bit flips per step even for a monotone path, so chi(Γ) can be positive without any backtracking or non-convexity. Consequently, the CantorNet and MNIST experiments in Section 6, which use all hidden layers and interpret chi as space folding, rest on an assumption the paper itself shows is violated. Please either compute chi per layer with the theoretical guarantee applied only to the first layer, or provide a separate justification for the deep-layer interpretation.
[Section 6.1 and Section 7] The reported Pearson correlation of 0.987 between network depth and aggregated median space folding is based on a single random seed (seed 4) and at most six architectures, and the text notes that only seeds 3 and 4 trained to satisfactory accuracy. This is too thin to support the general hypothesis in Section 7 that the maximal value of chi is associated with generalization capacity and increases with depth. Please report confidence intervals across seeds and architectures, and avoid selective reporting based on the best-performing seed.

minor comments (4)

[Section 5, Eq. (2)] The telescoping sum in Eq. (2) equals d_H(π_{i+1},π_1), so the maximum over i in {1,...,n-1} is max_{i in {2,...,n}} d_H(π_i,π_1), not max_{i in {1,...,n-1}} d_H(π_i,π_1); Eq. (4) uses the range {1,...,n}, so the two displays are inconsistent.
[Section 3, Definition 1] Definition 1 relies on the informal notion of 'observable' points; please define the set of observable binary vectors for a tessellation and state Definition 1 in terms of the induced subgraph of the Hamming cube.
[Section 4, Example 2] The example says 'we conclude that they are non-observable on the considered domain' after listing intermediate patterns; for clarity, specify the domain [0,1]^2 explicitly in the example statement, since the argument depends on the bounded domain.
[Section 6.1] The phrase '100 pairs, ~1M paths Γ for each pair' is hard to parse; clarify whether this is the number of test-image pairs per digit-pair and how the n intermediate points are chosen.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the folding measure is defined directly from Hamming distances, Lemma 1 is self-contained, and self-citations are attribution or benchmark sources only.

full rationale

The paper's central construction, the space folding measure χ(Γ), is defined directly from the Hamming distances along the mapped path (Eq. 4) with no fitted parameters and no dependence on any prior result. Lemma 1, the convexity equivalence, is argued in the paper from the geometry of hyperplane crossings; whatever the status of that proof, it does not reduce to an input of the measure. The self-citations (Moser et al. 2022 for the Hamming-convexity definition; Lewandowski et al. 2024 for CantorNet; Moser 2012/2014/2017 for range norms) are attribution, definitional context, and benchmark sources rather than load-bearing theorems: the definition is restated in the text, CantorNet is described in Appendix A, and the range norms are used only as inspiration. The acknowledged limitation that the one-bit-flip shortest-path assumption holds only for first-hidden-layer hyperplanes is a correctness or validity concern for deep-layer interpretation, not a circularity: χ itself remains an independently defined observable computed from the network's binarized activations. No equation in the paper sets a derived quantity equal to an input by construction, and no fitted value is renamed as a prediction. Therefore no significant circularity is present.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No new physical or mathematical entities are postulated; chi is a scalar measure defined from Hamming distances, and the theory rests on standard metric facts plus the first-layer hyperplane assumption. The free parameters are algorithmic choices, not fitted physical constants.

free parameters (2)

n (number of intermediate interpolation points) = user-specified; suggested upper bound N (total hidden neurons)
chi(Gamma) depends on the sampling density n along the line segment; no criterion for choosing n is given, and the authors note that equal spacing may be suboptimal (Sec. 5).
m1, m2 (number of clusters per class) = varied k in {1, 2, 5, 10, 20, 50, 100} in Appendix C
Introduced in Sec. 5 to reduce the computational cost of Eq. (6); they do not affect the definition of chi, but they change which paths are measured and the authors note the impact is only preliminarily studied.

assumptions (4)

standard math Triangle inequality in Hamming space
Used in Lemma 2 to establish the upper bound chi(Gamma) <= 1.
domain assumption First-layer hyperplanes intersect the entire input space
Assumed in Lemma 1; the authors state it holds only for the first hidden layer of a ReLU network (Sec. 4, discussion after Lemma 1).
domain assumption A straight line in input space crosses each hyperplane at most once, flipping one activation bit at a time, so the induced Hamming path is a shortest path
Assumed in the proof of Lemma 1 (Fig. 3); the authors acknowledge in Example 2 that for deeper layers adjacent regions can have Hamming distance greater than one, violating this premise outside the first layer.
ad hoc to paper Non-observable binary vectors are excluded from the convexity condition (Definition 1)
The 'observable' caveat is introduced in Definition 1 and used in Example 2 to declare certain pattern sets convex; this modifies the standard geodesic convexity notion of the hypercube and is central to the proof's interpretation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of On Space Folds of ReLU Neural Networks." pith.science (2026). https://pith.science/paper/PUBSE6QS

@misc{pith2026250209954,
  author       = {Pith},
  title        = {Pith review of: On Space Folds of ReLU Neural Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PUBSE6QS}},
  note         = {Machine review of arXiv:2502.09954}
}

read the original abstract

Recent findings suggest that the consecutive layers of ReLU neural networks can be understood geometrically as space folding transformations of the input space, revealing patterns of self-similarity. In this paper, we present the first quantitative analysis of this space folding phenomenon in ReLU neural networks. Our approach focuses on examining how straight paths in the Euclidean input space are mapped to their counterparts in the Hamming activation space. In this process, the convexity of straight lines is generally lost, giving rise to non-convex folding behavior. To quantify this effect, we introduce a novel measure based on range metrics, similar to those used in the study of random walks, and provide the proof for the equivalence of convexity notions between the input and activation spaces. Furthermore, we provide empirical analysis on a geometrical analysis benchmark (CantorNet) as well as an image classification benchmark (MNIST). Our work advances the understanding of the activation space in ReLU neural networks by leveraging the phenomena of geometric folding, providing valuable insights on how these models process input information.

Figures

Figures reproduced from arXiv: 2502.09954 by the authors.

**Figure 2.** Activation patterns πi of recursion-based representation of CantorNet (see Appendix A). We skip neurons with unchanged values. The colours are used for increased visibility; activation patterns {π1, π2, π3} are convex in the Hamming cube sense (see Ex. 2). The darker gray of π5 has been used to visually distinguish from π4 and π6. (Adapted from Lewandowski et al. (2024) with the authors’ approval.) Before introducin… view at source ↗

**Figure 3.** The shaded gray area illustrates a convex set in the Euclidean space. The hyperplanes [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (6 more)

**Figure 4.** Figure 4: Left: Straight line between x1 and x2 in the Euclidean space. Observe that, when mapped to the Hamming activation space (dotted arrows), the Hamming distance may decrease when following the path, i.e., it might happen that dH(π1, πn) < maxi dH(π1, πi). Right: An extrem…

**Figure 5.** Figure 5: Activation patterns πi of the recursion-based representation of CantorNet. For the computation of the space folding measure χ we can consider a subset of layers; left: Highlighted activations in the first layer, right: All layers. For a path Γ = (π6, π5, π4), the foldi…

**Figure 6.** Figure 6: Behaviour of dH(π1, ·) for respective πi (dotted line) and distance between neighboring patterns dH(πi , πi+1) (dashed line) on path Γ constructed between points x1 = (0, 3 4 ) and x2 = (1, 3 4 ) (indicated by arrows). Background represents CantorNet recursion-based re…

**Figure 7.** Figure 7: Median of maximas of non-zero space folding [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]

**Figure 8.** Figure 8: Preliminary results illustrate the robustness of the space folding measure [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]

**Figure 9.** Figure 9: We investigate the folding effects for larger networks. Interestingly, we do not observe much [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

63 extracted references · 55 canonical work pages

[1]

and Álvarez León, L

Alemán-Flores, M. and Álvarez León, L. (2004). Video segmentation through multiscale texture analysis. In Campilho, A. and Kamel, M., editors, Image Analysis and Recognition, ICIAR 2004, Lecture Notes in Computer Science , volume 3212, pages 339--346. Springer, Berlin, Heidelberg

work page 2004
[2]

Alexiewicz, A. (1948). Linear functionals on denjoy-integrable functions. Colloq. Math. , 1:289--293

work page 1948
[3]

Anil, C., Lucas, J., and Grosse, R. (2019). Sorting out L ipschitz function approximation. In Chaudhuri, K. and Salakhutdinov, R., editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 291--301. PMLR

work page 2019
[4]

On the Geometry of Deep Learning

Balestriero, R., Humayun, A. I., and Baraniuk, R. (2024). On the geometry of deep learning. arXiv preprint arXiv:2408.04809

work page Pith review arXiv 2024
[5]

and Hayes, B

Bern, M. and Hayes, B. (1996). The complexity of flat origami. In Proceedings of the Seventh Annual ACM-SIAM Symposium on Discrete Algorithms , pages 175--183, Philadelphia, PA, USA. Society for Industrial and Applied Mathematics

work page 1996
[6]

and Hayes, B

Bern, M. and Hayes, B. (2008). Origami embedding of piecewise-linear two-manifolds. In Laber, E. S., Bornstein, C., Nogueira, L. T., and Faria, L., editors, LATIN 2008: Theoretical Informatics , pages 617--629, Berlin, Heidelberg. Springer Berlin Heidelberg

work page 2008
[7]

and Yvinec, M

Boissonnat, J.-D. and Yvinec, M. (1998). Algorithmic Geometry . Cambridge University Press

work page 1998
[8]

Bonicelli, L., Boschini, M., Porrello, A., Spampinato, C., and Calderara, S. (2022). On the effectiveness of lipschitz-driven rehearsal in continual learning. Advances in Neural Information Processing Systems , 35:31886--31901

work page 2022

Show all 63 references

[9]

Cantor, G. (1883). \"U ber unendliche, lineare punktmannigfaltigkeiten. Math. Annalen , 21(4):545--591
[10]

Cisse, M., Bojanowski, P., Grave, E., Dauphin, Y., and Usunier, N. (2017). Parseval networks: Improving robustness to adversarial examples. In Precup, D. and Teh, Y. W., editors, Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of...

2017
[11]

H., Rolnick, D., and van de Panne, M

Cohan, S., Kim, N. H., Rolnick, D., and van de Panne, M. (2022). Understanding the evolution of linear regions in deep reinforcement learning. In Advances in Neural Information Processing Systems

2022
[12]

D., and Benini, L

Conti, F., Schiavone, P. D., and Benini, L. (2018). Xnor neural engine: a hardware accelerator ip for 21.6 fj/op binary neural network inference. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems , 37(11):2940 -- 2951

2018
[13]

Courbariaux, M., Bengio, Y., and David, J.-P. (2015). Binaryconnect: Training deep neural networks with binary weights during propagations. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2 , NIPS'15, page 3123–3131, Cambri...

2015
[14]

Crescenzi, P., Goldman, D., Papadimitriou, C., Piccolboni, A., and Yannakakis, M. (1998). On the complexity of protein folding. Journal of Computational Biology , 5(3):423--465

1998
[15]

A., and Mirny, L

Dekker, J., Marti-Renom, M. A., and Mirny, L. A. (2013). Exploring the three-dimensional organization of genomes: interpreting chromatin interaction data. Nature Reviews Genetics , 14(6):390--403

2013
[16]

Demaine, E. D. and Demaine, M. L. (2005). A survey of folding and unfolding in computational geometry. In Goodman, J. E., Pach, J., and Welzl, E., editors, Combinatorial and Computational Geometry , volume 52 of Mathematical Sciences Research Institute Publications , pages 167...

2005
[17]

D., Demaine, M

Demaine, E. D., Demaine, M. L., and Lubiw, A. (2000). Folding and cutting paper. In Akiyama, J., Kano, M., and Urabe, M., editors, Discrete and Computational Geometry , pages 104--118, Berlin, Heidelberg. Springer Berlin Heidelberg

2000
[18]

A., Ozkan, S

Dill, K. A., Ozkan, S. B., Shell, M. S., and Weikl, T. R. (2008). The protein folding problem. Annual Review of Biophysics

2008
[19]

Foote, J. (1999). Visualizing music and audio using self-similarity. In Proceedings of the Seventh ACM International Conference on Multimedia (Part 1) , pages 77--80

1999
[20]

Gamba, M., Azizpour, H., and Bj \"o rkman, M. (2023). On the lipschitz constant of deep networks and double descent. arXiv preprint arXiv:2301.12309

2023 arXiv
[21]

Gamba, M., Chmielewski-Anders, A., Sullivan, J., Azizpour, H., and Bjorkman, M. (2022). Are all linear regions created equal? In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics , volume 151 of Proceedings of Machine Learning Research ...

2022
[22]

Grigsby, E., Lindsey, K., and Rolnick, D. (2023). Hidden symmetries of R e LU networks. In International Conference on Machine Learning . PMLR

2023
[23]

Hanin, B., Jeong, R., and Rolnick, D. (2021). Deep relu networks preserve expected length. In International Conference on Learning Representations

2021
[24]

and Rolnick, D

Hanin, B. and Rolnick, D. (2019a). Complexity of linear regions in deep networks. In International Conference on Machine Learning , pages 2596--2604. PMLR

2019
[25]

and Rolnick, D

Hanin, B. and Rolnick, D. (2019b). Deep relu networks have surprisingly few activation patterns. In NeurIPS

2019
[26]

He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770--778

2016
[27]

and Orey, S

Jain, N. and Orey, S. (1968). On the range of random walk. Israel J. Math. , 6:373--380

1968
[28]

Jumper, J. et al. (2021). Highly accurate protein structure prediction with alphafold. Nature

2021
[29]

and Helias, M

Keup, C. and Helias, M. (2022). Origami in n dimensions: How feed-forward networks manufacture linear separability. arXiv preprint arXiv:2203.11355

2022 arXiv
[30]

Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In NeurIPS

2012
[31]

LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278--2324

1998
[32]

Lewandowski, M., Eghbal-zadeh, H., and A.Moser, B. (2024). Cantornet: A sandbox for testing topological and geometrical measures. In NeurIPS Workshop On Symmetry and Geometry in Neural Networks , Proceedings of Machine Learning Research

2024
[33]

L., Hannun, A

Maas, A. L., Hannun, A. Y., and Ng, A. Y. (2013). Rectifier nonlinearities improve neural network acoustic models. In ICML Workshop on Deep Learning for Audio, Speech and Language Processing

2013
[34]

Makhoul, J., El-Jaroudi, A., and Schwartz, R. (1991). Partitioning capabilities of two-layer neural networks. IEEE Transactions on Signal Processing , 39(6):1435--1440

1991
[35]

Makhoul, J., Schwartz, R., and El-Jaroudi, A. (1989). Classification capabilities of two-layer neural nets. In International Conference on Acoustics, Speech, and Signal Processing, , pages 635--638 vol.1

1989
[36]

Mandelbrot, B. B. (1983). The Fractal Geometry of Nature . Macmillan

1983
[37]

Marcus, G. (2018). Deep learning: A critical appraisal. arXiv

2018
[38]

and Papert, S

Minsky, M. and Papert, S. (1969). Perceptrons: An Introduction to Computational Geometry . MIT Press

1969
[39]

C., Tuzel, O., Samei, G., Rastegari, M., and Farajtabar, M

Mirzadeh, I., Alizadeh, K., Mehta, S., Del Mundo, C. C., Tuzel, O., Samei, G., Rastegari, M., and Farajtabar, M. (2023). Relu strikes back: Exploiting activation sparsity in large language models. arXiv preprint arXiv:2310.04564

2023 arXiv
[40]

F., Pascanu, R., Cho, K., and Bengio, Y

Mont \'u far, G. F., Pascanu, R., Cho, K., and Bengio, Y. (2014). On the number of linear regions of deep neural networks. In NeurIPS , volume 27

2014
[41]

Moser, B. A. (2012). Geometric characterization of weyl's discrepancy norm in terms of its n-dimensional unit balls. Discrete Comput. Geom. , 48(4):793--806

2012
[42]

Moser, B. A. (2014). The range of a simple random walk on z: An elementary combinatorial approach. Electron. J. Comb. , 21

2014
[43]

Moser, B. A. (2017). Similarity recovery from threshold-based sampling under general conditions. IEEE Transactions on Signal Processing , 65(17):4645--4654

2017
[44]

A., Lewandowski, M., Kargaran, S., Zellinger, W., Biggio, B., and Koutschan, C

Moser, B. A., Lewandowski, M., Kargaran, S., Zellinger, W., Biggio, B., and Koutschan, C. (2022). Tessellation-filtering relu neural networks. IJCAI

2022
[45]

Neyshabur, B., Bhojanapalli, S., McAllester, D., and Srebro, N. (2017). Exploring generalization in deep learning. Advances in neural information processing systems , 30

2017
[46]

and Lampert, C

Phuong, M. and Lampert, C. H. (2020). Functional vs. parametric equivalence of relu networks. In International Conference on Learning Representations

2020
[47]

and Tanner, J

Price, I. and Tanner, J. (2021). Trajectory growth lower bounds for random sparse deep relu networks. In 2021 20th IEEE International Conference on Machine Learning and Applications (ICMLA) , pages 1004--1009

2021
[48]

Raghu, M., Poole, B., Kleinberg, J., Ganguli, S., and Sohl-Dickstein, J. (2017). On the expressive power of deep neural networks. ICML

2017
[49]

Rastegari, M., Ordonez, V., Redmon, J., and Farhadi, A. (2016). Xnor-net: Imagenet classification using binary convolutional neural networks. In Computer Vision--ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11--14, 2016, Proceedings, Part IV , pages...

2016
[50]

Rosenblatt, F. (1958). The perceptron: A probabilistic model for information storage and organization in the brain. Psychological Review , 65(6):386--408

1958
[51]

Roy, A. J. and Stell, J. G. (2003). Convexity in discrete space. In Kuhn, W., Worboys, M. F., and Timpf, S., editors, Spatial Information Theory. Foundations of Geographic Information Science , pages 253--269, Berlin, Heidelberg. Springer Berlin Heidelberg

2003
[52]

Sejnowski, T. J. (2020). The unreasonable effectiveness of deep learning in artificial intelligence. Proceedings of the National Academy of Sciences , 117(48):30033--30038

2020
[53]

Serra, T., Tjandraatmadja, C., and Ramalingam, S. (2018). Bounding and counting linear regions of deep neural networks. In International Conference on Machine Learning , pages 4558--4566

2018
[54]

Shen, K., Guo, J., Tan, X., Tang, S., Wang, R., and Bian, J. (2023). A study on relu and softmax in transformer. arXiv preprint arXiv:2302.06461

2023 arXiv
[55]

Telgarsky, M. (2015). Representation benefits of deep feedforward networks. arXiv preprint arXiv:1509.08101

2015 arXiv
[56]

Telgarsky, M. (2016). Benefits of depth in neural networks. In Feldman, V., Rakhlin, A., and Shamir, O., editors, 29th Annual Conference on Learning Theory , volume 49 of Proceedings of Machine Learning Research , pages 1517--1539, Columbia University, New York, New York, USA. PMLR

2016
[57]

Tsuzuku, Y., Sato, I., and Sugiyama, M. (2018). Lipschitz-margin training: Scalable certification of perturbation invariance for deep neural networks. Advances in Neural Information Processing Systems , 31

2018
[58]

and Scaman, K

Virmaux, A. and Scaman, K. (2018). Lipschitz regularity of deep neural networks: analysis and efficient estimation. In Advances in Neural Information Processing Systems , volume 31

2018
[59]

Wang, X., Wang, K., and Lian, S. (2020). A survey on face data augmentation for the training of deep neural networks. Neural Computing and Applications , 32(19):15503--15531

2020
[60]

Weyl, H. (1916). Uber die gleichverteilung von zahlen. Eins.Mathematische Annalen , 77:331--352

1916
[61]

Xiong, H., Huang, L., Yu, M., Liu, L., Zhu, F., and Shao, L. (2020). On the number of linear regions of convolutional neural networks. In International Conference on Machine Learning . PMLR

2020
[62]

Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. (2017). Understanding deep learning requires rethinking generalization. In ICLR

2017
[63]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

and Álvarez León, L

Alemán-Flores, M. and Álvarez León, L. (2004). Video segmentation through multiscale texture analysis. In Campilho, A. and Kamel, M., editors, Image Analysis and Recognition, ICIAR 2004, Lecture Notes in Computer Science , volume 3212, pages 339--346. Springer, Berlin, Heidelberg

work page 2004

[2] [2]

Alexiewicz, A. (1948). Linear functionals on denjoy-integrable functions. Colloq. Math. , 1:289--293

work page 1948

[3] [3]

Anil, C., Lucas, J., and Grosse, R. (2019). Sorting out L ipschitz function approximation. In Chaudhuri, K. and Salakhutdinov, R., editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 291--301. PMLR

work page 2019

[4] [4]

On the Geometry of Deep Learning

Balestriero, R., Humayun, A. I., and Baraniuk, R. (2024). On the geometry of deep learning. arXiv preprint arXiv:2408.04809

work page Pith review arXiv 2024

[5] [5]

and Hayes, B

Bern, M. and Hayes, B. (1996). The complexity of flat origami. In Proceedings of the Seventh Annual ACM-SIAM Symposium on Discrete Algorithms , pages 175--183, Philadelphia, PA, USA. Society for Industrial and Applied Mathematics

work page 1996

[6] [6]

and Hayes, B

Bern, M. and Hayes, B. (2008). Origami embedding of piecewise-linear two-manifolds. In Laber, E. S., Bornstein, C., Nogueira, L. T., and Faria, L., editors, LATIN 2008: Theoretical Informatics , pages 617--629, Berlin, Heidelberg. Springer Berlin Heidelberg

work page 2008

[7] [7]

and Yvinec, M

Boissonnat, J.-D. and Yvinec, M. (1998). Algorithmic Geometry . Cambridge University Press

work page 1998

[8] [8]

Bonicelli, L., Boschini, M., Porrello, A., Spampinato, C., and Calderara, S. (2022). On the effectiveness of lipschitz-driven rehearsal in continual learning. Advances in Neural Information Processing Systems , 35:31886--31901

work page 2022