Pith. sign in

REVIEW 5 major objections 5 minor 43 references

Discrete Functional Geometry of ReLU Networks via ReLU Transition Graphs

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a ReLU network's generalization error is controlled by the spectral gap of its ReLU Transition Graph and by the average KL divergence across adjacent activation regions.

desk verdict The RTG extension is a reasonable idea, but the paper's central spectral and KL generalization bounds are asserted rather than derived, and Theorem IV.6 is contradicted by the paper's own Experiment 5. read the letter →

arxiv 2509.03056 v2 pith:GYNQAXRH submitted 2025-09-03 cs.LG

Sahil Rajesh Dhayalkar This is my paper · ORCID
classification cs.LG
keywords ReLUnetworksactivationregionsTransitionGraphspectralgapgeneralizationboundsregionentropyKLdivergenceexpandergraphs
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper extends the ReLU Transition Graph (RTG) framework, in which each node is a linear activation region of a ReLU network and each edge connects regions differing by a single ReLU flip. It argues that this graph's structure governs both expressivity and generalization: spectral gap bounds expected generalization error, mean edge-wise KL divergence gives a generalization–compression duality, and region entropy bounds effective capacity. If correct, these results turn the discrete geometry of activation regions into principled, graph-based capacity controls and diagnostics for deep ReLU networks. The paper validates the theory by explicitly constructing RTGs for small 2D-input networks and measuring expansion, degree distributions, entropy, spectral gap, and edge KL divergence.

What carries the argument

The ReLU Transition Graph (RTG) is the central object: an undirected graph whose vertices are the network's linear activation regions and whose edges connect regions whose activation patterns differ by exactly one ReLU flip, making it a subgraph of the hypercube. The arguments run through spectral graph theory (the normalized Laplacian's second eigenvalue and Cheeger's inequality), region entropy measured against the input distribution, and KL divergence between the local affine maps on adjacent regions. These tools translate functional variation across region boundaries into graph-theoretic quantities that bound capacity and generalization.

What would settle it

Train many ReLU networks of the same architecture on a fixed classification task with varying label noise or initialization, construct each RTG, and look for a network whose RTG has a large spectral gap and small mean edge KL divergence but a large generalization gap; under the paper's claimed control this combination should be impossible (up to the unspecified constants), so finding such a case would falsify the central generalization claim.

Watch

Extended reading notes

Core claim

The central claim is that the RTG is not just a combinatorial byproduct of ReLU activations but a load-bearing geometric object. The paper proves that RTGs at random initialization are expanders with binomial degree distributions, and states two generalization bounds: E[GenError(f)] ≤ C·$L_f^{2}$ / λ_2, where λ_2 is the second eigenvalue of the normalized Laplacian, and E[GenError(f)] ≤ (1/|E|) Σ_{(i,j)∈E} KL(f|_{R_i} ∥ f|_{R_j}). It also proves a capacity bound VCdim(f) = O(H(G_f)·L·m) via region entropy, with entropy saturating under overparameterization when data lie on a lower-dimensional manifold. Empirically, the paper reports that spectral gap correlates with perfect generalization in a simple classification task, that mean edge KL decreases with depth even under random-label training, and that entropy grows then saturates with width.

Load-bearing premise

The argument assumes that generalization error is genuinely controlled by how much the network's output varies between adjacent activation regions; the proof sketches assert this link rather than deriving it formally, so if functional variation across RTG edges does not drive test error, both main generalization bounds collapse.

Editorial extensions

If this is right

  • Networks with larger RTG spectral gaps should exhibit smoother functional variation across activation regions and therefore tighter generalization error, making spectral gap a candidate diagnostic or regularizer during training and architecture search.
  • The mean KL divergence across RTG edges provides a functional regularizer: encouraging representational consistency between adjacent activation patterns could improve generalization without relying solely on parameter-norm penalties.
  • Region entropy saturating under overparameterization gives a data-aware stopping criterion for width or depth scaling, since additional regions that miss the data manifold do not add usable expressivity.
  • The expander property implies that any activation region can be reached from any other through a small number of single-neuron flips, promoting rapid mixing and robust functional connectivity at random initialization.
  • The binomial degree distribution of RTGs explains variability in local Lipschitz behavior and functional sharpness, potentially informing initialization or training schedules for controlled local smoothness.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same graph metrics could be computed for convolutional, residual, or transformer architectures by defining activation patterns over feature maps rather than per-neuron binary states, a natural extension the paper does not test.
  • A testable extension would track RTG spectral gap and mean edge KL throughout training on higher-dimensional benchmarks and compare their predictive power for test error against weight-norm, margin, and NTK-based baselines.
  • If the generalization–compression duality holds, then explicitly regularizing mean edge KL during training should yield smoother decision boundaries and improved robustness to input perturbations, a consequence the author leaves implicit.
  • The entropy-saturation result suggests a principled pruning criterion: once region entropy plateaus, units or regions contributing negligible probability mass can be removed without sacrificing usable capacity, though such pruning is not demonstrated in the paper.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper extends the ReLU Transition Graph (RTG) framework of [5], modeling each linear activation region as a node and connecting regions that differ by a single ReLU flip. It claims several theoretical results: RTGs are expanders at random initialization (Lemma IV.1); the RTG spectral gap upper-bounds expected generalization error (Theorem IV.2); region entropy upper-bounds effective VC dimension (Theorem IV.3); region degrees are binomially distributed (Proposition IV.5); and mean edge KL divergence upper-bounds generalization error (Theorem IV.6). The paper then constructs RTGs for small 2D MLPs and reports experiments on expansion, spectral gap, entropy, degree distribution, and KL-based generalization–compression. The central assertion is that RTG structure, specifically the spectral gap and edge-wise KL divergence, governs generalization.

Significance. If the theoretical claims were established, the RTG framework would offer a finite-width, discrete set of generalization diagnostics for ReLU networks, complementing NTK and norm-based bounds. The paper has some positive elements: it gives an explicit graph construction, runs small experiments with reproducible protocol, and identifies a plausible object of study (region adjacency). However, the significance is conditional, because the main generalization bounds are proof sketches that do not derive the central bridge between region functional variation and generalization error, and the paper's own Experiment 5 contradicts Theorem IV.6. There are no machine-checked proofs, and the experiments are too small and too few to substantiate the claims on their own.

major comments (5)
  1. [Theorem IV.6, Experiment 5] The claimed bound is directly falsified by the paper's own data. The Experiment 5 table reports at depth 6 a generalization gap of 0.333 and a mean edge KL of 0.0020, so the stated inequality E[GenError(f)] <= (1/|E|) * sum_{(i,j) in E} KL(f|Ri || f|Rj) would require 0.333 <= 0.0020. If GenError denotes the generalization gap measured in the experiment, Theorem IV.6 is false as stated; if it denotes a different quantity, the experiment does not test the theorem. The proof sketch, which states that 'since sharp transitions hurt generalization... we upper-bound GenError by the mean edge discrepancy', asserts the bridge rather than deriving it, and the cited reference [30] is an information-theoretic bound with no RTG terms.
  2. [Theorem IV.2, proof sketch] The argument jumps from the spectral inequality E(phi) >= lambda_2 * Var(phi) to the claimed bound E[GenError(f)] <= C * L_f^2 / lambda_2. The proportionality between GenError and Var(phi), the appearance of L_f, and the unspecified constant C are never derived. The invocation 'generalization error is controlled by functional variation across regions (e.g., Raginsky et al., 2017)' does not supply the missing bridge; the cited result is an information-theoretic bound involving mutual information between data and hypothesis, with no dependence on region adjacency or the RTG Laplacian.
  3. [Theorem IV.3, proof sketch] The claimed bound VCdim(f)=O(H(Gf)*L*m) conflates the distribution-free VC dimension with the distribution-dependent quantity H(Gf). Since H(Gf) depends on the input distribution P, it cannot upper-bound a distribution-free capacity measure such as VCdim; for example, if P is concentrated on a single region, H(Gf)=0 but the VC dimension of the network function class is unchanged. Sauer's lemma, invoked in the proof sketch, bounds the number of dichotomies on n points by a polynomial in n and the VC dimension; it does not yield a bound of the form 2^{H(Gf)} on 'effectively distinguishable functions', a term that is never formalized.
  4. [Lemma IV.1, proof sketch] The proof sketch asserts that activation patterns are 'uniformly likely' and attributes this to Montufar et al. (2014). That reference counts linear regions; it does not establish uniformity of pattern probabilities under i.i.d. continuous initialization. The further assertion that a random induced subgraph of the hypercube formed by reachable regions is an expander with high probability is not a standard theorem; induced subgraphs of the hypercube can have poor expansion even when the vertex set is large.
  5. [Experiment 2, Theorem IV.2] The experiment reports a single network with perfect generalization and lambda_2 approx 0.0286. This is a single observation with no variation of the relevant quantities, and perfect generalization makes the test trivial; it does not test the quantitative content of Theorem IV.2, and the statement that the nonzero spectral gap 'confirms' the theorem is unsupported.
minor comments (5)
  1. [Proposition IV.5] The statement contains a typo: 'the the ReLU Transition Graph' should read 'the ReLU Transition Graph'.
  2. [Experiment 5, results paragraph] The text says 'as width increases', but the table varies depth d from 2 to 6 at fixed width m=256; the wording should be corrected.
  3. [Section IV, proof sketches] The text cites 'Raginsky et al., 2017', but the reference list entry [30] is 'Xu and Raginsky'; the citation should be corrected for consistency.
  4. [Experiment 5, KL definition] The theorem states KL between induced local output distributions, but the experimental formula uses KL of the expected outputs only; the relation between these objects is not specified.
  5. [End of manuscript] The author background section at the end of the manuscript is not part of a standard scientific article; it should be removed or moved to the cover letter.

Circularity Check

3 steps flagged · score 7.0 of 10

Theorem IV.6 and Theorem IV.2 derive their conclusions by assuming the very generalization-variation bridge they purport to establish, and Experiment 4 validates the binomial degree model by fitting p to the empirical mean.

  1. self definitional [Section IV, Theorem IV.6 (Generalization–Compression Duality), Proof Sketch]
    "Since sharp transitions hurt generalization (e.g., via margin theory), we upper-bound GenError(f) by the mean edge discrepancy."

    The theorem to be established is exactly E[GenError(f)] <= (1/|E|) * sum over edges of KL(f|Ri || f|Rj). The proof's only substantive step is 'we upper-bound GenError(f) by the mean edge discrepancy,' which restates the target inequality as its own premise. No argument connects KL to margin theory or to the generalization gap; the preceding averaging over edges only defines the right-hand side. Thus the claimed derivation is the conclusion re-entered as an assumption.

  2. other [Section IV, Theorem IV.2 (Spectral Gap Bounds Generalization), Proof Sketch]
    "Since generalization error is controlled by functional variation across regions (e.g., Raginsky et al., 2017), we obtain the stated bound with GenError(f) ∝ Var(ϕ) ≤ E(ϕ)/λ2."

    The theorem asserts E[GenError(f)] <= C L_f^2 / λ2. The spectral identity E(ϕ) >= λ2 Var(ϕ) is standard, but the bridge 'GenError(f) ∝ Var(ϕ)' is the substantive content of the theorem and is merely asserted. The cited Raginsky et al. 2017 result is an information-theoretic generalization bound with no RTG or functional-variation term, so it does not supply the missing proportionality. The generalization bound is therefore not derived from the graph spectrum; it assumes that generalization is controlled by region variation, which is the conclusion.

1 more flagged steps
  1. fitted input called prediction [Section V.E, Experiment 4 (Degree Distribution of the RTG)]
    "Empirical mean degree and estimated p = mean degree n • Overlay a fitted Binomial distribution Bin(n,p) ... This supports Proposition IV.5, confirming that each ReLU boundary intersects regions independently with low probability."

    Proposition IV.5 predicts deg(R) ~ Binomial(n,p) with p the per-neuron intersection probability. The experiment sets p = mean degree / n from the same degree data and then overlays the fitted Binomial as confirmation. Any empirical distribution is matched by construction in its first moment by Binomial(n, mean/n), so the 'prediction' of the mean is forced by the fitting procedure; the independence/Bernoulli model is not independently tested.

full rationale

The central generalization claims are not derived from independent premises. Theorem IV.6's proof restates the target inequality as 'we upper-bound GenError by the mean edge discrepancy,' and the paper's own Experiment 5 contradicts the bound (at depth 6, Gen Gap 0.333 versus mean KL 0.0020), which shows the theorem is imposed rather than entailed. Theorem IV.2 similarly imports the missing link 'GenError(f) ∝ Var(ϕ)' with a citation to Raginsky et al. 2017 that contains no such statement about RTG functional variation. Experiment 4 validates the binomial degree model by fitting p to the empirical mean degree, so that validation reduces to a moment match. These are load-bearing circularities: the main generalization bounds reduce to their intended conclusions by construction, and the empirical check of Proposition IV.5 is fitted. The RTG definitions, the entropy bound (Theorem IV.3), and entropy saturation (Corollary IV.4) are definitional or observational rather than circular, and Experiment 3 is not a fit. Because the two main generalization bounds and one of the validating experiments reduce to assumptions or fitted values, the paper's core derivation chain is substantially circular, though not the entire manuscript.

Assumptions & free parameters 2 free parameters · 6 assumptions · 0 invented entities

The paper introduces no new physical entities; the RTG itself is from prior work [5]. The central claims rest on several unproved domain assumptions: that generalization error is controlled by RTG functional variation, that activation patterns are uniformly distributed at initialization, that facet events are independent Bernoulli, and that entropy-weighted counting bounds the VC dimension. The only number fitted to data is the Bernoulli probability p in the degree-distribution experiment, where p is set to mean degree / n and then the binomial curve is presented as confirmation.

free parameters (2)
  • p (Bernoulli probability in Proposition IV.5) = 0.11, 0.05, 0.02 for widths 16, 32, 64 (mean degree / n)
    Fitted to the empirical mean degree in Experiment 4 and then used as the binomial parameter to 'validate' the degree distribution; the prediction is the fit.
  • C (unspecified constants in Theorems IV.2 and IV.3) = undefined
    Introduced ad hoc as 'some constant' depending on input distribution and region complexity; without specification the spectral-gap and entropy bounds are vacuous.
assumptions (6)
  • standard math Cheeger inequality and spectral gap property E(phi) >= lambda_2 Var(phi).
    Standard in spectral graph theory; used in Theorem IV.2 proof, but the following step from Var(phi) to GenError is unjustified.
  • domain assumption Generalization error is controlled by functional variation across activation regions.
    Invoked in Theorem IV.2 and Theorem IV.6 proofs; no formal bridge is supplied and the cited works do not establish it for RTGs.
  • domain assumption Activation patterns are uniformly likely under i.i.d. continuous random initialization.
    Invoked in Lemma IV.1 proof sketch; Montufar et al. 2014 does not state or prove uniformity.
  • ad hoc to paper Sauer's lemma can be applied to entropy-weighted 'effectively distinguishable' functions to bound VC dimension.
    Used in Theorem IV.3 proof; VC dimension is distribution-free, and the entropy-weighted counting is a nonstandard, unproven relaxation.
  • ad hoc to paper Each ReLU hyperplane intersects a given region independently with constant probability p.
    Used in Proposition IV.5 proof; the Xi are not independent and the probability varies across regions.
  • domain assumption Sharp transitions between adjacent regions hurt generalization, so mean edge discrepancy upper-bounds GenError.
    Used in Theorem IV.6 proof sketch; this assumption is effectively the conclusion restated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Discrete Functional Geometry of ReLU Networks via ReLU Transition Graphs." pith.science (2026). https://pith.science/paper/GYNQAXRH

@misc{pith2026250903056,
  author       = {Pith},
  title        = {Pith review of: Discrete Functional Geometry of ReLU Networks via ReLU Transition Graphs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GYNQAXRH}},
  note         = {Machine review of arXiv:2509.03056}
}
read the original abstract

We extend the ReLU Transition Graph (RTG) framework into a comprehensive graph-theoretic model for understanding deep ReLU networks. In this model, each node represents a linear activation region, and edges connect regions that differ by a single ReLU activation flip, forming a discrete geometric structure over the network's functional behavior. We prove that RTGs at random initialization exhibit strong expansion, binomial degree distributions, and spectral properties that tightly govern generalization. These structural insights enable new bounds on capacity via region entropy and on generalization via spectral gap and edge-wise KL divergence. Empirically, we construct RTGs for small networks, measure their smoothness and connectivity properties, and validate theoretical predictions. Our results show that region entropy saturates under overparameterization, spectral gap correlates with generalization, and KL divergence across adjacent regions reflects functional smoothness. This work provides a unified framework for analyzing ReLU networks through the lens of discrete functional geometry, offering new tools to understand, diagnose, and improve generalization.

Figures

Figures reproduced from arXiv: 2509.03056 by the authors.

Figure 2
Figure 2. Experiment 3: Region entropy H(Gf ) and number of activation regions versus width. Entropy grows with width but saturates as the input space becomes effectively partitioned. Results: Region entropy increases from 1.316 at width 4 to 9.210 at width 1024, while the number of activation regions grows from 8 to 9998 [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 1
Figure 1. Experiment 1: Histogram of estimated edge expansion [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 3
Figure 3. Experiment 4: Degree distributions for RTGs across [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 25 canonical work pages

  1. [5]

    Dhayalkar, Sahil. (2025). The Geometry of ReLU Networks through the ReLU Transition Graph. 10.48550/arXiv.2505.11692

    work page Pith review arXiv doi:10.48550/arxiv.2505.11692 2025

  2. [30]

    Information-theoretic analysis of gen- eralization capability of learning algorithms

    Aolin Xu and Maxim Raginsky. Information-theoretic analysis of gen- eralization capability of learning algorithms. In Advances in Neural Information Processing Systems, pages 2524–2533, 2017

    work page 2017

  3. [1]

    Vinod Nair and Geoffrey E. Hinton. 2010. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th Interna- tional Conference on International Conference on Machine Learning (ICML’10). Omnipress, Madison, WI, USA, 807–814

    work page 2010

  4. [2]

    On the number of linear regions of deep neural networks,

    G. Mont ´ufar, R. Pascanu, K. Cho, and Y . Bengio, “On the number of linear regions of deep neural networks,” arXiv preprint arXiv:1402.1869, 2014

    arXiv 2014

  5. [3]

    Deep ReLU networks have surprisingly few activation patterns,

    B. Hanin and D. Rolnick, “Deep ReLU networks have surprisingly few activation patterns,” in *Advances in Neural Information Processing Systems (NeurIPS)*, vol. 32, 2019

    work page 2019

  6. [4]

    Bounding and counting linear regions of deep neural networks,

    T. Serra, C. Tjandraatmadja, and S. Ramalingam, “Bounding and counting linear regions of deep neural networks,” in *Proc. 35th Int. Conf. Machine Learning (ICML)*, Stockholm, Sweden, Jul. 2018, pp. 4558–4566

    work page 2018

  7. [6]

    On the second eigenvalue of a graph and a network flow problem,

    J. Friedman, N. Linial, and M. Saks, “On the second eigenvalue of a graph and a network flow problem,” *Combinatorica*, vol. 11, no. 1, pp. 1–19, 1991

    work page 1991

  8. [7]

    Expander graphs,

    S. Hoory, N. Linial, and A. Wigderson, “Expander graphs,” *Bull. Amer. Math. Soc.*, vol. 43, no. 4, pp. 439–561, 2006

    work page 2006

Show all 43 references
  1. [8]

    Neural tangent kernel: Con- vergence and generalization in neural networks,

    A. Jacot, F. Gabriel, and C. Hongler, “Neural tangent kernel: Con- vergence and generalization in neural networks,” arXiv preprint arXiv:1806.07572, 2020

    2020 arXiv

  2. [9]

    Facing Up to Arrangements: Face-Count Formulas for Partitions of Space by Hyperplanes

    Zaslavsky, Thomas. “Facing Up to Arrangements: Face-Count Formulas for Partitions of Space by Hyperplanes.” (1975)

    1975

  3. [10]

    Understand- ing deep neural networks with rectified linear units,

    R. Arora, A. Basu, P. Mianjy, and A. Mukherjee, “Understand- ing deep neural networks with rectified linear units,” arXiv preprint arXiv:1611.01491, 2018

    2018 arXiv

  4. [11]

    Complexity of linear regions in deep networks,

    B. Hanin and D. Rolnick, “Complexity of linear regions in deep networks,” in *Proc. 36th Int. Conf. Machine Learning (ICML)*, Long Beach, CA, USA, Jun. 2019, pp. 2596–2604

    2019

  5. [12]

    On the expressive power of deep neural networks,

    M. Raghu, B. Poole, J. Kleinberg, S. Ganguli, and J. Sohl-Dickstein, “On the expressive power of deep neural networks,” in *Proc. 34th Int. Conf. Machine Learning (ICML)*, Sydney, Australia, Aug. 2017, pp. 2847–2854

    2017

  6. [13]

    Benefits of depth in neural networks,

    M. Telgarsky, “Benefits of depth in neural networks,” arXiv preprint arXiv:1602.04485, 2016

    2016 arXiv

  7. [14]

    Notes on the number of linear regions of deep neural networks,

    G. Mont ´ufar, “Notes on the number of linear regions of deep neural networks,” unpublished manuscript, Mar. 2017

    2017

  8. [15]

    Balestriero, R. (2021). Max-Affine Splines Insights into Deep Learning (Order No. 28736027). Available from ProQuest Dissertations & Theses Global. (2572635695)

    2021

  9. [16]

    Belkin, D

    M. Belkin, D. Hsu, S. Ma, & S. Mandal, Reconciling modern machine-learning practice and the classical bias–variance trade- off, Proc. Natl. Acad. Sci. U.S.A. 116 (32) 15849-15854, https://doi.org/10.1073/pnas.1903070116 (2019)

    2019 doi

  10. [17]

    Gradient descent on neural networks typically occurs at the edge of stability,

    J. M. Cohen, S. Kaur, Y . Li, J. Z. Kolter, and A. Talwalkar, “Gradient descent on neural networks typically occurs at the edge of stability,” arXiv preprint arXiv:2103.00065, 2022

    2022 arXiv

  11. [18]

    Ongie, A

    G. Ongie, A. Jalal, C. A. Metzler, R. G. Baraniuk, A. G. Dimakis and R. Willett, ”Deep Learning Techniques for Inverse Problems in Imaging,” in IEEE Journal on Selected Areas in Information Theory, vol. 1, no. 1, pp. 39-56, May 2020, doi: 10.1109/JSAIT.2020.2991563

    2020

  12. [19]

    On the information bottleneck theory of deep learning,

    A. Saxe, Y . Bansal, J. Dapello, M. Advani, A. Kolchinsky, B. Tracey, and D. Cox, “On the information bottleneck theory of deep learning,” unpublished manuscript, Feb. 2018

    2018

  13. [20]

    Bianchini and F

    M. Bianchini and F. Scarselli, ”On the Complexity of Neural Network Classifiers: A Comparison Between Shallow and Deep Architectures,” in IEEE Transactions on Neural Networks and Learning Systems, vol. 25, no. 8, pp. 1553-1565, Aug. 2014, doi: 10.1109/TNNLS.2013.2293637

    2014

  14. [21]

    Ramanujan graphs,

    A. Lubotzky, R. Phillips, and P. Sarnak, “Ramanujan graphs,” *Combi- natorica*, vol. 8, no. 3, pp. 261–277, 1988

    1988

  15. [22]

    The secret sharer: Evaluating and testing unintended memorization in neural networks,

    N. Carlini, C. Liu, ´U. Erlingsson, J. Kos, and D. Song, “The secret sharer: Evaluating and testing unintended memorization in neural networks,” in *Proc. 28th USENIX Security Symp.*, Santa Clara, CA, USA, Aug. 2019, pp. 267–284. [Online]. Available: https://www.usenix.org/con...

    2019

  16. [23]

    A Combinatorial Theory of Dropout: Subnetworks, Graph Geometry, and Generalization,

    Dhayalkar, Sahil. (2025) “A Combinatorial Theory of Dropout: Subnetworks, Graph Geometry, and Generalization,” arXiv preprint arXiv:2504.14762

    2025 arXiv

  17. [24]

    Flat minima,

    S. Hochreiter and J. Schmidhuber, “Flat minima,” *Neural Computa- tion*, vol. 9, no. 1, pp. 1–42, Jan. 1997

    1997

  18. [25]

    A PAC-Bayesian ap- proach to spectrally-normalized margin bounds for neural networks,

    B. Neyshabur, S. Bhojanapalli, and N. Srebro, “A PAC-Bayesian ap- proach to spectrally-normalized margin bounds for neural networks,” arXiv preprint arXiv:1707.09564, 2018

    2018 arXiv

  19. [26]

    Spectrally-normalized margin bounds for neural networks,

    P. Bartlett, D. J. Foster, and M. Telgarsky, “Spectrally-normalized margin bounds for neural networks,” arXiv preprint arXiv:1706.08498, 2017

    2017 arXiv

  20. [27]

    Distance-based regularisa- tion of deep networks for fine-tuning,

    H. Gouk, T. M. Hospedales, and M. Pontil, “Distance-based regularisa- tion of deep networks for fine-tuning,” arXiv preprint arXiv:2002.08253, 2021

    2002 arXiv

  21. [28]

    Visualizing the loss landscape of neural nets,

    H. Li, Z. Xu, G. Taylor, C. Studer, and T. Goldstein, “Visualizing the loss landscape of neural nets,” arXiv preprint arXiv:1712.09913, 2018

    2018 arXiv

  22. [29]

    Entropy-SGD: Biasing gradient descent into wide valleys,

    P. Chaudhari, A. Choromanska, S. Soatto, Y . LeCun, C. Baldassi, C. Borgs, J. Chayes, L. Sagun, and R. Zecchina, “Entropy-SGD: Biasing gradient descent into wide valleys,” *J. Stat. Mech. Theory Exp.*, vol. 2019, no. 12, p. 124018, Dec. 2019

    2019

  23. [31]

    Deep learning and the information bottleneck principle,

    N. Tishby and N. Zaslavsky, “Deep learning and the information bottleneck principle,” arXiv preprint arXiv:1503.02406, 2015

    2015 arXiv

  24. [32]

    Alessandro Achille and Stefano Soatto.Emergence of invariance and disentanglement in deep representations.Journal of Machine Learning Research, 19(50):1–34, 2018

    2018

  25. [33]

    Neural networks and principal component analysis: Learning from examples without local minima,

    P. Baldi and K. Hornik, “Neural networks and principal component analysis: Learning from examples without local minima,” *Neural Networks*, vol. 2, no. 1, pp. 53–58, 1989

    1989

  26. [34]

    Exponential expressivity in deep neural networks through transient chaos,

    B. Poole, S. Lahiri, M. Raghu, J. Sohl-Dickstein, and S. Ganguli, “Exponential expressivity in deep neural networks through transient chaos,” arXiv preprint arXiv:1606.05340, 2016

    2016 arXiv

  27. [35]

    Towards understanding the role of over-parametrization in generalization of neural networks,

    B. Neyshabur, Z. Li, S. Bhojanapalli, Y . LeCun, and N. Srebro, “Towards understanding the role of over-parametrization in generalization of neural networks,” arXiv preprint arXiv:1805.12076, 2018

    2018 arXiv

  28. [36]

    On the number of variables to use in principal component regression,

    J. Xu and D. Hsu, “On the number of variables to use in principal component regression,” in *Advances in Neural Information Processing Systems (NeurIPS)*, vol. 32, 2019

    2019

  29. [37]

    Neural tangents: Fast and easy infinite neural networks in python,

    R. Novak, L. Xiao, J. Hron, J. Lee, A. A. Alemi, J. Sohl-Dickstein, and S. S. Schoenholz, “Neural tangents: Fast and easy infinite neural networks in python,” arXiv preprint arXiv:1912.02803, 2019

    1912 arXiv

  30. [38]

    Geometric deep learning: Going beyond Euclidean data,

    M. M. Bronstein, J. Bruna, Y . LeCun, A. Szlam, and P. Vandergheynst, “Geometric deep learning: Going beyond Euclidean data,” *IEEE Signal Process. Mag.*, vol. 34, no. 4, pp. 18–42, Jul. 2017

    2017

  31. [39]

    Semi-supervised classification with graph convolutional networks,

    T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” arXiv preprint arXiv:1609.02907, 2017

    2017 arXiv

  32. [40]

    Shaoyun Shi, Hanxiong Chen, Weizhi Ma, Jiaxin Mao, Min Zhang, Yongfeng Zhang. 2020. Neural Logic Reasoning. In Proceedings of the 29th ACM International Conference on Information and Knowledge Management (CIKM’20)

    2020

  33. [41]

    Interpreting blackbox models via model extraction,

    O. Bastani, C. Kim, and H. Bastani, “Interpreting blackbox models via model extraction,” arXiv preprint arXiv:1705.08504, 2019

    2019 arXiv

  34. [42]

    Interpretable deep learning under fire,

    X. Zhang, N. Wang, H. Shen, S. Ji, X. Luo, and T. Wang, “Interpretable deep learning under fire,” arXiv preprint arXiv:1812.00891, 2019

    2019 arXiv

  35. [43]

    PyTorch: An imperative style, high- performance deep learning library,

    A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “PyTorch: An imperative style, high- p...

    2019

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.