Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

ResNets Are Deeper Than You Think

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Residual connections put networks in a different function space, and the paper argues this is why they keep beating feedforward nets even when training is no longer the bottleneck.

desk verdict A clever post-training comparison suggests variable-depth nets have a real advantage, but the headline claim of inductive bias beyond optimization rests on an untested assumption about the two extraction procedures; worth a serious review with requests for more evidence. read the letter →

arxiv 2506.14386 v1 pith:LTVRAKBX submitted 2025-06-17 cs.LG cs.AI

Christian H.X. Ali Mehmeti-Göpel , Michael Wand This is my paper · ORCID
classification cs.LGcs.AI
keywords residualconnectionstrainabilityinductivebiasfunctionspacevariable-depthnetworkspartiallinearizationnormalizedaveragepathlengthgeneralizationgap
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that residual connections are not just a training aid. Its central claim is that a residual network lives in a different function space from a feedforward network of the same width and depth, and that this difference carries an inductive bias that matches natural data. On the evidence side, the authors take a pretrained feedforward network, then in a short post-training phase linearize some of its nonlinear units either per channel (producing variable-depth networks with a mixture of long and short paths, like a ResNet) or per layer (producing fixed-depth networks). They find that the variable-depth extraction beats the fixed-depth extraction at matched average nonlinear depth, even though both start from the same converged model and differ only in the shape constraint. The upshot is that the long-standing performance gap between residual and feedforward networks may never be fully closed by better initialization or schedules.

What carries the argument

The load-bearing object is the partial-linearization procedure used to compare function spaces. Starting from a pretrained feedforward network, ReLU units are replaced by PReLU units with slopes $\alpha_i$, and a sparsity penalty $\sum_i |1-\alpha_i|^{0.5}$ pushes some slopes toward 1. Linearizing per channel yields variable-depth networks (each channel may become linear independently, so paths of different lengths emerge), while linearizing per layer yields fixed-depth networks; normalized average path length (NAPL) is used to match the resulting nonlinear depth. The analytical counterpart is the impossibility result for equivalent re-parametrization, which establishes that the two shapes genuinely span different function classes. The empirical machinery controls for trainability by extracting both shapes from the same converged network, so the only difference is the shape constraint itself.

What would settle it

Take the same pretrained feedforward model, run the channel-wise and layer-wise linearization many times with different random seeds and a much longer post-training phase, or with a solver that provably finds the best linear subset of units, and compare test accuracy at matched NAPL; if the layer-wise extraction ever catches up with the channel-wise one, the variable-depth advantage is an optimization artifact rather than an inductive bias.

Watch

Extended reading notes

Core claim

The discovery is that, once trainability is factored out, variable-depth networks still generalize better than fixed-depth networks. The paper proves a small negative result first: with a non-injective nonlinearity and square weight matrices, a residual block $R(x)=\phi(Wx+b)+x$ cannot be re-parametrized as an equivalent feedforward block $F(x)=\phi(Wx+b)$ of the same width and depth, because the feedforward function class cannot represent the identity map. It then shows experimentally that when a pretrained fixed-depth network is molded post-training into either a variable-depth or fixed-depth shape, the variable-depth shape wins on ImageNet, CIFAR-10, and CIFAR-100 for low normalized average path length, and the gap widens as networks get shallower. The authors argue this is an emergent property of optimization rather than an artifact: the extracted variable-depth networks display path-length histograms similar to those of standard ResNets, and the advantage mostly disappears when the same procedure starts from a pretrained residual network.

Load-bearing premise

The load-bearing premise is that the short post-training phase used to linearize networks does not systematically favor one extraction method: after the network has converged, the layer-wise and channel-wise procedures are equally likely to reach the best possible network of their respective shape, so the measured gap is a property of the shape rather than of optimization.

Editorial extensions

If this is right

  • If the claim is right, no amount of initialization tuning, learning-rate scheduling, or gradient shaping should fully close the gap between deep feedforward networks and ResNets, because the gap is not only an optimization effect.
  • Architectures that contain a mixture of long and short paths, with residual connections being one way to get them, should be preferred for natural data with multi-scale structure, and the advantage should grow as the target depth shrinks.
  • The advantage should transfer to new datasets roughly in proportion to dataset difficulty: the paper observes a lower break-even NAPL on easier datasets, so harder datasets should show a wider regime where variable-depth beats fixed-depth.
  • Post-training linearization gives a practical way to probe a network's effective depth and to compare architectures while holding optimization roughly constant.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: one could test the inductive-bias story directly by training from scratch architectures that are deliberately shaped as variable-depth networks without residual skip connections, for example by pruning channels to linearity during training; the paper's account predicts they would retain a generalization edge over fixed-depth networks of the same NAPL.
  • Beyond the paper: the channel-wise extraction result suggests model-compression techniques that convert nonlinear units to linear ones per channel may preserve accuracy better than layer-wise folding, because they retain variable-depth structure; that is a testable engineering consequence.
  • Beyond the paper: if the inductive bias is truly aligned with natural data, synthetic data with controlled scale structure, such as mixtures of low- and high-frequency components, should show a tunable version of the gap, with the break-even NAPL shifting with the depth of the data's own hierarchy.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper argues that residual connections confer performance advantages beyond optimization by placing networks in a different function space: variable-depth networks (containing both short and long paths) rather than fixed-depth feedforward networks. It first gives an analytical argument (Section 3 and Appendix A) that residual and feedforward blocks of the same width and depth are not reparametrizations of each other, since a residual block can represent the identity while a feedforward block with a non-injective nonlinearity cannot. The main empirical contribution is a post-training linearization experiment: starting from a pretrained RepVGG-A2 feedforward network, the authors regularize PReLU slopes toward 1 either channel-wise (allowing some channels to become linear and creating variable-depth subnetworks) or layer-wise (forcing whole layers to become linear and creating fixed-depth subnetworks). They compare test accuracy at matched normalized average path length (NAPL) on ImageNet, CIFAR-10, and CIFAR-100, and find that the channel-wise (variable-depth) networks outperform the layer-wise (fixed-depth) ones at low NAPL. This gap largely disappears when the starting architecture is a ResNet, which the authors interpret as evidence that the advantage is due to variable-depth structure rather than to other artifacts.

Significance. If the central claim holds, the paper would reframe the role of skip connections in deep learning: rather than merely improving trainability, residual connections impose an inductive bias aligned with natural data. The experimental protocol is creative and includes several thoughtful controls, such as parameter-count comparisons (Appendices B.3-B.5), alternative nonlinearity measures (Appendix B.2), and a ResNet starting-point control (Section 4.3). The analytical argument in Section 3 is simple but correct for the stated restricted setting. The paper also ships code and training details in the appendix, which supports reproducibility. The main risk is that the interpretation rests on an untested assumption about the equivalence of optimization difficulty between the two extraction procedures; if that assumption fails, the observed gap could be an optimization artifact rather than evidence for an inductive bias.

major comments (3)
  1. [Section 4.5 and Appendices B.4-B.5] The central claim that the observed gap reflects an inductive bias rather than an optimization artifact depends on the assumption, stated in Section 4.5, that the short post-training phases for channel-wise and layer-wise linearization are unlikely to differ in their final generalization because both start from a converged network. This assumption is load-bearing and currently unsupported. The layer-wise procedure imposes a much more rigid constraint (one slope per layer vs. one per channel), so at matched NAPL it may simply be harder to optimize; the loss landscapes of the two procedures are not the same. The controls in Appendix B.4 and B.5 do not resolve this: B.4 releases the layer-wise constraint only after the layer-wise solution has been found, so it cannot reveal whether a better layer-wise basin exists, and B.5 adds parameters while keeping the same rigid per-layer constraint during optimization. Please provide convergence diagnostics (training loss and validation curves for both procedures), multiple restarts, and ideally a within-optimization control in which the layer-wise constraint is relaxed during training (e.g., a run initialized from the channel-wise solution but trained under the layer-wise objective, or vice versa). Without such evidence, the interpretation as an inductive bias is not uniquely supported.
  2. [Figure 1 (left)] The ImageNet results, which are the headline comparison, are reported from a single run without error bars. Given that the paper makes a strong claim about a consistent performance gap, the ImageNet panel should be supported by at least a few independent seeds or by a clear statement of run-to-run variability, for example by reporting the spread observed in the CIFAR-100 repeated runs. As it stands, the ImageNet curve could be consistent with run-to-run noise at some NAPL values, particularly where the gap is small.
  3. [Section 3 and Appendix A.1] The analytical argument establishes that the identity function is representable by a residual block but not by a feedforward block with the same width, depth, and a non-injective elementwise nonlinearity. This is a valid existence proof, but it does not by itself predict the magnitude or even the sign of a generalization gap on natural data. The paper already phrases this as making the gap 'plausible,' which is appropriate, but the discussion in Sections 5 and 6 should more clearly separate this possibility argument from the empirical evidence, so that readers do not mistake the theory for a direct explanation of the measured accuracy differences.
minor comments (6)
  1. [Appendix A.1, Proposition 2 proof] The proof says that applying Lemma 1 to the first layer 'follows immediately,' but it should explicitly state that composing a non-injective first layer with arbitrary subsequent layers preserves non-injectivity; this is a trivial but necessary step.
  2. [Section 4.2] The phrase 'starting for networks with NAPL under 12' should be 'starting from networks with NAPL under 12'.
  3. [Section 4 and Figure 2 caption] There are typos: 'vey similar' in Section 4 and 'stating from a ResNet56' in the Figure 2 caption should be 'very similar' and 'starting from a ResNet56', respectively.
  4. [References] Several references are duplicated with identical titles and URLs: Saxe et al. 2014a/2014b, Yang et al. 2019a/2019b, and Mingard et al. 2021a/2021b. These should be disambiguated or merged.
  5. [Section 4.4] The text refers to 'Figure 4.4' when discussing the path-length histograms, but the figure is numbered as Figure 3 in the paper; the cross-reference is incorrect.
  6. [NeurIPS Checklist, item 5] The checklist states that full code and launching scripts are provided, but no URL or repository link appears in the manuscript; for a journal submission, please include a link or a statement of when the code will be made publicly available.

Circularity Check

0 steps flagged · score 0.0 of 10

No load-bearing circularity found; the empirical gap is observed, not derived from the hypothesis, and self-citations are instrumental rather than argumentative.

full rationale

The paper's central empirical claim—that variable-depth architectures retain a generalization advantage over fixed-depth networks even when optimization differences are minimized—rests on a controlled post-training experiment, not on an equation that presupposes the conclusion. No fitted parameter is renamed as a prediction, and no quantity appearing in the loss or in the NAPL matching is constructed from the test accuracy gap being explained. The theoretical function-space argument (Appendix A.1, Propositions 1–3) is independent of the experiments: it shows non-equivalence of residual and feedforward function classes under stated assumptions, without assuming the empirical performance gap. Self-citations to Ali Mehmeti-Göpel and Disselhoff [2023] and Ali Mehmeti-Göpel and Wand [2024] are used to import a linearization technique and prior evidence of residual/feedforward gaps; they do not supply the paper's conclusion as a premise. The acknowledged limitation in Section 4.5—that there is no guarantee the post-training extraction reaches the best possible test accuracy for the target shape, with the authors arguing optimization is 'unlikely to make a difference'—is an unverified optimization-equivalence assumption and a legitimate external-validity concern, but it is not circular: the assumption does not restate the inductive-bias conclusion, and Appendix B.4 and B.5 provide control experiments aimed at it. The histograms in Section 4.4 are compared against the external Veit et al. [2016] binomial prediction rather than being used to define the result. Overall, the derivation chain is self-contained and the empirical comparison is an independent observation, so no circular step can be exhibited.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central empirical claim does not rest on fitted constants; it is a measured comparison. The main axioms are methodological assumptions about the post-training linearization and about the composition argument in Proposition 2. The paper contains no invented entities.

free parameters (3)
  • regularization weight omega = swept (values not fully enumerated in text)
    Controls the NAPL of extracted sub-networks; the comparison is made across a range of omega, so no single fitted value is used. The chosen values affect the shape of the curves.
  • PReLU slope quantization threshold = 0.01
    Slopes with |alpha_i - 1| < 0.01 are frozen to 1, determining which channels count as linear; this hand-chosen threshold affects the measured NAPL and the extracted architecture.
  • sparsity penalty exponent = 0.5
    The L0.5 pseudo-norm is chosen to encourage sparsity; another exponent would change the linearization trajectories.
assumptions (4)
  • standard math Taylor expansion of phi at c in Proposition 1 is valid and the epsilon-to-0 limit gives an exact reparametrization
    Requires phi differentiable at c and bounded pre-activations; the authors list these as limitations in Section 3.
  • ad hoc to paper Composition of a non-injective first layer with arbitrary remaining layers is non-injective (unstated in Proposition 2)
    This is the load-bearing step in Appendix A.1; it is generally false without surjectivity of the remaining layers.
  • domain assumption Post-training partial linearization of a converged network has optimization dynamics that do not differentially bias final generalization between channel-wise and layer-wise variants
    Stated in Section 4.5 as 'we argue that optimization is unlikely to make a difference'; the entire empirical interpretation rests on this.
  • domain assumption ReLU reparametrization in Proposition 3 requires lower-bounded pre-activations and extra biases not present in standard ResNets
    Acknowledged in Appendix A.2; limits the generality of the possibility construction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ResNets Are Deeper Than You Think." pith.science (2026). https://pith.science/paper/LTVRAKBX

@misc{pith2026250614386,
  author       = {Pith},
  title        = {Pith review of: ResNets Are Deeper Than You Think},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LTVRAKBX}},
  note         = {Machine review of arXiv:2506.14386}
}
read the original abstract

Residual connections remain ubiquitous in modern neural network architectures nearly a decade after their introduction. Their widespread adoption is often credited to their dramatically improved trainability: residual networks train faster, more stably, and achieve higher accuracy than their feedforward counterparts. While numerous techniques, ranging from improved initialization to advanced learning rate schedules, have been proposed to close the performance gap between residual and feedforward networks, this gap has persisted. In this work, we propose an alternative explanation: residual networks do not merely reparameterize feedforward networks, but instead inhabit a different function space. We design a controlled post-training comparison to isolate generalization performance from trainability; we find that variable-depth architectures, similar to ResNets, consistently outperform fixed-depth networks, even when optimization is unlikely to make a difference. These results suggest that residual connections confer performance advantages beyond optimization, pointing instead to a deeper inductive bias aligned with the structure of natural data.

Figures

Figures reproduced from arXiv: 2506.14386 by the authors.

Figure 1
Figure 1. Comparing the test accuracies of partially linearized networks using a channel-wise or [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Comparing the test accuracies on Cifar100 of partially linearized networks using a channel [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Comparing the histograms of networks extracted via [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Comparing the test accuracies on Ci￾far10 of partially linearized networks using a channel-wise or layer-wise partial linearization approach [PITH_FULL_IMAGE:figures/full_fig_p017_4.png]
Figure 6
Figure 6. Figure 6: Comparing the test accuracies on Ci￾far100 of partially linearized networks using a channel-wise or layer-wise partial lineariza￾tion approach, using the percentage of disabled units measure [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 9
Figure 9. Figure 9: Comparing the test accuracies on Im￾ageNet of partially linearized networks using a channel-wise or layer-wise partial lineariza￾tion approach, with an additional channel-wise multiplication layer after each nonlinearity to account for missing parameters in the layer-w…
Figure 10
Figure 10. Figure 10: Details of the training regime. C Hardware and Training Details The experiments in the paper were made on computers running Arch Linux, Python 3.13.3, PyTorch Version 2.7.0. The GPUs used were NVIDIA GeForce GTX 1080 Ti, GeForce RTX 2080 Ti and GeForce RTX 4080 Ti. Th…
Figure 11
Figure 11. Figure 11: Details of the post-training regime. NeurIPS Paper Checklist 1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: We claim that ResNets span a different functi…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Differentially Private Natural Gradient Descent

    cs.LG 2026-07 conditional novelty 6.0 of 10

    DP-NGD enables second-order optimization under differential privacy by decoupling curvature estimation onto public data, performing isotropic DP operations in a whitened space, and dynamically clamping curvature eigen...

Reference graph

Works this paper leans on

42 extracted references · 30 canonical work pages · cited by 1 Pith paper

  1. [1]

    Christian H. X. Ali Mehmeti - G \" o pel and Jan Disselhoff. Nonlinear advantage: Trained networks might not be as complex as you think. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , volume...

    work page 2023

  2. [2]

    Christian H. X. Ali Mehmeti - G \" o pel , David Hartmann, and Michael Wand. Ringing relus: Harmonic distortion analysis of nonlinear feedforward networks. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021. URL https://openreview.net/forum?id=TaYhv-q1Xit

    work page 2021

  3. [3]

    Ali Mehmeti-G \"o pel and Michael Wand

    Christian H.X. Ali Mehmeti-G \"o pel and Michael Wand. On the weight dynamics of deep normalized networks. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=AzUCfhJ9Bs

    work page 2024

  4. [4]

    Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy P

    Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean - Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy P. Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael ...

    arXiv doi:10.48550/arxiv.2312.11805 2023

  5. [5]

    David Balduzzi, Marcus Frean, Lennox Leary, J. P. Lewis, Kurt Wan - Duo Ma, and Brian McWilliams. The shattered gradients problem: If resnets are the answer, then what is the question? In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017 , volum...

    work page 2017

  6. [6]

    Bach, Thomas Hofmann, and Aur \' e lien Lucchi

    Hadi Daneshmand, Jonas Moritz Kohler, Francis R. Bach, Thomas Hofmann, and Aur \' e lien Lucchi. Batch normalization provably avoids ranks collapse for randomly initialised deep networks. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria - Florina Balcan, and Hsuan - Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annu...

    work page 2020

  7. [7]

    Soham De and Samuel L. Smith. Batch normalization biases residual blocks towards the identity function in deep networks. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria - Florina Balcan, and Hsuan - Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIP...

    work page 2020

  8. [8]

    Repvgg: Making vgg-style convnets great again

    Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. Repvgg: Making vgg-style convnets great again. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021 , pages 13733--13742. Computer Vision Foundation / IEEE , 2021. doi:10.1109/CVPR46437.2021.01352. URL https://openaccess.thecv...

    arXiv 2021

Show all 42 references
  1. [9]

    Jevnisek

    Amir Ben Dror, Niv Zehngut, Avraham Raviv, Evgeny Artyomov, Ran Vitek, and Roy J. Jevnisek. Layer folding: Neural network depth reduction using activation linearization. CoRR, abs/2106.09309, 2021. URL https://arxiv.org/abs/2106.09309

    2021 arXiv

  2. [10]

    Understanding the difficulty of training deep feedforward neural networks

    Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Yee Whye Teh and D. Mike Titterington, editors, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2010, Chia L...

    2010

  3. [11]

    Delving deep into rectifiers: Surpassing human-level performance on imagenet classification

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015 , pages 1026--1034. IEE...

    2015 doi

  4. [12]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016 , pages 770--778. IEEE Computer Society, 2016. doi:10.1109/C...

    2016 doi

  5. [13]

    Why do deep residual networks generalize better than deep feedforward networks? - A neural tangent kernel perspective

    Kaixuan Huang, Yuqing Wang, Molei Tao, and Tuo Zhao. Why do deep residual networks generalize better than deep feedforward networks? - A neural tangent kernel perspective. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria - Florina Balcan, and Hsuan - Tien Lin, edi...

    2020

  6. [14]

    Decorrelated batch normalization

    Lei Huang, Dawei Yang, Bo Lang, and Jia Deng. Decorrelated batch normalization. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018 , pages 791--800. Computer Vision Foundation / IEEE Computer Society, 2018. ...

    2018

  7. [15]

    Iterative normalization: Beyond standardization towards efficient whitening

    Lei Huang, Yi Zhou, Fan Zhu, Li Liu, and Ling Shao. Iterative normalization: Beyond standardization towards efficient whitening. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019 , pages 4874--4883. Computer Vision ...

    2019

  8. [16]

    Batch normalization: Accelerating deep network training by reducing internal covariate shift

    Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Francis R. Bach and David M. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11...

    2015

  9. [17]

    Highly accurate protein structure prediction with alphafold

    John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Z \' dek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596 0 (7873): 0 583--589, 2021

    2021

  10. [18]

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In Peter L. Bartlett, Fernando C. N. Pereira, Christopher J. C. Burges, L \' e on Bottou, and Kilian Q. Weinberger, editors, Advances in Neural Information...

    2012

  11. [19]

    The llama 3 herd of models

    AI @ Meta Llama Team. The llama 3 herd of models. jul 2024

    2024

  12. [20]

    Dick, and Hidenori Tanaka

    Ekdeep Singh Lubana, Robert P. Dick, and Hidenori Tanaka. Beyond batchnorm: Towards a unified understanding of normalization in deep learning. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural In...

    2021

  13. [21]

    Why batch norm causes exploding gradients

    Kyle Luther. Why batch norm causes exploding gradients. Blog post, 2020. URL https://kyleluther.github.io/2020/02/18/batchnorm-exploding-gradients.html

    2020

  14. [22]

    James Martens and Roger B. Grosse. Optimizing neural networks with kronecker-factored approximate curvature, 2015. URL http://proceedings.mlr.press/v37/martens15.html

    2015

  15. [23]

    Schoenholz

    James Martens, Andy Ballard, Guillaume Desjardins, Grzegorz Swirszcz, Valentin Dalibard, Jascha Sohl - Dickstein, and Samuel S. Schoenholz. Rapid training of deep neural networks without skip connections or normalization layers using deep kernel shaping. CoRR, abs/2110.01765, ...

    arXiv

  16. [24]

    Srinivasan, Matthew Tancik, Jonathan T

    Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan - Michael Frahm, editors, Computer Vision - EC...

    2020 doi

  17. [25]

    Chris Mingard, Guillermo Valle P \' e rez, Joar Skalse, and Ard A. Louis. Is SGD a bayesian sampler? well, almost. J. Mach. Learn. Res., 22: 0 79:1--79:64, 2021 a . URL https://jmlr.org/papers/v22/20-676.html

    2021

  18. [26]

    Chris Mingard, Guillermo Valle P \' e rez, Joar Skalse, and Ard A. Louis. Is SGD a bayesian sampler? well, almost. J. Mach. Learn. Res., 22: 0 79:1--79:64, 2021 b . URL https://jmlr.org/papers/v22/20-676.html

    2021

  19. [27]

    Deep neural networks have an inbuilt occam's razor

    Chris Mingard, Henry Rees, Guillermo Valle-P \'e rez, and Ard A Louis. Deep neural networks have an inbuilt occam's razor. Nat. Commun., 16 0 (1): 0 220, jan 2025

    2025

  20. [28]

    Oyedotun, Kassem Al Ismaeil, and Djamila Aouada

    Oyebade K. Oyedotun, Kassem Al Ismaeil, and Djamila Aouada. Training very deep neural networks: Rethinking the role of skip connections. Neurocomputing, 441: 0 105--117, 2021. doi:10.1016/J.NEUCOM.2021.02.S004. URL https://doi.org/10.1016/j.neucom.2021.02.004

    2021 doi

  21. [29]

    Schoenholz, and Surya Ganguli

    Jeffrey Pennington, Samuel S. Schoenholz, and Surya Ganguli. Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, e...

    2017

  22. [30]

    How does batch normalization help optimization? In Samy Bengio, Hanna M

    Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normalization help optimization? In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicol \` o Cesa - Bianchi, and Roman Garnett, editors, Advances in Neural Information Pr...

    2018

  23. [32]

    Saxe, James L

    Andrew M. Saxe, James L. McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In Yoshua Bengio and Yann LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April...

    2014 arXiv

  24. [33]

    Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl - Dickstein

    Samuel S. Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl - Dickstein. Deep information propagation. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings . OpenReview.net, 2017. URL ht...

    2017

  25. [34]

    Very deep convolutional networks for large-scale image recognition

    Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedin...

    2015 arXiv

  26. [35]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, ...

    2017

  27. [36]

    Wilber, and Serge J

    Andreas Veit, Michael J. Wilber, and Serge J. Belongie. Residual networks behave like ensembles of relatively shallow networks. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, Advances in Neural Information Processing Systems...

    2016

  28. [37]

    Schoenholz, and Jeffrey Pennington

    Lechao Xiao, Yasaman Bahri, Jascha Sohl - Dickstein, Samuel S. Schoenholz, and Jeffrey Pennington. Dynamical isometry and a mean field theory of cnns: How to train 10, 000-layer vanilla convolutional neural networks. In Jennifer G. Dy and Andreas Krause, editors, Proceedings o...

    2018

  29. [38]

    Schoenholz

    Greg Yang and Samuel S. Schoenholz. Mean field residual networks: On the edge of chaos. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: An...

    2017

  30. [39]

    Schoenholz

    Greg Yang, Jeffrey Pennington, Vinay Rao, Jascha Sohl - Dickstein, and Samuel S. Schoenholz. A mean field theory of batch normalization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019 a . URL h...

    2019

  31. [40]

    Schoenholz

    Greg Yang, Jeffrey Pennington, Vinay Rao, Jascha Sohl - Dickstein, and Samuel S. Schoenholz. A mean field theory of batch normalization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019 b . URL h...

    2019

  32. [41]

    Learning strict identity mappings in deep residual networks

    Xin Yu, Zhiding Yu, and Srikumar Ramalingam. Learning strict identity mappings in deep residual networks. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018 , pages 4432--4440. Computer Vision Foundation / I...

    2018

  33. [42]

    Deep learning without shortcuts: Shaping the kernel with tailored rectifiers

    Guodong Zhang, Aleksandar Botev, and James Martens. Deep learning without shortcuts: Shaping the kernel with tailored rectifiers. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022. URL https://...

    2022

  34. [43]

    Dauphin, and Tengyu Ma

    Hongyi Zhang, Yann N. Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019. URL https://openreview.net/forum?id=...

    2019

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.