Pith. sign in

REVIEW 5 major objections 4 minor 52 references

The impact of allocation strategies in subset learning on the expressive power of neural networks

T0 review · 5 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A fixed budget of learnable weights can be maximally expressive or completely useless depending only on how the weights are spread across rows and columns.

desk verdict A clean benchmark and solid linear results, but the main maximality theorem for recurrent weights has a real, likely repairable proof gap. read the letter →

arxiv 2502.06300 v1 pith:KWETOWKZ submitted 2025-02-10 cs.LG

Ofir Schlisselberg , Ran Darshan This is my paper · ORCID
classification cs.LG
keywords allocationstrategysubsetlearningexpressivepowerteacher-studentsetupmatchprobabilitylinearrecurrentneuralnetworkparameter-efficient
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks a resource-constrained question: if only r of a network's weights can be learned and the rest stay frozen at random values, which choices of learnable positions maximize the chance that the student exactly reproduces a teacher's outputs on m samples? It answers with a benchmark, the match probability MP(A), and a set of conditions in linear recurrent and feedforward networks. The central finding is that allocations can be maximal (match probability 1, exact reproduction) or minimal (match probability 0) depending only on how the r weights are spread across rows and columns, with a sharp transition between the two. The paper also argues that, as a rule, distributing the same fixed budget over more neurons raises expressive power, and it gives empirical evidence for this in shallow ReLU networks.

What carries the argument

The machinery is the student-teacher match probability MP(A,m), defined as the probability that a student whose learnable positions are fixed by allocation A and whose other weights are random can exactly match the teacher's outputs on m samples. The proof engine is Lemma 3.3, which rewrites the degree-T polynomial matching equations as a linear-quadratic system in the recurrent weights W and auxiliary variables F; when whole rows or columns of W are frozen, the auxiliary equations become linear. Counting the ranks of these linear systems with Kronecker products (the tensor product operation that turns matrix equations into ordinary linear systems) shows when F is forced to a unique value, after which the learnable blocks in W are determined by square linear systems.

What would settle it

Take a small linear recurrent network satisfying the hypotheses of Theorem 3.5, form the coefficient matrices of equations (B.5) and (B.6) from the proof, and compute the rank of the stacked system: if the rank is less than $nTb$, the unique-solution step fails and a maximal allocation could still miss the teacher on a set of positive probability.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that in a linear recurrent network, learning the recurrent matrix W with r = dm learnable weights is maximal whenever each active row of W contains exactly Tb learnable weights, or each active column contains exactly Td learnable weights, and no allocation with any row above Tb or any column above Td can match the teacher. This structure turns a system of high-degree polynomial equations into a forced linear system, so a student with only r learnable weights reproduces all m teacher outputs with probability 1. The same block-counting logic gives exact conditions for the decoder (m weights per row), the encoder (no row above Tm and no column above Td), and intermediate feedforward layers (no row above m and no column above d).

Load-bearing premise

The maximality proof adds up the number of independent equations in two separate linear systems and assumes no equation is shared between them; if the two systems overlap, the unique-solution argument collapses.

Editorial extensions

If this is right

  • A fixed budget r that concentrates too many learnable weights in one row or column makes the student unable to match the teacher at all, whereas the same budget spread one block per row or column succeeds with probability 1.
  • The decoder result is rigid: in a linear RNN readout, every row must have exactly m learnable weights, so moving a single weight between rows flips the match probability from 1 to 0.
  • For recurrent connections, maximality is reached at the minimal allowed number of active rows, r/(Tb), so the transition from zero to full expressivity is sharp at that threshold.
  • In linear feedforward networks, the same row and column conditions hold per layer, with all allocations that violate them being minimal.
  • If the paper's conjecture about random polynomial systems is right, larger networks should make even suboptimal allocations approach match probability 1, so resource constraints become less damaging as model size grows.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable prediction follows for neuroscience: if the brain has a fixed budget of synaptic changes, behavioral competence should show near-threshold transitions as the fraction of modified neurons crosses a row or column bound, rather than smooth improvement.
  • The row and column bounds could give a first-order estimate for parameter-efficient fine-tuning: when a constrained update fits data exactly, it should be visible as a square linear system after the same auxiliary-variable reduction used here.
  • Because match probability only asks whether a solution exists, not whether gradient descent finds it, the practical gain from spreading weights may be smaller than the theory suggests; the gap between exact solvability and trainability is a natural next benchmark.
  • A concrete extension of the paper's conjecture would hold r fixed and vary only the number of rows used by the allocation: match probability should increase monotonically with the row count in nonlinear networks as well.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. This paper studies subset learning: given a network and a fixed budget r of learnable weights, which allocation of those weights maximizes expressive power? The authors define a teacher–student benchmark, match probability, and evaluate it at m = r/d so that the number of output constraints equals the number of learnable weights. They prove exact maximal/minimal allocation conditions for a linear estimator, for the encoder, decoder, and recurrent weights of linear RNNs, and for intermediate layers of linear feedforward networks. For allocations that fall between the extremes they propose heuristic principles, which are extended to shallow ReLU networks and tested numerically. The central claim is that spreading the same number of learnable weights over more rows or columns generally increases expressive power.

Significance. If the proofs are completed, this is a valuable new benchmark for subset-learning expressivity. The paper cleanly separates the counting/existence question from optimization, and it makes falsifiable predictions: allocations satisfying Theorem 3.5 should reach match probability 1, while allocations violating Theorem 3.4 should have match probability 0. The numerical experiments and included code strengthen the empirical part. The main theorems, however, depend on appendix arguments that are not yet rigorous; in particular, the proof of Theorem 3.5 has two unproven linear-algebra steps, and the encoder/feedforward theorems rest on an informal lemma. The results are plausible and likely fixable, but the manuscript in its current form does not fully support the central maximality claim.

major comments (5)
  1. [Theorem B.6 (proof of Theorem 3.5)] The proof computes ranks dm and nTb−dm for Equations (B.5) and (B.6) and concludes that the two systems jointly give nTb linearly independent equations in the nTb entries of F. This conclusion requires that the row spaces of the two systems have trivial intersection; individual rank computations do not imply this. The same unsupported step appears in Theorem B.9. Since the uniqueness of F is the first load-bearing step of the maximality proof, Theorem 3.5 is not established by the argument given.
  2. [Theorem B.6, step after Eq. (B.3)] Once F is fixed, the proof substitutes it into W_TOP gB(F) = F_TOP and asserts that each learned row is solvable because there are Tb equations and Tb variables per row. Solvability of a square linear system is not guaranteed by equal dimensions: the Tb×Tb submatrix of gB(F) selected by the learned columns must be shown invertible, or the right-hand side must be shown to lie in its column space. No such generic-invertibility argument is provided. The analogous step in Theorem B.9 has the same gap.
  3. [Lemma B.12] The second direction of Lemma B.12 is not written as a rigorous proof. It uses an induction on the number of distinct entries, but invokes 'uncorrelated' in places that require genuine independence, and it discusses probability-zero events without a measure-theoretic justification. This lemma is the foundation for Theorem B.14 (Theorem 3.2) and Theorem C.1 (Theorem 4.1), so the encoder and feedforward classifications are currently conditional on an unproved statement.
  4. [Theorem B.14 / Lemma B.13] Lemma B.13 shows that correlations between entries of powers of W are either zero or tend to zero as n→∞, whereas Theorem 3.2 is a fixed-n statement asserting MP equal to 0 or 1. Applying Lemma B.12 requires the entries of U_i to be uncorrelated at finite n, which has not been shown. Either a finite-n argument or an explicitly asymptotic reformulation of Theorem 3.2 is needed.
  5. [Theorem B.5 (first part of Theorem 3.4)] In the reduction from a row with ri>Tb learnable weights to a smaller allocation, the proof solves for the remaining Tb learnable weights by inverting the selected Tb×Tb submatrix of gB(F*). Invertibility of this submatrix is asserted without proof, and the relation between the constants in the original and reduced allocations is not specified. The claim that every solvable instance of A yields a solvable instance of A2 therefore does not follow as written.
minor comments (4)
  1. [Theorem 4.1] The phrase 'follows one of the following' should likely be 'satisfies both of the following'; under the literal OR reading, an allocation with no row exceeding m but one column exceeding d would be claimed maximal, which contradicts the proof in Theorem C.1.
  2. [Theorem 3.5] The theorem statement should explicitly note that condition 1 requires r to be divisible by Tb, and condition 2 requires r to be divisible by Td, for the respective clause to be applicable.
  3. [Appendix E.1] The choice of the number of fsolve initializations is described as based on a manually observed threshold; this is acceptable as a heuristic, but it should be labeled as such in the main text.
  4. [Definition 2.2] The complement of an allocation is written as an overline on A, which is easy to confuse with a closure; a more explicit symbol such as A^c would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the derivation chain is self-contained against an external benchmark; the appendix rank-sum gap is a proof flaw, not a circular step.

full rationale

The paper's central benchmark, match probability, is an external probability over random teacher weights, fixed weights, and samples; maximality is not defined as the paper's allocation conditions but as MP=1. The theorems are derived by exhibiting solutions to linear or polynomial systems (e.g., Lemma 3.3 proves equivalence of matching with equations (B.1)-(B.2), and Theorem B.6 attempts a constructive rank argument), rather than by assuming the conclusion. Heuristic principles for suboptimal allocations are explicitly labeled conjectures and are tested numerically, not fitted and then reported as predictions. The only self-citations (e.g., Kim et al. 2023 in the neuroscience motivation and Appendix F) are motivational and not used to establish any theorem. The appendix does contain a genuine proof gap: Theorem B.6/B.9 infer joint linear independence of two equation blocks from individual rank computations without proving the row spaces intersect trivially, and the later 'Tb equations, Tb variables' solvability claim needs an invertibility argument. This is a technical correctness concern, not circular reasoning, since no load-bearing step reduces to its own input by construction.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

The central claims rest on generic-position assumptions about Gaussian weights and inputs, the choice m = r/d as the benchmark operating point, and two unproven technical steps (joint full-rank of the F subsystems and the basis lemma for Kronecker products). None of these are fitted to data.

assumptions (6)
  • domain assumption Weights, teacher weights, and samples are drawn from continuous uncorrelated distributions such that matrices are invertible and diagonalizable with probability 1.
    Stated in Section 2 'Assumptions'; needed for generic position in the linear algebra and for Lemma B.4.
  • domain assumption Any m samples and any m teacher output vectors are linearly independent.
    Stated in Section 2; ensures square input matrices are invertible, as used in the warm-up and in Theorem B.6.
  • ad hoc to paper The benchmark is evaluated only at m = r/d, so the number of output constraints equals the number of learnable weights.
    Section 2 states 'In this paper we investigate MP(A, r/d)'; all theorems and experiments use this operating point, making the counting arguments give r = dm equations.
  • standard math For polynomial systems with Gaussian coefficients, zero-measure sets imply unsolvability with fewer than dm variables.
    Invoked in Lemma B.4 via Azas and Wschebor Theorem 6.8; the applicability to these specific polynomial systems is asserted.
  • ad hoc to paper The two linear subsystems in F (equations B.5 and B.6) have disjoint row spaces, so their ranks add to nTb.
    Used in the proof of Theorem B.6 (and similarly B.9) to conclude a unique solution for F; not proven.
  • ad hoc to paper Lemma B.12: stacked outer-product matrices form a basis almost surely under the stated row and column occurrence bounds.
    Central to Theorem 3.2 and Theorem 4.1; the proof's second direction relies on informal null-set and subspace arguments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The impact of allocation strategies in subset learning on the expressive power of neural networks." pith.science (2026). https://pith.science/paper/KWETOWKZ

@misc{pith2026250206300,
  author       = {Pith},
  title        = {Pith review of: The impact of allocation strategies in subset learning on the expressive power of neural networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KWETOWKZ}},
  note         = {Machine review of arXiv:2502.06300}
}
read the original abstract

In traditional machine learning, models are defined by a set of parameters, which are optimized to perform specific tasks. In neural networks, these parameters correspond to the synaptic weights. However, in reality, it is often infeasible to control or update all weights. This challenge is not limited to artificial networks but extends to biological networks, such as the brain, where the extent of distributed synaptic weight modification during learning remains unclear. Motivated by these insights, we theoretically investigate how different allocations of a fixed number of learnable weights influence the capacity of neural networks. Using a teacher-student setup, we introduce a benchmark to quantify the expressivity associated with each allocation. We establish conditions under which allocations have maximal or minimal expressive power in linear recurrent neural networks and linear multi-layer feedforward networks. For suboptimal allocations, we propose heuristic principles to estimate their expressivity. These principles extend to shallow ReLU networks as well. Finally, we validate our theoretical findings with empirical experiments. Our results emphasize the critical role of strategically distributing learnable weights across the network, showing that a more widespread allocation generally enhances the network's expressive power.

Figures

Figures reproduced from arXiv: 2502.06300 by the authors.

Figure 1
Figure 1. Schema of the student-teacher setup. The match probability (MP) estimates the expressive [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. a. Estimation of MP for allocations in the recurrent weights of LRNN with [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Estimation of MP for FFNs. MP increases when distributing the weights. a. Estimation [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 28 canonical work pages

  1. [1]

    Thirst regulates motivated behavior through modulation of brainwide neural population dynamics

    William E Allen, Michael Z Chen, Nandini Pichamoorthy, Rebecca H Tien, Marius Pachitariu, Liqun Luo, and Karl Deisseroth. Thirst regulates motivated behavior through modulation of brainwide neural population dynamics. Science, 364 0 (6437): 0 eaav3932, 2019

    work page 2019

  2. [2]

    Level sets and extrema of random processes and fields

    Jean-Marc Azas and Mario Wschebor. Level sets and extrema of random processes and fields. Level Sets and Extrema of Random Processes and Fields, 01 2009. doi:10.1002/9780470434642

    work page doi:10.1002/9780470434642 2009

  3. [3]

    Recurrent neural networks as versatile tools of neuroscience research

    Omri Barak. Recurrent neural networks as versatile tools of neuroscience research. Current opinion in neurobiology, 46: 0 1--6, 2017

    work page 2017

  4. [4]

    A map of anticipatory activity in mouse motor cortex

    Tsai-Wen Chen, Nuo Li, Kayvon Daie, and Karel Svoboda. A map of anticipatory activity in mouse motor cortex. Neuron, 94 0 (4): 0 866--879, 2017

    work page 2017

  5. [5]

    On the expressive power of deep learning: A tensor analysis, 2016

    Nadav Cohen, Or Sharir, and Amnon Shashua. On the expressive power of deep learning: A tensor analysis, 2016. URL https://arxiv.org/abs/1509.05009

    arXiv 2016

  6. [6]

    Capacity and trainability in recurrent neural networks, 2017

    Jasmine Collins, Jascha Sohl-Dickstein, and David Sussillo. Capacity and trainability in recurrent neural networks, 2017. URL https://arxiv.org/abs/1611.09913

    arXiv 2017

  7. [7]

    Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition

    Thomas M Cover. Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition. IEEE transactions on electronic computers, 0 (3): 0 326--334, 1965

    work page 1965

  8. [8]

    George V. Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals and Systems, 2: 0 303--314, 1989. URL https://api.semanticscholar.org/CorpusID:3958369

    work page 1989

Show all 52 references
  1. [9]

    Targeted photostimulation uncovers circuit motifs supporting short-term memory

    Kayvon Daie, Karel Svoboda, and Shaul Druckmann. Targeted photostimulation uncovers circuit motifs supporting short-term memory. Nature neuroscience, 24 0 (2): 0 259--265, 2021

    2021

  2. [10]

    Finding structure in time

    Jeffrey L Elman. Finding structure in time. Cognitive science, 14 0 (2): 0 179--211, 1990

    1990

  3. [11]

    Elsayed, Ian Goodfellow, and Jascha Sohl-Dickstein

    Gamaleldin F. Elsayed, Ian Goodfellow, and Jascha Sohl-Dickstein. Adversarial reprogramming of neural networks, 2018. URL https://arxiv.org/abs/1806.11146

    2018 arXiv

  4. [12]

    Adversarial reprogramming revisited, 2022

    Matthias Englert and Ranko Lazic. Adversarial reprogramming revisited, 2022. URL https://arxiv.org/abs/2206.03466

    2022 arXiv

  5. [13]

    Connectivity underlying motor cortex activity during naturalistic goal-directed behavior

    Arseny Finkelstein, Kayvon Daie, Marton Rozsa, Ran Darshan, and Karel Svoboda. Connectivity underlying motor cortex activity during naturalistic goal-directed behavior. bioRxiv, pages 2023--11, 2023

    2023

  6. [14]

    The lottery ticket hypothesis: Finding sparse, trainable neural networks, 2019

    Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks, 2019. URL https://arxiv.org/abs/1803.03635

    2019 arXiv

  7. [15]

    Three unfinished works on the optimal storage capacity of networks

    E Gardner and B Derrida. Three unfinished works on the optimal storage capacity of networks. Journal of Physics A: Mathematical and General, 22 0 (12): 0 1983, jun 1989. doi:10.1088/0305-4470/22/12/004. URL https://dx.doi.org/10.1088/0305-4470/22/12/004

    1983 doi

  8. [16]

    The expressive power of tuning only the normalization layers, 2023

    Angeliki Giannou, Shashank Rajput, and Dimitris Papailiopoulos. The expressive power of tuning only the normalization layers, 2023. URL https://arxiv.org/abs/2302.07937

    2023 arXiv

  9. [17]

    Rush, and Yoon Kim

    Demi Guo, Alexander M. Rush, and Yoon Kim. Parameter-efficient transfer learning with diff pruning, 2021. URL https://arxiv.org/abs/2012.07463

    2021 arXiv

  10. [18]

    Labelling and optical erasure of synaptic memory traces in the motor cortex

    Akiko Hayashi-Takagi, Sho Yagishita, Mayumi Nakamura, Fukutoshi Shirai, Yi I Wu, Amanda L Loshbaugh, Brian Kuhlman, Klaus M Hahn, and Haruo Kasai. Labelling and optical erasure of synaptic memory traces in the motor cortex. Nature, 525 0 (7569): 0 333--338, 2015

    2015

  11. [19]

    Heij, A.C.M

    C. Heij, A.C.M. Ran, and F. van Schagen. Introduction to Mathematical Systems Theory: Linear Systems, Identification and Control. Birkh \"a user Basel, 2006. ISBN 9783764375492. URL https://books.google.co.il/books?id=NJ_5Enwpk58C

    2006

  12. [20]

    Neural networks and physical systems with emergent collective computational abilities

    John J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79 0 (8): 0 2554--2558, 1982

    1982

  13. [21]

    Horn and Charles R

    Roger A. Horn and Charles R. Johnson. Topics in Matrix Analysis. Cambridge University Press, 1991

    1991

  14. [22]

    Multilayer feedforward networks are universal approximators

    Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural Networks, 2 0 (5): 0 359--366, 1989. ISSN 0893-6080. doi:https://doi.org/10.1016/0893-6080(89)90020-8. URL https://www.sciencedirect.com/science/article/pii...

    1989

  15. [23]

    The next generation of approaches to investigate the link between synaptic plasticity and learning

    Yann Humeau and Daniel Choquet. The next generation of approaches to investigate the link between synaptic plasticity and learning. Nature neuroscience, 22 0 (10): 0 1536--1543, 2019

    2019

  16. [24]

    Bci learning phenomena can be explained by gradient-based optimization

    Peter C Humphreys, Kayvon Daie, Karel Svoboda, Matthew Botvinick, and Timothy P Lillicrap. Bci learning phenomena can be explained by gradient-based optimization. bioRxiv, pages 2022--12, 2022

    2022

  17. [25]

    Joshi, Cristian Bodnar, Simon V

    Chaitanya K. Joshi, Cristian Bodnar, Simon V. Mathis, Taco Cohen, and Pietro Liò. On the expressive power of geometric graph neural networks, 2024. URL https://arxiv.org/abs/2301.09308

    2024 arXiv

  18. [26]

    Expressive power of recurrent neural networks, 2018

    Valentin Khrulkov, Alexander Novikov, and Ivan Oseledets. Expressive power of recurrent neural networks, 2018. URL https://arxiv.org/abs/1711.00811

    2018 arXiv

  19. [27]

    Kim, Arseny Finkelstein, Carson C

    Christopher M. Kim, Arseny Finkelstein, Carson C. Chow, Karel Svoboda, and Ran Darshan. Distributing task-related neural activity across a cortical network through task-independent connections. Nature Communications, 14 0 (1): 0 2851, May 2023. ISSN 2041-1723. doi:10.1038/s414...

    2023 doi

  20. [28]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. URL https://arxiv.org/abs/1412.6980

    2017 arXiv

  21. [29]

    Proving the lottery ticket hypothesis: Pruning is all you need, 2020

    Eran Malach, Gilad Yehudai, Shai Shalev-Shwartz, and Ohad Shamir. Proving the lottery ticket hypothesis: Pruning is all you need, 2020. URL https://arxiv.org/abs/2002.00585

    2020 arXiv

  22. [30]

    Packnet: Adding multiple tasks to a single network by iterative pruning, 2018

    Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning, 2018. URL https://arxiv.org/abs/1711.05769

    2018 arXiv

  23. [31]

    Piggyback: Adapting a single network to multiple tasks by learning to mask weights, 2018

    Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting a single network to multiple tasks by learning to mask weights, 2018. URL https://arxiv.org/abs/1801.06519

    2018 arXiv

  24. [32]

    Cortical layer--specific critical dynamics triggering perception

    James H Marshel, Yoon Seok Kim, Timothy A Machado, Sean Quirin, Brandon Benson, Jonathan Kadmon, Cephra Raja, Adelaida Chibukhchyan, Charu Ramakrishnan, Masatoshi Inoue, et al. Cortical layer--specific critical dynamics triggering perception. Science, 365 0 (6453): 0 eaaw5202, 2019

    2019

  25. [33]

    Synaptic plasticity and memory: an evaluation of the hypothesis

    Stephen J Martin, Paul D Grimwood, and Richard GM Morris. Synaptic plasticity and memory: an evaluation of the hypothesis. Annual review of neuroscience, 23 0 (1): 0 649--711, 2000

    2000

  26. [34]

    On the number of linear regions of deep neural networks, 2014

    Guido Montúfar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the number of linear regions of deep neural networks, 2014. URL https://arxiv.org/abs/1402.1869

    2014 arXiv

  27. [35]

    Partial observation can induce mechanistic mismatches in data-constrained models of neural dynamics

    William Qian, Jacob A Zavatone-Veth, Benjamin S Ruben, and Cengiz Pehlevan. Partial observation can induce mechanistic mismatches in data-constrained models of neural dynamics. bioRxiv, pages 2024--05, 2024

    2024

  28. [36]

    On the expressive power of deep neural networks, 2017

    Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl-Dickstein. On the expressive power of deep neural networks, 2017. URL https://arxiv.org/abs/1606.05336

    2017 arXiv

  29. [37]

    Targeted activation of hippocampal place cells drives memory-guided spatial behavior

    Nick TM Robinson, Lucie AL Descamps, Lloyd E Russell, Moritz O Buchholz, Brendan A Bicknell, Georgy K Antonov, Joanna YN Lau, Rebecca Nutbrown, Christoph Schmidt-Hieber, and Michael H \"a usser. Targeted activation of hippocampal place cells drives memory-guided spatial behavi...

    2020

  30. [38]

    An analytical theory of curriculum learning in teacher-student networks

    Luca Saglietti, Stefano Mannelli, and Andrew Saxe. An analytical theory of curriculum learning in teacher-student networks. Advances in Neural Information Processing Systems, 35: 0 21113--21127, 2022

    2022

  31. [39]

    Saxe, James L

    Andrew M. Saxe, James L. McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks, 2014. URL https://arxiv.org/abs/1312.6120

    2014 arXiv

  32. [40]

    Overcoming catastrophic forgetting with hard attention to the task, 2018

    Joan Serrà, Dídac Surís, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task, 2018. URL https://arxiv.org/abs/1801.01423

    2018 arXiv

  33. [41]

    Siegelmann and E.D

    H.T. Siegelmann and E.D. Sontag. On the computational power of neural nets. Journal of Computer and System Sciences, 50 0 (1): 0 132--150, 1995. ISSN 0022-0000. doi:https://doi.org/10.1006/jcss.1995.1013. URL https://www.sciencedirect.com/science/article/pii/S0022000085710136

    1995

  34. [42]

    Mathematical problems for the next century

    Steve Smale. Mathematical problems for the next century. The Mathematical Intelligencer, 20 0 (2a): 0 7--15, Mar 1998. ISSN 0343-6993. doi:10.1007/BF03025291. URL https://doi.org/10.1007/BF03025291

    1998 doi

  35. [43]

    Distributed coding of choice, action and engagement across the mouse brain

    Nicholas A Steinmetz, Peter Zatka-Haas, Matteo Carandini, and Kenneth D Harris. Distributed coding of choice, action and engagement across the mouse brain. Nature, 576 0 (7786): 0 266--273, 2019

    2019

  36. [44]

    Concentration for the zero set of large random polynomial systems, 2024

    Eliran Subag. Concentration for the zero set of large random polynomial systems, 2024. URL https://arxiv.org/abs/2303.11924

    2024 arXiv

  37. [45]

    Training neural networks with fixed sparse masks, 2021

    Yi-Lin Sung, Varun Nair, and Colin Raffel. Training neural networks with fixed sparse masks, 2021. URL https://arxiv.org/abs/2111.09839

    2021 arXiv

  38. [46]

    Spdf: sparse pre-training and dense fine-tuning for large language models

    Vithursan Thangarasa, Abhay Gupta, William Marshall, Tianda Li, Kevin Leong, Dennis DeCoste, Sean Lie, and Shreyas Saxena. Spdf: sparse pre-training and dense fine-tuning for large language models. In Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Inte...

    2023

  39. [47]

    Optical interrogation of multi-scale neuronal plasticity underlying behavioral learning

    Shinichiro Tsutsumi and Akiko Hayashi-Takagi. Optical interrogation of multi-scale neuronal plasticity underlying behavioral learning. Current Opinion in Neurobiology, 67: 0 8--15, 2021

    2021

  40. [48]

    A comprehensive survey of continual learning: Theory, method and application, 2024

    Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application, 2024. URL https://arxiv.org/abs/2302.00487

    2024 arXiv

  41. [49]

    Supermasks in superposition, 2020

    Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, and Ali Farhadi. Supermasks in superposition, 2020. URL https://arxiv.org/abs/2006.14769

    2020 arXiv

  42. [50]

    Adahessian: An adaptive second order optimizer for machine learning

    Zhewei Yao, Amir Gholami, Sheng Shen, Kurt Keutzer, and Michael W Mahoney. Adahessian: An adaptive second order optimizer for machine learning. AAAI (Accepted), 2021

    2021

  43. [51]

    Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models, 2022

    Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models, 2022. URL https://arxiv.org/abs/2106.10199

    2022

  44. [52]

    The expressive power of low-rank adaptation, 2024

    Yuchen Zeng and Kangwook Lee. The expressive power of low-rank adaptation, 2024. URL https://arxiv.org/abs/2310.17513

    2024 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.