pith. sign in

arxiv: 1907.01588 · v1 · pith:F5J2KQDFnew · submitted 2019-07-02 · 🧬 q-bio.NC

Reverse engineering neural networks from many partial recordings

Elahe Arani , Sofia Triantafillou , Konrad P. Kording This is my paper

Pith reviewed 2026-05-25 10:12 UTC · model grok-4.3

classification 🧬 q-bio.NC
keywords reverse engineeringpartial recordingsneural networksneuroscienceMNISTinput-output functionscaling analysissimultaneous recording
0
0 comments X

The pith

A neural network can be reverse-engineered from partial recordings of far fewer neurons than the total number.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks whether recovering the input-output function of a neural network requires recording most or all neurons at the same time. It uses an artificial network trained on MNIST as a stand-in for a biological system and measures how well the original mapping can be recovered when only subsets of neurons are observed. The analysis shows that a number of neurons considerably smaller than the full population already allows meaningful recovery. It also finds that repeating the experiment many times on different small random subsets produces surprisingly accurate results. This framing is meant to guide how neuroscientists should scale up recordings when they cannot observe every neuron simultaneously.

Core claim

Reverse engineering of the nonlinear neural network is meaningfully possible if a sufficiently large number of neurons is simultaneously recorded but that this number can be considerably smaller than the number of neurons. Moreover, recording many times from small random subsets of neurons yields surprisingly good performance. Application in neuroscience suggests to approximate the I/O function of an actual neural system, we need to record from a much larger number of neurons. The kind of scaling analysis we perform here can, and arguably should be used to calibrate approaches that can dramatically scale up the size of recorded data sets in neuroscience.

What carries the argument

Scaling analysis of reverse-engineering quality versus the number of simultaneously recorded neurons, using an MNIST-trained artificial neural network as the test system.

If this is right

  • A number of simultaneously recorded neurons considerably smaller than the total already permits meaningful recovery of the network's input-output function.
  • Repeating recordings on many different small random subsets produces performance close to that of larger simultaneous recordings.
  • Approximating the input-output function of a biological neural system requires recording from a much larger number of neurons than is typical today.
  • Scaling analyses of this form should be used to decide how to allocate effort when designing larger-scale recording experiments.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Neuroscientists could design experiments that deliberately sample many overlapping small subsets rather than pursuing simultaneous coverage of the entire population.
  • Optical techniques that repeatedly target chosen subsets of neurons may be sufficient for functional reconstruction if the sampling is dense enough across sessions.
  • The same scaling logic could be applied to other model systems, such as recurrent networks or networks trained on different tasks, to test whether the partial-recording advantage generalizes.

Load-bearing premise

An artificial neural network trained on MNIST provides a representative test case for the challenges of reverse-engineering biological neural systems from partial recordings.

What would settle it

Applying the identical partial-recording protocol to a real neural circuit whose full connectivity and input-output behavior are independently known and finding that the recovered function deviates substantially even when many neurons are recorded.

Figures

Figures reproduced from arXiv: 1907.01588 by Elahe Arani, Konrad P. Kording, Sofia Triantafillou.

Figure 1
Figure 1. Figure 1: An example of three recording subsets. (a) The ground-truth neural network. (b,c,d): An example of three [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Unsurprisingly, having more data leads to better reconstructions. RMSE over 5 iterations as a function of sample size per setting from the first and the third hidden layer of the NN, using ANN-MVI. 1 2 4 8 16 32 64 128 #Observed neurons (Layer 1) 0.0 0.1 0.2 0.3 0.4 RMSE ANN-MP ANN-MVI ANN-SI XGB-MP XGB-MVI XGB-SI XGB-G 1 2 4 8 16 32 64 128 #Observed neurons (Layer 3) 0.0 0.1 0.2 0.3 0.4 RMSE ANN-MP ANN-MV… view at source ↗
Figure 3
Figure 3. Figure 3 [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Simultaneously recording more neurons helps reverse engineering for a fixed number of samples, just as well for shallow and deep networks. Left: RMSE for RE as a function of observed subset size using ANN-MVI in layers 1, 3 of the NN and in layers 1, 3, 6 of the DNN. Right: Performance (as a percentage of maximum decrease in RMSE) vs percentage of observed neurons for each layer. 1 2 4 8 16 32 64 128 1 3 9… view at source ↗
Figure 5
Figure 5. Figure 5: Sequentially recording from many subsets of neurons can compensate for recording only small subsets at a time. RMSE for RE using ANN-MVI for a different number of sequentially recorded subsets K, keeping the total number observed values approximately constant. Results are shown for layers 1, 3 of the NN and 1, 3, 6 of the DNN. with data simulated from a deeper neural network (DNN) as the ground truth netwo… view at source ↗
Figure 6
Figure 6. Figure 6: Noise has little effect but mean value imputation beats soft imputation. RMSE for RE as a function of the observed subset size using ANN-MVI and ANN-SI with various additive noise levels. 1 2 4 8 16 32 64 128 172 # observed neurons 0.054 0.056 0.058 0.060 0.062 RMSE [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Recording from larger subsets of neurons in PMd helps for predicting hand kinematics. RMSE for RE as a function of the observed subset size using XGBoost-MVI on spiking data. The recordings of neural networks include several sources of noise. The effective noise in the recordings has a component that comes from measurement, e.g. Johnson noise in the electrode, and a component that comes from the brain itse… view at source ↗
read the original abstract

Much of neuroscience aims at reverse engineering the brain, but we only record a small number of neurons at a time. We do not currently know if reverse engineering the brain requires us to simultaneously record most neurons or if multiple recordings from smaller subsets suffice. This is made even more important by the development of novel techniques that allow recording from selected subsets of neurons, e.g. using optical techniques. To get at this question, we analyze a neural network, trained on the MNIST dataset, using only partial recordings and characterize the dependency of the quality of our reverse engineering on the number of simultaneously recorded "neurons". We find that reverse engineering of the nonlinear neural network is meaningfully possible if a sufficiently large number of neurons is simultaneously recorded but that this number can be considerably smaller than the number of neurons. Moreover, recording many times from small random subsets of neurons yields surprisingly good performance. Application in neuroscience suggests to approximate the I/O function of an actual neural system, we need to record from a much larger number of neurons. The kind of scaling analysis we perform here can, and arguably should be used to calibrate approaches that can dramatically scale up the size of recorded data sets in neuroscience.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The manuscript analyzes reverse engineering of the input-output function of a feedforward neural network (784-256-10 architecture) trained on MNIST, using only partial recordings of its units. It reports that meaningful reverse engineering is possible when a sufficiently large but sub-total number of units is recorded simultaneously, and that repeated recordings from small random subsets can also yield good performance. The work concludes with a suggestion that similar scaling analyses should be used to calibrate the number of neurons that must be recorded in biological systems.

Significance. If the scaling relations hold beyond the specific architecture tested, the paper supplies a concrete, quantitative example of how partial-recording strategies can be evaluated, which could help experimentalists decide between simultaneous large-population recordings and repeated smaller-subset recordings when approximating neural I/O functions.

major comments (2)
  1. [Methods and Results (simulation setup and scaling experiments)] The load-bearing assumption that results obtained on a static, acyclic, feedforward network with MNIST inputs generalize to biological circuits is not tested. The manuscript contains no ablation or comparison network that introduces recurrence, continuous-time dynamics, or high-dimensional naturalistic inputs whose unobserved variables affect the recorded subset; without such controls the claim that 'recording many times from small random subsets yields surprisingly good performance' may be architecture-specific rather than a general property of nonlinear networks.
  2. [Abstract and Results] The quantitative definition of 'quality of reverse engineering' (including the precise loss or reconstruction metric, error bars, and data-exclusion criteria) is not supplied in sufficient detail to allow independent verification of the reported scaling; the abstract states the main finding but the supporting figures and tables must be checked against these omissions.
minor comments (1)
  1. [Abstract] The abstract would benefit from a one-sentence statement of the network architecture and the exact performance metric used.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments. We address each major point below and indicate the revisions we will make.

read point-by-point responses

  1. Referee: [Methods and Results (simulation setup and scaling experiments)] The load-bearing assumption that results obtained on a static, acyclic, feedforward network with MNIST inputs generalize to biological circuits is not tested. The manuscript contains no ablation or comparison network that introduces recurrence, continuous-time dynamics, or high-dimensional naturalistic inputs whose unobserved variables affect the recorded subset; without such controls the claim that 'recording many times from small random subsets yields surprisingly good performance' may be architecture-specific rather than a general property of nonlinear networks.

    Authors: We agree that the reported results are obtained on one specific feedforward architecture and that no ablations introducing recurrence or continuous-time dynamics were performed. The work is presented as a controlled proof-of-concept on a fully known, acyclic network to isolate the effect of partial recordings. The manuscript does not assert that the scaling relations are universal; it concludes by recommending that analogous scaling analyses be performed on biological data. We will revise the Discussion to explicitly qualify the architectural assumptions and to note that recurrence or unobserved inputs could alter the observed scaling. No new simulations will be added. revision: partial

  2. Referee: [Abstract and Results] The quantitative definition of 'quality of reverse engineering' (including the precise loss or reconstruction metric, error bars, and data-exclusion criteria) is not supplied in sufficient detail to allow independent verification of the reported scaling; the abstract states the main finding but the supporting figures and tables must be checked against these omissions.

    Authors: We will expand the Methods section to state the precise metric (test-set classification accuracy of the recovered network), the loss used during recovery (cross-entropy), how error bars are obtained (standard deviation across five independent random seeds), and the absence of any data-exclusion criteria. Cross-references from the Results and abstract to these details will be added so that the scaling curves can be reproduced. revision: yes

Circularity Check

0 steps flagged

No circularity; claims rest on independent simulation outcomes

full rationale

The paper's central results derive from running a trained feedforward network on MNIST inputs and measuring reconstruction quality under varying partial-recording regimes. No equation, parameter fit, or uniqueness claim reduces the reported scaling to a definition or self-citation by construction. The simulation is externally falsifiable (different architectures, inputs, or recording strategies can be substituted) and contains no load-bearing self-citations. This is the normal case of a self-contained empirical study.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Only the abstract is available; no information is provided on free parameters, background axioms, or new entities introduced by the analysis.

pith-pipeline@v0.9.0 · 5739 in / 1054 out tokens · 36950 ms · 2026-05-25T10:12:11.976099+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

22 extracted references · 22 canonical work pages

  1. [1]

    Spatio-temporal correlations and visual signalling in a complete neuronal population

    Jonathan W Pillow, Jonathon Shlens, Liam Paninski, Alexander Sher, Alan M Litke, EJ Chichilnisky, and Eero P Simoncelli. Spatio-temporal correlations and visual signalling in a complete neuronal population. Nature, 454 (7207):995–999, 2008

    work page 2008

  2. [2]

    Computational models in the age of large datasets

    Timothy O’Leary, Alexander C Sutton, and Eve Marder. Computational models in the age of large datasets. Current opinion in neurobiology, 32:87–94, 2015

    work page 2015

  3. [3]

    A 1024-channel cmos microelectrode-array system with 26’400 electrodes for recording and stimulation of electro-active cells in-vitro

    M Ballini, J Mueller, P Livi, Y Chen, U Frey, A Shadmani, IL Jones, W Gong, M Fiscella, M Radivojevic, et al. A 1024-channel cmos microelectrode-array system with 26’400 electrodes for recording and stimulation of electro-active cells in-vitro. In VLSI Circuits (VLSIC), 2013 Symposium on , pages C54–C55. IEEE, 2013

    work page 2013

  4. [4]

    Imaging in vivo: watching the brain in action

    Jason ND Kerr and Winfried Denk. Imaging in vivo: watching the brain in action. Nature Reviews Neuroscience, 9(3):195–205, 2008

    work page 2008

  5. [5]

    Imaging calcium in neurons

    Christine Grienberger and Arthur Konnerth. Imaging calcium in neurons. Neuron, 73(5):862–885, 2012

    work page 2012

  6. [6]

    Optogenetics

    Karl Deisseroth. Optogenetics. Nature methods, 8(1):26–29, 2011

    work page 2011

  7. [7]

    Neural characterization in partially observed populations of spiking neurons

    Jonathan W Pillow and Peter E Latham. Neural characterization in partially observed populations of spiking neurons. In NIPS, pages 1161–1168, 2007. 8 A PREPRINT - J ULY 4, 2019

    work page 2007

  8. [8]

    Linear readout from a neural population with partial correlation data

    Adrien Wohrer, Ranulfo Romo, and Christian K Machens. Linear readout from a neural population with partial correlation data. In Advances in Neural Information Processing Systems , pages 2469–2477, 2010

    work page 2010

  9. [9]

    Inferring neural population dynamics from multiple partial recordings of the same neural circuit

    Srini Turaga, Lars Buesing, Adam M Packer, Henry Dalgleish, Noah Pettit, Michael Hausser, and Jakob Macke. Inferring neural population dynamics from multiple partial recordings of the same neural circuit. In Advances in Neural Information Processing Systems, pages 539–547, 2013

    work page 2013

  10. [10]

    Automatic discovery of cell types and microcircuitry from neural connectomics

    Eric Jonas and Konrad Kording. Automatic discovery of cell types and microcircuitry from neural connectomics. ELife, 4:e04250, 2015

    work page 2015

  11. [11]

    Could a neuroscientist understand a microprocessor? PLOS Computational Biology, 13(1):e1005268, 2017

    Eric Jonas and Konrad Paul Kording. Could a neuroscientist understand a microprocessor? PLOS Computational Biology, 13(1):e1005268, 2017

    work page 2017

  12. [12]

    Analysis of neural networks with redundancy

    Yoshio Izui and Alex Pentland. Analysis of neural networks with redundancy. Neural Comput., 2(2):226–238, April 1990. ISSN 0899-7667. doi: 10.1162/neco.1990.2.2.226. URL http://dx.doi.org/10.1162/neco. 1990.2.2.226

    work page doi:10.1162/neco.1990.2.2.226 1990

  13. [13]

    Predicting parameters in deep learning

    Misha Denil, Babak Shakibi, Laurent Dinh, Nando de Freitas, et al. Predicting parameters in deep learning. In Advances in Neural Information Processing Systems , pages 2148–2156, 2013

    work page 2013

  14. [14]

    Yu, Rogerio S

    Yu Cheng, Felix X. Yu, Rogerio S. Feris, Sanjiv Kumar, Alok Choudhary, and Shi-Fu Chang. An exploration of parameter redundancy in deep networks with circulant projections. In The IEEE International Conference on Computer Vision (ICCV), December 2015

    work page 2015

  15. [15]

    Distilling the knowledge in a neural network

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. In In Deep Learning and Representation Learning Workshop, NIPS, 2015

    work page 2015

  16. [16]

    Inference and missing data

    Donald B Rubin. Inference and missing data. Biometrika, pages 581–592, 1976

    work page 1976

  17. [17]

    Multiple imputation after 18+ years

    Donald B Rubin. Multiple imputation after 18+ years. Journal of the American statistical Association , 91(434): 473–489, 1996

    work page 1996

  18. [18]

    Spectral regularization algorithms for learning large incomplete matrices

    Rahul Mazumder, Trevor Hastie, and Robert Tibshirani. Spectral regularization algorithms for learning large incomplete matrices. Journal of machine learning research, 11(Aug):2287–2322, 2010

    work page 2010

  19. [19]

    Constraint-based causal discovery from multiple interventions over overlapping variable sets

    Sofia Triantafillou and Ioannis Tsamardinos. Constraint-based causal discovery from multiple interventions over overlapping variable sets. Journal of Machine Learning Research, 16:2147–2205, 2015

    work page 2015

  20. [20]

    Statistical matching: Theory and practice

    Marcello D’Orazio, Marco Di Zio, and Mauro Scanu. Statistical matching: Theory and practice . John Wiley & Sons, 2006

    work page 2006

  21. [21]

    Xgboost: A scalable tree boosting system

    Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22Nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 785–794. ACM, 2016

    work page 2016

  22. [22]

    Statistical assessment of the stability of neural movement representations

    Ian H Stevenson, Anil Cherian, Brian M London, Nicholas A Sachs, Eric Lindberg, Jacob Reimer, Marc W Slutzky, Nicholas G Hatsopoulos, Lee E Miller, and Konrad P Kording. Statistical assessment of the stability of neural movement representations. Journal of neurophysiology, 106(2):764–774, 2011. 9

    work page 2011