Pith. sign in

REVIEW 3 major objections 4 minor 75 references

Time-multiplexed layer reuse for physical neural networks

T0 review · 3 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Physical neural networks gain depth by switching fixed weight banks.

desk verdict The hardware-motivated idea of periodic layer reuse is real and clearly presented, but the abstract's equal-parameter scaling claim is contradicted by the paper's own fixed-budget experiment. read the letter →

arxiv 2511.00044 v3 pith:OSJCPSOV submitted 2025-10-28 cs.LG nlin.AO

Kohei Tsuchiyama , Andre Roehm , Takatomo Mihana , Ryoichi Horisaki This is my paper · ORCID
classification cs.LGnlin.AO
keywords physicalneuralnetworkstimemultiplexingweighttyingrecurrentparameterefficiencyphotoniccomputingspintronicsneuromorphichardware
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Physical neural networks have two very different speeds: a fast forward pass and weights that are slow to retune. The paper's claim is that a PNN can gain the benefits of a deeper network without fast weight reconfiguration by switching periodically among a small bank of fixed, trainable weight matrices—an architecture it calls ReLaX-Net. In numerical experiments on digit images and character-level next-token prediction, ReLaX-Net with two or more reused weight sets beats a recurrent network that repeats a single fixed layer, and repeating layers improves accuracy even when no parameters are added. A fixed-budget comparison also shows that one wide layer repeated several times outperforms splitting the same parameters into many smaller distinct layers. If the hardware premise holds, this gives a practical, low-modification route to larger effective physical computers built on photonic, spintronic, or other slow-tuning substrates.

What carries the argument

The central mechanism is the periodic switching schedule θ[t] = θ_{t mod L_W} inside the update h[t+1] = αh[t] + f(W_xh x[t] + W_hh[t] h[t] + b_h[t]). It turns one physical nonlinear element and L_W sets of slow-tunable weights into an L_T-layer network by time multiplexing, keeping all components except switches on slow timescales. The work it does is to generate effective depth from temporal repetition: reused layers form a function between a stateless RNN and a fully untied DNN, and because the same layer appears at several times, gradients from the different occurrences add together during training, which the paper credits for its stability against vanishing or exploding gradients.

What would settle it

A hardware measurement: run the same ReLaX-Net schedule on a photonic or spintronic testbed with real switches, and compare end-to-end accuracy and throughput against a stateless RNN and an untied DNN with equal parameters. If switch reconfiguration time or insertion loss makes L_T forward steps slower than a single repeated layer, the claimed scaling fails.

Watch

Extended reading notes

Core claim

The load-bearing discovery is a middle architecture between a recurrent and a feedforward network. Instead of reconfiguring weights at every time step or keeping one fixed matrix, ReLaX-Net defines a small number L_W of distinct hidden-layer parameter sets and cycles through them periodically over L_T time steps, with a residual connection and optional input at each step. The nonlinear activation is shared; only the weight and bias elements are duplicated, and fast switches route the hidden state between the fixed parameter banks. The paper's numerical findings are that, for 1 < L_W < L_T, this scheme beats a pure stateless RNN at equal hidden-parameter count on an image classification task

Load-bearing premise

The whole advantage rests on fast switches being able to route the hidden state between fixed weight banks at forward-pass speed without appreciable latency, loss, or added noise; if switching is slow or lossy, the gain over simply duplicating layers disappears.

Editorial extensions

If this is right

  • Adding fast switches to an existing PNN with slow weight tuning yields a deeper effective network without fast weight reconfiguration; the only new hardware elements are the switches.
  • On the tested benchmarks, ReLaX-Net with more than one weight set exceeds a same-size stateless RNN, and repeating layers improves performance even when the total parameter count stays fixed.
  • Under a fixed parameter budget, the best observed configuration is a single wide layer repeated several times, not many narrow distinct layers, so ReLaX-Net is most useful when physical constraints cap the width of each trainable layer.
  • The scheme is compatible with photonic and spintronic weight banks and with at least two different nonlinearities, suggesting it is a general way to scale intermediate-size physical neural networks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • This suggests a testable corollary: an aperiodic or learned switching schedule should outperform the fixed periodic schedule used in the paper, since the periodic choice is presented as the simplest option rather than the optimal one.
  • The fixed-budget result points to a design rule for intermediate-scale PNNs: keep each trainable layer as wide as fabrication allows and use repetition for depth, rather than splitting width into more distinct layers.
  • The same layer-reuse principle may transfer to other weight-constrained analog systems and to training methods that do not use backpropagation-through-time, because only the forward-path routing has to be fast; the paper's gradient analysis is one training route, not a requirement.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes ReLaX-Net, a physical-neural-network architecture in which a small set of L_W fixed trainable weight matrices is periodically switched over L_T time steps, interpolating between a stateless RNN (L_W=1) and a fully unrolled DNN (L_W=L_T). The claimed advantage is that only fast switches need to operate at forward-pass speed, while all weight parameters can be reconfigured slowly. The authors validate the idea numerically on SVHN image classification and Shakespeare character-level next-token prediction, with supporting experiments on oscillator-based activations and MNIST in the appendices. The central advertised claim is favorable scaling: ReLaX-Nets 'exceed the performance of equivalent traditional RNNs or DNNs with the same number of parameters' (abstract).

Significance. If the central claim were supported, ReLaX-Net would be a useful intermediate design point for PNNs, allowing extra effective depth without fast weight reconfiguration. The paper has clear strengths: it reports direct training results on two benchmark classes, includes ablations on repetition (L_T) and parameter budget (L_W), and is unusually honest in Section 4.3 where the fixed-budget experiment actually contradicts the abstract's equal-parameter claim. The architecture is simple, plausible, and the empirical setup is largely reproducible from the description. The value of the paper, however, depends on which claim is taken as the contribution: the narrow claim (under a fixed per-layer width cap, a few switched distinct layers improve over a same-width RNN) is supported by Figs. 5–7; the broad equal-parameter scaling claim is not.

major comments (3)
  1. [Abstract; §4.3, Table 1, Fig. 8] The abstract's central claim that ReLaX-Nets 'exceed the performance of equivalent traditional RNNs or DNNs with the same number of parameters' is not supported by the paper's own controlled test. In §4.3, the total hidden parameter budget is held fixed (roughly 33k) while H is traded against L_W (Table 1). Fig. 8a reports the best test error at L_W=1, L_T>1 — the standard single-weight RNN limit. Thus, at equal parameter count, the wide stateless RNN outperforms all switched multi-layer configurations. Figures 5–6 increase parameters with L_W, so they do not provide an equal-parameter comparison, and no same-parameter DNN baseline appears anywhere. Please replace the equal-parameter claim with the supported claim (fixed per-layer width cap: a few switched distinct layers outperform a same-width RNN; repeated use at fixed L_W helps), or supply a proper same-parameter baseline that overtu
  2. [§4.1–4.2, Figs. 5–7] Main performance figures lack error bars and seed counts. Fig. 7 shows differences of roughly 1–2% in test loss between configurations; without repeated runs or confidence intervals the improvement from repetition is not statistically established. Supplementary Fig. C1 reports five trials for the NLP experiment, so multi-seed evaluation is feasible. Please add error bars or at least state the number of seeds for all main quantitative comparisons.
  3. [§3, Appendix B] The hardware premise is load-bearing: the advantage over simply duplicating layers evaporates if the fast switches are slow, lossy, or have overhead that scales with hidden-state dimension. The text mentions DMD, fiber, and current switches but provides no system-level model of latency, insertion loss, crosstalk, or control power. I ask for either an explicit analysis (even a simple cost model) or a softened statement that practical gains are conditional on switch overhead being negligible.
minor comments (4)
  1. [Title/Abstract] The abstract refers to 'Time-Indexed Deep Alternating Layers Network (TIDAL-Net)', while the main text and title use 'ReLaX-Net'. Please unify the nomenclature throughout.
  2. [Cross-references] References to 'Supplemental Information Section 3.8/3.9' in §4 correspond to Appendix C.8/C.9; the numbering is inconsistent and should be fixed.
  3. [Eq. (3), Eq. (6)–(7)] The relation between W_hh[t] in Eq. (3) and θ[t] in Eq. (7) could be made explicit; currently the periodic switching rule appears only in Eq. (7), which may confuse readers about whether Eq. (3) allows arbitrary time variation.
  4. [Terminology] The label 'stateless RNN' for Eq. (5) is unconventional; a fixed-weight RNN with recurrent connections is stateful. Consider renaming to 'single-weight RNN' or 'fixed-weight RNN'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: ReLaX-Net's claims are empirical training results; the paper's own fixed-budget experiment undercuts the abstract's scaling claim, which is an inconsistency, not a circular reduction.

full rationale

ReLaX-Net is presented as a numerical architecture study, not a derivation. The core update rule (Eq. 6) and periodic switching (Eq. 7) are explicit architectural assumptions, and performance is measured against standard RNN/DNN limits rather than derived from them. There are no fitted constants subsequently renamed as predictions, and no uniqueness theorem is invoked. The few self-citations (Fit-DNN [47], adjoint-based training [70]) are used as illustrative comparisons or compatibility remarks and are not load-bearing. The closest potential issue is the abstract's claim that ReLaX-Nets 'exceed the performance of equivalent traditional RNNs or DNNs with the same number of parameters', but the paper's own controlled fixed-budget experiment (Table 1, Fig. 8) reports the opposite: performance is best at L_W=1, the standard wide RNN limit. That is an internal inconsistency/overclaim, not a case where the result is equivalent to its inputs by construction. The paper explicitly acknowledges the trade-off ('Surprisingly, "dividing" a large physical trainable layer into multiple smaller ones ... is not worth the trade-off according to our numerical simulations') and candidly states that similar ideas may have been considered before ('We would be surprised to find that they had not been considered at all before'). No circular reduction exists.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper contains no analytic derivation; the central claim rests on numerical training plus the hardware premise that fast switches can multiplex entire layers. The trainable weights themselves are ordinary network parameters, not free constants introduced to force a result; the hand-chosen hyperparameters L_W, L_T, H are listed because the scaling conclusions depend on them.

free parameters (3)
  • Residual connection strength alpha = trained; initialized to 1.0
    Trainable scalar in Eq. (6); part of the model and not independently justified.
  • Layer-switching hyperparameters (L_T, L_W, hidden width H) = L_T=12; L_W=1..12; H=64/128; fixed-budget widths in Table 1
    Chosen by hand per experiment; the central scaling conclusions depend on these values and no theory predicts optimal settings.
  • Oscillator coefficients (gamma_p, gamma_nl, Delta_t, eta) = trained; eta fixed to 3
    Used only in the Appendix G oscillator-network check; not central to the main claim.
assumptions (4)
  • domain assumption Fast switches can route entire hidden-state vectors between fixed weight banks at forward-pass speed without significant latency, loss, or noise penalty.
    Sec. 3 and Appendix B; the hardware motivation depends on this, but no system-level analysis of switching cost is provided.
  • domain assumption Slow weight reconfiguration is the common bottleneck in PNNs, justifying the timescale-separation design.
    Sec. 2 and Appendix B; true for many cited MRR/MZI/MTJ platforms, but not universal across all PNN substrates.
  • standard math Backpropagation through time with standard autograd can train the periodically switched graph stably.
    Appendix C.2; the gradient equations are standard chain-rule applications.
  • domain assumption SVHN and Shakespeare character prediction are representative benchmarks for PNN-relevant tasks.
    Sec. 4; reasonable but limited evidence for the broad claim that ReLaX-Net improves PNN performance generally.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Time-multiplexed layer reuse for physical neural networks." pith.science (2026). https://pith.science/paper/OSJCPSOV

@misc{pith2026251100044,
  author       = {Pith},
  title        = {Pith review of: Time-multiplexed layer reuse for physical neural networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OSJCPSOV}},
  note         = {Machine review of arXiv:2511.00044}
}
read the original abstract

Physical neural networks (PNNs) are promising candidates for next-generation computing, but existing demonstrations remain several orders of magnitude smaller than modern digital neural networks, whose recent advances have been driven by rapid growth in trainable parameters. This situation resembles the constraints of early digital neural networks, which led to ideas around parameter reuse. We investigate what similarly efficient hardware architectures may look like, focusing specifically on the common bottleneck of slow re-adjustment of the weights in PNNs. We propose the Time-Indexed Deep Alternating Layers Network (TIDAL-Net), which occupies an intermediate regime between recurrent and deep neural networks, specifically aimed at the scales and restrictions of common PNN prototypes. TIDAL-Net leverages the timescale separation found in many PNNs between fast forward dynamics and slowly trainable weights and biases, using layer-by-layer time multiplexing to increase effective depth while limiting implementation cost. Numerical experiments on image classification and natural language processing tasks show that TIDAL-Net improves performance with only minor modifications to conventional PNNs.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

75 extracted references · 8 linked inside Pith

  1. [1]

    Deep learning.nature, 521(7553):436–444, 2015

    Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning.nature, 521(7553):436–444, 2015

    2015

  2. [2]

    Very deep convolutional networks for large-scale image recognition.arXiv preprint arXiv:1409.1556, 2014

    Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition.arXiv preprint arXiv:1409.1556, 2014

    arXiv 2014

  3. [3]

    You only look once: Unified, real-time object detection

    Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 779–788, 2016

    2016

  4. [4]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016

    2016

  5. [5]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  6. [6]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the associ- ation for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171–4186, 2019

    2019

  7. [7]

    Inference in artificial intelligence with deep optics and photonics.Nature, 588(7836):39–47, 2020

    Gordon Wetzstein, Aydogan Ozcan, Sylvain Gigan, Shanhui Fan, Dirk Englund, Marin Soljaˇ ci´ c, Cornelia Denz, David AB Miller, and Demetri Psaltis. Inference in artificial intelligence with deep optics and photonics.Nature, 588(7836):39–47, 2020

    2020

  8. [8]

    Information processing using a single dynamical node as complex system

    Lennert Appeltant, Miguel Cornelles Soriano, Guy Van der Sande, Jan Danckaert, Serge Massar, Joni Dambre, Benjamin Schrauwen, Claudio R Mirasso, and Ingo Fischer. Information processing using a single dynamical node as complex system. Nature communications, 2(1):468, 2011

    2011

Show all 75 references
  1. [9]

    Experimental demonstration of reservoir computing on a silicon photonics chip.Nature communications, 5(1):3541, 2014

    Kristof Vandoorne, Pauline Mechet, Thomas Van Vaerenbergh, Martin Fiers, Geert Morthier, David Verstraeten, Benjamin Schrauwen, Joni Dambre, and Peter Bienstman. Experimental demonstration of reservoir computing on a silicon photonics chip.Nature communications, 5(1):3541, 2014

    2014

  2. [10]

    Deep learning with coherent nanophotonic circuits.Nature photonics, 11(7):441– 446, 2017

    Yichen Shen, Nicholas C Harris, Scott Skirlo, Mihika Prabhu, Tom Baehr-Jones, Michael Hochberg, Xin Sun, Shijie Zhao, Hugo Larochelle, Dirk Englund, et al. Deep learning with coherent nanophotonic circuits.Nature photonics, 11(7):441– 446, 2017. 16

    2017

  3. [11]

    Neuromorphic photonic networks using silicon photonic weight banks.Scientific reports, 7(1):7430, 2017

    Alexander N Tait, Thomas Ferreira De Lima, Ellen Zhou, Allie X Wu, Mitchell A Nahmias, Bhavin J Shastri, and Paul R Prucnal. Neuromorphic photonic networks using silicon photonic weight banks.Scientific reports, 7(1):7430, 2017

    2017

  4. [12]

    Reinforcement learning in a large-scale photonic recurrent neural network.Optica, 5(6):756–760, 2018

    Julian Bueno, Sheler Maktoobi, Luc Froehly, Ingo Fischer, Maxime Jacquot, Lau- rent Larger, and Daniel Brunner. Reinforcement learning in a large-scale photonic recurrent neural network.Optica, 5(6):756–760, 2018

    2018

  5. [13]

    All-optical machine learning using diffractive deep neural networks.Science, 361(6406):1004–1008, 2018

    Xing Lin, Yair Rivenson, Nezih T Yardimci, Muhammed Veli, Yi Luo, Mona Jarrahi, and Aydogan Ozcan. All-optical machine learning using diffractive deep neural networks.Science, 361(6406):1004–1008, 2018

    2018

  6. [14]

    Silicon photonics for artificial intelligence acceleration: Hotchips

    Carl Ramey. Silicon photonics for artificial intelligence acceleration: Hotchips

  7. [15]

    A crossbar array of magnetoresistive memory devices for in-memory computing

    Seungchul Jung, Hyungwoo Lee, Sungmeen Myung, Hyunsoo Kim, Seung Keun Yoon, Soon-Wan Kwon, Yongmin Ju, Minje Kim, Wooseok Yi, Shinhee Han, et al. A crossbar array of magnetoresistive memory devices for in-memory computing. Nature, 601(7892):211–216, 2022

    2022

  8. [16]

    Wright, Tatsuhiro Onodera, Martin M

    Logan G. Wright, Tatsuhiro Onodera, Martin M. Stein, Tianyu Wang, Darren T. Schachter, Zoey Hu, and Peter L. McMahon. Deep physical neural networks trained with backpropagation.Nature, 601:549–555, 2022

    2022

  9. [17]

    Neuromorphic computing with nanoscale spintronic oscillators

    Jacob Torrejon, Mikael Riou, Fabio AP Araujo, Shunsuke Tsunegi, Guru Khalsa, Damien Querlioz, Paolo Bortolotti, Vincent Cros, Kay Yakushiji, Akio Fukushima, et al. Neuromorphic computing with nanoscale spintronic oscillators. Nature, 547(7664):428–431, 2017

    2017

  10. [18]

    Training a multilayer dynamical spintronic network with stan- dard machine-learning tools to perform time-series classification.Physical Review Applied, 23(3):034051, mar 2025

    Erwan Plouet, D´ edalo Sanz Hern´ andez, Aymeric Vecchiola, Julie Grollier, and Alice Mizrahi. Training a multilayer dynamical spintronic network with stan- dard machine-learning tools to perform time-series classification.Physical Review Applied, 23(3):034051, mar 2025

    2025

  11. [19]

    Vowel recognition with four coupled spin-torque nano- oscillators.Nature, 563(7730):230–234, 2018

    Miguel Romera, Philippe Talatchian, Sumito Tsunegi, Flavio Abreu Araujo, Vincent Cros, Paolo Bortolotti, Juan Trastoy, Kay Yakushiji, Akio Fukushima, Hitoshi Kubota, et al. Vowel recognition with four coupled spin-torque nano- oscillators.Nature, 563(7730):230–234, 2018

    2018

  12. [20]

    A pro- grammable chemical computer with memory and pattern recognition.Nature communications, 11(1):1442, 2020

    Juan Manuel Parrilla-Gutierrez, Abhishek Sharma, Soichiro Tsuda, Geoffrey JT Cooper, Gerardo Aragon-Camarasa, Kevin Donkers, and Leroy Cronin. A pro- grammable chemical computer with memory and pattern recognition.Nature communications, 11(1):1442, 2020

    2020

  13. [21]

    Phys- ical implementation of reservoir computing through electrochemical reaction

    Shaohua Kan, Kohei Nakajima, Tetsuya Asai, and Megumi Akai-Kasaya. Phys- ical implementation of reservoir computing through electrochemical reaction. Advanced Science, 9(6):2104076, 2022

    2022

  14. [22]

    Imagenet classification with deep convolutional neural networks

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. InAdvances in Neural Information Processing Systems, volume 25, pages 1097–1105, 2012

    2012

  15. [23]

    Improv- ing language understanding by generative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improv- ing language understanding by generative pre-training. 2018

    2018

  16. [24]

    Megatron-lm: Training multi-billion parameter 17 language models using model parallelism.arXiv preprint arXiv:1909.08053, 2019

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter 17 language models using model parallelism.arXiv preprint arXiv:1909.08053, 2019

    1909 arXiv

  17. [25]

    Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    2019

  18. [26]

    Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020

    2020

  19. [27]

    Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. InAdvances in Neural Information Processing Systems, volume 33, pages 1877–1...

    1901

  20. [28]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.Journal of Machine Learning Research, 23(120):1–39, 2022

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.Journal of Machine Learning Research, 23(120):1–39, 2022

    2022

  21. [29]

    Palm: Scaling language modeling with pathways.Journal of Machine Learning Research, 24(240):1–113, 2023

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebas- tian Gehrmann, et al. Palm: Scaling language modeling with pathways.Journal of Machine Learning Research, 24(240):1–113, 2023

    2023

  22. [30]

    All-optical spiking neurosynaptic networks with self- learning capabilities.Nature, 569(7755):208–214, 2019

    Johannes Feldmann, Nathan Youngblood, C David Wright, Harish Bhaskaran, and Wolfram HP Pernice. All-optical spiking neurosynaptic networks with self- learning capabilities.Nature, 569(7755):208–214, 2019

    2019

  23. [31]

    Fully hardware-implemented memristor convolutional neural network.Nature, 577(7792):641–646, 2020

    Peng Yao, Huaqiang Wu, Bin Gao, Jianshi Tang, Qingtian Zhang, Wenqiang Zhang, J Joshua Yang, and He Qian. Fully hardware-implemented memristor convolutional neural network.Nature, 577(7792):641–646, 2020

    2020

  24. [32]

    IEEE Computer Society, 2020

    In2020 IEEE hot chips 32 symposium (HCS), pages 1–26. IEEE Computer Society, 2020

    2020

  25. [33]

    Novel nondelay-based reservoir computing with a single micromechanical nonlinear resonator for high-efficiency information processing.Microsystems & Nanoengineering, 7(1):83, 2021

    Jie Sun, Wuhao Yang, Tianyi Zheng, Xingyin Xiong, Yunfei Liu, Zheng Wang, Zhitian Li, and Xudong Zou. Novel nondelay-based reservoir computing with a single micromechanical nonlinear resonator for high-efficiency information processing.Microsystems & Nanoengineering, 7(1):83, 2021

    2021

  26. [34]

    Collective and synchronous dynamics of photonic spiking neurons.Nature communications, 12(1):2325, 2021

    Takahiro Inagaki, Kensuke Inaba, Timoth´ ee Leleu, Toshimori Honjo, Takuya Ikuta, Koji Enbutsu, Takeshi Umeki, Ryoichi Kasahara, Kazuyuki Aihara, and Hiroki Takesue. Collective and synchronous dynamics of photonic spiking neurons.Nature communications, 12(1):2325, 2021

    2021

  27. [35]

    Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  28. [36]

    Machine learning model sizes and the parameter gap.arXiv preprint arXiv:2207.02852, 2022

    Pablo Villalobos, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, Anson Ho, and Marius Hobbhahn. Machine learning model sizes and the parameter gap.arXiv preprint arXiv:2207.02852, 2022

    2022 arXiv

  29. [37]

    Diffractive optical computing in free space.Nature Communications, 15(1):1525, 2024

    Jingtian Hu, Deniz Mengu, Dimitrios C Tzarouchis, Brian Edwards, Nader Engheta, and Aydogan Ozcan. Diffractive optical computing in free space.Nature Communications, 15(1):1525, 2024

    2024

  30. [38]

    Scaling up silicon photonic-based accelerators: Challenges and opportunities.APL Photonics, 7(2), 2022

    MA Al-Qadasi, L Chrostowski, BJ Shastri, and S Shekhar. Scaling up silicon photonic-based accelerators: Challenges and opportunities.APL Photonics, 7(2), 2022. 18

    2022

  31. [39]

    On the effect of the thermal cross-talk in a photonic feed-forward neural network based on silicon microresonators.Frontiers in Physics, 10:1093191, 2022

    Stefano Biasi, Riccardo Franchi, Davide Bazzanella, and Lorenzo Pavesi. On the effect of the thermal cross-talk in a photonic feed-forward neural network based on silicon microresonators.Frontiers in Physics, 10:1093191, 2022

    2022

  32. [40]

    Neuromorphic spintronics.Nature electronics, 3(7):360–370, 2020

    Julie Grollier, Damien Querlioz, Kerem Y Camsari, Karin Everschor-Sitte, Shun- suke Fukami, and Mark D Stiles. Neuromorphic spintronics.Nature electronics, 3(7):360–370, 2020

    2020

  33. [41]

    The physics of optical computing.Nature Reviews Physics, 5(12):717–734, 2023

    Peter L McMahon. The physics of optical computing.Nature Reviews Physics, 5(12):717–734, 2023

    2023

  34. [42]

    Handwritten digit recognition with a back-propagation network.Advances in neural information processing systems, 2, 1989

    Yann LeCun, Bernhard Boser, John Denker, Donnie Henderson, Richard Howard, Wayne Hubbard, and Lawrence Jackel. Handwritten digit recognition with a back-propagation network.Advances in neural information processing systems, 2, 1989

    1989

  35. [43]

    Neural networks and principal component analysis: Learning from examples without local minima.Neural Networks, 2(1):53–58, 1989

    Pierre Baldi and Kurt Hornik. Neural networks and principal component analysis: Learning from examples without local minima.Neural Networks, 2(1):53–58, 1989

    1989

  36. [44]

    Balanced and deterministic weight-sharing helps network performance

    Oscar Chang and Hod Lipson. Balanced and deterministic weight-sharing helps network performance. InInternational Conference on Artificial Neural Networks, pages 41–50, 2018

    2018

  37. [45]

    Quantum computing in the nisq era and beyond.Quantum, 2:79, 2018

    John Preskill. Quantum computing in the nisq era and beyond.Quantum, 2:79, 2018

    2018

  38. [46]

    Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. InAdvances in Neural Information Processing Systems (NeurIPS), volume 31, 2018

    2018

  39. [47]

    Behavioral classification of sequential neural activity using time varying recurrent neural networks.IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2025

    Yongxu Zhang, Catalin Mitelut, David J Arpin, David Vaillancourt, Timothy Murphy, and Shreya Saxena. Behavioral classification of sequential neural activity using time varying recurrent neural networks.IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2025

    2025

  40. [48]

    Deep neural networks using a single neuron: folded-in-time architecture using feedback-modulated delay loops.Nature communications, 12(1):5164, 2021

    Florian Stelzer, Andr´ e R¨ ohm, Raul Vicente, Ingo Fischer, and Serhiy Yanchuk. Deep neural networks using a single neuron: folded-in-time architecture using feedback-modulated delay loops.Nature communications, 12(1):5164, 2021

    2021

  41. [49]

    Hypernetworks.arXiv preprint arXiv:1609.09106, 2016

    David Ha, Andrew Dai, and Quoc V Le. Hypernetworks.arXiv preprint arXiv:1609.09106, 2016

    2016 arXiv

  42. [50]

    Wave physics as an analog recurrent neural network.Science advances, 5(12):eaay6946, 2019

    Tyler W Hughes, Ian AD Williamson, Momchil Minkov, and Shanhui Fan. Wave physics as an analog recurrent neural network.Science advances, 5(12):eaay6946, 2019

    2019

  43. [51]

    Dynamic filter networks.Advances in neural information processing systems, 29, 2016

    Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Luc V Gool. Dynamic filter networks.Advances in neural information processing systems, 29, 2016

    2016

  44. [52]

    Using fast weights to attend to the recent past.Advances in neural information processing systems, 29, 2016

    Jimmy Ba, Geoffrey E Hinton, Volodymyr Mnih, Joel Z Leibo, and Catalin Ionescu. Using fast weights to attend to the recent past.Advances in neural information processing systems, 29, 2016

    2016

  45. [53]

    Feedback control for microring weight banks.Optics express, 26(20):26422–26443, 2018

    Alexander N Tait, Hasitha Jayatilleka, Thomas Ferreira De Lima, Philip Y Ma, Mitchell A Nahmias, Bhavin J Shastri, Sudip Shekhar, Lukas Chrostowski, and Paul R Prucnal. Feedback control for microring weight banks.Optics express, 26(20):26422–26443, 2018

    2018

  46. [54]

    Large-scale photonic processors 19 and their applications.npj Nanophotonics, 2(1):32, 2025

    Daniel P´ erez-L´ opez and Luis Torrijos-Mor´ an. Large-scale photonic processors 19 and their applications.npj Nanophotonics, 2(1):32, 2025

    2025

  47. [55]

    Generating sequences with recurrent neural networks.arXiv preprint arXiv:1308.0850, 2013

    Alex Graves. Generating sequences with recurrent neural networks.arXiv preprint arXiv:1308.0850, 2013

    2013 arXiv

  48. [56]

    A corre- spondence between random neural networks and statistical field theory.arXiv preprint arXiv:1710.06570, 2017

    Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. A corre- spondence between random neural networks and statistical field theory.arXiv preprint arXiv:1710.06570, 2017

    2017 arXiv

  49. [57]

    Digital light processing and mems: timely convergence for a bright future.Proceedings of SPIE, 2639:2–14, 1997

    Larry J Hornbeck. Digital light processing and mems: timely convergence for a bright future.Proceedings of SPIE, 2639:2–14, 1997

    1997

  50. [58]

    Femotosecond switching in a dual-core-fiber nonlinear coupler.Optics Letters, 13(10):904–906, 1988

    Stephen R Friberg, Andrew M Weiner, YARON Silberberg, BRUNO G Sfez, and PS Smith. Femotosecond switching in a dual-core-fiber nonlinear coupler.Optics Letters, 13(10):904–906, 1988

    1988

  51. [59]

    Silicon microring resonators.Laser & photonics reviews, 6(1):47–73, 2012

    Wim Bogaerts, Peter De Heyn, Thomas Van Vaerenbergh, Katrien De Vos, Shankar Kumar Selvaraja, Tom Claes, Pieter Dumon, Peter Bienstman, Dries Van Thourhout, and Roel Baets. Silicon microring resonators.Laser & photonics reviews, 6(1):47–73, 2012

    2012

  52. [60]

    Reading digits in natural images with unsupervised feature learning

    Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. InNIPS workshop on deep learning and unsupervised feature learning, volume 2011, page 4. Granada, 2011

    2011

  53. [61]

    Delving deep into rectifiers: Surpassing human-level performance on imagenet classification

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026–1034, 2015

    2015

  54. [62]

    A theoretically grounded application of dropout in recurrent neural networks.Advances in neural information processing systems, 29, 2016

    Yarin Gal and Zoubin Ghahramani. A theoretically grounded application of dropout in recurrent neural networks.Advances in neural information processing systems, 29, 2016

    2016

  55. [63]

    Shakespeare’s plays, sonnets and poems from the fol- ger shakespeare

    William Shakespeare. Shakespeare’s plays, sonnets and poems from the fol- ger shakespeare. https://folger.edu/explore/shakespeares-works/all-works, n.d. Accessed: 2025-06-10

    2025

  56. [64]

    On the difficulty of training recurrent neural networks

    Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. InInternational conference on machine learning, pages 1310–1318. Pmlr, 2013

    2013

  57. [65]

    Learning long-term depen- dencies with gradient descent is difficult.IEEE transactions on neural networks, 5(2):157–166, 1994

    Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Learning long-term depen- dencies with gradient descent is difficult.IEEE transactions on neural networks, 5(2):157–166, 1994

    1994

  58. [66]

    van der Wiel

    Herbert Jaeger, Beatriz Noheda, and Wilfred G. van der Wiel. Toward a formal theory for computing machines made out of whatever physics offers.Nature Communications, 14(1):4911, 2023

    2023

  59. [67]

    Infor- mation processing capacity of dynamical systems.Scientific reports, 2(1):514, 2012

    Joni Dambre, David Verstraeten, Benjamin Schrauwen, and Serge Massar. Infor- mation processing capacity of dynamical systems.Scientific reports, 2(1):514, 2012

    2012

  60. [68]

    On the number of linear regions of deep neural networks

    Guido F Mont´ ufar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the number of linear regions of deep neural networks. InAdvances in neural information processing systems, volume 27, 2014

    2014

  61. [69]

    Adam: A method for stochastic optimization.arXiv preprint 20 arXiv:1412.6980, 2014

    Diederik P Kingma. Adam: A method for stochastic optimization.arXiv preprint 20 arXiv:1412.6980, 2014

    2014 arXiv

  62. [70]

    Physical deep learning with biologically inspired training method: gradient-free approach for physical hardware.Nature Communications, 13(1):7847, 2022

    Mitsumasa Nakajima, Katsuma Inoue, Kenji Tanaka, Yasuo Kuniyoshi, Toshikazu Hashimoto, and Kohei Nakajima. Physical deep learning with biologically inspired training method: gradient-free approach for physical hardware.Nature Communications, 13(1):7847, 2022

    2022

  63. [71]

    Blending optimal control and biologically plausible learning for noise-robust physical neural networks.Phys

    Satoshi Sunada, Tomoaki Niiyama, Kazutaka Kanno, Rin Nogami, Andr´ e R¨ ohm, Takato Awano, and Atsushi Uchida. Blending optimal control and biologically plausible learning for noise-robust physical neural networks.Phys. Rev. Lett., 134:017301, Jan 2025

    2025

  64. [72]

    Boris T. Polyak. Some methods of speeding up the convergence of iteration methods.USSR Computational Mathematics and Mathematical Physics, 4(5):1– 17, 1964

    1964

  65. [73]

    Lecture 6e: Rmsprop — divide the gradient by a running average of its recent magnitude

    Geoffrey Hinton. Lecture 6e: Rmsprop — divide the gradient by a running average of its recent magnitude. Coursera: Neural Networks for Machine Learning, 2012. http://www.cs.toronto.edu/∼tijmen/csc321/slides/lecture slides lec6.pdf

    2012

  66. [74]

    Gradient-based learning applied to document recognition.Proceedings of the IEEE, 86(11):2278– 2324, 1998

    Yann LeCun, L´ eon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition.Proceedings of the IEEE, 86(11):2278– 2324, 1998

    1998

  67. [75]

    Input (Hidden, Output)

    Junhe Zhou, Qiqi Wang, and Chenweng Huang. Recurrent diffractive deep neural networks.Optics Express, 32(27):48093–48104, 2024. Acknowledgements We thank Felix K¨ oster for fruitful feedback and discussions. This study was supported in part by a Grant-in-Aid for Transformative...

    2024

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.