Pith. sign in

REVIEW 3 major objections 4 minor 79 references

Understanding and controlling the geometry of memory organization in RNNs

T0 review · 3 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read RNNs learn abruptly because their phase-space slow points first restructure, often without a bifurcation; a temporal-consistency regularizer induces this restructuring sooner, and its online local form builds cue memories without…

desk verdict Worth a serious referee, but the flagship GR-mechanism claim needs a real metric and code before you trust the causal story. read the letter →

arxiv 2502.07256 v1 pith:ZKCM7ZWV submitted 2025-02-11 q-bio.NC

Udith Haputhanthri , Liam Storan , Yiqi Jiang , Tarun Raheja , Adam Shai , Orhun Akengin , Nina Miolane , Mark J. Schnitzer
show 2 more authors
Fatih Dinc Hidenori Tanaka
This is my paper · ORCID
classification q-bio.NC
keywords abruptlearningrecurrentneuralnetworksgeometricrestructuringslowpointslineattractorstemporalconsistencyregularizationstronglyconnectedRNNsworkingmemory
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to explain the abrupt, staircase-shaped jumps that recurrent neural networks show during training on short-term memory tasks, and to show the jumps can be deliberately triggered. Its central claim is that the jump is preceded by a geometric restructuring of the network's phase space: the landscape of 'slow points,' states of near-zero but not necessarily zero dynamics, rearranges within a few epochs, and this restructuring equips the network with the attractor-like structures it needs to hold and manipulate information. The claim is made in two complementary registers — numerical energy minimization in general piecewise-linear RNNs, and an exact one-dimensional 'latent circuit' analysis in rank-one RNNs that shows an approximate line attractor can be born without any bifurcation. The paper then shows that a temporal consistency regularization (TCR), which simply asks a subset of neurons to change slowly from one time step to the next, induces the beneficial restructuring earlier, shortens the search phase across architectures and tasks, and even allows training in the strongly connected, chaotic regime where ordinary gradient training fails. If correct, this reframes the frustrating plateau phase as a search through weight space for a geometry that the gradient alone cannot find, and suggests that small, local, goal-agnostic mechanisms can substitute for much of the global optimization effort.

What carries the argument

Three pieces carry the argument. The detector: slow points are found by locally minimizing the kinetic energy $E(x) = \|\partial x[t]/\partial t\|_2^2$ from many randomly chosen initial states (Eq. 2), which identifies states with near-zero speed without assuming they are fixed points — a distinction that matters because the structures that emerge as memory substrates can be slow points rather than true equilibria. The window: in rank-one RNNs the recurrent weight is constrained to a rank-one matrix, which collapses the network onto a one-dimensional latent variable $\kappa(t)$ whose evolution is a closed equation; this makes the growth of flat regions ('approximate line attractors') analytically visible and quantifiable through a flatness score that measures the total $\kappa$-interval where both $\dot{\kappa}$ and its derivative with respect to $\kappa$ are small. The intervention: temporal consistency regularization, $L_{\mathrm{TCR}} = \frac{1}{T}\sum_{t=1}^{T}\sum_{i=1}^{N_{\mathrm{reg}}} (x_i[t] - x_i[t-1])^2$, applied to a chosen subset of 'memory neurons,' together with its degenerate online form $\Delta W_{ij}[t] = -\lambda_{\mathrm{TCR}} (r_i[t] - r_i[t-\Delta t])\, r_j[t-\Delta t]$, which implements 'freeze-in training' of cue-responsive fixed points using only local signals. Together these let the authors detect restructuring, explain it without invoking bifurcations, and induce it deliberately.

What would settle it

Train fresh PLRNNs on the delayed addition task while sampling the slow-point landscape by energy minimization every epoch and computing the latent-circuit flatness score in rank-one versions. The paper's account predicts that in nearly every network the flatness score and slow-point configuration change measurably in the same epoch as the accuracy jump, and that no network reaches competence while its slow-point landscape is statistically indistinguishable from the preceding plateau, a check that should be verified with an independent Jacobian-based fixed-point continuation. A single well-trained network whose accuracy jumps while its flatness score and slow-point count stay at plateau levels would refute the claim that geometric restructuring is the causal precursor of abrupt skill acquisition.

Watch

Extended reading notes

Core claim

The core discovery is that abrupt learning in RNNs is a geometric event before it is an accuracy event. A few epochs before the loss collapses, the network's slow-point landscape — the positions and shapes of the states where trajectories barely move — changes sharply, and this change, not a gradual accumulation of gradient steps, is what unlocks the task. In full-rank piecewise-linear RNNs the paper shows this concretely: the restructuring happens in only a few epochs, the weight matrix changes abruptly but permanently during it, and gradients destabilize then recover. In rank-one RNNs, where the dynamics reduce to a one-dimensional latent circuit, the story is exact: learning grows a flat region of the latent velocity field that approximates a line attractor over the full output range, the flatness score rises in lockstep with test accuracy, and roughly one fifth of well-trained networks never change their number of fixed points — so no bifurcation was needed. The paper's second contribution is that the event can be induced: temporal consistency regularization, which penalizes $(x_i[t] - x_i[t-1])^2$ on a subset of 'memory neurons,' shortens the search phase in PLRNNs, LSTMs, and leaky firing-rate RNNs, and in the strongly connected regime 14 of 20 TCR-trained networks solved an evidence accumulation task that none of 20 unregularized networks could solve in 30,000 epochs. Run as a purely local online rule, the same principle creates 25 distinct cue-responsive fixed points without any global error signal.

Load-bearing premise

The load-bearing premise is that the slow-point landscape recovered by the non-convex kinetic-energy minimization faithfully represents the structures that actually perform the memory computation: if those local minima are artifacts of initialization or sampling, or if skill acquisition is driven by structures the method cannot see, then the claim that geometric restructuring is the causal precursor of the accuracy jump is not established.

Editorial extensions

If this is right

  • The long plateau in RNN training is not primarily a vanishing-gradient problem: before a geometric restructuring event the gradient direction itself is misaligned with the final solution, so fixing gradient magnitude alone will not remove the plateau — the network needs a signal that promotes restructuring.
  • Bifurcation-based analyses of RNN training will miss the central mechanism: skill acquisition can occur through the growth of approximate line attractors while the number of fixed points stays constant, so counting bifurcations is neither necessary nor sufficient to detect the onset of competence.
  • Temporal consistency regularization offers a route around the chaos obstacle that motivated reservoir computing and the FORCE algorithm: it trains strongly connected recurrent networks without the global shrinkage of the weight spectrum that weight decay imposes.
  • The online, purely local form of the rule can assemble associative memories — cue-responsive fixed points — with no supervision or global loss, making it a candidate mechanism for self-organizing memory in biological circuits; the paper's testable prediction is a specific short-term synaptic plasticity that implements temporal consistency ('changed rates change synaptic weights').
  • The benefit is bounded, as the paper states: TCR helps most when initialization is far from the desired weight subspace, confers little when the network already starts near a solution, and can slow learning for tasks that require rapid activity changes, so it is best viewed as a search-phase accelerator.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A diagnostic use the paper does not develop: monitoring the count and shape of slow points during training could serve as an early-warning signal that an accuracy jump is about to occur, well before the loss curve moves — a tool that transfers to any recurrent architecture whose internal states are observable.
  • The plateau-then-jump pattern described here resembles 'grokking' reported in large non-recurrent models; if slow-point geometry can be measured in those settings, the GR framework might supply a common dynamical-systems account of delayed generalization across very different architectures.
  • The paper flags that freeze-in training can converge to the trivial solution $W = 0$ and leaves the question open; a direct next step is to test the proposed controls — stochastic weight updates or a time-varying regularization strength $\lambda_{\mathrm{TCR}}(t)$ — and to measure how many cues can be stored before interference between fixed points limits capacity.
  • Because the online freeze-in rule uses only pre- and post-synaptic activities, it is compatible with neuromorphic or on-chip learning where global error backpropagation is unavailable; testing how the rule scales with network size and cue count would clarify its practical reach.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This manuscript studies abrupt learning in recurrent neural networks trained on short-term memory tasks. It claims that a geometric restructuring (GR) of the phase-space slow-point landscape precedes the abrupt accuracy jump, that this restructuring can occur without classical bifurcations, and that a temporal consistency regularization (TCR) that penalizes rapid changes in a subset of neurons accelerates training, promotes attractor formation, and enables learning in strongly connected networks. The evidence includes full-rank PLRNN experiments with energy-minimization slow-point extraction, rank-one RNN latent-circuit analyses using a flatness score, and TCR experiments on PLRNNs, leaky firing-rate RNNs, and LSTMs across several tasks, together with an online 'freeze-in' rule for generating cue-responsive fixed points.

Significance. The paper's strongest contribution is empirical: TCR consistently shortens the search phase across multiple architectures and tasks, and the training-speed comparisons are reported over multiple seeds. The analytic rank-one latent-circuit analysis is a useful demonstration that approximate line attractors can emerge without bifurcations. If the mechanistic interpretation is supported by a quantitative full-rank GR analysis, the paper would provide a practical regularizer and a testable biological prediction. However, the central causal language in the abstract is not yet supported: the full-rank GR detection is largely qualitative, and the link between TCR and GR events rests on a single example. No code or data availability statement is included, which is a reproducibility concern for a purely computational study.

major comments (3)
  1. [II.B-C, Eq. (2), App. S1.4] The full-rank GR event is not quantitatively defined. The paper states that for the 19-network set the authors 'visually confirmed GR events (data not shown)' (Sec. II.C), and the only reported statistics are singular-value changes of gradients and weights (Fig. S1C-D), which reflect learning instability rather than phase-space geometry. Since Eq. (2) is a non-convex minimization initialized from 500 states sampled from the current epoch's trial activities, an apparent change in the extracted slow-point set could be produced by a change in the initialization distribution even if the vector field is unchanged. Please provide a quantitative slow-point geometry metric (e.g., count, location, or pairwise distance between slow-point sets), apply it to all networks, and verify robustness by re-running the minimization from fixed reference initializations across epochs. In addition, Figs. 1B-C place both the restructuring and the accuracy jump at the same epoch, so 'prior to the drop' is not demonstrated at the available temporal resolution.
  2. [II.D and Methods S1.6] The flatness score and the Spearman correlations in Sec. II.D rest on user-defined thresholds (|κ˙|<0.2 and |∂κ κ˙|<0.2) and on aligning all networks to the end of the search phase. The reported correlation of 0.99±0.01 between flatness score and relative epoch to the GR event is then partly structural, since both quantities are aligned to the same event and both increase monotonically after it. Please report raw, unaligned trajectories and a threshold-sensitivity analysis, and state which conclusions survive if the flatness thresholds are varied by an order of magnitude.
  3. [II.G-H and Fig. 6] The paper shows that TCR accelerates training and enables strongly connected training, but it does not establish the mechanistic claim that TCR works by promoting GR events. Because LTCR (Eq. 5) directly penalizes per-step changes in a subset of activities, its benefit for memory tasks could arise from a simple reduction in effective recurrent gain or from a smoothing of the loss landscape, rather than from the specific 'geometric restructuring' described for unregularized networks. The only direct evidence for TCR-induced GR is a single example in Fig. 5B. Please compare TCR against control regularizers (e.g., L2 penalty on activities, or penalty on output changes) and measure slow-point geometry quantitatively in regularized networks across multiple seeds; if such a comparison shows no difference, the abstract should be weakened from 'promotes these GR events' to a statement about accelerated training.
minor comments (4)
  1. [Throughout] The manuscript does not include a code or data availability statement; releasing code and data would allow readers to verify the visual GR assessments and reproduce the training curves.
  2. [II.I and Discussion] Eq. (7) is described as biologically plausible, but the update requires the derivative of the post-synaptic activity, and the paper itself notes that a plasticity process capable of tracking that derivative has not been identified; the word 'bioplausible' in the abstract is therefore stronger than the evidence presented.
  3. [II.E, Fig. 4B] The 'optimal learning direction' uses the final trained weights W_f, which are unavailable during training; the learning-signal-quality analysis should be framed explicitly as a retrospective, oracle-based diagnostic rather than as a property of the gradient available to the learning algorithm.
  4. [Methods S1.9] The definition of training speed relies on thresholds (e.g., 0.08 fraction correct) and on 'visual inspection' to determine the onset of the comprehension phase; please report the threshold rule and a sensitivity check for the statistical comparisons.

Circularity Check

1 steps flagged · score 2.0 of 10

Minor definitional circularity in the claim that TCR induces GR events; the paper's central empirical measurements are independent.

  1. self definitional [Section II.G, Eq. (5); Section II.B, Eq. (2); Section III.B]
    "We now propose the temporal consistency regularization (TCR), a (potentially) biologically plausible mechanism that is agnostic to the network architecture and the details of the desired geometric landscapes: L_TCR = 1/T Σ_{t=1}^T Σ_{i=1}^{Nreg} (x_i[t] − x_i[t − 1])^2. (5) Intuitively, TCR encourages slow time dynamics for a subset (Nreg out of N) of neurons, which we term as 'memory neurons.' ... These insights led to our development of temporal consistency regularization, an attractor-agnostic, architecture-agnostic, and biologically plausible mechanism to induce GR events."

    Eq. (2) defines slow points by minimizing the squared temporal derivative ||∂x/∂t||^2 in state space. Eq. (5) is a time-averaged sum of squared one-step temporal differences (x_i[t]−x_i[t−1])^2 for the regularized neurons, which is the same discrete kinetic energy. Thus TCR is literally a training objective whose minimization creates the slow-point condition used to define GR events. The statement that TCR 'induce[s] GR events' or 'facilitates attractor formation' restates the loss definition rather than providing an independent mechanism. The empirical speedups and strongly-connected training results are separate, measured outcomes and are not implied by the loss alone, so the overall circularity is limited.

full rationale

The main empirical chain is self-contained and not fit-derived: the observation that slow-point landscapes restructure before accuracy jumps is a direct measurement from network checkpoints; the TCR speedup comparisons are direct training measurements; and the strongly-connected training result is benchmarked against unregularized networks that fail. The flatness-score analysis in Fig. 3 is a mild alignment concern because networks are aligned on the end of the search phase, so a monotone flatness score correlates with relative epoch; however, the flatness score itself is an independent latent-circuit measure, not a fitted predictor of accuracy. The non-convexity and initialization dependence of the energy minimization in Eq. (2) and App. S1.4 are validity limitations, not circularity, since the extracted slow points are not used as fitted inputs to predict the loss jump. The rank-one latent-circuit equations are derived in the Methods, so the reliance on the self-citation [47] is methodological rather than a load-bearing uniqueness claim. The only definitional reduction is the TCR claim identified above; it is minor because the paper's central contribution—accelerated training and training in strongly connected networks—is empirical and independent of that tautology. Overall circularity is therefore low.

Assumptions & free parameters 5 free parameters · 4 assumptions · 1 invented entities

The paper is an empirical study, so the ledger captures the user-chosen thresholds and modeling assumptions on which the central claims depend. There is no theorem whose inputs can be audited.

free parameters (5)
  • TCR regularization strength lambda_TCR = 1e-3 to 1 (chosen by hand)
    Learning curves and search-phase speed depend on this strength; no principled selection, only sweeps (Figs. 5C, 6D, S6).
  • Number of memory neurons Nreg = 20/40 PLRNN, 25/50 lfRNN, 20/40 LSTM
    Subset size is chosen by hand in every experiment; central to TCR's effect.
  • Flatness score thresholds = |kappa_dot| < 0.2 and |d_kappa kappa_dot| < 0.2
    The correlation between flatness and accuracy (0.87) depends on this user-defined threshold; no sensitivity analysis is provided.
  • Energy minimization hyperparameters = alpha=0.1, 1000 steps, 500 init points
    Slow-point landscapes in Figs. 1-2 are produced by these choices; non-convex minimization may depend on them.
  • Strong-connection strength g = g=4 (and g=3 for flip-flop)
    The strongly connected regime claim is tied to this chosen initialization scale.
assumptions (4)
  • domain assumption Local minima of kinetic energy E(x)=||dx/dt||^2 identify behaviorally relevant slow points in RNN phase space
    This underpins all GR identification (Eq. 2 and Methods). If the energy landscape is not representative or minimization artifacts dominate, GR events are not established.
  • domain assumption The rank-one latent variable kappa(t)=n^T r/N captures the computational geometry relevant to delayed addition
    Used in Section II D to infer absence of bifurcations from fixed-point counts of the 1D latent circuit.
  • domain assumption Zero-crossing count of kappa_dot on a 100-point grid detects all relevant bifurcations except transcritical
    Explicitly stated in Section II D; approximate and could miss tangencies that create flat regions.
  • ad hoc to paper TCR is a proxy for a biological plasticity mechanism and can be implemented locally
    The biological plausibility claims and the prediction of a TCR-like synaptic plasticity mechanism depend on this assumption (Discussion D).
invented entities (1)
  • Short-term synaptic plasticity mechanism implementing temporal consistency regularization
    purpose: To explain how biological networks could implement temporal consistency regularization
    Predicted in Discussion D as a testable biological prediction; no experimental evidence or quantitative handle beyond the existence claim.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Understanding and controlling the geometry of memory organization in RNNs." pith.science (2026). https://pith.science/paper/ZKCM7ZWV

@misc{pith2026250207256,
  author       = {Pith},
  title        = {Pith review of: Understanding and controlling the geometry of memory organization in RNNs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZKCM7ZWV}},
  note         = {Machine review of arXiv:2502.07256}
}
read the original abstract

Training recurrent neural networks (RNNs) is a high-dimensional process that requires updating numerous parameters. Therefore, it is often difficult to pinpoint the underlying learning mechanisms. To address this challenge, we propose to gain mechanistic insights into the phenomenon of \emph{abrupt learning} by studying RNNs trained to perform diverse short-term memory tasks. In these tasks, RNN training begins with an initial search phase. Following a long period of plateau in accuracy, the values of the loss function suddenly drop, indicating abrupt learning. Analyzing the neural computation performed by these RNNs reveals geometric restructuring (GR) in their phase spaces prior to the drop. To promote these GR events, we introduce a temporal consistency regularization that accelerates (bioplausible) training, facilitates attractor formation, and enables efficient learning in strongly connected networks. Our findings offer testable predictions for neuroscientists and emphasize the need for goal-agnostic secondary mechanisms to facilitate learning in biological and artificial networks.

Figures

Figures reproduced from arXiv: 2502.07256 by the authors.

Figure 1
Figure 1. FIG. 1 [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. FIG. 2 [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. FIG. 3 [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: FIG. 4 [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: FIG. 5 [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: FIG. 6 [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: FIG. 7 [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

79 extracted references · 60 canonical work pages

  1. [1]

    Reconstructing computational dynamics from neural measurements with recurrent neural networks

    Daniel Durstewitz, Georgia Koppe, and Max Ingo Thurm. Reconstructing computational dynamics from neural measurements with recurrent neural networks. bioRxiv, 2022

    work page 2022

  2. [2]

    Inferring brain-wide interactions using data- constrained recurrent neural network models

    Matthew G Perich, Charlotte Arlt, Sofia Soares, Megan E Young, Clayton P Mosher, Juri Minxha, Eugene Carter, Ueli Rutishauser, Peter H Rudebeck, Christopher D Har- vey, et al. Inferring brain-wide interactions using data- constrained recurrent neural network models. bioRxiv, pages 2020–12, 2021

    work page 2020

  3. [3]

    A neural network that finds a naturalistic solution for the production of muscle activ- ity

    David Sussillo, Mark M Churchland, Matthew T Kauf- man, and Krishna V Shenoy. A neural network that finds a naturalistic solution for the production of muscle activ- ity. Nature neuroscience, 18(7):1025–1033, 2015

    work page 2015

  4. [4]

    Extracting computational mechanisms from neural data using low-rank rnns

    Adrian Valente, Jonathan W Pillow, and Srdjan Ostojic. Extracting computational mechanisms from neural data using low-rank rnns. Advances in Neural Information Processing Systems, 35:24072–24086, 2022

    2022

  5. [5]

    Attractor dynamics gate cortical information flow during decision-making

    Arseny Finkelstein, Lorenzo Fontolan, Michael N Economo, Nuo Li, Sandro Romani, and Karel Svoboda. Attractor dynamics gate cortical information flow during decision-making. Nature Neuroscience , 24(6):843–850, 15 2021

    work page 2021

  6. [6]

    A unifying perspective on neural manifolds and circuits for cognition

    Christopher Langdon, Mikhail Genkin, and Tatiana A Engel. A unifying perspective on neural manifolds and circuits for cognition. Nature Reviews Neuroscience , pages 1–15, 2023

    work page 2023

  7. [7]

    Recurrent neural networks are universal approx- imators

    Anton Maximilian Sch¨ afer and Hans Georg Zimmer- mann. Recurrent neural networks are universal approx- imators. In Artificial Neural Networks–ICANN 2006: 16th International Conference, Athens, Greece, Septem- ber 10-14, 2006. Proceedings, Part I 16 , pages 632–640. Springer, 2006

    work page 2006

  8. [8]

    Shap- ing dynamics with multiple populations in low-rank re- current networks

    Manuel Beiran, Alexis Dubreuil, Adrian Valente, Francesca Mastrogiuseppe, and Srdjan Ostojic. Shap- ing dynamics with multiple populations in low-rank re- current networks. Neural Computation, 33(6):1572–1615, 2021

    2021

Show all 79 references
  1. [9]

    Context-dependent computation by recurrent dynamics in prefrontal cortex

    Valerio Mante, David Sussillo, Krishna V Shenoy, and William T Newsome. Context-dependent computation by recurrent dynamics in prefrontal cortex. nature, 503(7474):78–84, 2013

    2013

  2. [10]

    Backpropagation algo- rithms and reservoir computing in recurrent neural net- works for the forecasting of complex spatiotemporal dy- namics

    P R Vlachas, J Pathak, B R Hunt, T P Sapsis, M Gir- van, E Ott, and P Koumoutsakos. Backpropagation algo- rithms and reservoir computing in recurrent neural net- works for the forecasting of complex spatiotemporal dy- namics. Neural Netw, 126:191–217, March 2020

    2020

  3. [11]

    Circuit mech- anisms for the maintenance and manipulation of in- formation in working memory

    Nicolas Y Masse, Guangyu R Yang, H Francis Song, Xiao-Jing Wang, and David J Freedman. Circuit mech- anisms for the maintenance and manipulation of in- formation in working memory. Nature neuroscience , 22(7):1159–1167, 2019

    2019

  4. [12]

    Peeking inside the black-box: a survey on explainable artificial intelli- gence (xai)

    Amina Adadi and Mohammed Berrada. Peeking inside the black-box: a survey on explainable artificial intelli- gence (xai). IEEE access, 6:52138–52160, 2018

    2018

  5. [13]

    Golub, Surya Ganguli, and David Sussillo

    Niru Maheswaranathan, Alex Williams, Matthew D. Golub, Surya Ganguli, and David Sussillo. Reverse en- gineering recurrent networks for sentiment classification reveals line attractor dynamics, 2019

    2019

  6. [14]

    Opening the black box: low-dimensional dynamics in high-dimensional recurrent neural networks

    David Sussillo and Omri Barak. Opening the black box: low-dimensional dynamics in high-dimensional recurrent neural networks. Neural computation , 25(3):626–649, 2013

    2013

  7. [15]

    Golub and David Sussillo

    Matthew D. Golub and David Sussillo. Fixedpointfinder: A tensorflow toolbox for identifying and characterizing fixed points in recurrent neural networks. Journal of Open Source Software, 3(31):1003, 2018

    2018

  8. [16]

    Rethinking brain- wide interactions through multi-region ‘network of net- works’ models

    Matthew G Perich and Kanaka Rajan. Rethinking brain- wide interactions through multi-region ‘network of net- works’ models. Current opinion in neurobiology , 65:146– 151, 2020

    2020

  9. [17]

    Mikail Khona and Ila R. Fiete. Attractor and integra- tor networks in the brain. Nature Reviews Neuroscience, 23(12):744–766, Dec 2022

    2022

  10. [18]

    Alvarez, Asohan Amaras- ingham, Habiba Azab, Zhe S

    Daniel Levenstein, Veronica A. Alvarez, Asohan Amaras- ingham, Habiba Azab, Zhe S. Chen, Richard C. Gerkin, Andrea Hasenstaub, Ramakrishnan Iyer, Renaud B. Jo- livet, Sarah Marzen, Joseph D. Monaco, Astrid A. Prinz, Salma Quraishi, Fidel Santamaria, Sabyasachi Shivku- mar, Mat...

    2023

  11. [19]

    The simplicity bias in multi-task rnns: Shared attractors, reuse of dynamics, and geometric representation

    Elia Turner and Omri Barak. The simplicity bias in multi-task rnns: Shared attractors, reuse of dynamics, and geometric representation. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, ed- itors, Advances in Neural Information Processing Sys- tems, volum...

    2023

  12. [20]

    Buice, Caswell Barry, Robin Hayman, Neil Burgess, and Ila R

    KiJung Yoon, Michael A. Buice, Caswell Barry, Robin Hayman, Neil Burgess, and Ila R. Fiete. Specific evidence of low-dimensional continuous attractor dynamics in grid cells. Nature Neuroscience, 16(8):1077–1084, Aug 2013

    2013

  13. [21]

    The intrinsic attractor mani- fold and population dynamics of a canonical cognitive circuit across waking and sleep

    Rishidev Chaudhuri, Berk Ger¸ cek, Biraj Pandey, Adrien Peyrache, and Ila Fiete. The intrinsic attractor mani- fold and population dynamics of a canonical cognitive circuit across waking and sleep. Nature Neuroscience, 22(9):1512–1520, Sep 2019

    2019

  14. [22]

    A coupled attractor model of the rodent head direc- tion system

    A David Redish, Adam N Elga, and David S Touret- zky. A coupled attractor model of the rodent head direc- tion system. Network: Computation in Neural Systems , 7(4):671, nov 1996

    1996

  15. [23]

    Recurrent network models of sequence generation and memory

    Kanaka Rajan, Christopher D Harvey, and David W Tank. Recurrent network models of sequence generation and memory. Neuron, 90(1):128–142, 2016

    2016

  16. [24]

    Dynamics on the manifold: Identifying computational dynamical activity from neural population recordings

    Lea Duncker and Maneesh Sahani. Dynamics on the manifold: Identifying computational dynamical activity from neural population recordings. Current opinion in neurobiology, 70:163–170, 2021

    2021

  17. [25]

    Recurrent dynamics of prefrontal cortex during context-dependent decision- making

    Zach Cohen, Brian DePasquale, Mikio C Aoi, and Jonathan W Pillow. Recurrent dynamics of prefrontal cortex during context-dependent decision- making. bioRxiv, pages 2020–11, 2020

    2020

  18. [26]

    On the difficulty of learning chaotic dynamics with rnns

    Jonas Mikhaeil, Zahra Monfared, and Daniel Durstewitz. On the difficulty of learning chaotic dynamics with rnns. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Informa- tion Processing Systems, volume 35, pages 11297–11312. Cu...

    2022

  19. [27]

    On the difficulty of training recurrent neural networks

    Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning , pages 1310–1318. Pmlr, 2013

    2013

  20. [28]

    Bifurcations in the learning of recurrent neural networks 3

    Kenji Doya et al. Bifurcations in the learning of recurrent neural networks 3. learning (RTRL), 3:17, 1992

    1992

  21. [29]

    Identifying non- linear dynamical systems with multiple time scales and long-range dependencies

    Dominik Schmidt, Georgia Koppe, Zahra Monfared, Max Beutelspacher, and Daniel Durstewitz. Identifying non- linear dynamical systems with multiple time scales and long-range dependencies. In International Conference on Learning Representations, 2021

    2021

  22. [30]

    Identifying nonlinear dy- namical systems via generative recurrent neural networks with applications to fmri

    Georgia Koppe, Hazem Toutounji, Peter Kirsch, Stefanie Lis, and Daniel Durstewitz. Identifying nonlinear dy- namical systems via generative recurrent neural networks with applications to fmri. PLoS computational biology , 15(8):e1007263, 2019

    2019

  23. [31]

    The role of population structure in computations through neural dynamics

    Alexis Dubreuil, Adrian Valente, Manuel Beiran, Francesca Mastrogiuseppe, and Srdjan Ostojic. The role of population structure in computations through neural dynamics. Nature Neuroscience, pages 1–12, 2022

    2022

  24. [32]

    Task representations in neural networks trained to perform many cognitive tasks

    Guangyu Robert Yang, Madhura R Joglekar, H Francis Song, William T Newsome, and Xiao-Jing Wang. Task representations in neural networks trained to perform many cognitive tasks. Nature neuroscience, 22(2):297– 306, 2019

    2019

  25. [33]

    Bifurcations and loss jumps in rnn training

    Lukas Eisenmann, Zahra Monfared, Niclas Alexander G¨ oring, and Daniel Durstewitz. Bifurcations and loss jumps in rnn training. arXiv preprint arXiv:2310.17561 , 2023

    2023 arXiv

  26. [34]

    On the dynamics of learning time- aware behavior with recurrent neural networks

    Peter DelMastro, Rushiv Arora, Edward Rietman, and 16 Hava T Siegelmann. On the dynamics of learning time- aware behavior with recurrent neural networks. arXiv preprint arXiv:2306.07125, 2023

    2023 arXiv

  27. [35]

    Monfared and D

    Z. Monfared and D. Durstewitz. Existence of n-cycles and border-collision bifurcations in piecewise-linear continu- ous maps with applications to recurrent neural networks. Nonlinear Dynamics, 101(2):1037–1052, Jul 2020

    2020

  28. [36]

    Robert Haschke and Jochen J. Steil. Input space bifur- cation manifolds of recurrent neural networks. Neuro- computing, 64:25–38, 2005. Trends in Neurocomputing: 12th European Symposium on Artificial Neural Networks 2004

    2005

  29. [37]

    Generalized teacher forcing for learn- ing chaotic dynamics

    Florian Hess, Zahra Monfared, Manuel Brenner, and Daniel Durstewitz. Generalized teacher forcing for learn- ing chaotic dynamics. arXiv preprint arXiv:2306.04406 , 2023

    2023 arXiv

  30. [38]

    Nonlinear dynamics and chaos: With applications to physics, biology, chemistry, and engineer- ing

    Steven H Strogatz. Nonlinear dynamics and chaos: With applications to physics, biology, chemistry, and engineer- ing. CRC press, 2018

    2018

  31. [39]

    Preventing gradient explosions in gated recur- rent units

    Sekitoshi Kanai, Yasuhiro Fujiwara, and Sotetsu Iwa- mura. Preventing gradient explosions in gated recur- rent units. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett, editors, Advances in Neural Information Processing Systems, v...

    2017

  32. [40]

    The effect of the forget gate on bifurcation boundaries and dynamics in re- current neural networks and its implications for gradient- based optimization

    Alexander Rehmer and Andreas Kroll. The effect of the forget gate on bifurcation boundaries and dynamics in re- current neural networks and its implications for gradient- based optimization. In 2022 International Joint Confer- ence on Neural Networks (IJCNN) , pages 01–08, 2022

    2022

  33. [41]

    Ribeiro, Koen Tiels, Luis A

    Ant´ onio H. Ribeiro, Koen Tiels, Luis A. Aguirre, and Thomas Sch¨ on. Beyond exploding and vanishing gradi- ents: analysing rnn training using attractors and smooth- ness. In Silvia Chiappa and Roberto Calandra, edi- tors, Proceedings of the Twenty Third International Con- fe...

    2020

  34. [42]

    Chaos in random neural networks

    Haim Sompolinsky, Andrea Crisanti, and Hans-Jurgen Sommers. Chaos in random neural networks. Physical review letters, 61(3):259, 1988

    1988

  35. [43]

    Real-time computing without stable states: A new framework for neural computation based on per- turbations

    Wolfgang Maass, Thomas Natschl¨ ager, and Henry Markram. Real-time computing without stable states: A new framework for neural computation based on per- turbations. Neural computation, 14(11):2531–2560, 2002

    2002

  36. [44]

    Harnessing nonlinear- ity: Predicting chaotic systems and saving energy in wire- less communication

    Herbert Jaeger and Harald Haas. Harnessing nonlinear- ity: Predicting chaotic systems and saving energy in wire- less communication. science, 304(5667):78–80, 2004

    2004

  37. [45]

    Generating coherent patterns of activity from chaotic neural networks

    David Sussillo and Larry F Abbott. Generating coherent patterns of activity from chaotic neural networks. Neu- ron, 63(4):544–557, 2009

    2009

  38. [46]

    full-force: A target- based method for training recurrent networks

    Brian DePasquale, Christopher J Cueva, Kanaka Rajan, G Sean Escola, and LF Abbott. full-force: A target- based method for training recurrent networks. PloS one, 13(2):e0191527, 2018

    2018

  39. [47]

    A ghost mechanism: An analytical model of abrupt learning

    Fatih Dinc, Ege Cirakman, Yiqi Jiang, Mert Yuksek- gonul, Mark J Schnitzer, and Hidenori Tanaka. A ghost mechanism: An analytical model of abrupt learning. arXiv preprint arXiv:2501.02378 , 2025

    2025 arXiv

  40. [48]

    A line attractor encoding a persistent internal state requires neuropeptide signal- ing

    George Mountoufaris, Aditya Nair, Bin Yang, Dong- Wook Kim, Amit Vinograd, Samuel Kim, Scott W Lin- derman, and David J Anderson. A line attractor encoding a persistent internal state requires neuropeptide signal- ing. Cell, 2024

    2024

  41. [49]

    Ring attractor dynamics in the drosophila central brain

    Sung Soo Kim, Herv´ e Rouault, Shaul Druckmann, and Vivek Jayaraman. Ring attractor dynamics in the drosophila central brain. Science, 356(6340):849–853, 2017

    2017

  42. [50]

    Toroidal topology of population activity in grid cells

    Richard J Gardner, Erik Hermansen, Marius Pachitariu, Yoram Burak, Nils A Baas, Benjamin A Dunn, May- Britt Moser, and Edvard I Moser. Toroidal topology of population activity in grid cells. Nature, 602(7895):123– 128, 2022

    2022

  43. [51]

    Generalized teacher forcing for learning chaotic dynamics

    Florian Hess, Zahra Monfared, Manuel Brenner, and Daniel Durstewitz. Generalized teacher forcing for learning chaotic dynamics. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th Internati...

    2023

  44. [52]

    Beyond exploding and vanishing gradi- ents: analysing rnn training using attractors and smooth- ness

    Antˆ onio H Ribeiro, Koen Tiels, Luis A Aguirre, and Thomas Sch¨ on. Beyond exploding and vanishing gradi- ents: analysing rnn training using attractors and smooth- ness. In International conference on artificial intelligence and statistics , pages 2370–2380. PMLR, 2020

    2020

  45. [53]

    Fixedpointfinder: A tensorflow toolbox for identifying and characterizing fixed points in recurrent neural networks

    Matthew D Golub and David Sussillo. Fixedpointfinder: A tensorflow toolbox for identifying and characterizing fixed points in recurrent neural networks. Journal of Open Source Software, 3(31):1003, 2018

    2018

  46. [54]

    Automatic differentiation in pytorch

    Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zach DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS 2017 Workshop Au- todiff Decision Program Chairs, 2017

    2017

  47. [55]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  48. [56]

    Low- dimensional dynamics for working memory and time en- coding

    Christopher J Cueva, Alex Saez, Encarni Marcos, Aldo Genovesio, Mehrdad Jazayeri, Ranulfo Romo, C Daniel Salzman, Michael N Shadlen, and Stefano Fusi. Low- dimensional dynamics for working memory and time en- coding. Proceedings of the National Academy of Sciences, 117(37):230...

    2020

  49. [57]

    Ramping activity is a cor- tical mechanism of temporal control of action

    Nandakumar S Narayanan. Ramping activity is a cor- tical mechanism of temporal control of action. Current opinion in behavioral sciences , 8:226–230, 2016

    2016

  50. [58]

    Cornn: Convex optimization of recurrent neural networks for rapid inference of neural dynam- ics

    Fatih Dinc, Adam Shai, Mark Schnitzer, and Hide- nori Tanaka. Cornn: Convex optimization of recurrent neural networks for rapid inference of neural dynam- ics. Advances in Neural Information Processing Systems, 36:51273–51301, 2023

    2023

  51. [59]

    Neural networks and physical systems with emergent collective computational abilities

    John J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Pro- ceedings of the national academy of sciences, 79(8):2554– 2558, 1982

    1982

  52. [60]

    Domain adaptive video segmentation via temporal consistency regularization, 2021

    Dayan Guan, Jiaxing Huang, Aoran Xiao, and Shijian Lu. Domain adaptive video segmentation via temporal consistency regularization, 2021

    2021

  53. [61]

    Preserving semantic and temporal consistency for unpaired video- to-video translation

    Kwanyong Park, Sanghyun Woo, Dahun Kim, Donghyeon Cho, and In So Kweon. Preserving semantic and temporal consistency for unpaired video- to-video translation. In Proceedings of the 27th ACM International Conference on Multimedia . ACM, oct 2019

    2019

  54. [62]

    Coherent online video style transfer, 2017

    Dongdong Chen, Jing Liao, Lu Yuan, Nenghai Yu, and Gang Hua. Coherent online video style transfer, 2017

    2017

  55. [63]

    Real- time neural style transfer for videos

    Haozhi Huang, Hao Wang, Wenhan Luo, Lin Ma, Wen- 17 hao Jiang, Xiaolong Zhu, Zhifeng Li, and Wei Liu. Real- time neural style transfer for videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017

    2017

  56. [64]

    Artistic style transfer for videos

    Manuel Ruder, Alexey Dosovitskiy, and Thomas Brox. Artistic style transfer for videos. In Lecture Notes in Computer Science , pages 26–36. Springer International Publishing, 2016

    2016

  57. [65]

    Finally, most relevantly, [3] has utilized a penalty term that regularized the first-order dynamics in RNNs to incentivize RNNs to find simpler final solutions

    has argued that enforcing smaller first derivatives could allow taking large steps to simulate dynamical sys- tems faster. Finally, most relevantly, [3] has utilized a penalty term that regularized the first-order dynamics in RNNs to incentivize RNNs to find simpler final solu...

  58. [66]

    Learning differential equations that are easy to solve

    Jacob Kelly, Jesse Bettencourt, Matthew J Johnson, and David K Duvenaud. Learning differential equations that are easy to solve. Advances in Neural Information Pro- cessing Systems, 33:4370–4380, 2020

    2020

  59. [67]

    Understanding the difficulty of training deep feedforward neural networks

    Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics , pages 249–256. JMLR Workshop and Conference Proceedings, 2010

    2010

  60. [68]

    Chaotic recurrent neural networks for brain modelling: A review

    Andrea Mattera, Valerio Alfieri, Giovanni Granato, and Gianluca Baldassarre. Chaotic recurrent neural networks for brain modelling: A review. Neural Networks , page 107079, 2024

    2024

  61. [69]

    Robust timing and motor patterns by taming chaos in recurrent neural networks

    Rodrigo Laje and Dean V Buonomano. Robust timing and motor patterns by taming chaos in recurrent neural networks. Nature neuroscience, 16(7):925–933, 2013. Appendix S1: Methods

    2013

  62. [70]

    PLRNN dynamical systems equa- tion is as follows

    Network architectures Piece-wise linear recurrent neural networks We considered piece-wise linear recurrent networks (PLRNNs) [29, 30] for most of our experiments unless otherwise specified. PLRNN dynamical systems equa- tion is as follows. x[t] = Ax[t − 1] + W ϕ(x[t − 1]) + C...

  63. [71]

    Input to each trial, U = {(u1 1, u2 1), (u1 2, u2 2), ...,(u1 T , u2 T )}, has the shape of T × 2, where T is the trial length

    T ask details Delayed addition task The delayed addition task consists of several train and test trials. Input to each trial, U = {(u1 1, u2 1), (u1 2, u2 2), ...,(u1 T , u2 T )}, has the shape of T × 2, where T is the trial length. Here, u1 t ∼ U[0, 1) (S5) is sampled randoml...

  64. [72]

    Manifold attractor regularization Recent work has demonstrated the promise in promot- ing attractor formation [29]. In this framework, memory units are regularized to form a line-attractor subspace, while computation units remain unregularized, resulting in enhanced memory cap...

  65. [73]

    We first picked a data batch with 100 trials from the test set

    Energy minimization To extract the slow points of the network, the core im- plementation is based on previously published work [14], though we modified it slightly by replacing the minimiza- tion algorithm with a Pytorch based solver to speed up the process (similar to a later...

  66. [74]

    To fit the data to PCA, we first fed 100 trials from the test set to the fully- trained PLRNN and obtained the corresponding activa- tions x[t]s

    Visualization analysis To visualize attractor manifolds, we reduced the di- mensionality of x[t] and xe[t] into 3 with Principle Component Analysis (PCA). To fit the data to PCA, we first fed 100 trials from the test set to the fully- trained PLRNN and obtained the correspondi...

  67. [75]

    Latent circuit analysis in rank-one recurrent neural networks We investigated rapid skill acquisition mechanisms using interpretable rank-one recurrent neural networks (RNNs). The network dynamics are governed by: τ ˙r(t) = −r(t) + tanh(W r(t) + Cu(t) + h) (S9) where r(t) ∈ RN...

  68. [76]

    We picked 10 unregularized PLRNNs and 10 PLRNNs trained with MAR and MAI for the delayed addition task from the networks shown in Fig

    Stimulus Decoding Experiments To test the memory capabilities of attractor- incentivized networks compared to unregularized PLRNNs, we conducted cue and stimulus decoding experiments. We picked 10 unregularized PLRNNs and 10 PLRNNs trained with MAR and MAI for the delayed addi...

  69. [77]

    To conduct this analysis, we selected 3 representative networks from Fig

    F requency Analysis We analyzed frequency distributions of x[t] belong- ing to unregularized PLRNNs, PLRNNs with MAI, and PLRNNs with MAI+MAR. To conduct this analysis, we selected 3 representative networks from Fig. S4 A, with T = 20. We first concatenated 50 trials with T = ...

  70. [78]

    epoch b is computed based on the stalling points in the loss function during training

    T raining speed during the search phase We calculated the training speed during the search phase by taking 1 epochb . epoch b is computed based on the stalling points in the loss function during training. Specifically, we computed the first epoch that achieves a particular val...

  71. [79]

    (S11) Here, ri is the activity or firing rate of neuron i and zi is the total input current to neuron i

    Cue-responsive fixed point generation (online and local) For our online experiments, we once again used leaky firing rate RNNs: τ dri(t) dt = −ri(t) + f (zi(t)), zi(t) = NrecX j=1 Wijrj(t) + NinX j=1 Cijsj(t) + ϵi(t), ∀i ∈ [Nrec]. (S11) Here, ri is the activity or firing rate ...

    2000

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.