pith. sign in

arxiv: 2601.19019 · v2 · submitted 2026-01-26 · 🧬 q-bio.NC · cs.LG

Embedding of Low-Dimensional Sensory Dynamics in Recurrent Networks: Implications for the Geometry of Neural Representation

Vikas N. O'Reilly-Shah , Alessandro Maria Selvitella This is my paper

Pith reviewed 2026-05-16 10:29 UTC · model grok-4.3

classification 🧬 q-bio.NC cs.LG
keywords recurrent neural networksneural manifoldssensory dynamicsgeneralized synchronizationdelay embeddingpredictionrepresentational geometry
0
0 comments X

The pith

Recurrent networks develop smooth internal manifolds embedding low-dimensional sensory dynamics when neuron count exceeds twice the input dimension.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shows that recurrent neural networks driven by simple periodic sensory inputs such as circles and tori form smooth internal manifolds that capture the structure of those inputs. This embedding occurs generically in contracting networks once the number of neurons passes a modest threshold of more than twice the sensory dimension. The work further links prediction performance to representational geometry by showing that accurate prediction requires network states to separate sufficiently different inputs, which in turn creates categorical boundaries and sets discrimination limits. A sympathetic reader would care because the account ties the observed low-dimensional organization of sensory cortical activity directly to basic properties of recurrent dynamics and the need to predict.

Core claim

Contracting recurrent networks generically develop smooth internal manifolds embedding the sensory dynamics with N>2d neurons sufficient, where d is the intrinsic sensory dimension. Accurate prediction forces state separation up to a resolution set by prediction error, yielding categorical boundaries, metameric equivalence, and discrimination thresholds. Numerical experiments with trained tanh RNNs recover ring- and torus-shaped hidden manifolds, and state separation improves sharply at the 2d+1 threshold. Training pushes networks beyond strict contraction yet embedding persists.

What carries the argument

generalized synchronization combined with delay-embedding theory applied to contracting recurrent networks receiving low-dimensional regular sensory inputs

If this is right

  • Networks with more than twice the sensory dimension can form faithful internal copies of the input manifold.
  • Prediction accuracy directly sets the resolution at which states separate, producing metameric equivalence for close inputs.
  • Trained networks can exceed strict contraction while the embedding structure remains intact.
  • State separation sharpens once neuron count crosses the 2d+1 threshold.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If biological sensory areas approximate recurrent dynamics with low-dimensional inputs, their manifold geometries may arise from the same embedding and prediction requirements.
  • The account predicts that disrupting accurate prediction in a trained network should reduce state separation and blur category boundaries.
  • Testing would involve checking whether observed manifold dimensionality in cortex scales with twice the intrinsic dimension of the relevant sensory input.

Load-bearing premise

Real cortical circuits behave like recurrent networks driven by clean low-dimensional periodic sensory dynamics so that generalized synchronization and delay embedding apply directly.

What would settle it

Trained recurrent networks with N greater than 2d failing to produce smooth embedding manifolds for regular inputs such as circles or tori, or prediction accuracy showing no systematic relation to the degree of state separation.

Figures

Figures reproduced from arXiv: 2601.19019 by Alessandro Maria Selvitella, Vikas N. O'Reilly-Shah.

Figure 1
Figure 1. Figure 1: Generalized synchronization as a commutative diagram. The environment evolves via ϕ (top); the neural state evolves via the recurrent map F (bottom). The synchronization function f : M → R N (dashed blue) embeds environmental states into neural state space. The observation function ω : M → R (green) provides sensory input to the recurrent dynamics. Commutativity implies that evolving the environmental stat… view at source ↗
read the original abstract

Neural population activity in sensory cortex is organized on low-dimensional manifolds, but why such manifolds arise and what determines their geometry remain unclear. We model cortical populations as recurrent circuits driven by low-dimensional regular sensory dynamics (circles, tori). Combining generalized synchronization and delay-embedding theory, we show that contracting recurrent networks generically develop smooth internal manifolds embedding the sensory dynamics. The dimensional requirement is modest: N>2d suffices, where d is the intrinsic sensory dimension (compatible with Whitney and Takens bounds). We prove a prediction-separation result linking representational geometry to predictive performance without assuming contraction: accurate prediction forces state separation up to a resolution set by prediction error, yielding categorical boundaries, metameric equivalence, and discrimination thresholds. Numerical experiments with trained tanh RNNs recover ring- and torus-shaped hidden manifolds; state separation improves sharply at the 2d+1 threshold. Training pushes networks beyond strict contraction, yet embedding persists, indicating sufficient but not necessary conditions. These results provide a mechanistic account of why sensory manifolds emerge in recurrent circuits and how prediction constrains their resolution.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 2 minor

Summary. The manuscript claims that recurrent networks driven by low-dimensional regular sensory dynamics (circles, tori) generically develop smooth internal manifolds embedding the input via generalized synchronization and delay-embedding theory, with N > 2d neurons sufficient. It proves a prediction-separation result (independent of contraction) showing that accurate prediction enforces state separation up to a resolution set by prediction error, producing categorical boundaries, metameric equivalence, and discrimination thresholds. Numerical experiments with trained tanh RNNs recover the predicted ring- and torus-shaped hidden manifolds and show sharp improvement in state separation at the 2d+1 threshold, even after training pushes networks beyond strict contraction.

Significance. If the results hold, the work supplies a mechanistic account of why low-dimensional sensory manifolds arise in recurrent circuits and how predictive performance constrains their geometry and resolution. The use of established embedding theorems together with a contraction-independent separation argument, plus supporting numerics that survive training, constitutes a clear advance linking dynamical systems to neural representation.

major comments (1)
  1. [Prediction-separation theorem] Prediction-separation theorem (abstract and §4): the claim that separation distance is set by prediction error is central to the categorical-boundary prediction, yet the manuscript provides no explicit functional bound (e.g., separation ≥ f(prediction error)) that could be compared directly with psychophysical discrimination thresholds; this step must be made quantitative for the result to be falsifiable.
minor comments (2)
  1. [Numerical experiments] Methods section: the precise RNN architecture (neuron count N, input dimension d, training protocol, and contraction-rate diagnostics) is only summarized; explicit parameter values and code availability are required for independent reproduction of the manifold-recovery and threshold results.
  2. [Dimensional requirement] §3: the compatibility statement with Whitney/Takens bounds is noted but would benefit from a short paragraph contrasting the generalized-synchronization embedding dimension with the classical delay-embedding requirement to clarify the modest N > 2d claim.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the positive assessment and constructive comment. We address the major comment below.

read point-by-point responses

  1. Referee: [Prediction-separation theorem] Prediction-separation theorem (abstract and §4): the claim that separation distance is set by prediction error is central to the categorical-boundary prediction, yet the manuscript provides no explicit functional bound (e.g., separation ≥ f(prediction error)) that could be compared directly with psychophysical discrimination thresholds; this step must be made quantitative for the result to be falsifiable.

    Authors: We agree that an explicit functional lower bound would make the result more directly comparable to psychophysical data. The current theorem (Section 4) shows that accurate prediction implies state separation up to a resolution controlled by the prediction error, but does not state a specific inequality of the form separation distance ≥ f(prediction error). We will revise the manuscript to derive and include such a quantitative bound from the existing proof, updating Section 4 and the abstract accordingly. This addresses the falsifiability concern. revision: yes

Circularity Check

0 steps flagged

Minor self-citation but central claims rest on external results and independent proof

full rationale

The derivation combines generalized synchronization and delay-embedding theorems cited from external literature with a separate prediction-separation argument that explicitly avoids assuming contraction. The N>2d bound is presented as compatible with Whitney/Takens embedding theorems rather than derived internally. Numerical results come from standard RNN training on clean low-dimensional inputs without parameter fitting that would enforce the claimed manifold geometry by construction. No load-bearing step reduces to a self-defined quantity or a self-citation chain whose validity depends on the present paper. This yields a low but non-zero score reflecting normal self-citation without circular reduction of the main results.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The central claims rest on two established mathematical frameworks plus standard RNN assumptions. No free parameters are introduced to fit the manifold geometry. No new entities are postulated.

axioms (2)
  • domain assumption Generalized synchronization and delay-embedding theory apply to contracting recurrent networks driven by low-dimensional regular inputs.
    Invoked to guarantee smooth internal manifolds when N>2d.
  • domain assumption Cortical populations can be modeled as recurrent circuits receiving clean low-dimensional sensory dynamics.
    Stated as the modeling premise for the entire account.

pith-pipeline@v0.9.0 · 5494 in / 1538 out tokens · 38700 ms · 2026-05-16T10:29:24.779823+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Neural Manifolds as Crystallized Embeddings: A Synthesis of the Free Energy Principle, Generalized Synchronization, and Hebbian Plasticity

    q-bio.NC 2026-05 unverdicted novelty 5.0

    Neural manifolds arise as embeddings from generalized synchronization in recurrent circuits driven by sensory input and are crystallized by Hebbian plasticity into continuous attractor networks.

Reference graph

Works this paper leans on

46 extracted references · 46 canonical work pages · cited by 1 Pith paper

  1. [1]

    B Mindlin

    G Uribarri and G. B Mindlin. Dynamical time series embeddings in recurrent neural networks. Chaos, Solitons & Fractals, 154:111612, 2022. doi: 10.1016/j.chaos.2021.111612

    work page doi:10.1016/j.chaos.2021.111612 2022

  2. [2]

    Delay embedding theory of neural sequence models

    Mitchell Ostrow, Adam Eisen, and Ila Fiete. Delay embedding theory of neural sequence models.International Conference on Machine Learning (ICML) workshop on Next Generation Sequence Modeling, 2024. URLhttps://arxiv.org/abs/2406.11993v1

    work page arXiv 2024

  3. [3]

    A. G Hart. Generic and isometric embeddings in reservoir computers.Chaos, 35:111103, 2025. doi: 10.1063/5.0288408

    work page doi:10.1063/5.0288408 2025

  4. [4]

    Detecting strange attractors in turbulence

    F Takens. Detecting strange attractors in turbulence. In D. Rand and L.-S. Young, editors, Dynamical Systems and Turbulence, Warwick 1980, volume 898 ofLecture Notes in Mathematics, pages 366–381. Springer, 1981

    work page 1980

  5. [5]

    A Yorke, and M Casdagli

    T Sauer, J. A Yorke, and M Casdagli. Embedology.Journal of Statistical Physics, 65(3–4): 579–616, 1991. doi: 10.1007/BF01053745

    work page doi:10.1007/bf01053745 1991

  6. [6]

    Delay coordinate embedding as neuronally implemented information processing: The state space theory of consciousness.Journal of Consciousness Studies, 32(1–2): 132–164, 2025

    Vikas O’Reilly-Shah. Delay coordinate embedding as neuronally implemented information processing: The state space theory of consciousness.Journal of Consciousness Studies, 32(1–2): 132–164, 2025

    work page 2025

  7. [7]

    State space theory as a unifying framework for consciousness.Nonlinear Dynamics, Psychology, and Life Sciences, forthcoming/scheduled, April 2026

    Vikas O’Reilly-Shah. State space theory as a unifying framework for consciousness.Nonlinear Dynamics, Psychology, and Life Sciences, forthcoming/scheduled, April 2026. URLhttps: //philpapers.org/rec/ORESST

    work page 2026

  8. [8]

    Visuo-motor coordination and internal models for object interception.Experimental Brain Research, 192(4):571–604, 2009

    M Zago, J McIntyre, P Senot, and F Lacquaniti. Visuo-motor coordination and internal models for object interception.Experimental Brain Research, 192(4):571–604, 2009. doi: 10.1007/s00221-008-1691-3

    work page doi:10.1007/s00221-008-1691-3 2009

  9. [9]

    J. A. S Kelso.Dynamic patterns: The self-organization of brain and behavior. MIT Press, 1995

    work page 1995

  10. [10]

    D Golub, D Sussillo, and K

    S Vyas, M. D Golub, D Sussillo, and K. V Shenoy. Computation through neural pop- ulation dynamics.Annual Review of Neuroscience, 43:249–275, 2020. doi: 10.1146/ annurev-neuro-092619-094115

    work page 2020

  11. [11]

    L Saltzman and K

    E. L Saltzman and K. G Munhall. A dynamical approach to gestural patterning in speech production.Ecological Psychology, 1(4):333–382, 1989

    work page 1989

  12. [12]

    R. F Port. Meter and speech.Journal of Phonetics, 31(3–4):599–611, 2003

    work page 2003

  13. [13]

    W Large and J

    E. W Large and J. F Kolen. Resonance and the perception of musical meter.Connection Science, 6(2–3):177–208, 1994

    work page 1994

  14. [14]

    A Wandell.Foundations of vision

    B. A Wandell.Foundations of vision. Sinauer Associates, 1995

    work page 1995

  15. [15]

    H Brainard and W

    D. H Brainard and W. T Freeman. Bayesian color constancy.Journal of the Optical Society of America A, 14(7):1393–1411, 1997

    work page 1997

  16. [16]

    The free-energy principle: a unified brain theory? , volume =

    K Friston. The free-energy principle: A unified brain theory?Nature Reviews Neuroscience, 11 (2):127–138, 2010. doi: 10.1038/nrn2787. 25

    work page doi:10.1038/nrn2787 2010

  17. [17]

    Whatever next? predictive brains, situated agents, and the future of cognitive science

    A Clark. Whatever next? predictive brains, situated agents, and the future of cognitive science. Behavioral and Brain Sciences, 36(3):181–204, 2013

    work page 2013

  18. [18]

    M Churchland, J

    M. M Churchland, J. P Cunningham, M. T Kaufman, J. D Foster, P Nuyujukian, S. I Ryu, and K. V Shenoy. Neural population dynamics during reaching.Nature, 487(7405):51–56, 2012. doi: 10.1038/nature11129

    work page doi:10.1038/nature11129 2012

  19. [19]

    Activation of the primary visual cortex by Braille reading in blind subjects.Nature, 380(6574): 526–528, 1996

    N Sadato, A Pascual-Leone, J Grafman, V Ibañez, M.-P Deiber, G Dold, and M Hallett. Activation of the primary visual cortex by Braille reading in blind subjects.Nature, 380(6574): 526–528, 1996

    work page 1996

  20. [20]

    G Cohen, P Celnik, A Pascual-Leone, B Corwell, L Faiz, J Dambrosia, M Honda, N Sadato, C Gerloff, M

    L. G Cohen, P Celnik, A Pascual-Leone, B Corwell, L Faiz, J Dambrosia, M Honda, N Sadato, C Gerloff, M. D Catalá, and M Hallett. Functional relevance of cross-modal plasticity in blind humans.Nature, 389(6647):180–183, 1997

    work page 1997

  21. [21]

    W Śliwińska, A Amedi, and M Szwed

    K Siuda-Krzywicka, A Marchewka, M. W Śliwińska, A Amedi, and M Szwed. Massive cortical reorganization in sighted Braille readers.eLife, 5:e10762, 2016. doi: 10.7554/eLife.10762

    work page doi:10.7554/elife.10762 2016

  22. [22]

    N Whitehead.Process and Reality

    A. N Whitehead.Process and Reality. Macmillan, 1929

    work page 1929

  23. [23]

    G Husserl.On the phenomenology of the consciousness of internal time (1893-1917) (J

    E. G Husserl.On the phenomenology of the consciousness of internal time (1893-1917) (J. B. Brough, Trans.). Kluwer Academic Publishers, 1991

    work page 1917

  24. [24]

    Theyieldcurveandpredictingrecessions

    Vikas O’Reilly-Shah. Computational dynamic monism: Process metaphysics for the state space theory of consciousness.SSRN Electronic Journal (PREPRINT), dec 2025. doi: 10.2139/ssrn. 6042716. URLhttps://ssrn.com/abstract=6042716

    work page doi:10.2139/ssrn 2025

  25. [25]

    Delay embeddings for forced systems

    J Stark. Delay embeddings for forced systems. i. deterministic forcing.Journal of Nonlinear Science, 9(2):255–332, 1999

    work page 1999

  26. [26]

    Embedding and approximation theorems for echo state networks

    A Hart, J Hook, and J Dawes. Embedding and approximation theorems for echo state networks. Neural Networks, 128:234–247, 2020

    work page 2020

  27. [27]

    Regularity of invariant graphs for forced systems.Ergodic Theory and Dynamical Systems, 19(1):155–199, 1999

    J Stark. Regularity of invariant graphs for forced systems.Ergodic Theory and Dynamical Systems, 19(1):155–199, 1999

    work page 1999

  28. [28]

    , author Plenz, D

    J. M Beggs and D Plenz. Neuronal avalanches in neocortical circuits.Journal of Neuroscience, 23(35):11167–11177, 2003. doi: 10.1523/JNEUROSCI.23-35-11167.2003

    work page doi:10.1523/jneurosci.23-35-11167.2003 2003

  29. [29]

    Real-time computation at the edge of chaos in recurrent neural networks.Neural Computation, 16(7):1413–1436, 2004

    N Bertschinger and T Natschläger. Real-time computation at the edge of chaos in recurrent neural networks.Neural Computation, 16(7):1413–1436, 2004

    work page 2004

  30. [30]

    J Bruce, and P

    S Funahashi, C. J Bruce, and P. S Goldman-Rakic. Mnemonic coding of visual space in the monkey’s dorsolateral prefrontal cortex.Journal of Neurophysiology, 61(2):331–349, 1989

    work page 1989

  31. [31]

    Synaptic reverberation underlying mnemonic persistent activity.Trends in Neurosciences, 24(8):455–463, 2001

    X.-J Wang. Synaptic reverberation underlying mnemonic persistent activity.Trends in Neurosciences, 24(8):455–463, 2001. doi: 10.1016/S0166-2236(00)01868-3

    work page doi:10.1016/s0166-2236(00)01868-3 2001

  32. [32]

    K Inagaki, L Fontolan, S Romani, and K Svoboda

    H. K Inagaki, L Fontolan, S Romani, and K Svoboda. Discrete attractor dynamics underlies persistent activity in the frontal cortex.Nature, 566(7743):212–217, 2019. doi: 10.1038/ s41586-018-0783-x. 26

    work page 2019

  33. [33]

    D Murray, A Bernacchia, D

    J. D Murray, A Bernacchia, D. J Freedman, R Romo, J. D Wallis, X Cai, C Padoa-Schioppa, T Pasternak, H Seo, D Lee, and X.-J Wang. A hierarchy of intrinsic timescales across primate cortex.Nature Neuroscience, 17(12):1661–1663, 2014

    work page 2014

  34. [34]

    Echo state property linked to an input: Exploring a fundamental characteristic of recurrent neural networks.Neural Computation, 25(3):671–696, 2013

    G Manjunath and H Jaeger. Echo state property linked to an input: Exploring a fundamental characteristic of recurrent neural networks.Neural Computation, 25(3):671–696, 2013. doi: 10.1162/NECO_a_00489

    work page doi:10.1162/neco_a_00489 2013

  35. [35]

    B Yildiz, H Jaeger, and S

    I. B Yildiz, H Jaeger, and S. J Kiebel. Re-visiting the echo state property.Neural Networks, 35:1–9, 2012. doi: 10.1016/j.neunet.2012.07.005

    work page doi:10.1016/j.neunet.2012.07.005 2012

  36. [36]

    Distributed fading memory for stimulus properties in the primary visual cortex.PLOS Biology, 7(12):e1000260, 2009

    D Nikolić, S Häusler, W Singer, and W Maass. Distributed fading memory for stimulus properties in the primary visual cortex.PLOS Biology, 7(12):e1000260, 2009. doi: 10.1371/ journal.pbio.1000260

    work page 2009

  37. [37]

    B Rubin, M Lengyel, and K

    G Hennequin, Y Ahmadian, D. B Rubin, M Lengyel, and K. D Miller. The dynamical regime of sensory cortex: Stable dynamics around a single stimulus-tuned attractor account for patterns of noise variability.Neuron, 98(4):846–860, 2018

    work page 2018

  38. [38]

    High-dimensional geometry of population responses in visual cortex

    C Stringer, M Michaelos, D Tsyboulski, S. E Lindo, and M Pachitariu. High-dimensional geometry of population responses in visual cortex.Nature, 571(7765):361–365, 2019. doi: 10.1038/s41586-019-1346-5

    work page doi:10.1038/s41586-019-1346-5 2019

  39. [39]

    H Jafri, and R Ramaswamy

    G Keller, H. H Jafri, and R Ramaswamy. Nature of weak generalized synchronization in chaotically driven maps.Physical Review E, 87(4):042913, 2013

    work page 2013

  40. [40]

    D Murray, A Bernacchia, N

    J. D Murray, A Bernacchia, N. A Roy, C Constantinidis, R Romo, and X.-J Wang. Stable populationcodingforworkingmemorycoexistswithheterogeneousneuraldynamicsinprefrontal cortex.Proceedings of the National Academy of Sciences, 114(2):394–399, 2017

    work page 2017

  41. [41]

    Oxford University Press, New York, NY, May 2007

    Robert Gilmore and Christophe Letellier.The symmetry of chaos. Oxford University Press, New York, NY, May 2007

    work page 2007

  42. [42]

    H Herzog

    A Doerig, A Schurger, K Hess, and M. H Herzog. The unfolding argument: Why IIT and other causal structure theories cannot explain consciousness.Consciousness and Cognition, 72:49–59,

    work page

  43. [43]

    doi: 10.1016/j.concog.2019.03.003

    work page doi:10.1016/j.concog.2019.03.003 2019

  44. [44]

    A caveat regarding the unfolding argument: Implications of plasticity for computational theories of consciousness

    Vikas O’Reilly-Shah, Alessandro Selvitella, and Aaron Schurger. A caveat regarding the unfolding argument: Implications of plasticity for computational theories of consciousness. bioRxiv (PREPRINT), 2025. doi: 10.1101/2025.11.04.686457. URL https://www.biorxiv. org/content/early/2025/11/05/2025.11.04.686457.1

    work page doi:10.1101/2025.11.04.686457 2025

  45. [45]

    S Broomhead, M

    J Stark, D. S Broomhead, M. E Davies, and J Huke. Delay embeddings for forced systems. ii. stochastic forcing.Journal of Nonlinear Science, 13(6):519–577, 2003

    work page 2003

  46. [46]

    J Gibson.The ecological approach to visual perception

    J. J Gibson.The ecological approach to visual perception. Houghton Mifflin, 1979. 27

    work page 1979