pith. machine review for the scientific record. sign in

arxiv: 2605.13370 · v1 · submitted 2026-05-13 · 💻 cs.LG · cs.CL

Recognition: 2 theorem links

· Lean Theorem

Phasor Memory Networks: Stable Backpropagation Through Time for Scalable Explicit Memory

Sungwoo Goo , Hwi-yeol Yun , Sangkeun Jung
Authors on Pith no claims yet

Pith reviewed 2026-05-14 20:32 UTC · model grok-4.3

classification 💻 cs.LG cs.CL
keywords explicit memoryphasor dynamicsgradient stabilitybackpropagation through timelong contextunitary constraintsneural networkssequence modeling
0
0 comments X

The pith

Constraining recurrent states to phase rotations on the complex unit circle preserves gradient norms in explicit memory networks.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shows that explicit memory networks can avoid catastrophic gradient instability during backpropagation through time by using unitary phasor dynamics. Instead of scaling up models or using special tricks, PMNet rotates memory states as phases on a unit circle, keeping gradient magnitudes stable by design. This allows a hierarchical memory structure to retrieve information over very long sequences in a controlled byte-level task. The result is a compact model that performs as well as much larger ones on long-context robustness without divergence issues.

Core claim

PMNet resolves the long-standing gradient instability in explicit memory architectures like the Neural Turing Machine by enforcing unitary phasor dynamics, where state updates are constrained to phase rotations on the complex unit circle, and by using hierarchical learnable anchors. This structural approach preserves gradient norms inherently, enabling stable training on long sequences and effective use of an 85-slot hierarchical memory tree for near-perfect retrieval beyond local attention windows.

What carries the argument

Unitary Phasor Dynamics: constraining recurrent state updates to phase rotations on a complex unit circle to preserve gradient norms and prevent divergence.

If this is right

  • Explicit memory can be trained stably without specialized initialization or gradient clipping.
  • A hierarchical memory tree enables exact retrieval across temporal distances exceeding local sliding windows.
  • Compact 119M parameter models can achieve long-context performance comparable to three-times-larger models.
  • The historical failure of explicit memory was due to structural misalignment rather than fundamental limitations.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Hybrid models combining phasor memory with attention might handle even longer contexts more efficiently.
  • Testing on diverse natural language benchmarks would reveal if the synthetic task success translates without additional tuning.
  • Similar phase-based constraints could stabilize other recurrent architectures prone to vanishing or exploding gradients.

Load-bearing premise

The phase rotation constraint and hierarchical anchors maintain enough model expressivity to handle natural language without new failure modes or heavy hyperparameter tuning.

What would settle it

A demonstration of gradient divergence or significantly degraded performance on a real-world long-context language modeling task would falsify the claim that the unitary phasor approach overcomes the structural issues.

Figures

Figures reproduced from arXiv: 2605.13370 by Hwi-yeol Yun, Sangkeun Jung, Sungwoo Goo.

Figure 1
Figure 1. Figure 1: Evaluation of Zero-shot Generalization on PG-19. Comparison of Byte-level Perplexity (BPB, lower is better) on the PG-19 test set. (Green) PMNet (119M) (Ours): Trained on FineWeb￾Edu. Despite the dual disadvantage of being 3× smaller than the baseline and evaluated in a zero-shot setting, PMNet maintains stable extrapolation up to 512k bytes. This confirms that our Phasor Dynamics capture universal linguis… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the PMNet Architecture. The model integrates local processing with a global hierarchical memory system. The global context is organized as a sparse tree structure. At each hierarchy level h, a routing mechanism selects a specific active memory group index gh, allowing the total memory capacity to scale exponentially with depth while maintaining O(1) access complexity per layer. 5 [PITH_FULL_IM… view at source ↗
Figure 3
Figure 3. Figure 3: Controlled Architectural Comparison. Evaluation of ≈30M parameter models trained from scratch on 0.5B FineWeb-Edu tokens. PMNet consistently outperforms parameter-matched SmolLM and Mamba baselines. We ablate memory and recurrence: No Recurrence disables recur￾rence only during evaluation; No Memory / No Recurrence serves as a pure SWA baseline (disabled in both phases); and No Memory / Recurrence disables… view at source ↗
Figure 4
Figure 4. Figure 4: Mechanistic Validation via Copy-Paste Accuracy. The SWA-only baseline (No Memory) exhibits a catastrophic collapse in accuracy at sequence lengths N > 768 as the required lookback distance (2N) exceeds its receptive field limit of 1,536 bytes (12 layers × 128 window size). In contrast, despite being trained on a variable distribution of lengths N ∈ [10, 1024]—corresponding to a maximum lookback distance of… view at source ↗
Figure 5
Figure 5. Figure 5: Training dynamics on the copy-paste task for PMNet (No memory) and PMNet. PMNet (No [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: (1) Monotonic Power-Law Convergence: PMNet follows a smooth power-law scaling curve, indicating that the hierarchical memory is effectively compressing information rather than accumulating noise. (2) Bounded Gradient Norms: As shown in Figure 6b, gradient norms remain strictly bounded throughout the entire 18.8B training process, empirically validating that the exploding gradient problem has been structura… view at source ↗
Figure 7
Figure 7. Figure 7: Structural Stability via Hierarchical Embeddings. Cumulative Delta BPB over a 128k sequence (PG-19) under zero-shot ablation. Removing all embeddings (Orange) causes linear addressing drift. The stark contrast between ablating the root (Red) and leaf (Green) validates our design: roots form the critical addressing schema, while leaves offer scalable, fine-grained detail storage. A critical challenge in exp… view at source ↗
read the original abstract

For over a decade, explicit memory architectures like the Neural Turing Machine have remained theoretically appealing yet practically intractable for language modeling due to catastrophic gradient instability during Backpropagation Through Time. In this work, we break this stalemate with \textit{Phasor Memory Network} (PMNet), a novel architecture that structurally resolves memory volatility through \textit{Unitary Phasor Dynamics} and \textit{Hierarchical Learnable Anchors}. Rather than relying on brute-force scaling, we present a mechanistic proof-of-concept in a controlled byte-level setting. By constraining recurrent state updates to phase rotations on a complex unit circle, PMNet preserves gradient norms and inherently prevents divergence without the need for specialized initialization. We empirically demonstrate the active actuation of the memory module through a synthetic Copy-Paste task, where PMNet utilizes an expansive \textit{85-slot hierarchical memory tree} ($=\sum^{4}_{h=1}4^{h-1}$) to achieve near 100\% exact retrieval across temporal distances that completely exceed the local sliding window attention's receptive field. Furthermore, despite being a compact 119M parameter model trained on 18.8B tokens, PMNet matches the zero-shot long-context robustness of a Mamba model that is three times larger. Our ablation studies and gradient analyses confirm that the historical failure of explicit memory was a structural alignment problem, which PMNet effectively overcomes, providing a theoretically grounded foundation for scalable sequence modeling.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The manuscript introduces Phasor Memory Networks (PMNet) that enforce unitary phasor dynamics by constraining recurrent state updates to phase rotations on the complex unit circle, augmented by hierarchical learnable anchors, to achieve stable backpropagation through time in explicit memory architectures. It reports near-100% exact retrieval on a synthetic Copy-Paste task using an 85-slot hierarchical memory tree and claims to match the long-context robustness of a three-times-larger Mamba model with a compact 119M-parameter network trained on 18.8B tokens.

Significance. If the unitary constraint and anchor mechanism can be shown to preserve both gradient norms and sufficient representational capacity, the approach would provide a structural alternative to ad-hoc stabilization techniques for explicit memory, potentially enabling scalable integration of read-write memory into sequence models without catastrophic divergence during BPTT.

major comments (3)
  1. [Unitary Phasor Dynamics and Hierarchical Learnable Anchors] The central stability claim (Abstract) rests on phase rotations preserving gradient norms under exact unitarity, yet the Hierarchical Learnable Anchors are introduced without a demonstration that their parameter updates remain strictly unitary; any effective magnitude scaling or non-isometric transformation during retrieval would invalidate the norm-preservation guarantee for the full forward pass.
  2. [Representational Capacity] By fixing every memory slot to |z|=1, the architecture removes any mechanism for continuous amplitude-based modulation or decay of stored values; all information must be encoded purely in phase, which constitutes a stricter representational bottleneck than standard complex or real-valued memory cells and is not addressed in the ablation studies.
  3. [Empirical Evaluation] The synthetic Copy-Paste results claim near-100% retrieval across distances exceeding the local attention window, but the text provides neither error bars, full training details, nor an ablation isolating the contribution of the unitary constraint versus the hierarchical tree structure, leaving the attribution of success to the phasor dynamics unverifiable.
minor comments (2)
  1. [Abstract] The abstract refers to a 'mechanistic proof-of-concept' and 'theoretically grounded foundation,' yet no explicit derivations, gradient-norm proofs, or stability theorems appear in the provided text; a dedicated analysis section would strengthen the presentation.
  2. [Model Architecture] Clarify how the 85-slot memory tree (sum from h=1 to 4 of 4^{h-1}) is parameterized within the total 119M count and whether the branching factor and depth are treated as free hyperparameters.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the detailed and constructive feedback. We address each major comment point-by-point below. Revisions will be incorporated to strengthen the claims regarding unitarity preservation, representational considerations, and empirical rigor.

read point-by-point responses

  1. Referee: [Unitary Phasor Dynamics and Hierarchical Learnable Anchors] The central stability claim (Abstract) rests on phase rotations preserving gradient norms under exact unitarity, yet the Hierarchical Learnable Anchors are introduced without a demonstration that their parameter updates remain strictly unitary; any effective magnitude scaling or non-isometric transformation during retrieval would invalidate the norm-preservation guarantee for the full forward pass.

    Authors: The hierarchical anchors are implemented as fixed-magnitude phase vectors on the unit circle, with updates performed via phase-only rotations (i.e., multiplication by complex exponentials of learnable angles). This construction ensures that anchor retrieval and write operations remain isometric by design. We will add a short appendix subsection with the explicit update rule and a short proof that the composite forward pass (phasor dynamics + anchor lookup) preserves Euclidean norm of the state vector at every step. This directly addresses the concern about potential magnitude scaling. revision: yes

  2. Referee: [Representational Capacity] By fixing every memory slot to |z|=1, the architecture removes any mechanism for continuous amplitude-based modulation or decay of stored values; all information must be encoded purely in phase, which constitutes a stricter representational bottleneck than standard complex or real-valued memory cells and is not addressed in the ablation studies.

    Authors: We agree that the unit-magnitude constraint is a deliberate design choice that trades amplitude modulation for guaranteed stability. Phase encoding can still represent continuous values (via angular resolution) and supports the exact retrieval observed on the Copy-Paste task. To address the gap, we will expand the discussion section to explain this trade-off and add a new ablation that relaxes the constraint (allowing learnable magnitudes) and shows that the non-unitary variant diverges during BPTT while the phasor version remains stable. This will be reported as an additional row in Table 2. revision: partial

  3. Referee: [Empirical Evaluation] The synthetic Copy-Paste results claim near-100% retrieval across distances exceeding the local attention window, but the text provides neither error bars, full training details, nor an ablation isolating the contribution of the unitary constraint versus the hierarchical tree structure, leaving the attribution of success to the phasor dynamics unverifiable.

    Authors: We will revise the experimental section to include: (i) error bars computed over five independent random seeds for all Copy-Paste accuracy curves; (ii) a complete hyperparameter table and training schedule moved to Appendix B; and (iii) a new ablation that disables the unitary constraint (replacing phase rotations with standard complex linear updates) while keeping the identical hierarchical tree. The non-unitary variant exhibits gradient explosion and <20% retrieval accuracy, confirming the contribution of the phasor dynamics. These additions will appear in the revised Figure 3 and Table 3. revision: yes

Circularity Check

0 steps flagged

No significant circularity; stability follows from explicit unitary constraint

full rationale

The paper defines PMNet via an explicit architectural constraint (recurrent updates restricted to phase rotations on the complex unit circle) and states that this preserves gradient norms as a direct mathematical property of unitary transformations. This is a standard result from linear algebra and prior unitary RNN literature, not a quantity fitted to the paper's own outputs or reduced to a self-referential definition. No equations or claims are shown to equate the reported performance (e.g., Copy-Paste retrieval or Mamba-scale comparison) back to parameters defined by the result itself. Hierarchical anchors are introduced as additional learnable components without any indication that their updates are forced to match the stability claim by construction. The work is therefore self-contained against external benchmarks and does not rely on load-bearing self-citations or ansatz smuggling.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 1 invented entities

Central claim rests on the assumption that phase rotations on the unit circle preserve both stability and expressivity; hierarchical anchors are introduced without independent justification beyond the synthetic result.

free parameters (1)
  • memory tree branching factor and depth
    The 85-slot tree is defined as sum 4^{h-1} for h=1 to 4; the specific base-4 structure is chosen rather than derived.
axioms (1)
  • domain assumption Recurrent updates can be restricted to unitary phase rotations without loss of modeling power for sequence tasks.
    Invoked to guarantee gradient-norm preservation.
invented entities (1)
  • Hierarchical Learnable Anchors no independent evidence
    purpose: Organize the memory tree for distant retrieval
    New component introduced to structure the 85-slot memory; no external evidence provided.

pith-pipeline@v0.9.0 · 5567 in / 1320 out tokens · 29190 ms · 2026-05-14T20:32:48.647889+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

21 extracted references · 21 canonical work pages · 10 internal anchors

  1. [1]

    B., Lozhkov, A., Bakouch, E., von Werra, L., and Wolf, T

    Allal, L. B., Lozhkov, A., Bakouch, E., von Werra, L., and Wolf, T. (2024). Smollm-blazingly fast and remarkably powerful.Hugging Face Blog, 16

    work page 2024

  2. [2]

    Arjovsky, M., Shah, A., and Bengio, Y . (2016). Unitary evolution recurrent neural networks. InInternational conference on machine learning, pages 1120–1128. PMLR

    work page 2016

  3. [3]

    Cheng, X., Zeng, W., Dai, D., Chen, Q., Wang, B., Xie, Z., Huang, K., Yu, X., Hao, Z., Li, Y ., et al. (2026). Conditional memory via scalable lookup: A new axis of sparsity for large language models.arXiv preprint arXiv:2601.07372

    work page arXiv 2026

  4. [4]

    Child, R. (2019). Generating long sequences with sparse transformers.arXiv preprint arXiv:1904.10509

    work page internal anchor Pith review Pith/arXiv arXiv 2019

  5. [5]

    G., Le, Q., and Salakhutdinov, R

    Dai, Z., Yang, Z., Yang, Y ., Carbonell, J. G., Le, Q., and Salakhutdinov, R. (2019). Transformer-xl: Attentive language models beyond a fixed-length context. InProceedings of the 57th annual meeting of the association for computational linguistics, pages 2978–2988

    work page 2019

  6. [6]

    Graves, A., Wayne, G., and Danihelka, I. (2014). Neural turing machines.arXiv preprint arXiv:1410.5401

    work page internal anchor Pith review Pith/arXiv arXiv 2014

  7. [7]

    G., Grefenstette, E., Ramalho, T., Agapiou, J., et al

    Graves, A., Wayne, G., Reynolds, M., Harley, T., Danihelka, I., Grabska-Barwi´nska, A., Colmenarejo, S. G., Grefenstette, E., Ramalho, T., Agapiou, J., et al. (2016). Hybrid computing using a neural network with dynamic external memory.Nature, 538(7626):471–476

    work page 2016

  8. [8]

    and Dao, T

    Gu, A. and Dao, T. (2024). Mamba: Linear-time sequence modeling with selective state spaces. InFirst conference on language modeling

    work page 2024

  9. [9]

    Gu, A., Goel, K., and Ré, C. (2021). Efficiently modeling long sequences with structured state spaces.arXiv preprint arXiv:2111.00396

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  10. [10]

    Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. (2020). Transformers are rnns: Fast autoregressive transformers with linear attention. InInternational conference on machine learning, pages 5156–5165. PMLR

    work page 2020

  11. [11]

    Decoupled Weight Decay Regularization

    Loshchilov, I. and Hutter, F. (2017). Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  12. [12]

    A., V on Werra, L., Wolf, T., et al

    Penedo, G., Kydlí ˇcek, H., Lozhkov, A., Mitchell, M., Raffel, C. A., V on Werra, L., Wolf, T., et al. (2024). The fineweb datasets: Decanting the web for the finest text data at scale.Advances in Neural Information Processing Systems, 37:30811–30849

    work page 2024

  13. [13]

    Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Biderman, S., Cao, H., Cheng, X., Chung, M., Grella, M., et al. (2023). Rwkv: Reinventing rnns for the transformer era.arXiv preprint arXiv:2305.13048. 10

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  14. [14]

    Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation

    Press, O., Smith, N. A., and Lewis, M. (2021). Train short, test long: Attention with linear biases enables input length extrapolation.arXiv preprint arXiv:2108.12409

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  15. [15]

    W., Potapenko, A., Jayakumar, S

    Rae, J. W., Potapenko, A., Jayakumar, S. M., and Lillicrap, T. P. (2019). Compressive transformers for long-range sequence modelling.arXiv preprint arXiv:1911.05507

    work page arXiv 2019

  16. [16]

    Simon, J., Kunin, D., Atanasov, A., Boix-Adserà, E., Bordelon, B., Cohen, J., Ghosh, N., Guth, F., Jacot, A., Kamb, M., et al. (2026). There will be a scientific theory of deep learning.arXiv preprint arXiv:2604.21691

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  17. [17]

    Sim- plified state space layers for sequence modeling.arXiv preprint arXiv:2208.04933, 2022

    Smith, J. T., Warrington, A., and Linderman, S. W. (2022). Simplified state space layers for sequence modeling.arXiv preprint arXiv:2208.04933

    work page arXiv 2022

  18. [18]

    Su, J., Lu, Y ., Pan, S., Murtadha, A., Wen, B., and Liu, Y . (2021). Roformer: enhanced transformer with rotary position embedding. arxiv.arXiv preprint arXiv:2104.09864

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  19. [19]

    Sun, Y ., Dong, L., Huang, S., Ma, S., Xia, Y ., Xue, J., Wang, J., and Wei, F. (2023). Retentive network: A successor to transformer for large language models.arXiv preprint arXiv:2307.08621

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  20. [20]

    Deep Complex Networks

    Trabelsi, C., Bilaniuk, O., Zhang, Y ., Serdyuk, D., Subramanian, S., Santos, J. F., Mehri, S., Rostamzadeh, N., Bengio, Y ., and Pal, C. J. (2017). Deep complex networks.arXiv preprint arXiv:1705.09792

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  21. [21]

    N., Hutchins, D., and Szegedy, C

    Wu, Y ., Rabe, M. N., Hutchins, D., and Szegedy, C. (2022). Memorizing transformers.arXiv preprint arXiv:2203.08913. 11 A Notations Table 1: Summary of notations and tensor shapes used in PMNet. Symbol Description l, tLayer index, Time step h, pHierarchy level, Write period LTotal layers HMemory hierarchy depth, L/P dDimension of hidden state dm Dimension...

    work page arXiv 2022