Pith. sign in

REVIEW 1 major objections 3 minor 29 references

The paper argues that orthogonalizing the mLSTM memory read during training is a removable training scaffold, not a memory-capacity improvement.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 08:57 UTC pith:EOUREI2Z

load-bearing objection A genuinely interesting mechanism study: the orthogonalized read is a removable training scaffold, and the escape-hazard reframing is well supported; weaknesses are localized to the storage probe and small seed counts. the 1 major comments →

arxiv 2607.19390 v2 pith:EOUREI2Z submitted 2026-07-02 cs.LG

The Orthogonalized Read Is a Removable Training Scaffold for Recurrent Memory

Keston Aquino-Michaels This is my paper
classification cs.LG
keywords recurrent memorymLSTMassociative recallorthogonalizationNewton-Schulztraining dynamicsescape hazardemergence
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to show that a recently reported accuracy gain—orthogonalizing the mLSTM memory matrix at read time with five Newton-Schulz iterations—is not a memory improvement. The training process on noisy associative recall is a long chance plateau followed by a sharp escape, and the orthogonalized read re-conditions that learning problem during the plateau rather than adding storage capacity. The paper demonstrates that the benefit is removable: anneal the orthogonalized read away after escape, and a numerically stock mLSTM retains 95–100% accuracy. A sympathetic reader would care because it implies that some recall-benchmark gains measure trainability and schedule luck, not architectural memory capacity.

Core claim

The central discovery is that the orthogonalized read works by making the optimizer's read-path gradients more coherent during the plateau—read-path gradient coherence roughly doubles—and by multiplying escape hazard roughly six-fold, independent of hardness. It is not read-time denoising: applying it to failed models at inference rescues none, and removing it from solved models after training costs little; an escape-gated anneal yields 13/16 seeds finishing as stock mLSTMs at mean 99.6% accuracy. Direct probing of memory shows failed models carry about 52% linearly decodable, key-specific associations while at chance behavior, so the plateau is a readout failure over half-written storage. T

What carries the argument

The central object is the orthogonalized read: replacing the mLSTM's matched-filter read q_t^T C_t with q_t^T NS5(C_t), where NS5 is five Newton-Schulz iterations (with normalization) toward the nearest orthogonal matrix, trained through. This is a self-consistent whitening of the read that changes the optimization problem, not the memory content. The argument is carried by survival analysis of escape times, an elimination table of read variants, and a ridge-decoding probe that reads stored associations out of the raw memory matrix.

Load-bearing premise

The claim that the plateau is a readout failure over half-written storage depends on ridge-decoding a single slice of the memory matrix being a faithful measure of what is stored; if the task-relevant association is not linearly decodable from that slice, the storage/readout dissociation is less clean.

What would settle it

Freeze the memory writes of a behaviorally failed model at chance, then train only the readout parameters; if the model still cannot escape to high accuracy, storage is not sufficient and the readout-failure claim fails. Conversely, if any seed trained with the scaffold collapses to chance when the orthogonalized read is removed at inference despite full training, the scaffold is not removable and the core claim is wrong.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Training with the scaffold then annealing it away yields a stock mLSTM with full accuracy and zero inference overhead.
  • Solved-rate on a single budget and schedule is a censored measurement of escape hazard; rankings can flip by changing the learning-rate schedule alone.
  • The orthogonalized read widens the workable learning-rate corridor and rescues the too-cold edge, but it acts as a uniform escape-hazard multiplier, not a capacity addition.
  • The original vocab-96 headline result required large batch as a gradient-noise condition, not as a capacity condition.
  • Failed models' memories are half-written and fully unread, so write-side conditioning fixes cannot resolve the plateau; readout learning is the bottleneck.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If this pattern holds broadly, other conditioning interventions—optimizer orthogonalization, normalization, or reparameterized reads—may be removable scaffolds; the anneal-away test is a cheap way to check.
  • The plateau-escape shape may be common in small-scale recall benchmarks, meaning architecture selection should report escape-time survival curves and schedule/batch robustness before crediting capacity.
  • The storage/readout dissociation offers a template for testing 'emergent' thresholds in larger models: probe internal structure while behavior is at chance, instead of inferring discontinuity from the metric alone.
  • A testable extension: on a different memory architecture or task without plateau-escape dynamics, the same orthogonalized read should show no benefit; if it does, the scaffold story would need revision.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 3 minor

Summary. This paper studies the 'orthogonalized read' (five Newton-Schulz iterations of the mLSTM memory matrix at read time) on MAD noisy associative recall. The authors replicate the reported benefit, then present evidence that the benefit is not an increase in memory capacity but a training scaffold: it reconditions the optimization problem during a long chance plateau. The evidence includes replication and a schedule confound (§3–4), an elimination table showing that self-consistent whitening (ns5 or RLS) is necessary while straight-through, delta-rule, frozen-key, and normalization variants fail (§5), swap and anneal experiments showing that the read can be removed after escape with no loss (§6), a hazard analysis giving a ~6.4x uniform escape-hazard multiplier with no hardness interaction (§7), a batch-size analysis separating heat (elasticity +3.0) from gradient noise (−1.65) (§8), and a ridge-decode probe suggesting failed models store about half of their associations in linearly decodable form (§9). The paper draws implications for benchmark-driven architecture selection and the emergent-abilities debate.

Significance. The result, if it holds, is significant: it turns a seemingly clean architectural win into a case study in optimization-controlled benchmarks. The paper's empirical discipline is a strength: per-seed traces are released, escape-time survival analysis and exact tests are used, the schedule confound is demonstrated with a paired McNemar flip, and the scaffold-removal claim is tested with an escape-gated anneal rather than a single checkpoint. The limitations are candid. The storage/readout probe is the softest link, but the central removability claim is independent of it. This paper deserves publication after a moderate local revision.

major comments (1)
  1. [§9, Table 2] The manuscript states that 'failed models' memories are half-written and fully unread' and that 'storage was never the binding constraint.' The only direct evidence is ridge decoding of k_u^T C_t at the write position. The wrong-key control (5.2% vs 52.3%) supports specificity, but §9 also reports that oracle substitution q_t := k_u fails on solved models (93–97% to 19–40%), implying the learned query geometry is essential. If so, linear decodability under a fixed key slice may not measure task-relevant storage; it could over- or under-estimate functional associations. This does not threaten the scaffold claim (established by §6 swap/anneal), but the 'half-written' wording and the storage conclusion go beyond what the probe establishes. Please add a control that decodes using the trained query features, or soften the claim to 'linearly decodable under a fixed key-matched probe.'
minor comments (3)
  1. [§7] The v96/s768 4,000-step comparison (baseline 1/4 vs ns5 4/4) seems to use 4 seeds rather than the 8 described for the grid; please clarify the seed count or report n explicitly.
  2. [§8] The escape law is fit on AdamW, but the lr^2/B noise scale is standard for SGD. Please qualify the 'gradient-noise' attribution or provide a small SGD comparison to justify the causal language.
  3. [§4] The phrase 'ns5's ~12-minute budget (12,000 constant-LR steps)' is confusing; clarify that the 12,000 steps are baseline steps at matched wall-clock, not ns5 steps.

Circularity Check

0 steps flagged

No significant circularity: central claim is tested against external benchmarks and independent variants; fitted quantities are reported as fits, not predictions.

full rationale

The derivation chain is self-contained and empirically falsifiable. The central conclusion that the orthogonalized read is a removable training scaffold is supported by external comparisons: schedule-stretch flips (Section 4), constant-LR and wall-clock baselines (Section 4), elimination variants including straight-through, delta-rule, frozen keys, normalization, and an exact RLS read (Section 5, Table 1), inference-time swap evaluations and escape-gated anneals leaving stock mLSTMs at 99.6% (Section 6), a 144-run hazard grid (Section 7), a batch/learning-rate hazard decomposition (Section 8), and ridge-decoding with wrong-key and oracle-query controls (Section 9). The only fitted quantities (6.4x hazard multiplier, +3.0/-1.65 elasticities) are presented as empirical fits and are not used to manufacture predictions; the original vocab-96 gain is reproduced under its own batch-64 regime, not predicted from the fits. The statement that solved-rate at a fixed budget is a censored measurement of escape time is a definitional consequence of the paper's explicitly bimodal outcome convention (Section 2), but it is not the source of the empirical content; the schedule, batch, and anneal experiments stand independently. The storage probe's adequacy is an interpretive assumption (a validity concern, not circularity), and no load-bearing self-citations or imported uniqueness claims appear; the only cited prior intervention [23] is external and independently replicated.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 0 invented entities

The central claim rests on a modest number of fitted statistical quantities (hazard multiplier, heat/noise elasticities, escape threshold) and on domain assumptions about bimodality, independent censoring, SGD noise scaling, and the ridge-decoding probe. No new physical or architectural entities are postulated; 'scaffold' and 'escape hazard' are interpretive labels for existing objects. The paper is transparent about most of these.

free parameters (5)
  • escape threshold = 80% accuracy
    Definition of 'escaped' in all survival/hazard analyses (§2). Outcomes are bimodal, so results are not very sensitive, but the threshold is chosen, not derived.
  • ns5 escape-hazard multiplier = 6.4 (95% CI [3.5, 11.7])
    Fitted in discrete-time hazard model over 1,633 run-intervals with 70 escapes (§7); used to claim the scaffold is a uniform multiplier.
  • heat elasticity = 3.0 (95% CI [2.0, 4.1])
    Fitted coefficient of log lr in the escape-hazard regression (§8); central to the heat/noise escape law.
  • gradient-noise elasticity = -1.65 (95% CI [-2.1, -1.2])
    Fitted coefficient of log(lr^2/B) in the same regression (§8); cancellation with heat elasticity explains the learning-rate corridor and batch-size effects.
  • anneal gate parameters = dwell 200 steps, ramp 500 steps
    Hand-chosen schedule for the escape-gated anneal (§6); the fixed-window variant shows qualitative robustness, but exact values are not derived.
axioms (6)
  • domain assumption Outcomes are bimodal: runs either stay at chance (~3%) or escape sharply to ≥90%.
    §2; the escape-hazard analysis and the censored-measurement interpretation depend on this discreteness.
  • domain assumption Independent censoring in hazard models, including treating destabilization as an independent competing risk.
    §7–8; acknowledged in §13 as potentially refinable with a joint competing-risks model.
  • domain assumption SGD gradient noise scales as lr^2/B.
    §8; standard scale used to separate heat and noise terms; not derived in the paper.
  • standard math Newton–Schulz iterations converge toward the nearest orthogonal matrix and are differentiable/trained through.
    §2, from [6,10]; defines the ns5 intervention and the whitening family.
  • domain assumption Linear ridge decoding of k_u^T C_t measures stored associations.
    §9; the storage/readout dissociation rests on this probe; the wrong-key control supports it but does not prove it.
  • domain assumption The MAD noisy-recall generator is a valid recall benchmark.
    §2; vendored from [19]; the paper's benchmark-level claims inherit this assumption.

pith-pipeline@v1.3.0-alltime-deepseek · 13788 in / 15460 out tokens · 155579 ms · 2026-08-02T08:57:00.331744+00:00 · methodology

0 comments
read the original abstract

A recent report finds that orthogonalizing the mLSTM memory matrix at read time (five Newton-Schulz iterations, trained through) substantially improves noisy associative recall. The effect replicates, but it is not a memory improvement. Training on this task is a long chance plateau followed by a sharp escape, and the orthogonalized read acts by re-conditioning the learning problem during the plateau. Three properties establish this. It must be self-consistent: an exact recursive least-squares read (the Mesa layer) reproduces it, while straight-through halves, delta-rule writes, frozen random keys, and plain normalization all fail. It is uniform: across a learning-rate x hardness grid it multiplies the escape hazard roughly six-fold with no detectable hardness dependence, widening the workable learning-rate corridor that narrows for the baseline. And it is removable: applied to failed models at inference it rescues none, and annealed away on an escape-triggered schedule it leaves numerically stock mLSTMs at full accuracy. Much of the published gain needs no architecture at all: solved-rate at a fixed budget measures escape hazard, which follows a heat/noise law (learning-rate elasticity +3.0, gradient-noise elasticity -1.65) under which the original vocab-96 result is a large-batch noise condition rather than a capacity one. Decoding the memory state directly shows failed models carry roughly half their associations in linearly recoverable form: the plateau is a readout failure over half-written storage. Two conclusions travel beyond the intervention: recall benchmarks used for architecture selection partly measure trainability, and the system is a fully instrumented model organism of "emergence," in which a sharp behavioral threshold demonstrably arises from a censored metric over gradually accumulating structure.

Figures

Figures reproduced from arXiv: 2607.19390 by Keston Aquino-Michaels.

Figure 1
Figure 1. Figure 1: Escape curves at the original budget of Tambde [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Baseline mLSTM escape curves under schedule variation (v80/s512; same 16 seeds per [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Escape fraction against each run’s measured wall-clock on identical L4 GPUs. Each family [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Effective rank of the raw memory C (final position, first validation batch) over training, for baseline, ns5, and delta runs; one line per seed, colored by outcome. Claims: §5. Swap evaluations. Re-evaluating trained checkpoints under the other read: bolting NS5 onto failed baselines at inference rescues none from chance (12/12 at-chance failures stay at chance; the single borderline seed, at 78%, edges to… view at source ↗
Figure 5
Figure 5. Figure 5: Whitening dose response: fraction of seeds solved vs. Newton–Schulz iteration count at [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Accuracy traces under the fixed-window scaffold anneal (window shaded): the orthogonal [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Basin map: solved seeds out of 8 per cell (constant LR [PITH_FULL_IMAGE:figures/full_fig_p011_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Baseline solved fractions across batch size [PITH_FULL_IMAGE:figures/full_fig_p012_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

29 extracted references · 15 linked inside Pith

  1. [1]

    P. D. Allison. Discrete-time methods for the analysis of event histories.Sociological Methodology, 13:61–98, 1982

    1982

  2. [2]

    Arora, S

    S. Arora, S. Eyuboglu, A. Timalsina, I. Johnson, M. Poli, J. Zou, A. Rudra, and C. Ré. Zoology: Measuring and improving recall in efficient language models.arXiv preprint arXiv:2312.04927, 2023. 15

    Pith/arXiv arXiv 2023

  3. [3]

    Arora, S

    S. Arora, S. Eyuboglu, M. Zhang, A. Timalsina, S. Alberti, D. Zinsley, J. Zou, A. Rudra, and C. Ré. Simple linear attention language models balance the recall-throughput tradeoff.arXiv preprint arXiv:2402.18668, 2024

    Pith/arXiv arXiv 2024

  4. [4]

    M. Beck, K. Pöppel, M. Spanring, A. Auer, O. Prudnikova, M. Kopp, G. Klambauer, J. Brand- stetter, and S. Hochreiter. xLSTM: Extended long short-term memory.arXiv preprint arXiv:2405.04517, 2024

    Pith/arXiv arXiv 2024

  5. [5]

    Bernstein and L

    J. Bernstein and L. Newhouse. Old optimizer, new norm: An anthology.arXiv preprint arXiv:2409.20325, 2024

    Pith/arXiv arXiv 2024

  6. [6]

    Björck and C

    Å. Björck and C. Bowie. An iterative algorithm for computing the best estimate of an orthogonal matrix.SIAM Journal on Numerical Analysis, 8(2):358–364, 1971

    1971

  7. [7]

    I. T. Ersoy and K. Wiesner. Noise-driven escape from metastable phases explains grokking in deep neural networks.arXiv preprint arXiv:2606.17120, 2026

    Pith/arXiv arXiv 2026

  8. [8]

    Gu and T

    A. Gu and T. Dao. Mamba: Linear-time sequence modeling with selective state spaces.arXiv preprint arXiv:2312.00752, 2023

    Pith/arXiv arXiv 2023

  9. [9]

    Hafner, T

    D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi. Dream to control: Learning behaviors by latent imagination. InInternational Conference on Learning Representations, 2020

    2020

  10. [10]

    N. J. Higham.Functions of Matrices: Theory and Computation. SIAM, 2008

    2008

  11. [11]

    Jacob, S

    B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018

    2018

  12. [12]

    Javaloy and A

    A. Javaloy and A. Vergari. An embarrassingly simple way to optimize orthogonal matrices at scale.arXiv preprint arXiv:2602.14656, 2026

    Pith/arXiv arXiv 2026

  13. [13]

    Jordan, Y

    K. Jordan, Y. Jin, V. Boza, J. You, F. Cesista, L. Newhouse, and J. Bernstein. Muon: An optimizer for hidden layers in neural networks.https://kellerjordan.github.io/posts/ muon/, 2024

    2024

  14. [14]

    E. L. Kaplan and P. Meier. Nonparametric estimation from incomplete observations.Journal of the American Statistical Association, 53(282):457–481, 1958

    1958

  15. [15]

    Loshchilov and F

    I. Loshchilov and F. Hutter. Decoupled weight decay regularization. InInternational Conference on Learning Representations, 2019

    2019

  16. [16]

    N. Mantel. Evaluation of survival data and two new rank order statistics arising in its consideration.Cancer Chemotherapy Reports, 50(3):163–170, 1966

    1966

  17. [17]

    Pandey and G

    V. Pandey and G. Singh. Variational linear attention: Stable associative memory for long-context transformers.arXiv preprint arXiv:2605.11196, 2026

    Pith/arXiv arXiv 2026

  18. [18]

    B. Peng, E. Alcaide, Q. Anthony, et al. RWKV: Reinventing RNNs for the transformer era. arXiv preprint arXiv:2305.13048, 2023

    Pith/arXiv arXiv 2023

  19. [19]

    M. Poli, A. W. Thomas, E. Nguyen, P. Ponnusamy, B. Deiseroth, K. Kersting, T. Suzuki, B. Hie, S. Ermon, C. Ré, C. Zhang, and S. Massaroli. Mechanistic design and scaling of hybrid architectures.arXiv preprint arXiv:2403.17844, 2024. 16

    Pith/arXiv arXiv 2024

  20. [20]

    Power, Y

    A. Power, Y. Burda, H. Edwards, I. Babuschkin, and V. Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets.arXiv preprint arXiv:2201.02177, 2022

    Pith/arXiv arXiv 2022

  21. [21]

    Schaeffer, B

    R. Schaeffer, B. Miranda, and S. Koyejo. Are emergent abilities of large language models a mirage? InAdvances in Neural Information Processing Systems, 2023

    2023

  22. [22]

    Schlag, K

    I. Schlag, K. Irie, and J. Schmidhuber. Linear transformers are secretly fast weight programmers. InInternational Conference on Machine Learning, 2021

    2021

  23. [23]

    A. Tambde. Matrix orthogonalization improves memory in recurrent models. Blog post, June 2026. URL https://ayushtambde.com/blog/ matrix-orthogonalization-improves-memory-in-recurrent-models . Accessed 2026- 07-01

    2026

  24. [24]

    von Oswald, M

    J. von Oswald, M. Schlegel, A. Meulemans, S. Kobayashi, E. Niklasson, N. Zucchet, N. Scherrer, N. Miller, M. Sandler, B. Agüera y Arcas, M. Vladymyrov, R. Pascanu, and J. Sacramento. Uncovering mesa-optimization algorithms in transformers.arXiv preprint arXiv:2309.05858, 2023

    Pith/arXiv arXiv 2023

  25. [25]

    von Oswald, N

    J. von Oswald, N. Scherrer, S. Kobayashi, L. Versari, S. Yang, S. Mittal, M. Schlegel, K. Maile, Y. Schimpf, O. Sieberling, A. Meulemans, R. A. Saurous, G. Lajoie, C. Frenkel, R. Pascanu, B. Agüera y Arcas, and J. Sacramento. MesaNet: Sequence modeling by locally optimal test-time training.arXiv preprint arXiv:2506.05233, 2025

    Pith/arXiv arXiv 2025

  26. [26]

    K. A. Wang, J. Shi, and E. B. Fox. Test-time regression: A unifying framework for designing sequence models with associative memory.arXiv preprint arXiv:2501.12352, 2025

    Pith/arXiv arXiv 2025

  27. [27]

    S. Wang, F. Zhang, J. Li, C. Du, C. Du, T. Pang, Z. Yang, M. Hong, and V. Y. F. Tan. Muon outperforms Adam in tail-end associative memory learning.arXiv preprint arXiv:2509.26030, 2025

    arXiv 2025

  28. [28]

    J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, E. H. Chi, T. Hashimoto, O. Vinyals, P. Liang, J. Dean, and W. Fedus. Emergent abilities of large language models.Transactions on Machine Learning Research, 2022

    2022

  29. [29]

    S. Yang, B. Wang, Y. Zhang, Y. Shen, and Y. Kim. Parallelizing linear transformers with the delta rule over sequence length.arXiv preprint arXiv:2406.06484, 2024. 17

    Pith/arXiv arXiv 2024