Pith. sign in

REVIEW 3 major objections 4 minor 64 references

Associative memory inspires improvements for in-context learning using a novel attention residual stream architecture

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Adding a residual connection that passes attention values from each head to the same-index head in the next layer improves in-context learning in Transformers across scales from a two-layer synthetic task to 1-billion-parameter language…

desk verdict Useful small-scale result; the 1B scale-up claim rests on a single run per architecture and a statistically invalid paired t-test, so treat the headline as unproven. read the letter →

arxiv 2412.15113 v2 pith:EK4NJR3K submitted 2024-12-19 cs.NE cs.AIcs.CL

Thomas F Burns , Tomoki Fukai , Christopher J Earls This is my paper · ORCID
classification cs.NEcs.AIcs.CL
keywords in-contextlearningassociativememoryresidualvaluesstreamattentiontransformerlanguagemodelsindirectobjectidentificationneuroscience-inspiredarchitecture
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models can adapt to tasks and patterns that appear only in their input context, an ability called in-context learning (ICL). This paper argues that ICL can be understood as associative memory, and constructs a one-layer associative memory model, AMICL, that performs an object–label ICL task in a single forward pass by letting attention values directly stand for the remembered tokens. From that model the paper derives an architectural prescription: let the attention values of each head in one layer flow directly into the same-index head of the next layer through a residual values stream. In a two-layer Transformer this modification reaches 0.95 accuracy on in-context probes about 24 percent faster than the classic network, and it also improves ICL-related evaluations in language models with 8 million and 1 billion parameters. If the claim is right, a small wiring change that adds no parameters could make Transformers acquire few-shot adaptation earlier in training and carry it to larger, more naturalistic settings.

What carries the argument

The load-bearing object is the residual values stream: a residual connection, present in every layer but the first, that feeds the value vectors $V_{n-1}$ of each attention head into the values $V_n$ of the same-index head in the next layer, so that $V_n = W_v^n X_n + V_{n-1}$. It adds no trainable parameters. In AMICL, the values are the input tokens themselves, so the stream is the Transformer analogue of retaining the remembered pattern; in a trained network it preserves 'look-back' information in the answers that attention produces, acting as a memory pathway across layers. The argument is framed through the universal associative memory operation $\mathrm{projection}(\mathrm{separation}(\mathrm{similarity}(K,Q)),V)$, with the value stream carrying the projection term that determines what is retrieved.

What would settle it

Train the 1B classic and residual-values-stream architectures from several fresh random seeds, holding data order and compute budget fixed, and compare the distributions of final accuracy on ARC, OpenBookQA, and IOI-Hard; if the residual stream's advantage does not exceed run-to-run variance, the scaling claim fails. As a mechanism check at small scale, scramble the head-index mapping of the residual stream: if the IOI gain persists under scrambling, the improvement is not about preserving specific head values.

Watch

Extended reading notes

Core claim

The paper's central claim is that residual attention value streams improve in-context learning. It first defines AMICL, an associative memory model in which each token's key and query are a weighted average of the previous and current token, the value is the token itself, and next-token prediction is the universal associative memory operation $\mathrm{projection}(\mathrm{separation}(\mathrm{similarity}(K,Q)),V)$. Because values equal inputs, retrieval returns the original stored label, which the paper identifies as auto-association. Training experiments show that deleting value projections outright destabilizes learning, so the paper retains learned value matrices and instead adds the previous layer's values to the current layer's values, $V_2 = W_v^2 X + V_1$, with multi-head models adding same-index heads. In the two-layer synthetic Transformer, residual values streams cross the 0.95-accuracy threshold on the in-context probes IC and IC2 about 24 percent sooner than the classic architecture and beat residual streams applied to queries or keys. In 8M-parameter language models trained on TinyStories, an indirect-object-identification probe improves from about 7 percent to about 41 percent correct; in 1B-parameter Llama-style models, average single- and five-shot accuracy on ARC, PIQA, OpenBookQA, and HellaSwag is moderately higher, with the largest gains on ARC, OpenBookQA, and a 1,000-prompt IOI-Hard probe.

Load-bearing premise

The 1-billion-parameter comparison rests on a single training run for each architecture, and the reported significance test compares evaluation checkpoints of those same two runs, so the observed advantage could shrink or vanish when the models are retrained from different random seeds.

Editorial extensions

If this is right

  • In the two-layer synthetic setup, residual values streams make in-context probes IC and IC2 cross the 0.95-accuracy threshold about 24 percent sooner than the classic network, and the ordering holds at lower thresholds of 0.5 and 0.9.
  • The benefit is tied to values rather than to any skip connection: residual streams on queries or keys also speed learning relative to classic, but residual values streams are significantly faster on IC and IC2.
  • At 8M parameters, the residual values stream lifts correct next-token probability on an indirect-object-identification probe from about 7 percent to about 41 percent while reducing confusion between the subject and the indirect object.
  • At 1B parameters, the modification yields modest but sustained average accuracy gains across ARC, PIQA, OpenBookQA, and HellaSwag, with a large gain on the IOI-Hard probe, suggesting the effect transfers to more naturalistic data and larger scale.
  • Because the modification adds no parameters, it is a pure wiring change that can be layered onto existing Transformer stacks without altering parameter count.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the mechanism is preservation of input-identity information, then weighting the added value stream (analogous to AMICL's parameter $a$ or a successor-representation discount factor) could tune how far across layers a contextual association persists; the paper leaves that weight fixed at 1.
  • The same-index-head coupling is a convenience choice; a learned or random permutation of head indices between layers would test whether the benefit comes from preserving particular head values or from any preserved value information.
  • The 1B result is one training run per architecture, so a natural next experiment is to train both architectures from several random seeds and compare final accuracies; the paper's paired t-test over checkpoints does not capture seed variance.
  • If value streams make intermediate activations more stable, they may also aid interpretability by creating a linear path through layers that can be inspected to see where in-context associations are formed.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces AMICL, a one-layer associative memory model that performs an in-context classification task through an attention-like update, and uses it as inspiration for a residual values stream: in each Transformer layer after the first, the values of each attention head are added to the values of the same-index head in the next layer. The authors report that in a two-layer Transformer on the synthetic ICL task, the residual values stream reaches 0.95 accuracy on held-out in-context probes roughly 24% faster than the classic architecture, with t-tests over four seeds. They then apply the same modification to 8M-parameter and 1B-parameter language models; the 8M models show higher accuracy on an IOI probe, and the single pair of 1B models shows modest average gains on ARC/OpenBookQA and a large gain on a constructed IOI-Hard task. The central claim is that this simple, parameter-neutral architectural change improves in-context learning and that the improvement scales to naturalistic language-model settings.

Significance. If the 1B-scale result were secure, this would be a practically interesting result: a one-line residual connection between same-index attention head values, requiring no additional parameters, that improves ICL at scale. The paper has real strengths: the code is released, the two-layer experiments use four seeds with standard deviations and t-tests on held-out probes, and the 8M experiments use three matched seeds. The AMICL construction is a useful conceptual bridge between associative memory and attention, although it is a constructed model rather than a derivation, and the smoothing parameter a is hand-set after a grid scan; this does not create circularity for the architecture experiments because those are evaluated on tasks not used to tune a. The main weakness is that the scale-up claim rests on a single pair of 1B training runs and an invalid paired t-test over correlated checkpoints, which is the load-bearing evidence for the abstract's claim of improved performance at larger scales.

major comments (3)
  1. [§5.2, Figures 5 and 6, t-test paragraph] The 1B-scale conclusion rests on a single training run per architecture, so there is no estimate of seed or initialization variance. The paired two-sample t-test is applied to evaluation checkpoints from these same two runs; the checkpoints are highly autocorrelated, and the authors themselves report a 0.993 correlation between the two accuracy trajectories. The effective sample size is therefore far smaller than the number of checkpoints, and the reported p-values do not provide valid evidence that the improvement would replicate across runs. Because this is the only evidence at the 1B scale, the abstract's claim of improved performance 'at these larger and more naturalistic scales' is not secured. Please provide multiple independent runs per architecture, or analyze final-checkpoint differences with appropriate uncertainty, and calibrate the language of the claims.
  2. [§5.2, Figure 5 and Appendix Figure 14] The reported significance test is applied to average accuracy over training steps, which conflates learning speed with final performance. If the residual model learns faster but converges to the same final accuracy, the average-over-training would show a significant difference even though the final deployed model is no better. Please report final-checkpoint accuracies (with uncertainty if multiple runs become available) separately from the learning-speed comparison, and state whether the observed gap persists at the end of training.
  3. [§4.2, Table 2, and §5.2, Figure 6] The two natural-language ICL evaluations lack adequate inferential support for the strength of the claims. Table 2 reports large per-sentence standard deviations (for example, classic Sentence 1 correct probability 11.73 ± 15.62) but no significance test across the three seeds, and Figure 6 shows IOI-Hard curves with no error bars because each curve is a single run. Since these evaluations carry the scaling narrative from the synthetic task to language models, please add per-seed breakdowns or appropriate tests, or explicitly label the comparisons as descriptive.
minor comments (4)
  1. [§5.2] Please report the actual t-statistics, degrees of freedom, and p-values rather than 'much smaller than 0.05', and state whether the test was one-tailed or two-tailed in the main text.
  2. [Figure 5 caption] Please state explicitly what is averaged in the 'average accuracy': four tasks, two shot settings, and which checkpoints, and whether the average is unweighted.
  3. [Appendix E, notation table] There are minor typographical issues: 'langauge models' and the extra space in 'V ariables'; these should be corrected.
  4. [§6.3] The limitations paragraph appropriately acknowledges that the gains are modest and task-specific, but this sits in tension with the abstract and Section 5.2's concluding sentence; the strength of the claims should be aligned across the abstract, the results, and the limitations.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the architecture experiments are evaluated on held-out tasks and benchmarks, and the AMICL model is a constructive demonstration rather than a fitted prediction.

full rationale

The paper's derivation chain is constructive rather than derivational, and no load-bearing step reduces to its own inputs by the paper's own equations or by self-citation. AMICL is an explicitly built associative-memory-style model: keys/queries are pairwise averages of adjacent tokens and values are the input tokens themselves, so its success on the synthetic ICL task is a demonstration of a designed mechanism, not a prediction extracted from a fitted quantity. The scalar a is tuned on that same task ('after testing within the range [0, 2], is set as a = 2'), but no subsequent claim depends on that fit: the residual-value-stream modification uses an implicit weight of 1 and the two-layer, 8M, and 1B comparisons are trained architectures evaluated on data and tasks not used to select a. The residual stream adds no learned parameters, so the improvements cannot be explained by additional capacity fitted to the evaluation sets. The 1B results are assessed on ARC, PIQA, OpenBookQA, HellaSwag, and IOI-Hard, none of which were used for architectural hyperparameter selection. Citations to the authors' own prior work (e.g., Burns 2024 for hetero-association, Burns et al. 2022 and Burns & Fukai 2023 for associative-memory capacity) are contextual or illustrative, not load-bearing; there is no imported uniqueness theorem and no ansatz smuggled in via a self-citation. The main weakness flagged by the skeptical reading — a paired t-test over autocorrelated checkpoints from a single run per architecture — is a statistical-validity limitation, not a circular reduction, and the paper's own limitations section explicitly concedes that the gains are modest and task-specific. No step meets the required evidence threshold of exhibiting Eq. X = Eq. Y by construction or a fitted parameter renamed as a prediction.

Assumptions & free parameters 2 free parameters · 4 assumptions · 1 invented entities

The central claim rests on the synthetic task design, the proxy tasks for ICL, and on the statistical treatment of the 1B comparison. The 'a' and 'epsilon' are hand-set hyperparameters of the AMICL demonstration; the residual stream itself adds no fitted constants.

free parameters (2)
  • AMICL smoothing parameter a = 2
    Chosen after testing a in [0,2]; performance is perfect for a >= 1.5 so the exact value is not critical, but it is hand-set.
  • object inter-instance variability epsilon = 0.1
    Set to 0.1 in the synthetic task; not varied in reported experiments.
assumptions (4)
  • domain assumption Synthetic ICL task token embeddings are i.i.d. Gaussian with variance 1/e
    Underlies AMICL analysis and the two-layer Transformer training.
  • domain assumption The universal associative memory framework (Millidge et al.) correctly generalizes attention
    Used to interpret AMICL as an attention-like memory update.
  • domain assumption IOI and IOI-Hard tasks measure in-context learning rather than memorization
    Used as proxy for ICL at 8M and 1B scales; not demonstrated to be free of shortcut solutions.
  • ad hoc to paper Paired t-test across evaluation checkpoints is valid inference for architecture comparison
    Applied to 1B results with one model per architecture; checkpoints of the same run are not independent samples.
invented entities (1)
  • residual attention stream (residual values stream) independent evidence
    purpose: Carries attention values from one layer's heads to same-index heads in the next layer to improve ICL
    Tested in two-layer, 8M, and 1B experiments; predictions of improved ICL are falsifiable and partly confirmed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Associative memory inspires improvements for in-context learning using a novel attention residual stream architecture." pith.science (2026). https://pith.science/paper/EK4NJR3K

@misc{pith2026241215113,
  author       = {Pith},
  title        = {Pith review of: Associative memory inspires improvements for in-context learning using a novel attention residual stream architecture},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EK4NJR3K}},
  note         = {Machine review of arXiv:2412.15113}
}
read the original abstract

Large language models (LLMs) demonstrate an impressive ability to utilise information within the context of their input sequences to appropriately respond to data unseen by the LLM during its training procedure. This ability is known as in-context learning (ICL). Humans and non-human animals demonstrate similar abilities, however their neural architectures differ substantially from LLMs. Despite this, a critical component within LLMs, the attention mechanism, resembles modern associative memory models, widely used in and influenced by the computational neuroscience community to model biological memory systems. Using this connection, we introduce an associative memory model capable of performing ICL. We use this as inspiration for a novel residual stream architecture which allows information to directly flow between attention heads. We test this architecture during training within a two-layer Transformer and show its ICL abilities manifest more quickly than without this modification. We then apply our architecture in small language models with 8 million and 1 billion parameters, focusing on attention head values, with results also indicating improved performance at these larger and more naturalistic scales.

Figures

Figures reproduced from arXiv: 2412.15113 by the authors.

Figure 1
Figure 1. Depiction of the AMICL algorithm on label–object pairs, consisting of three steps: (a) consider the [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Partial diagrams of (a) the AMICL model and (b) our residual attention stream architecture with [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Accuracies and losses for the Test, IW, IC, and IC2 tasks over training time for the classic [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Training loss over steps for the 1B models. [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Average accuracy across training steps for [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Accuracy across training steps for 1,000 com [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Performance of a single (value-absent) attention layer followed by three ReLU layers over training [PITH_FULL_IMAGE:figures/full_fig_p030_7.png]
Figure 8
Figure 8. Figure 8: Performance of a model with three ReLU layers followed by a single (value-absent) attention layer [PITH_FULL_IMAGE:figures/full_fig_p030_8.png]
Figure 9
Figure 9. Figure 9: Performance of a model with two ReLU layers, followed by a single (value-absent) attention layer, [PITH_FULL_IMAGE:figures/full_fig_p030_9.png]
Figure 10
Figure 10. Figure 10: Performance of a model with three ReLU layers, followed by a single (value-absent) attention [PITH_FULL_IMAGE:figures/full_fig_p031_10.png]
Figure 11
Figure 11. Figure 11: Attention matrices for label–object pairs in AMICL, using different similarity and separation [PITH_FULL_IMAGE:figures/full_fig_p031_11.png]
Figure 12
Figure 12. Figure 12: Proportion of correct trials, tested over [PITH_FULL_IMAGE:figures/full_fig_p032_12.png]
Figure 13
Figure 13. Figure 13: Proportion of correct trials, tested over [PITH_FULL_IMAGE:figures/full_fig_p032_13.png]
Figure 14
Figure 14. Figure 14: Accuracy across training steps for our evaluations of the 1B model, single- and five-shot. [PITH_FULL_IMAGE:figures/full_fig_p033_14.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

64 extracted references · 45 canonical work pages

  1. [1]

    Learning patterns and pattern sequences by self-organizing nets of threshold elements

    Shun'ichi Amari. Learning patterns and pattern sequences by self-organizing nets of threshold elements. IEEE Transactions on Computers, C-21 0 (11): 0 1197--1206, 1972. doi:10.1109/T-C.1972.223477

    work page arXiv 1972

  2. [2]

    Modeling brain function: The world of attractor neural networks

    Daniel J Amit. Modeling brain function: The world of attractor neural networks. Cambridge university press, 1989

    work page 1989

  3. [3]

    Attractor neural networks and biological reality: associative memory and learning

    Daniel J Amit. Attractor neural networks and biological reality: associative memory and learning. Future Generation Computer Systems, 6 0 (2): 0 111--119, 1990

    work page 1990

  4. [4]

    Cognitive maps and novel inferences: a flexibility hierarchy

    Erie D Boorman, Sarah C Sweigart, and Seongmin A Park. Cognitive maps and novel inferences: a flexibility hierarchy. Current Opinion in Behavioral Sciences, 38: 0 141--149, 2021. ISSN 2352-1546. doi:https://doi.org/10.1016/j.cobeha.2021.02.017. URL https://www.sciencedirect.com/science/article/pii/S2352154621000395. Computational cognitive neuroscience

    work page doi:10.1016/j.cobeha.2021.02.017 2021

  5. [5]

    Attention approximates sparse distributed memory

    Trenton Bricken and Cengiz Pehlevan. Attention approximates sparse distributed memory. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=WVYzd7GvaOM

    work page 2021

  6. [6]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

  7. [7]

    Semantically-correlated memories in a dense associative model

    Thomas F Burns. Semantically-correlated memories in a dense associative model. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp.\ 4936--4970. PM...

    work page 2024

  8. [8]

    Simplicial hopfield networks

    Thomas F Burns and Tomoki Fukai. Simplicial hopfield networks. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=_QLsH8gatwx

    work page 2023

Show all 64 references
  1. [9]

    Multiscale and extended retrieval of associative memory structures in a cortical model of local-global inhibition balance

    Thomas F Burns, Tatsuya Haga, and Tomoki Fukai. Multiscale and extended retrieval of associative memory structures in a cortical model of local-global inhibition balance. eNeuro, 9 0 (3), 2022. doi:10.1523/ENEURO.0023-22.2022. URL https://www.eneuro.org/content/9/3/ENEURO.0023-22.2022

    2022 doi

  2. [10]

    Neural syntax: Cell assemblies, synapsembles, and readers

    György Buzsáki. Neural syntax: Cell assemblies, synapsembles, and readers. Neuron, 68 0 (3): 0 362--385, 2010. ISSN 0896-6273. doi:https://doi.org/10.1016/j.neuron.2010.09.023. URL https://www.sciencedirect.com/science/article/pii/S0896627310007658

    2010 doi

  3. [11]

    Trade-off between multiple constraints enables simultaneous formation of modules and hubs in neural systems

    Yuhan Chen, Shengjun Wang, Claus C Hilgetag, and Changsong Zhou. Trade-off between multiple constraints enables simultaneous formation of modules and hubs in neural systems. PLoS Comput. Biol., 9 0 (3): 0 e1002937, March 2013

    2013

  4. [12]

    A memory-based theory of emotional disorders

    Rivka T Cohen and Michael Jacob Kahana. A memory-based theory of emotional disorders. Psychological Review, 129 0 (4): 0 742, 2022

    2022

  5. [13]

    Hypothalamic and other connections with dorsal ca2 area of the mouse hippocampus

    Zhenzhong Cui, Charles R Gerfen, and W Scott Young 3rd. Hypothalamic and other connections with dorsal ca2 area of the mouse hippocampus. Journal of comparative neurology, 521 0 (8): 0 1844--1866, 2013

    2013

  6. [14]

    Resvit: residual vision transformers for multimodal medical image synthesis

    Onat Dalmaz, Mahmut Yurt, and Tolga C ukur. Resvit: residual vision transformers for multimodal medical image synthesis. IEEE Transactions on Medical Imaging, 41 0 (10): 0 2598--2614, 2022

    2022

  7. [15]

    Exploring the relationship between in-context learning and instruction tuning, 2023

    Hanyu Duan, Yixuan Tang, Yi Yang, Ahmed Abbasi, and Kar Yan Tam. Exploring the relationship between in-context learning and instruction tuning, 2023. URL https://arxiv.org/abs/2311.10367

    2023 arXiv

  8. [16]

    Rediscovering area ca2: unique properties and functions

    Serena M Dudek, Georgia M Alexander, and Shannon Farris. Rediscovering area ca2: unique properties and functions. Nature Reviews Neuroscience, 17 0 (2): 0 89--102, 2016

    2016

  9. [17]

    Tinystories: How small can language models be and still speak coherent english?, 2023

    Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english?, 2023. URL https://arxiv.org/abs/2305.07759

    2023 arXiv

  10. [18]

    A mathematical framework for transformer circuits

    Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1 0 (1): 0 12, 2021

    2021

  11. [19]

    The successor representation and temporal context

    Samuel J Gershman, Christopher D Moore, Michael T Todd, Kenneth A Norman, and Per B Sederberg. The successor representation and temporal context. Neural Computation, 24 0 (6): 0 1553--1568, 2012

    2012

  12. [20]

    Vinod Goel and Raymond J. Dolan. Anatomical Segregation of Component Processes in an Inductive Inference Task . Journal of Cognitive Neuroscience, 12 0 (1): 0 110--119, 01 2000. ISSN 0898-929X. doi:10.1162/08989290051137639. URL https://doi.org/10.1162/08989290051137639

    doi

  13. [21]

    The organization of behavior: A neuropsychological theory

    Donald Olding Hebb. The organization of behavior: A neuropsychological theory. Wiley, 1949

    1949

  14. [22]

    Linearity of relation decoding in transformer language models

    Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. Linearity of relation decoding in transformer language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://op...

    2024

  15. [23]

    Neural networks and physical systems with emergent collective computational abilities

    John J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences, 79 0 (8): 0 2554--2558, 1982. doi:10.1073/pnas.79.8.2554. URL https://www.pnas.org/doi/abs/10.1073/pnas.79.8.2554

    1982 doi

  16. [24]

    The temporal context model in spatial navigation and relational learning: toward a common explanation of medial temporal lobe function across domains

    Marc W Howard, Mrigankka S Fotedar, Aditya V Datey, and Michael E Hasselmo. The temporal context model in spatial navigation and relational learning: toward a common explanation of medial temporal lobe function across domains. Psychological review, 112 0 (1): 0 75, 2005

    2005

  17. [25]

    Scalelong: Towards more stable training of diffusion model via scaling network long skip connection

    Zhongzhan Huang, Pan Zhou, Shuicheng YAN, and Liang Lin. Scalelong: Towards more stable training of diffusion model via scaling network long skip connection. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=0N73P8pH2l

    2023

  18. [26]

    Gamma-models: Generative temporal difference learning for infinite-horizon prediction

    Michael Janner, Igor Mordatch, and Sergey Levine. Gamma-models: Generative temporal difference learning for infinite-horizon prediction. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp.\...

    2020

  19. [27]

    Zhou, Marcus K

    Li Ji-An, Corey Y. Zhou, Marcus K. Benna, and Marcelo G. Mattar. Linking in-context learning in transformers to human episodic memory, 2024. URL https://arxiv.org/abs/2405.14992

    2024 arXiv

  20. [28]

    Do llms dream of elephants (when told not to)? latent concept association and associative memory in transformers, 2024

    Yibo Jiang, Goutham Rajendran, Pradeep Ravikumar, and Bryon Aragam. Do llms dream of elephants (when told not to)? latent concept association and associative memory in transformers, 2024. URL https://arxiv.org/abs/2406.18400

    2024 arXiv

  21. [29]

    The perceiver architecture is a functional global workspace

    Arthur Juliani, Ryota Kanai, and Shuntaro Sasai Sasai. The perceiver architecture is a functional global workspace. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 44, 2022

    2022

  22. [30]

    Nonoptimal component placement, but short processing paths, due to long-distance projections in neural systems

    Marcus Kaiser and Claus C Hilgetag. Nonoptimal component placement, but short processing paths, due to long-distance projections in neural systems. PLoS Comput. Biol., 2 0 (7): 0 e95, July 2006

    2006

  23. [31]

    Attractor and integrator networks in the brain

    Mikail Khona and Ila R Fiete. Attractor and integrator networks in the brain. Nature Reviews Neuroscience, 23 0 (12): 0 744--766, 2022

    2022

  24. [32]

    Kastanenka, and Dmitry Krotov

    Leo Kozachkov, Ksenia V. Kastanenka, and Dmitry Krotov. Building T ransformers from neurons and astrocytes. bioRxiv, 2022. doi:10.1101/2022.10.12.511910. URL https://www.biorxiv.org/content/early/2022/10/15/2022.10.12.511910

    2022 doi

  25. [33]

    Learning to learn: Single session acquisition of new rules by freely moving mice

    Amir Levi, Noam Aviv, and Eran Stark. Learning to learn: Single session acquisition of new rules by freely moving mice . PNAS Nexus, 3 0 (5): 0 pgae203, 05 2024. ISSN 2752-6542. doi:10.1093/pnasnexus/pgae203. URL https://doi.org/10.1093/pnasnexus/pgae203

    2024 doi

  26. [34]

    Transformers as algorithms: Generalization and stability in in-context learning

    Yingcong Li, Muhammed Emrullah Ildiz, Dimitris Papailiopoulos, and Samet Oymak. Transformers as algorithms: Generalization and stability in in-context learning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Pr...

    2023

  27. [35]

    The existence of persistent states in the brain

    William A Little. The existence of persistent states in the brain. Mathematical Biosciences, 19 0 (1): 0 101--120, 1974. ISSN 0025-5564. doi:https://doi.org/10.1016/0025-5564(74)90031-5. URL https://www.sciencedirect.com/science/article/pii/0025556474900315

    1974

  28. [36]

    Expanding the scope of memory search: Modeling intralist and interlist effects in free recall

    Lynn J Lohnas, Sean M Polyn, and Michael J Kahana. Expanding the scope of memory search: Modeling intralist and interlist effects in free recall. Psychological review, 122 0 (2): 0 337, 2015

    2015

  29. [37]

    Analysis of the activity of the chains of internuncial neurons

    Rafael Lorente de Nó. Analysis of the activity of the chains of internuncial neurons. Journal of Neurophysiology, 1 0 (3): 0 207--244, 1938

    1938

  30. [38]

    Language conditioned imitation learning over unstructured data, 2021

    Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data, 2021. URL https://arxiv.org/abs/2005.07648

    2021 arXiv

  31. [39]

    Emergence of abstract rules in the primate brain

    Farshad Alizadeh Mansouri, David J Freedman, and Mark J Buckley. Emergence of abstract rules in the primate brain. Nature Reviews Neuroscience, 21 0 (11): 0 595--610, 2020

    2020

  32. [40]

    A logical calculus of the ideas immanent in nervous activity

    Warren S McCulloch and Walter Pitts. A logical calculus of the ideas immanent in nervous activity. The bulletin of mathematical biophysics, 5: 0 115--133, 1943

    1943

  33. [41]

    Distributed representations of words and phrases and their compositionality, 2013

    Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality, 2013. URL https://arxiv.org/abs/1310.4546

    2013 arXiv

  34. [42]

    An integrative theory of prefrontal cortex function

    Earl K Miller and Jonathan D Cohen. An integrative theory of prefrontal cortex function. Annual review of neuroscience, 24 0 (1): 0 167--202, 2001

    2001

  35. [43]

    Universal hopfield networks: A general framework for single-shot associative memory models

    Beren Millidge, Tommaso Salvatori, Yuhang Song, Thomas Lukasiewicz, and Rafal Bogacz. Universal hopfield networks: A general framework for single-shot associative memory models. In International Conference on Machine Learning, pp.\ 15561--15583. PMLR, 2022

    2022

  36. [44]

    Large language models as general pattern machines, 2023

    Suvir Mirchandani, Fei Xia, Pete Florence, Brian Ichter, Danny Driess, Montserrat Gonzalez Arenas, Kanishka Rao, Dorsa Sadigh, and Andy Zeng. Large language models as general pattern machines, 2023. URL https://arxiv.org/abs/2307.04721

    2023 arXiv

  37. [45]

    Associatron-a model of associative memory

    Kaoru Nakano. Associatron-a model of associative memory. IEEE Transactions on Systems, Man, and Cybernetics, SMC-2 0 (3): 0 380--388, 1972. doi:10.1109/TSMC.1972.4309133

    1972

  38. [46]

    In-context learning and induction heads

    Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kam...

    2022

  39. [47]

    a fl, Johannes Lehner, Philipp Seidl, Michael Widrich, Lukas Gruber, Markus Holzleitner, Thomas Adler, David Kreil, Michael K Kopp, G \

    Hubert Ramsauer, Bernhard Sch \"a fl, Johannes Lehner, Philipp Seidl, Michael Widrich, Lukas Gruber, Markus Holzleitner, Thomas Adler, David Kreil, Michael K Kopp, G \"u nter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. H opfield networks is all you need. In Internat...

    2021

  40. [48]

    Prefrontal cortex and episodic memory: Integrating findings from neuropsychology and functional brain imaging

    Charan Ranganath and Robert T Knight. Prefrontal cortex and episodic memory: Integrating findings from neuropsychology and functional brain imaging. The cognitive neuroscience of memory: Encoding and retrieval, 1: 0 83, 2002

    2002

  41. [49]

    The mechanistic basis of data dependence and abrupt learning in an in-context classification task

    Gautam Reddy. The mechanistic basis of data dependence and abrupt learning in an in-context classification task. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=aN4Jf6Cx69

    2024

  42. [50]

    Hippocampal area ca2: properties and contribution to hippocampal function

    Vincent Robert, Sadiyah Cassim, Vivien Chevaleyre, and Rebecca A Piskorowski. Hippocampal area ca2: properties and contribution to hippocampal function. Cell and tissue research, 373: 0 525--540, 2018

    2018

  43. [51]

    Mice in a labyrinth show rapid learning, sudden insight, and efficient exploration

    Matthew Rosenberg, Tony Zhang, Pietro Perona, and Markus Meister. Mice in a labyrinth show rapid learning, sudden insight, and efficient exploration. eLife, 10: 0 e66175, jul 2021. ISSN 2050-084X. doi:10.7554/eLife.66175. URL https://doi.org/10.7554/eLife.66175

    2021 doi

  44. [52]

    Principles of neurodynamics: Perceptrons and the theory of brain mechanisms, 1961

    Frank Rosenblatt. Principles of neurodynamics: Perceptrons and the theory of brain mechanisms, 1961

    1961

  45. [53]

    Prefrontal cortex and flexible cognitive control: Rules without symbols

    Nicolas P Rougier, David C Noelle, Todd S Braver, Jonathan D Cohen, and Randall C O'Reilly. Prefrontal cortex and flexible cognitive control: Rules without symbols. Proceedings of the National Academy of Sciences, 102 0 (20): 0 7338--7343, 2005

    2005

  46. [54]

    Human curriculum effects emerge with in-context learning in neural networks

    Jacob Russin, Ellie Pavlick, and Michael J Frank. Human curriculum effects emerge with in-context learning in neural networks. ArXiv, 2024

    2024

  47. [55]

    Riechers, Lucas Teixeira, Alexander Gietelink Oldenziel, and Sarah Marzen

    Adam Shai, Paul M. Riechers, Lucas Teixeira, Alexander Gietelink Oldenziel, and Sarah Marzen. Transformers represent belief state geometry in their residual stream. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.ne...

    2024

  48. [56]

    Learning to predict by the methods of temporal differences

    Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3: 0 9--44, 1988

    1988

  49. [57]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Info...

    2017

  50. [58]

    Mrsic-Flogel

    Ivan Voitov and Thomas D. Mrsic-Flogel. Cortical feedback loops bind distributed representations of working memory. Nature, 608 0 (7922): 0 381--389, Aug 2022. ISSN 1476-4687. doi:10.1038/s41586-022-05014-3. URL https://doi.org/10.1038/s41586-022-05014-3

    2022 doi

  51. [59]

    Transformers learn in-context by gradient descent

    Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, Joao Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jona...

    2023

  52. [60]

    Interpretability in the wild: a circuit for indirect object identification in GPT -2 small

    Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in GPT -2 small. In The Eleventh International Conference on Learning Representations, 2023. URL https://openrevi...

    2023

  53. [61]

    Nonparametric classification on low dimensional manifolds using overparameterized convolutional residual networks

    Zixuan Zhang, Kaiqi Zhang, Minshuo Chen, Yuma Takeda, Mengdi Wang, Tuo Zhao, and Yu-Xiang Wang. Nonparametric classification on low dimensional manifolds using overparameterized convolutional residual networks. In The Thirty-eighth Annual Conference on Neural Information Proce...

    2024

  54. [62]

    In-context exemplars as clues to retrieving from large associative memory

    Jiachen Zhao. In-context exemplars as clues to retrieving from large associative memory. In Associative Memory & Hopfield Networks in 2023 , 2023. URL https://openreview.net/forum?id=pgPAsSv5ga

    2023

  55. [63]

    Episodic retrieval for model-based evaluation in sequential decision tasks, 2023

    Corey Y Zhou, Deborah Talmi, Nathaniel Daw, and Marcelo G Mattar. Episodic retrieval for model-based evaluation in sequential decision tasks, 2023

    2023

  56. [64]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.