Pith. sign in

REVIEW 5 major objections 6 minor 45 references

Emergence of Episodic Memory in Transformers: Characterizing Changes in Temporal Structure of Attention Scores During Training

T0 review · 5 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Trained GPT-2 attention heads develop human-like temporal memory effects, and induction heads are the mechanism behind the model's tendency to recall list items in order.

desk verdict A useful, incremental study of temporal contiguity in transformer attention with a clean induction-head ablation, but the downstream recall probe is underspecified enough that the central causal claim needs another pass. read the letter →

arxiv 2502.06902 v1 pith:HU6YEBKE submitted 2025-02-09 cs.LG cs.AIcs.CL

Deven Mahesh Mistry , Anooshka Bajaj , Yash Aggarwal , Sahaj Singh Maini , Zoran Tiganj This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords episodicmemoryin-contextlearninginductionheadstemporalcontiguityrecencyandprimacylag-CRPserialrecallattention
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that, as GPT-2 models are trained, the temporal structure of attention comes to resemble human episodic memory: attention heads exhibit primacy, recency, and a contiguity effect in which attention falls off gradually with distance (lag) from a repeated token. It further claims that transformer outputs show a measurable tendency toward in-context serial recall, visible as a roughly tenfold increase in next-token probability at lag $+1$ after a list-with-middle-token prompt. The paper's central causal assertion is that induction heads are the driving force behind the contiguity effect: ablating the heads with induction scores above $0.01$ eliminates the effect in outputs, while ablating matched non-induction heads does not. If correct, this links a specific circuit-level mechanism in transformers to a hallmark behavioral signature of human episodic memory and clarifies what kind of temporal organization in-context learning actually produces.

What carries the argument

The lag-conditional recall probability (lag-CRP) curve is the measuring tool: for each probe item, it records the attention mass or output probability placed on items at a given distance (lag) in the sequence, averaged over many token permutations to reduce semantic similarity. The load-bearing mechanism is the induction head, an attention head that finds the previous occurrence of the current token and attends to the token that followed it; heads are identified by an induction matching score, and ablating them by setting their attention scores to $-\infty$ removes the contiguity effect in downstream outputs. A decaying-exponential fit to positive lags, after removing a linear recency term, yields the time constant that summarizes how many tokens of context are retrieved.

What would settle it

Renormalize the output distribution over the 500 list tokens only, and recompute the lag-CRP with and without induction-head ablation. If the lag $+1$ peak disappears under renormalization, or if it survives induction-head ablation in that cleaner protocol, the paper's central claim about induction heads driving serial recall would be falsified.

Watch

Extended reading notes

Core claim

On the paper's own terms: trained GPT-2 small and GPT-2 medium models, across Wikitext-103 and two larger web-text samples, develop attention-head lag-CRP curves with a gradual falloff over positive lags, a linear recency slope, and fitted time constants concentrated in a narrow range of 2–4 tokens. These are the same measures used in human free-recall experiments, and the paper interprets them as temporal contiguity, primacy, and recency effects in the attention heads. In a downstream free-recall-style probe, the next-token distribution shows a strong preference for the item immediately following the cue in the prior list. When induction heads are ablated by setting their attention logits to negative infinity, this contiguity peak disappears even though the ablated heads are only about 5% of the total; layer-matched ablation of non-induction heads produces a much smaller change. The paper concludes that induction heads are the mechanism that produces the serial-recall bias in transformer outputs.

Load-bearing premise

The downstream free-recall probe assumes that a peak in the next-token probability at lag $+1$, computed after a 500-token list and a middle-token cue, reflects retrieval of the list item, but the paper never states how the full vocabulary is projected onto the 500 list tokens; a token-frequency prior could in principle create or amplify the reported profile.

Editorial extensions

If this is right

  • The serial-recall bias in transformer outputs is attributable to a small, identifiable set of induction heads, so editing or ablating those heads can add or remove temporal copying behavior in a targeted way.
  • Because fitted time constants are short (2–4 lags), transformers should be expected to retrieve only a narrow temporal neighborhood around a repeated token, in contrast to human contiguity effects that extend broadly over time.
  • The dependence of induction strength on positional-encoding magnitude predicts that models with no positional encoding will still acquire a weak recency effect through causal masking alone, which the paper observes.
  • Across two model sizes, induction heads form a roughly constant fraction of all heads (12–14%), suggesting that the qualitative temporal structure reported here will appear in other from-scratch transformer trainings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A cleaner downstream protocol would renormalize the 50,257-token output distribution over the 500 list tokens before computing lag-CRP; if the lag $+1$ peak survives that renormalization and frequency controls, the episodic-retrieval interpretation would be on much firmer ground.
  • The same lag-CRP machinery could be run on instruction-tuned or much larger models to see whether fine-tuning preserves, strengthens, or overwrites the induction-driven serial-recall bias; the authors themselves flag that only relatively small, from-scratch models were tested.
  • If induction heads are truly the causal mechanism, then steering their attention weights directly (rather than ablating) should be able to amplify or suppress the lag $+1$ peak in a graded way, which would be a direct behavioral readout of circuit function.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper trains GPT-2 small and GPT-2 medium models from scratch on Wikitext-103 and two FineWeb subsets, and then applies lag-conditional recall probability (lag-CRP) analysis, borrowed from cognitive science, to characterize temporal biases in attention heads and in next-token predictions. The authors report that attention heads show temporal contiguity, primacy, and recency, that induction heads have lag-CRP time constants concentrated around 2-4 tokens, and that model outputs exhibit a strong lag+1 serial-recall peak that is eliminated when heads with induction scores above 0.01 are ablated. They also study how positional-encoding magnitude and model size affect these temporal profiles. The central causal claim is that induction heads are the driving force behind the contiguity effect in the model's outputs.

Significance. If the downstream serial-recall result is valid, the paper would provide a concrete mechanistic link between a well-studied circuit motif (induction heads) and a behavioral bias (serial recall in next-token probabilities), and would demonstrate a useful transfer of cognitive-science methods to mechanistic interpretability. The attention-level descriptive findings—peaked lag-CRP curves with exponential falloff—are plausible and align with prior work on induction heads. The design of the ablation with a layer-matched control for non-induction heads is a good practice. However, the validity of the headline causal claim depends on a downstream probe that is not specified precisely enough to rule out a token-frequency confound, and Eq. (1) appears to contain an error in the lag-CRP normalization. Once these are corrected, the paper could make a modest but solid contribution; in its current form, the central claim is not established.

major comments (5)
  1. [Section 3.5] The downstream free-recall probe is not specified sufficiently to support the causal claim that induction heads drive serial recall. The text states that the model is 'probed ... with a sequence of 500 randomly ordered tokens ... followed by a middle token' and that 'the probability of the next token as a function of lag' is quantified, but it does not state how the 50,257-token output distribution is projected onto the 500 list tokens, whether the probabilities are renormalized over the list, how the middle token is chosen (from the list or an external token), or how many permutations and cue positions are aggregated. Because the probe tokens are the 500 most frequent tokens in the dataset, their unigram probabilities are far above the vocabulary average, so an unnormalized lag-conditional probability can be dominated by token frequency rather than by retrieval of the successor in the list. The observed lag+1 peak, and its elimination after induction-head ablation, are therefore not yet a valid measure of serial recall. The authors should specify the computation precisely and, ideally, renormalize over the 500 list tokens and compare against a unigram baseline.
  2. [Section 2.2, Eq. (1)] Equation (1) appears to contain an error in the summation bounds and normalization. For a lag l, the valid source positions j=s+l in a length-N sequence are s=1,...,N-l (for l>0) or s=1-l,...,N (for l<0), giving N-|l| valid destination positions. The formula, however, sums over |l|<s≤N-|l|, which for l>0 contains only N-2|l| terms and excludes positions s=1,...,|l| that have valid source indices. The factor 2 is also unexplained; for l=0 it would make S_0=2 rather than an average. Since the lag-CRP curves and the fitted time constants throughout the paper are computed from this quantity, the authors must correct Eq. (1) and recompute the curves, or clarify the intended normalization.
  3. [Section 2.3 / Section 3.1] The paper never defines the criterion used to classify a head as an 'induction head', although this classification is used to report the Number of Induction Heads in Tables 2-6, the average induction scores, and the ablation in Section 3.5. Section 2.3 defines the induction matching score I, but no threshold or selection procedure is given; Section 2.4 describes selecting heads by the lag-CRP score at l=1, which is a different criterion. The ablation in Section 3.5 uses 'induction scores above 0.01', which is not derived from the earlier definitions. Specify the exact threshold(s) and how they were applied, or the numbers in the tables are not reproducible.
  4. [Section 2.4, Tables 2, 5, 6] The fitted exponential time constants are reported only as averages (e.g., 'Average Time Constant' in Tables 2, 5, 6) with no error bars, standard deviations, or the number of heads per entry. The claim in Section 3.4 that 'the time constants are mainly concentrated in the narrow range of 2-4 lags' is based on Fig. 9, which shows a histogram, but the figure lacks error bars and the sample size per configuration is not stated. Given that the lag-CRP curves are averaged over only 10 random permutations (Section 2.2), the uncertainty in the time constants is likely substantial; report per-head fits with dispersion measures so the reader can judge the 2-4 lag range.
  5. [Sections 2.2-2.3] The attention-level lag-CRP at lag +1 and the induction matching score measure essentially the same quantity: attention to the token following the previous occurrence of the current token. The paper does not discuss this overlap, which creates an appearance of circularity when it concludes that the contiguity effect is driven by induction heads. The authors should clarify what additional information the lag-CRP provides beyond the induction score (e.g., the shape of the falloff for |l|>1) and should present the downstream ablation as the decisive causal test rather than presenting the two attention measures as independent evidence.
minor comments (6)
  1. [Introduction and References] There are typographical errors (e.g., 'transfromer' in the introduction, 'V o' in the reference list) and inconsistent capitalization of 'WikiText' / 'Wikitext'; a careful copyedit is needed.
  2. [Figures 10, 11, A1, A2] The downstream CRP plots have no axis labels (the y-axis is ambiguous), no error bars, and no statement of the number of probe runs; the figure captions should define what 'CRP' means in this downstream context.
  3. [Section 3.5] The example 'middle token (e.g., GRDBTHMB)' is not explained; the authors should clarify whether this token is drawn from the 500-token list or is an external cue, and how the choice of middle token affects the lag axis.
  4. [Section 2.4] The head selection procedure is ambiguous: 'we choose a subset of heads that have the highest lag-CRP score at l = 1' does not state how many heads are chosen or whether the selection is per layer; specify the exact number and the lag window used.
  5. [Table 1] Table 1 presents perplexity values without comparison to the cited baselines; the statement that the models 'converged to perplexities similar to those of models with comparable size' is not supported by a comparison table or reference values.
  6. [Equation (2)] In the definition of t_{i,j}, the phrase 'the token at position i matches the token before position j' should be rephrased as 'the token at position i matches the token at position j-1' (or the analogous index offset) to avoid ambiguity.

Circularity Check

1 steps flagged · score 4.0 of 10

Attention-level contiguity at lag +1 is the induction score by construction; the downstream ablation of induction heads is an independent causal test, so the central output-level claim is not forced.

  1. self definitional [Section 2.2 Eq. (1), Section 2.3 Eq. (2), and Section 2.4/3.4 head selection]
    "The lag-CRP score for lag l was calculated as the average attention score between the tokens in the destination sequence, and a different token placed l positions away from the corresponding token in the source sequence. ... We then construct a target matrix that records matches based on the induction rule: if the token at a destination position matches the token before a particular source position, the corresponding entry in the target matrix is set to 1."

    Under the repeated source/destination protocol, destination position s+N contains the same token as source position s. The lag +1 CRP term averages a_{s+N,s+1}; the induction target t_{s+N,s+1} is 1 exactly because the token at s+N equals the token before s+1. Thus S_1 in Eq. 1 and the numerator of I in Eq. 2 sum the same attention weights over the same pairs. Calling the l=1 lag-CRP peak a contiguity effect and then attributing it to induction heads is therefore a restatement of the same measurement, not an independent causal finding. The downstream ablation of induction heads is a separate, non-circular test, so the output-level serial-recall claim still has independent content.

full rationale

The paper makes two distinct claims. The attention-level claim is that trained GPT-2 heads exhibit a contiguity effect in lag-CRP curves and that induction heads drive this effect. That claim is partially definitional: in the repeated source/destination construction, Eq. 1 at lag +1 and the induction matching numerator of Eq. 2 select the same attention weights, and the heads are selected by their lag-CRP score at l=1. The identification of contiguity with induction at the attention level is therefore close to tautological. The output-level claim is that transformer outputs show a serial-recall bias at lag +1 and that ablating induction heads eliminates it. This is an independent causal manipulation: the paper compares ablated induction heads with non-induction heads and reports a much smaller effect in the control condition. I do not find load-bearing self-citation; the cited prior work on induction heads and lag-CRP is external and not authored by the present authors. The downstream probe is underspecified in that the paper does not state how the 50,257-token output distribution is projected onto the 500 probe tokens, whether non-list tokens are marginalized, or how the middle token is selected. That is a validity and reproducibility concern about whether the lag+1 peak could reflect token-frequency priors, but it is not a circularity: the reported elimination after ablation is not equivalent to the input by construction. Overall, the attention-level equivalence gives a partial circularity, while the central ablation-based output claim remains empirically independent, supporting a moderate score of 4.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The central analysis relies on several fitted parameters for curve descriptions and chosen thresholds, and on domain assumptions about what attention scores and output probabilities mean in a free-recall framework. No new entities are introduced.

free parameters (5)
  • Exponential time constant tau = 3.1 (GPT-2 small, Wikitext-103, PE magnitude 1)
    Fitted to the positive-lag lag-CRP falloff after removing recency, used to characterize temporal extent in Tables 2, 5, 6.
  • Exponential amplitude a = not reported
    Free amplitude in the ae^{-t/tau} fit, necessary to fit the decay.
  • Induction head ablation threshold = 0.01
    Heads with induction matching score above 0.01 are ablated in Section 3.5; threshold is chosen without sensitivity analysis.
  • Lag window for head selection = -10 to +10
    Heads are selected as induction heads based on highest lag-CRP score at l=1 in this window (Section 2.4).
  • Recency slope exclusion band = lags -50 to +50 removed
    Linear recency fit uses only lags outside -50 to +50, an arbitrary choice in Section 2.4.
assumptions (3)
  • domain assumption Random permutations of 500 tokens remove semantic similarity from attention lag-CRP curves
    Section 2.2 states ten random permutations are used to reduce semantic effects; this assumes token semantics are sufficiently decorrelated by ten permutations.
  • domain assumption The next-token distribution after the probe list can be interpreted as a free-recall response distribution
    Section 3.5 treats output probabilities at each lag as recall probabilities without specifying how the 50k-vocabulary output is restricted to the 500 list items.
  • domain assumption Causal masking and positional encoding jointly shape recency, with PE magnitude as the main manipulation
    Section 3.2 interprets changes in recency slope and induction scores as caused by the PE magnitude multiplier, relying on Haviv et al. for the claim that masking alone gives weak position information.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Emergence of Episodic Memory in Transformers: Characterizing Changes in Temporal Structure of Attention Scores During Training." pith.science (2026). https://pith.science/paper/HU6YEBKE

@misc{pith2026250206902,
  author       = {Pith},
  title        = {Pith review of: Emergence of Episodic Memory in Transformers: Characterizing Changes in Temporal Structure of Attention Scores During Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HU6YEBKE}},
  note         = {Machine review of arXiv:2502.06902}
}
read the original abstract

We investigate in-context temporal biases in attention heads and transformer outputs. Using cognitive science methodologies, we analyze attention scores and outputs of the GPT-2 models of varying sizes. Across attention heads, we observe effects characteristic of human episodic memory, including temporal contiguity, primacy and recency. Transformer outputs demonstrate a tendency toward in-context serial recall. Importantly, this effect is eliminated after the ablation of the induction heads, which are the driving force behind the contiguity effect. Our findings offer insights into how transformers organize information temporally during in-context learning, shedding light on their similarities and differences with human memory and learning.

Figures

Figures reproduced from arXiv: 2502.06902 by the authors.

Figure 1
Figure 1. Attention scores as a function of lag for all attention heads in GPT-2 small after 4000 iterations on the [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 3
Figure 3. Example of the same induction head L7H3 during different stages of training. A. After 300 itera￾tions. B. After 4000 iterations. is consistent with the hypothesis that vector sim￾ilarity induced by positional encoding creates a temporal link needed for the induction heads to identify the token that followed the current token in the previous sequence. To further test this hypothe￾sis, we plotted the Pearson correlati… view at source ↗
Figure 2
Figure 2. Two induction heads before (top row) and after [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Induction scores for five checkpoints throughout the training of GPT-2 small on Wikitext-103 dataset. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Correlation in positional encoding vectors [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Example of the same induction head L7H3 for different magnitudes of positional encoding. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Induction scores for five different magnitudes of positional encoding (GPT-2 small trained on Wikitext-103 [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Induction scores for two different models. [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 10
Figure 10. Figure 10: CRP during downstream evaluation showing impact of induction head ablation. [PITH_FULL_IMAGE:figures/full_fig_p009_10.png]
Figure 11
Figure 11. Figure 11: CRP as a function of training iteration in [PITH_FULL_IMAGE:figures/full_fig_p009_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 23 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Hemali Angne, Charlotte Cornell, and Qiong Zhang. 2023. Why two heads together are worse than apart: A context-based account of collaborative inhibition in memory search. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 46

    work page 2023

  4. [4]

    Atkinson and Richard M

    Richard C. Atkinson and Richard M. Shiffrin. 1968. Human memory: A proposed system and its control processes. 2:89--195

    work page 1968

  5. [5]

    Tom B Brown. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165

    arXiv 2020

  6. [6]

    Joy Crosbie and Ekaterina Shutova. 2024. Induction heads as an essential mechanism for pattern matching in in-context learning. arXiv preprint arXiv:2407.07011

    arXiv 2024

  7. [7]

    Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. https://doi.org/10.18653/v1/P19-1285 Transformer- XL : Attentive language models beyond a fixed-length context . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978--2988, Florence, Italy. Association for Compu...

    doi:10.18653/v1/p19-1285 2019

  8. [8]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171--4186

    2019

Show all 45 references
  1. [9]

    Chris Donkin and Robert M Nosofsky. 2012. A power-law model of psychological memory strength in short- and long-term recognition. Psychological science, 23(6):625--634

    2012

  2. [10]

    Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. 2021. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1(1):12

    2021

  3. [11]

    Sarah Folkerts, Ueli Rutishauser, and Marc W Howard. 2018. Human episodic memory retrieval is accompanied by a neural contiguity effect. Journal of Neuroscience, 38(17):4200--4211

    2018

  4. [12]

    Zafeirios Fountas, Martin A Benfeghoul, Adnan Oomerjee, Fenia Christopoulou, Gerasimos Lampouras, Haitham Bou-Ammar, and Jun Wang. 2024. Human-like episodic memory for infinite context llms. arXiv preprint arXiv:2407.09450

    2024

  5. [13]

    Daniel Y Fu, Tri Dao, Khaled Kamal Saab, Armin W Thomas, Atri Rudra, and Christopher Re. 2023. https://openreview.net/forum?id=COZDy0WYGg Hungry hungry hippos: Towards language modeling with state space models . In The Eleventh International Conference on Learning Representations

    2023

  6. [14]

    Murray Glanzer and Anita R. Cunitz. 1966. Two storage mechanisms in free recall. Journal of Verbal Learning and Verbal Behavior, 5(4):351--360

    1966

  7. [15]

    Xiaobo Guo and Soroush Vosoughi. 2024. Serial position effects of large language models. arXiv preprint arXiv:2406.15981

    2024 arXiv

  8. [16]

    Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, and Omer Levy. 2022. Transformer language models without positional encodings still learn positional information. arXiv preprint arXiv:2203.16634

    2022 arXiv

  9. [17]

    Marc W Howard and Michael J Kahana. 2002. Contextual variability and serial position effects in free recall. Journal of Experimental Psychology: Learning, Memory, and Cognition, 28(4):923

    2002

  10. [18]

    jump back in time

    Marc W Howard, Indre V Viskontas, Karthik H Shankar, and Itzhak Fried. 2012. Ensembles of human mtl neurons “jump back in time” in response to a repeated stimulus. Hippocampus, 22(9):1833--1847

    2012

  11. [19]

    Marc W Howard, Tess E Youker, and Vijay S Venkatadass. 2008. The persistence of memory: Contiguity effects across hundreds of seconds. Psychonomic Bulletin & Review, 15:58--63

    2008

  12. [20]

    HuggingFaceFW . 2024. https://doi.org/10.57967/hf/2493 fineweb (revision af075be)

    2024 doi

  13. [21]

    Romuald A Janik. 2023. Aspects of human memory and large language models. arXiv preprint arXiv:2311.03839

    2023 arXiv

  14. [22]

    Lila J Jenkins and Charan Ranganath. 2010. Prefrontal and medial temporal lobe activity at encoding predicts temporal context memory. Journal of Neuroscience, 30(45):15596--15603

    2010

  15. [23]

    Haozhe Ji, Rongsheng Zhang, Zhenyu Yang, Zhipeng Hu, and Minlie Huang. 2022. https://doi.org/10.18653/v1/2022.naacl-main.422 L a M emo: Language modeling with look-ahead memory . In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa...

    2022 doi

  16. [24]

    Li Ji-An, Corey Y Zhou, Marcus K Benna, and Marcelo G Mattar. 2024. Linking in-context learning in transformers to human episodic memory. arXiv preprint arXiv:2405.14992

    2024 arXiv

  17. [25]

    Michael J Kahana. 1996. Associative retrieval processes in free recall. Memory & Cognition, 24(1):103--109

    1996

  18. [26]

    Andrej Karpathy. 2022. https://github.com/karpathy/nanoGPT

    2022

  19. [27]

    Kenneth Levenberg. 1944. A method for the solution of certain non-linear problems in least squares. Quarterly of applied mathematics, 2(2):164--168

    1944

  20. [28]

    Moufan Li, Kristopher T Jensen, Qihong Lu, Qiong Zhang, and Marcelo G Mattar. 2024. Modeling multiplicity of strategies in free recall with neural networks

    2024

  21. [29]

    Laura J Lohnas and Alice F Healy. 2021. The role of context in episodic memory: Behavior and neurophysiology. Psychology of Learning and Motivation, 75:157--203

    2021

  22. [30]

    Donald W Marquardt. 1963. An algorithm for least-squares estimation of nonlinear parameters. Journal of the society for Industrial and Applied Mathematics, 11(2):431--441

    1963

  23. [31]

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models

    2016

  24. [32]

    Bennet B. Jr. Murdock. 1962. The serial position effect of free recall. Journal of Experimental Psychology, 64(5):482--488

    1962

  25. [33]

    Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. 2022. In-context learning and induction heads. arXiv preprint arXiv:2209.11895

    2022 arXiv

  26. [34]

    Alexander Peysakhovich and Adam Lerer. 2023. Attention sorting combats recency bias in long context language models. arXiv preprint arXiv:2310.01427

    2023 arXiv

  27. [35]

    Vo, Qinyuan Wu, Jianing Mu, Javier S

    Mathis Pink, Vy A. Vo, Qinyuan Wu, Jianing Mu, Javier S. Turek, Uri Hasson, Kenneth A. Norman, Sebastian Michelmann, Alexander Huth, and Mariya Toneva. 2024. https://arxiv.org/abs/2410.08133 Assessing episodic memory in llms with sequence order recall tasks . Preprint, arXiv:2...

    2024 arXiv

  28. [36]

    Sean M Polyn, Kenneth A Norman, and Michael J Kahana. 2009. A context maintenance and retrieval model of organizational processes in free recall. Psychological review, 116(1):129

    2009

  29. [37]

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. https://api.semanticscholar.org/CorpusID:160025533 Language models are unsupervised multitask learners

    2019

  30. [38]

    David C Rubin and Amy E Wenzel. 1996. One hundred years of forgetting: A quantitative description of retention. Psychological review, 103(4):734

    1996

  31. [39]

    Aaditya K Singh, Ted Moskovitz, Felix Hill, Stephanie CY Chan, and Andrew M Saxe. 2024. What needs to go right for an induction head? a mechanistic study of in-context learning circuits and their formation. arXiv preprint arXiv:2404.07129

    2024 arXiv

  32. [40]

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063

    2024

  33. [41]

    A Vaswani. 2017. Attention is all you need. Advances in Neural Information Processing Systems

    2017

  34. [42]

    Benyou Wang, Lifeng Shang, Christina Lioma, Xin Jiang, Hao Yang, Qun Liu, and Jakob Grue Simonsen. 2021. https://openreview.net/forum?id=onxoVA9FxMw On position embeddings in \ bert \ . In International Conference on Learning Representations

    2021

  35. [43]

    Yiwei Wang, Yujun Cai, Muhao Chen, Yuxuan Liang, and Bryan Hooi. 2023. Primacy effect of chatgpt. arXiv preprint arXiv:2310.13206

    2023 arXiv

  36. [44]

    John T Wixted and Ebbe B Ebbesen. 1991. On the form of forgetting. Psychological science, 2(6):409--415

    1991

  37. [45]

    Zexuan Zhong, Tao Lei, and Danqi Chen. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.382 Training language models with memory augmentation . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5657--5673, Abu Dhabi, United Arab ...

    2022 doi

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.