Pith. sign in

REVIEW 5 major objections 5 minor 2 cited by

LM2: Large Memory Models

T0 review · 5 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces LM2, a decoder-only Transformer with an auxiliary gated memory bank, and claims it beats RMT by 37.1% and Llama-3.2-1.2B by 86.3% on BABILong while improving MMLU by 5.0%.

desk verdict A cleanly described memory-augmented Transformer with one solid same-data comparison, but the headline gains are not tied to the memory module because no baseline isolates it. read the letter →

arxiv 2502.06049 v1 pith:PFBW57AP submitted 2025-02-09 cs.CL cs.AI

Jikun Kang , Wenqi Wu , Filippos Christianos , Alex J. Chan , Fraser Greenlee , George Thomas , Marvin Purtorab , Andy Toulis This is my paper · ORCID
classification cs.CLcs.AI
keywords largememorymodelmemory-augmentedtransformercross-attentionbankgatedupdatelong-contextreasoningmulti-hopinferenceBABILongbenchmarkMMLU
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the Transformer's single attention stream is a bottleneck for long-context reasoning, because facts must be re-derived from the context every time they are needed, and that an explicit, persistently updated memory can relieve that bottleneck. To test this, it builds LM2, a decoder-only model that keeps the standard attention path and adds a parallel memory pathway: 2,048 memory slots queried by cross attention and rewritten by input, forget, and output gates. On the BABILong benchmark, LM2-1.7B reports an average accuracy of 39.9% at 8K–128K context lengths, and the paper claims average gains of 37.1% over the Recurrent Memory Transformer (RMT) baseline and 86.3% over Llama-3.2-1.2B. On MMLU, LM2 scores 29.4% versus 28.0% for a vanilla model pretrained on the same data, which the paper reads as evidence that memory can be added without degrading general abilities. The pith of the paper is that a gated, content-addressable memory bank, kept separate from the attention stream, is what enables these gains.

What carries the argument

The central object is the gated memory bank: an auxiliary set of 2,048 learnable slots that the model reads and writes at every decoder block. Reading is cross attention, with input embeddings as queries and the memory bank as keys and values; the resulting attention output $E_{mem}$ is passed through a sigmoid output gate and added to the normal self-attention output via a skip connection, so the original information flow is preserved. Writing follows a gated update, $M_{t+1} = g_{in} \cdot \tanh(E_{mem}) + g_{forget} \cdot M_t$, where the input gate $g_{in} = \sigma(E_t W_{in})$ and the forget gate $g_{forget} = \sigma(E_{mem} W_{forget})$ decide what to store and what to erase. This mechanism gives the model a persistent, content-addressable store that can hold facts across long contexts and be selectively rewritten, and it is the component the paper credits for the BABILong and MMLU results.

What would settle it

Run a controlled comparison in which LM2 and RMT share the same pretraining data, fine-tuning status (both fine-tuned on BABILong training sets or both zero-shot), context-chunking rule, and scoring, and report per-task accuracy at 0K, 1K, 4K, and 128K; if vanilla-Llama-1.7B then matches LM2 on BABILong or RMT matches LM2 on MMLU, the memory module is not the cause of the reported gains.

Watch

Extended reading notes

Core claim

On its own terms, the discovery is that an auxiliary memory bank integrated into every decoder block improves long-context multi-step reasoning without sacrificing general knowledge. The memory bank is queried by the input embeddings through cross attention, the retrieved content is blended into the normal attention output through a learned output gate, and the bank itself is updated with input and forget gates. The model is pretrained from scratch on the SmolLM-Corpus (28B synthetic textbook tokens plus 220B FineWeb-Edu tokens), reaching 1.7B parameters total, with the memory module adding 0.5B. On BABILong, LM2-1.7B outperforms RMT-1.7B at every context-length bucket and outperforms Llama-3.2-1.2B by a wide margin, with the largest edge in counting, single-step, and multi-step reasoning tasks. On MMLU, LM2's 29.4% average is above the vanilla model's 28.0% and well above RMT's 26.5%, which the authors present as showing that memory augmentation need not trade away general-task performance. The paper also reports that more memory blocks lower perplexity, that memory slots specialize in factual versus structural roles, and that cross-attention heatmaps shift toward question-relevant tokens during test-time inference.

Load-bearing premise

The load-bearing premise is that the comparisons isolate the memory module: if LM2 and RMT differ in pretraining data, fine-tuning, or context chunking, the reported gains cannot be attributed to memory, and the paper does not state these details.

Editorial extensions

If this is right

  • If the reported gains hold under controlled comparison, a separate, gated memory bank is a more effective way to extend Transformers to long-context multi-hop reasoning than recurrent memory tokens or retrieval-augmented generation on these tasks.
  • Because LM2's MMLU score does not fall below its vanilla twin, explicit memory can be added to a Transformer without the general-task degradation seen with the RMT baseline, making memory augmentation a safer modification for general-purpose models.
  • The result that memory in all 16 blocks yields lower perplexity than memory in fewer blocks implies that memory capacity should be distributed across the whole depth of the network, not concentrated in one layer.
  • The finding that memory slots specialize and that test-time memory updates shift attention toward question-relevant tokens suggests the bank acts as a content-addressable store that can be inspected during inference, not just a learned black-box state.
  • If memory helps most at 8K–128K contexts, explicit memory could become a standard component for models deployed on long documents, reducing the need to summarize or chunk input.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: a natural next experiment is an ablation that varies only the memory module while holding pretraining data, model size, fine-tuning, chunking, and evaluation protocol fixed; that experiment would directly measure how much of the reported gain is attributable to memory.
  • Editorial inference: the same gated memory bank could be attached to larger or instruction-tuned backbones, since the paper only demonstrates it on a 1.2B-parameter base; if gains persist, explicit memory becomes a more broadly useful component.
  • Editorial inference: because memory slots appear to specialize in distinct roles, a natural extension is to read or edit individual slots during inference to steer what the model recalls; the paper analyzes the specialization but does not attempt intervention.
  • Editorial inference: the reported MMLU scores (28–29%) are those of a 1.7B model, so the no-degradation claim is demonstrated at small scale; whether it carries to larger models is an open question.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes LM2, a decoder-only Transformer augmented with an auxiliary memory bank of N slots that interacts with the input via cross attention and is updated through input, forget, and output gates. The memory pathway is added to each of the 16 decoder blocks of a Llama-3-based 1.2B-parameter backbone, producing a 1.7B-parameter model pre-trained from scratch on 248B tokens of SmolLM-Corpus (excluding Python). The authors evaluate LM2 on BABILong at context lengths from 0K to 128K and on MMLU, reporting average gains of 37.1% over RMT and 86.3% over Llama-3.2, plus a 5.0% MMLU improvement over a vanilla same-data model. They also provide perplexity curves varying the number of memory blocks, a Neuron Explainer interpretability analysis, and cross-attention heatmaps illustrating test-time memory updates.

Significance. If the reported gains were robustly attributable to the memory mechanism, the paper would make a useful contribution: an explicit, gated memory bank that improves long-context reasoning without degrading general task performance is an attractive design. The manuscript has several genuine strengths: the architecture is described in sufficient detail to reimplement, a code link is provided, and Table 1 includes a same-data comparison of LM2 against a vanilla-Llama-1.7B baseline, where LM2 shows consistent gains at most context lengths. The perplexity analysis in Figure 5 also provides some evidence that adding memory blocks improves language modeling. However, the headline quantitative claims are not backed by controlled comparisons, the evaluation protocol is under-specified, and the MMLU evidence is weak because scores are near chance. The central claim is plausible but not yet established at the level claimed.

major comments (5)
  1. [Abstract and Section 4, Table 1] The headline gains of 37.1% over RMT and 86.3% over Llama-3.2 are not reproducible from Table 1. Averaging the five context-length rows (0K, 1K, 2K, 4K, and the aggregated >=8K row) gives LM2 a mean accuracy of 66.5%, RMT 49.9%, and Llama-3.2-1.2B 36.8%, corresponding to a 33.2% gain over RMT and 80.7% over Llama-3.2, not the stated 37.1% and 86.3%. The discrepancy should be corrected, and the averaging procedure should be stated explicitly.
  2. [Section 4, baselines paragraph] The comparison against RMT-1.7B does not isolate the memory mechanism: RMT is fine-tuned on the bAbI training data, while LM2 is pre-trained from scratch on 248B tokens and evaluated without any stated fine-tuning on BABILong. Similarly, Llama-3.2-1.2B was trained by Meta on a different corpus and token budget, and vanilla-Llama-1.7B is described only as 'scaled to 1.7 billion parameters' with no architectural recipe. No experiment varies only the presence of the memory module while holding the backbone, pretraining data, and training protocol fixed, so the reported advantages cannot be attributed to the memory module alone.
  3. [Sections 3 and 4.1] The evaluation protocol for LM2 on BABILong is under-specified: the paper does not state whether LM2 is evaluated zero-shot or fine-tuned, how sequences longer than the training context are chunked (particularly the 128K contexts), whether the optional top-k attention mentioned in Section 2.1 is used, or how the memory bank is initialized and reset across chunks. These details are necessary to reproduce Table 1 and to interpret the long-context results.
  4. [Section 4.3, Figure 5] The claim that integrating memory into all 16 decoder blocks 'empirically achieves the best performance' is supported only by perplexity curves, not by BABILong accuracy. Perplexity trends are not a substitute for downstream task performance, so the choice of memory placement is not validated for the benchmark on which the paper's central claims rest. Reporting BABILong accuracy for the 1-, 6-, 12-, and 16-block configurations would make this ablation load-bearing.
  5. [Section 4.2, Table 2] The MMLU result is near chance (LM2 29.4%, vanilla 28.0%, RMT 26.5% on a four-option benchmark), and no evaluation protocol is given: number of shots, example selection, answer extraction method, and whether accuracy is averaged over subjects or questions. A 1.4-point difference near chance is weak evidence for the 'no degradation' claim, and the paper does not report variance or significance. This should be stated as a limitation or replaced with a more discriminative general-knowledge benchmark.
minor comments (5)
  1. [Figure 1 and Section 2.1] The caption of Figure 1 says the gray curve shows the normal attention flow and the pink curve shows the extra memory flow, but the text in Section 2.1 refers to 'the memory information (gray path in Figure 1)' and 'the existing attention information flow (pink path in Figure 1)'. These color assignments are inconsistent and should be reconciled.
  2. [Section 4.1, paragraph 'Performance at Long Context Lengths (1K–4K)'] The text states that RMT-1.7B's average accuracy at 4K is '48.4%', but Table 1 reports 38.4% for that row; the number should be corrected.
  3. [Appendix B, Table 3] The row order within each context length is not consistent across blocks (e.g., in the 2K block LM2 appears before RMT, while in the 4K block RMT appears before LM2), which makes the table harder to read. The rows should be ordered consistently.
  4. [Appendix B and Table 1] The appendix table includes a Llama-3.2-3.2B row at 0K that is not present in Table 1, and the main text does not discuss it; this is confusing and should either be explained or removed.
  5. [Throughout] There are several minor typographical issues, including 'expeirments' in the Appendix B heading and inconsistent use of 'LLama' versus 'Llama'. These do not affect the technical content but should be cleaned up.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: LM2's memory mechanism is defined independently of the benchmarks, and reported gains are measured on external held-out tasks.

full rationale

The paper's derivation chain is self-contained: the memory module is specified by explicit equations (cross attention in Eq. 1, gates in Eqs. 2, 4, 5, update in Eq. 6) with learnable parameters trained on the SmolLM-Corpus, not fitted to BABILong or MMLU. The central empirical claims are comparisons on external benchmarks (BABILong, MMLU) against baselines; no target quantity is used to define the architecture or to set its parameters. The only design choice made by empirical results is the number of memory blocks, and Section 4.3 selects this via perplexity curves rather than the downstream accuracy metric, so it does not reduce the reported gains to a fit. The self-citation to Kang et al. (2024) appears once in the introduction as a general scaling-law remark and is not load-bearing. Concerns about baseline comparability (e.g., RMT being fine-tuned on bAbI while LM2 is pretrained from scratch, and vanilla-Llama-1.7B being scaled by an unspecified recipe) are experimental confounds that affect attribution of the gains, but they do not constitute circularity in the derivation: the model output is not defined in terms of the benchmark, and no fitted constant is renamed as a prediction.

Assumptions & free parameters 3 free parameters · 3 assumptions · 2 invented entities

The paper is an empirical architecture paper, not a derivation, so the ledger records design choices and evaluation assumptions. The main uncharged premises are that 2,048 identity-initialized memory slots in every block are beneficial, that the gating update rule is a good inductive bias, and that BABILong and MMLU measure the intended abilities.

free parameters (3)
  • Memory slot count N = 2048
    Set to 2,048 slots for all 16 blocks; no ablation over slot count is shown, and memory cost scales linearly with N.
  • Memory module placement = all 16 decoder blocks
    Chosen because the 16-block configuration had the lowest perplexity in Section 4.3; no BABILong or MMLU ablation is reported.
  • Memory slot initialization = identity matrix
    Slots are initialized as I_{d x d}; the paper gives no comparison to random or learned initialization, so the effect of this choice is unknown.
assumptions (3)
  • domain assumption BABILong accuracy is a valid measure of long-context memory and reasoning ability
    The central evaluation treats BABILong task accuracy as the target evidence for memory capability (Section 4.1).
  • domain assumption RMT-1.7B is a faithful implementation of Recurrent Memory Transformer following Kuratov et al. and Ko et al.
    The RMT baseline numbers depend on correct reproduction of prior fine-tuning and evaluation methodology (Section 4 baseline description).
  • domain assumption SmolLM-Corpus pretraining data is of sufficient quality and is used identically for vanilla-Llama and LM2
    Both models are pretrained from scratch on the described corpus, but exact data splits and training details are not given (Section 3).
invented entities (2)
  • Memory bank M with N slots
    purpose: Stores and retrieves long-range contextual representations through cross attention and gated updates
    The memory bank is a new architectural component; its benefit is evidenced only by in-paper benchmark scores and qualitative heatmaps, not by an external falsifiable prediction.
  • Input, forget, and output gate pathway
    purpose: Controls writing, erasing, and reading of memory slots during each decoder block
    These gates are trainable mechanisms analogous to LSTM gates; no independent evidence outside the paper's evaluations is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LM2: Large Memory Models." pith.science (2026). https://pith.science/paper/PFBW57AP

@misc{pith2026250206049,
  author       = {Pith},
  title        = {Pith review of: LM2: Large Memory Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PFBW57AP}},
  note         = {Machine review of arXiv:2502.06049}
}
read the original abstract

This paper introduces the Large Memory Model (LM2), a decoder-only Transformer architecture enhanced with an auxiliary memory module that aims to address the limitations of standard Transformers in multi-step reasoning, relational argumentation, and synthesizing information distributed over long contexts. The proposed LM2 incorporates a memory module that acts as a contextual representation repository, interacting with input tokens via cross attention and updating through gating mechanisms. To preserve the Transformers general-purpose capabilities, LM2 maintains the original information flow while integrating a complementary memory pathway. Experimental results on the BABILong benchmark demonstrate that the LM2model outperforms both the memory-augmented RMT model by 37.1% and the baseline Llama-3.2 model by 86.3% on average across tasks. LM2 exhibits exceptional capabilities in multi-hop inference, numerical reasoning, and large-context question-answering. On the MMLU dataset, it achieves a 5.0% improvement over a pre-trained vanilla model, demonstrating that its memory module does not degrade performance on general tasks. Further, in our analysis, we explore the memory interpretability, effectiveness of memory modules, and test-time behavior. Our findings emphasize the importance of explicit memory in enhancing Transformer architectures.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Memory for Large Language Models

    cs.CL 2026-07 conditional novelty 4.0 of 10

    The paper organizes LLM memory architectures along three axes—implicit vs explicit representation, offline vs online update, and short vs long-term persistence—plus finer-grained update rules.

  2. Memory-Augmented Transformers: A Systematic Review from Neuroscience Principles to Enhanced Model Architectures

    cs.LG 2025-08 unverdicted novelty 3.0 of 10

    Memory-augmented Transformer research is organized into a three-axis taxonomy bridging neuroscience memory concepts to network designs, but no new result is produced.

Reference graph

Works this paper leans on

26 extracted references · 6 canonical work pages · cited by 2 Pith papers

  1. [1]

    ETC: encoding long and structured data in transformers

    Joshua Ainslie, Santiago Onta \ n \' o n, Chris Alberti, Philip Pham, Anirudh Ravula, and Sumit Sanghai. ETC: encoding long and structured data in transformers. CoRR, abs/2004.08483, 2020. URL https://arxiv.org/abs/2004.08483

    arXiv 2004

  2. [2]

    Peters, and Arman Cohan

    Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer. CoRR, abs/2004.05150, 2020. URL https://arxiv.org/abs/2004.05150

    arXiv 2004

  3. [3]

    Language models can explain neurons in language models

    Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in language models. https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html, 2023

    2023

  4. [4]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

  5. [5]

    Aydar Bulatov, Yuri Kuratov, and Mikhail S. Burtsev. Recurrent memory transformer, 2022. URL https://arxiv.org/abs/2207.06881

    arXiv 2022

  6. [6]

    Carbonell, Quoc V

    Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. CoRR, abs/1901.02860, 2019. URL http://arxiv.org/abs/1901.02860

    arXiv 1901

  7. [7]

    The archival advantage: Integrating archival expertise into management of born-digital library materials

    Jackie Dooley. The archival advantage: Integrating archival expertise into management of born-digital library materials. Archival Science Special Issue on Archiving Research Data, 7 0 (1), March 2007

    work page 2007

  8. [8]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020

    arXiv 2010

Show all 26 references
  1. [9]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  2. [10]

    GMAT: global memory augmentation for transformers

    Ankit Gupta and Jonathan Berant. GMAT: global memory augmentation for transformers. CoRR, abs/2006.03274, 2020. URL https://arxiv.org/abs/2006.03274

    2006 arXiv

  3. [11]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021

    2021

  4. [12]

    Think before you act: Decision transformers with working memory

    Jikun Kang, Romain Laroche, Xingdi Yuan, Adam Trischler, Xue Liu, and Jie Fu. Think before you act: Decision transformers with working memory. In ICML . OpenReview.net, 2024

    2024

  5. [13]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  6. [14]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT, volume 1, page 2. Minneapolis, Minnesota, 2019

    2019

  7. [15]

    Memreasoner: A memory-augmented llm architecture for multi-hop reasoning

    Ching-Yun Ko, Sihui Dai, Payel Das, Georgios Kollias, Subhajit Chaudhury, and Aurelie Lozano. Memreasoner: A memory-augmented llm architecture for multi-hop reasoning. In The First Workshop on System-2 Reasoning at Scale, NeurIPS'24, 2024

    2024

  8. [16]

    Babilong: Testing the limits of llms with long context reasoning-in-a-haystack, 2024

    Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. Babilong: Testing the limits of llms with long context reasoning-in-a-haystack, 2024

    2024

  9. [17]

    u ttler, Mike Lewis, Wen - tau Yih, Tim Rockt \

    Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \" u ttler, Mike Lewis, Wen - tau Yih, Tim Rockt \" a schel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. CoR...

    2005 arXiv

  10. [18]

    Laffi: Leveraging hybrid natural language feedback for fine-tuning language models

    Qianxi Li, Yingyue Cao, Jikun Kang, Tianpei Yang, Xi Chen, Jun Jin, and Matthew E Taylor. Laffi: Leveraging hybrid natural language feedback for fine-tuning language models. arXiv preprint arXiv:2401.00907, 2023

    2023 arXiv

  11. [19]

    Hierarchical transformers for multi-document summarization

    Yang Liu and Mirella Lapata. Hierarchical transformers for multi-document summarization. In Anna Korhonen, David Traum, and Llu \'i s M \`a rquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5070--5081, Florence, Ital...

    2019 doi

  12. [20]

    Small language models: Efficient, accessible, and effective

    Allal Loubna, Ben, Lozhkov Anton, and Bakouch Elie. Small language models: Efficient, accessible, and effective. https://huggingface.co/blog/smollm, 2023. Accessed: 2025-01-16

    2023

  13. [21]

    Multi-hop question answering, 2024

    Vaibhav Mavi, Anubhav Jangra, and Adam Jatowt. Multi-hop question answering, 2024. URL https://arxiv.org/abs/2204.09140

    2024 arXiv

  14. [22]

    The fineweb datasets: Decanting the web for the finest text data at scale

    Guilherme Penedo, Hynek Kydl \' c ek, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. arXiv preprint arXiv:2406.17557, 2024

    2024 arXiv

  15. [23]

    Associative recurrent memory transformer, 2024

    Ivan Rodkin, Yuri Kuratov, Aydar Bulatov, and Mikhail Burtsev. Associative recurrent memory transformer, 2024. URL https://arxiv.org/abs/2407.04841

    2024 arXiv

  16. [24]

    Towards ai-complete question answering: A set of prerequisite toy tasks

    Jason Weston, Antoine Bordes, Sumit Chopra, and Tom \' a s Mikolov. Towards ai-complete question answering: A set of prerequisite toy tasks. In ICLR (Poster) , 2016

    2016

  17. [25]

    Big bird: Transformers for longer sequences

    Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Onta \ n \' o n, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big bird: Transformers for longer sequences. CoRR, abs/2007.14062, 2020. URL https://arxiv.org/abs/2007.14062

    2007 arXiv

  18. [26]

    Incorporating bert into neural machine translation

    Jinhua Zhu, Yingce Xia, Lijun Wu, Di He, Tao Qin, Wengang Zhou, Houqiang Li, and Tie-Yan Liu. Incorporating bert into neural machine translation. arXiv preprint arXiv:2002.06823, 2020

    2002 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.