Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

TPTT: Transforming Pretrained Transformers into Titans

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that pretrained transformers can be retrofitted with linearized attention and internal memory gating using only LoRA fine-tuning, and that the retrofit preserves or improves one-shot MMLU accuracy.

desk verdict Table 3's baselines are below chance, so the headline accuracy gain is an evaluation artifact; the real value is the open framework and code. read the letter →

arxiv 2506.17671 v2 pith:RKEVB46S submitted 2025-06-21 cs.CL cs.AIcs.LG

Fabien Furfaro This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords TPTTlinearattentionMemoryasGateDeltaProductLoRApretrainedtransformersMMLUlong-contextlanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

TPTT proposes a practical upgrade path for already-trained transformer LLMs: inject a linear-attention branch alongside the original softmax attention, gate the two outputs together, and fine-tune only low-rank adapters. The paper's central claim is that this conversion works without retraining from scratch, and that on one-shot MMLU the converted models can match or beat their base counterparts. The strongest reported result is for Llama-3.2-1B, where the "Titans" variant reaches an Exact Match of 0.246 versus 0.007 for the base model. If this holds up, it matters because it would give a cheap route to long-context, resource-limited deployment of existing LLMs while preserving their pretrained knowledge.

What carries the argument

The load-bearing component is LiZAttention, a dual-path attention block that keeps the pretrained softmax attention and adds a parallel linearized-attention branch. The linear branch uses the DeltaProduct operator as its feature map, with chunkwise-parallel state updates and optional virtual-token expansion via derivative and rotary tricks. Memory as Gate (MaG) then fuses the two branch outputs with a weighting α that can be fixed, gradually increased, or cycled during training; sharing Q/K/V/O projections across branches keeps the added parameters small, and LoRA supplies the trainable low-rank updates. This design is what makes the retrofit cheap to apply and stable to train.

What would settle it

Re-run the one-shot MMLU evaluation on the untransformed Llama-3.2-1B and Qwen2.5-1.5B using the same prompt and answer parser as the paper; if the base Exact Match scores are far above the reported 0.007 and 0.000, the baselines are mismeasured and the headline TPTT gains are artifacts of the harness rather than the architecture.

Watch

Extended reading notes

Core claim

On its own terms, the paper reports a framework for transforming pretrained causal transformers into memory-augmented linear-attention models. The core finding is that a module called LiZAttention—which runs softmax attention and a DeltaProduct-based linear attention in parallel and combines them with a learnable Memory-as-Gate weight—can be inserted into existing LLMs and adapted with LoRA. Evaluated in a one-shot MMLU setting, the converted Llama-3.2-1B shows substantially higher Exact Match than the base model (0.246 vs 0.007), with other converted models showing mixed or null EM differences. The paper also reports that a purely linear-attention model can be produced (α = 1) and that the approach trains stably across seven model families, from 270M to 7B parameters.

Load-bearing premise

The conclusion that TPTT improves accuracy depends on the one-shot MMLU evaluation being correctly implemented; if the prompt template or answer parser is flawed, the reported gains over near-zero base-model scores are an artifact.

Editorial extensions

If this is right

  • Existing deployed transformers can be upgraded to linear-time attention and memory gating without full retraining, lowering the barrier to long-context and resource-limited deployment.
  • A LiZA retrofit with LoRA can improve one-shot MMLU Exact Match on a 1B parameter model, so efficiency gains need not come at the cost of accuracy.
  • With α = 1, DeltaProduct can fully replace softmax attention, yielding a purely linear-attention model that still trains stably.
  • The same recipe trains across model families from 270M to 7B parameters, so the approach is not tied to one architecture.
  • Because adaptation uses LoRA on 500 samples for 5 epochs, the practical cost is small enough to run on a single T4 GPU.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Extension, not in the paper: re-measuring the base models with a standard MMLU harness is the first test to run; a 1B model scoring 0.007 EM is implausibly low, so the reported improvement margin may shrink once baselines are corrected.
  • Extension, not in the paper: the efficiency benefit of linear attention should be tested on genuinely long sequences (e.g., more than 4K tokens) with wall-clock and memory scaling; the paper's MMLU evaluation at 512-token training does not directly demonstrate the long-context advantage.
  • Extension, not in the paper: the unused bidirectional linear-attention path suggests the same retrofit could apply to encoders or diffusion LLMs, but the paper reports no experiments in those settings.
  • Extension, not in the paper: ablating shared versus separate Q/K/V/O projections between the two branches would isolate whether the LoRA adapters or the gating mechanism drive the reported gains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes TPTT, a framework for retrofitting pretrained causal transformer LMs with linearized attention (based on DeltaProduct) and memory gating (Memory as Gate, MaG), combined with LoRA-based parameter-efficient fine-tuning. The method is implemented as a library compatible with Hugging Face Transformers. The main empirical claim is that TPTT variants, especially Titans-Llama-3.2-1B, outperform their base counterparts on one-shot MMLU, with the abstract reporting an 'up to 20% relative increase' in Exact Match for Llama-3.2-1B. The paper also reports training convergence on a small Alpaca subset and positions TPTT as a practical path to long-context efficiency without full retraining.

Significance. If the reported accuracy gains were reliable, TPTT would be a useful contribution: it packages existing components (DeltaProduct, LoRA, Titans-inspired gating) into a practical retrofit toolkit and provides open-source code and Hugging Face integration. The architectural ideas are plausible, and the paper is transparent about its reliance on prior work. However, the empirical validation is the weakest part. The base-model scores in Table 3 are far below chance for a 4-choice benchmark, and the abstract's headline number is inconsistent with the table. Since every accuracy conclusion is a comparison against those baselines, the central claim is currently unsubstantiated. The framework's efficiency and long-context benefits are also not directly measured. These issues outweigh the positive aspects of the open-source release.

major comments (4)
  1. [§5.4, Table 3] The one-shot MMLU baseline scores are implausibly low: Llama-3.2-1B EM = 0.007, Qwen2.5-1.5B EM = 0.000, and OLMo-1B-hf EM = 0.000. MMLU is a 4-choice multiple-choice benchmark, so random guessing alone gives EM ≈ 0.25, and even small 1B models typically score well above that. These near-zero values strongly suggest a bug in the evaluation pipeline, such as incorrect prompt formatting, answer parsing, dataset subset, or normalization. Since the claimed TPTT improvements are all relative to these baselines, the central accuracy result is invalid until the evaluation is corrected and the comparison is rerun.
  2. [Abstract and §5.4] The abstract states 'up to a 20% relative increase in Exact Match', but Table 3 shows Llama base EM 0.007 → Titans EM 0.246, which is a 3414% relative increase, not 20%. This internal inconsistency indicates that the numbers were not cross-checked. Furthermore, the claim that 'TPTT models ... consistently outperform their base counterparts' is contradicted by the same table: Titans-Qwen2.5-1.5B has the same EM (0.000) as the base but lower PEM (0.500 vs 0.598) and lower PQEM (0.583 vs 0.690); for OLMo, EM is 0.000 for both, with only small PEM/PQEM gains. The evidence does not support the stated conclusions.
  3. [§5.1 and §5.4] The experimental setup is under-specified to the point of non-reproducibility. The paper does not state which subset of MMLU was used (the full 14,042 questions or a sample), what the one-shot prompt template was, how model outputs were extracted and matched to the 4 choices, how many independent runs produced the reported mean±std values, or what seeds were used. The training setup (500 Alpaca samples, 5 epochs) is described, but the connection between instruction tuning on Alpaca and MMLU performance is not justified. Without these details, Table 3 cannot be interpreted or replicated.
  4. [§6 and §5] The paper repeatedly claims efficiency and long-context benefits (e.g., 'quadratic to linear', 'long-context capable'), but no inference-time measurements are provided. Table 2 reports only training throughput, not wall-clock inference time, memory usage, or any long-sequence benchmark (≥8k tokens). The efficiency advantage is therefore entirely unsupported by the reported experiments. A quantitative comparison of inference cost between base and TPTT models on long sequences is needed to substantiate this core motivation.
minor comments (4)
  1. [Figure 2] The legend labels in Figure 2 appear to have missing Greek symbols (alpha, Delta, etc.), leaving empty spaces in model names such as 'Llama+LiZA( , k, + , nh = 2, = 0.5)'. This makes the figure very hard to read; the labels should be typeset properly.
  2. [§3.2 vs Algorithm 1] Section 3.2 states that α is scheduled externally and not optimized by backpropagation, but Algorithm 1 line 5 describes a 'learnable coefficient α ∈ [0,1]'. This inconsistency should be reconciled in the text.
  3. [§5.1] The paper says 'All experiments were performed on NVIDIA Tesla T4 GPUs (Kaggle platform)' but does not report the number of runs or any statistical test comparing Titans and base models; the standard deviations in Table 3 appear to be over what is currently unclear.
  4. [References] Reference [14] is listed as 'Albert Q. Jiang and all Mistral team' which is informal; the actual author list of the Mistral 7B technical report should be used.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: TPTT's claims are empirical comparisons on MMLU, with no derivation that reduces to its inputs.

full rationale

This paper does not contain a circular derivation. The central claims are empirical: TPTT models are fine-tuned on Alpaca and evaluated on MMLU, and the reported accuracy results are measurements, not predictions derived from the same data. The framework borrows DeltaProduct and Titans as external components; citing prior work for these components is transparent dependency, not circularity. MaG's alpha is scheduled externally rather than fitted to the evaluation set, so the gating mechanism is not a fitted-input-called-prediction. The statement that DeltaProduct of order n = 2 achieves expressivity comparable to Titans is cited from prior work and is not used to establish TPTT's empirical superiority, so it does not constitute a load-bearing self-citation. The implausibly low base MMLU EM scores (e.g., Llama-3.2-1B at 0.007) and the abstract's internally inconsistent '20% relative increase' claim are serious empirical-validity concerns, but they are evaluation-pipeline or reporting issues, not cases where a result reduces to its inputs by construction. No equation in the paper defines a prediction in terms of the benchmark outcome it claims to predict. Therefore, the circularity score is 0.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claims rest on the correctness of the DeltaProduct chunkwise update (cited, not re-derived), the assumption that pretrained weights remain useful when a linear branch is added, and the validity of the MMLU evaluation setup. The latter is not established and is contradicted by implausibly low baseline scores. Several hyperparameters (alpha, LoRA rank, nh) are set by hand.

free parameters (6)
  • MaG weight alpha (schedule) = 0.5 constant (also 0.125, 1.0, cyclic in variants)
    Set by hand or schedule, not learned; controls the balance between softmax and linear attention outputs. Central to the architecture.
  • Householder steps nh = 1 or 2
    Chosen per variant; increases expressivity of the memory state update.
  • LoRA rank = 8
    Standard low-rank dimension for parameter-efficient fine-tuning.
  • LoRA scaling alpha = 16
    LoRA scaling factor, set by convention.
  • LoRA dropout = 0.05
    Regularization hyperparameter.
  • Training budget = 500 samples, 5 epochs
    Very small fine-tuning budget; the effect of this choice on the reported results is not analyzed.
assumptions (5)
  • standard math DeltaProduct chunkwise update equations (18)-(21) are correct and equivalent to the sequential delta rule.
    Taken from [24]; the paper does not prove these, it uses them as the backbone of the linear attention branch.
  • domain assumption Adding a linear attention branch to a frozen pretrained transformer, training only LoRA adapters, preserves pretrained knowledge.
    This is the core premise of the framework, but it is only validated by training loss curves and a dubious MMLU evaluation, not by a controlled ablation.
  • domain assumption The one-shot MMLU harness measures model capability correctly.
    The baseline EM scores in Table 3 are near zero for 1B models, so this assumption is likely false.
  • standard math Householder product orthogonality makes the state update numerically stable.
    Uses the stability property from [24] without additional analysis in this paper.
  • ad hoc to paper The alpha scheduling policy (constant, gradual, or cyclic) is an acceptable way to set the gating weight without learning it.
    The paper states alpha was not optimized by backpropagation; the choice of schedule is a design decision whose effects are reported but not theoretically justified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TPTT: Transforming Pretrained Transformers into Titans." pith.science (2026). https://pith.science/paper/RKEVB46S

@misc{pith2026250617671,
  author       = {Pith},
  title        = {Pith review of: TPTT: Transforming Pretrained Transformers into Titans},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RKEVB46S}},
  note         = {Machine review of arXiv:2506.17671}
}
read the original abstract

Transformer-based large language models (LLMs) have achieved strong performance across many natural language processing tasks. Nonetheless, their quadratic computational and memory requirements, particularly in self-attention layers, pose challenges for efficient inference on long contexts and for deployment in resource-limited environments. We present TPTT (Transforming Pretrained Transformers into Titans), a framework designed to augment pretrained Transformers with linearized attention (LiZA) and internal memory gating via Memory as Gate (MaG), applied without full retraining. TPTT supports parameter-efficient fine-tuning (LoRA) and integrates with standard toolkits such as Hugging Face Transformers. We evaluated TPTT on several pretrained models, including Llama-1B, OlMoE-1B-7B, Qwen2.5-1.5B, Gemma3-270m, OpenELM-1.3B, and Mistral-7B, in order to assess applicability across architectures of different scales. Experiments on models with approximately 1 billion parameters, evaluated primarily on the MMLU benchmark, suggest potential improvements in both efficiency and accuracy compared to baseline models. For example, Titans-Llama-1B exhibited up to a 20\% relative increase in Exact Match scores in one-shot evaluation. An additional finding is that it is possible to convert a quadratic-attention model into a purely linear-attention model using the DeltaProduct mechanism. All training runs were carried out with modest computational resources. These preliminary findings indicate that TPTT may help adapt pretrained LLMs for long-context tasks with limited overhead. Further studies on larger models and a broader set of benchmarks will be necessary to evaluate the generality and robustness of the framework. Code is available at https://github.com/fabienfrfr/tptt . Python package at https://pypi.org/project/tptt/ .

Figures

Figures reproduced from arXiv: 2506.17671 by the authors.

Figure 1
Figure 1. Overview of the TPTT architecture. On the left, the diagram illustrates a decoder-only architecture [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Experimental training curves for TPTT models on the Alpaca dataset. The plots show training loss [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Black-Mamba: Biologically-Inspired Leaky Accumulation for Conceptual Knowledge under Distribution Drift

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Gating test-time memory writes on leaky accumulated surprisal preserves most of the adaptation benefit while roughly halving the number of updates.

Reference graph

Works this paper leans on

32 extracted references · 6 canonical work pages · cited by 1 Pith paper

  1. [1]

    Qwen technical report.arXiv preprint arXiv:2309.16609 , 2023

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report.arXiv preprint arXiv:2309.16609 , 2023

    arXiv 2023

  2. [2]

    It’s all connected: A journey through test-time memorization, attentional bias, retention, and online optimization

    Ali Behrouz, Meisam Razaviyayn, Peilin Zhong, and Vahab Mirrokni. It’s all connected: A journey through test-time memorization, attentional bias, retention, and online optimization. arXiv preprint arXiv:2504.13173, 2025

    arXiv 2025

  3. [3]

    Titans: Learning to memorize at test time.arXiv preprint arXiv:2501.00663, 2024

    Ali Behrouz, Peilin Zhong, and Vahab Mirrokni. Titans: Learning to memorize at test time.arXiv preprint arXiv:2501.00663, 2024

    arXiv 2024

  4. [4]

    Rethinking attention with performers

    Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009.14794 , 2020

    arXiv 2009

  5. [5]

    Flashattention-2: Faster attention with better parallelism and work partitioning.arXiv preprint arXiv:2307.08691, 2023

    Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning.arXiv preprint arXiv:2307.08691, 2023

    arXiv 2023

  6. [6]

    Transformers are ssms: Generalized models and efficient algorithms through structured state space duality.arXiv preprint arXiv:2405.21060 , 2024

    Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality.arXiv preprint arXiv:2405.21060 , 2024

    arXiv 2024

  7. [7]

    Bert: Pre-training of deep bidi- rectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidi- rectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers) , pages 4171–4186, 2019

    work page 2019

  8. [8]

    An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929 , 2020

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929 , 2020

    arXiv 2010

Show all 32 references
  1. [9]

    Lora - hugging face peft documentation

    Hugging Face. Lora - hugging face peft documentation. https://huggingface.co/docs/peft/main/ conceptual_guides/lora, 2024

    2024

  2. [10]

    Olmo: Accelerating the science of language models

    Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. Olmo: Accelerating the science of language models. arXiv preprint arXiv:2402.00838 , 2024

    2024 arXiv

  3. [11]

    Mamba: Linear-time sequence modeling with selective state spaces.arXiv preprint arXiv:2312.00752, 2023

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces.arXiv preprint arXiv:2312.00752, 2023. 12

    2023 arXiv

  4. [12]

    Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300 , 2020

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Stein- hardt. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300 , 2020

    2009 arXiv

  5. [13]

    Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022

    2022

  6. [14]

    Jiang and all Mistral team

    Albert Q. Jiang and all Mistral team. Mistral 7b, 2023

    2023

  7. [15]

    Transformers are rnns: Fast autoregressive transformers with linear attention

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. InInternational conference on machine learning , pages 5156–5165. PMLR, 2020

    2020

  8. [16]

    Liger: Linearizing large language models to gated recurrent structures.arXiv preprint arXiv:2503.01496 , 2025

    Disen Lan, Weigao Sun, Jiaxi Hu, Jusen Du, and Yu Cheng. Liger: Linearizing large language models to gated recurrent structures.arXiv preprint arXiv:2503.01496 , 2025

    2025 arXiv

  9. [17]

    Language models are few-shot learners.arXiv preprint arXiv:2005.14165 , 1:3, 2020

    Ben Mann, N Ryder, M Subbiah, J Kaplan, P Dhariwal, A Neelakantan, P Shyam, G Sastry, A Askell, S Agarwal, et al. Language models are few-shot learners.arXiv preprint arXiv:2005.14165 , 1:3, 2020

    2005 arXiv

  10. [18]

    Openelm: An efficient language model family with open-source training and inference framework.arXiv e-prints, pages arXiv–2404, 2024

    Sachin Mehta, Mohammad Hossein Sekhavat, Qingqing Cao, Maxwell Horton, Yanzi Jin, Chenfan Sun, Iman Mirzadeh, Mahyar Najibi, Dmitry Belenko, Peter Zatloukal, et al. Openelm: An efficient language model family with open-source training and inference framework.arXiv e-prints, pa...

    2024

  11. [19]

    Linearizing large language models.arXiv preprint arXiv:2405.06640 , 2024

    Jean Mercat, Igor Vasiljevic, Sedrick Keh, Kushal Arora, Achal Dave, Adrien Gaidon, and Thomas Kollar. Linearizing large language models.arXiv preprint arXiv:2405.06640 , 2024

    2024 arXiv

  12. [20]

    The illusion of state in state-space models.arXiv preprint arXiv:2404.08819, 2024

    William Merrill, Jackson Petty, and Ashish Sabharwal. The illusion of state in state-space models.arXiv preprint arXiv:2404.08819, 2024

    2024 arXiv

  13. [21]

    Olmoe: Open mixture-of-experts language models

    Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, et al. Olmoe: Open mixture-of-experts language models. arXiv preprint arXiv:2409.02060 , 2024

    2024 arXiv

  14. [22]

    Rwkv: Reinventing rnns for the transformer era.arXiv preprint arXiv:2305.13048, 2023

    Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, et al. Rwkv: Reinventing rnns for the transformer era.arXiv preprint arXiv:2305.13048, 2023

    2023 arXiv

  15. [23]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 , 2024

    2024 arXiv

  16. [24]

    Deltaproduct: Improving state-tracking in linear rnns via householder products

    Julien Siems, Timur Carstensen, Arber Zela, Frank Hutter, Massimiliano Pontil, and Riccardo Grazzi. Deltaproduct: Improving state-tracking in linear rnns via householder products. arXiv preprint arXiv:2502.10297, 2025

    2025

  17. [25]

    Alpaca: A strong, replicable instruction-following model.Stanford Center for Research on Foundation Models

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpaca: A strong, replicable instruction-following model.Stanford Center for Research on Foundation Models. https://crfm. stanford. edu/2023/03/13/alp...

    2023

  18. [26]

    Gemma 3 technical report.arXiv preprint arXiv:2503.19786, 2025

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report.arXiv preprint arXiv:2503.19786, 2025

    2025 arXiv

  19. [27]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023

    2023 arXiv

  20. [28]

    Attention is all you need.Advances in neural information processing systems , 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems , 30, 2017

    2017

  21. [29]

    Linformer: Self-attention with linear complexity

    Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768 , 2020

    2006 arXiv

  22. [30]

    Parallelizing linear transformers with the delta rule over sequence length.arXiv preprint arXiv:2406.06484 , 2024

    Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length.arXiv preprint arXiv:2406.06484 , 2024

    2024 arXiv

  23. [31]

    Dream 7b: Diffusion large language models.arXiv preprint arXiv:2508.15487 , 2025

    Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7b: Diffusion large language models.arXiv preprint arXiv:2508.15487 , 2025. 13

    2025 arXiv

  24. [32]

    Lolcats: On low-rank linearizing of large language models.arXiv preprint arXiv:2410.10254, 2024

    Michael Zhang, Simran Arora, Rahul Chalamala, Alan Wu, Benjamin Spector, Aaryan Singhal, Krithik Ramesh, and Christopher Ré. Lolcats: On low-rank linearizing of large language models.arXiv preprint arXiv:2410.10254, 2024. 14

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.