Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

Residual Matrix Transformers: Scaling the Size of the Residual Stream

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Replacing the transformer's residual stream with an outer-product memory matrix creates a nearly free scaling axis: widening the memory consistently lowers loss while cutting the FLOPs, parameters, and training tokens needed to reach a…

desk verdict A genuinely new residual-stream memory architecture with solid small-scale experiments, but the headline FLOP savings rest on a formula that undercounts the very operation being scaled. read the letter →

arxiv 2506.22696 v1 pith:CUHEUNVH submitted 2025-06-28 cs.LG cs.CL

Brian Mak , Jeffrey Flanigan This is my paper · ORCID
classification cs.LGcs.CL
keywords residualmatrixtransformerouterproductmemorystreamscalinglanguagemodelefficiencylawsmomentpropagationdata
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a transformer variant that replaces the per-token residual vector with a small outer-product memory matrix, so the residual stream can be made much wider without adding meaningful parameter or FLOP cost. The central claim is that this width becomes a new, nearly free scaling axis: fixing model size, dataset size, and compute budget, enlarging the residual stream consistently lowers validation loss. In head-to-head pretraining comparisons, the variant reaches the same loss as the transformer baseline using 58% fewer FLOPs, 25% fewer parameters, and 41% fewer training tokens, and it beats a 33% larger transformer on downstream tasks. The authors also derive closed-form variance propagation expressions showing that their storage and retrieval operations behave well at initialization, with one exception in the attention storage path. If these results hold at larger scale, they point to a cheaper way to build language models by giving the model more memory bandwidth rather than more parameters.

What carries the argument

The central object is the outer product memory matrix, or residual matrix: for each token the network stores a matrix $M = \sum_{h=1}^{R} w^{(h)} \otimes x^{(h)}$, where the $w^{(h)}$ are learned key vectors and the $x^{(h)}$ are data vectors, and it retrieves a feature by contracting a key vector against the first dimension of $M$. This replaces the transformer's linear read and write matrices with rank-one outer products, and it is what decouples the residual stream size $D_k$ from the parameter and FLOP counts of the core layers. The supporting theoretical machinery is a moment propagation analysis of the storage and retrieval operations, comparing their mean and variance behavior at initialization with the linear operations they replace.

What would settle it

Train matched transformer and RMT models on identical hardware with a kernel-optimized implementation of the RMT's tensor contractions, measuring wall-clock time and energy to reach a fixed loss; if the RMT does not reach that loss in no more time and energy than the transformer, the central efficiency claim is refuted.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the transformer's residual stream, normally a vector per token that must be resized along with every weight matrix, can be replaced by an outer-product memory matrix of shape $D_k \times D_v$ per token. Reading and writing this memory uses key vectors and tensor contractions rather than full matrices, and because the memory size is governed by $D_k$ while the core computations depend on $D_v$ and the layer rank $R$, increasing the memory size has a less than one percent effect on parameter count and FLOP count at typical model dimensions. Empirically, this makes the residual stream an independent scaling axis: models with residual streams four times larger than the baseline reach lower loss with 58% fewer FLOPs, 25% fewer parameters, and 41% fewer training tokens, and they win on every downstream task tested. The theoretical analysis shows that the storage and retrieval operations have mean and variance propagation ratios at initialization that are at least as good as the linear operations they replace, with the one exception of the attention storage operation.

Load-bearing premise

The headline efficiency claims rest on the paper's FLOP-count formulas rather than measured runtime, so if FLOP savings do not translate into time or energy savings on actual hardware, the practical efficiency gain shrinks or disappears.

Editorial extensions

If this is right

  • Scaling the residual stream size $D_k$ at fixed model size, dataset size, and compute budget monotonically decreases dev loss in the tested range (384 to 4096), and the 4096-wide model matches the 768-wide model's final loss using 23% fewer FLOPs and 25% fewer tokens.
  • The RMT reaches the same training loss as the transformer baseline with 58% fewer FLOPs, 25% fewer parameters, and 41% fewer training tokens in the largest (405M-class) comparison.
  • The RMT outperforms a transformer that is 33% larger on all zero-shot downstream tasks evaluated, despite having fewer parameters and using fewer FLOPs during training.
  • When the residual stream is not expanded, RMT performance is about the same as the transformer, indicating the gains come from the larger memory rather than the architecture change alone.
  • The paper presents residual stream scaling as a new scaling-law axis and argues it is orthogonal to prior residual stream modifications and other transformer efficiency techniques, so those could be combined with it.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The FLOP and token savings are computed from the paper's FLOP-count formulas, and the paper's own timing shows the RMT is 4% slower to reach the same loss and 43% slower to train the full 6B-token budget; whether the claimed savings materialize as real energy or cost reductions depends on closing that runtime gap with optimized kernels for the tensor contractions.
  • If the monotonic gain from wider residual streams persists at larger scale, it suggests scaling-law analyses should add a memory-bandwidth term to the usual parameter-data-compute triad, with the shape of diminishing returns still to be mapped.
  • A testable extension is to vary $D_k$ and $D_v$ independently to separate memory capacity from value dimensionality, since the current experiments scale $D_k$ while holding $D_v$ fixed.
  • Because the residual-matrix scheme replaces only the read/write mechanism of the residual stream, it should compose with orthogonal efficiency techniques such as mixture-of-experts or alternative attention kernels, which the paper does not test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. This paper introduces the Residual Matrix Transformer (RMT), a transformer variant in which the residual stream vectors are replaced by per-token outer-product memory matrices, allowing the residual stream dimension (Dk) to be varied without resizing the main weight matrices. The authors report that the RMT matches the transformer's loss with 58% fewer FLOPS, 25% fewer parameters, and 41% fewer training tokens, and that scaling Dk improves performance while changing the FLOP count by less than 1%. The claims are supported by scaling-law experiments over four model sizes, a moment propagation analysis, a residual-stream scaling study, and downstream evaluations. The paper ships code and uses µParam transfer for hyperparameter tuning, which are strengths. However, the paper's FLOP and parameter formulas contain arithmetic errors that are load-bearing for the headline claims, and the wall-clock runtime results show the RMT is slower than the transformer, a fact not reflected in the abstract.

Significance. If the reported numbers were correct, the RMT would be a meaningful contribution: it proposes a new scaling axis (residual stream size) and shows improvements in data and parameter efficiency. The theoretical moment propagation analysis is a useful first-principles contribution, and the empirical methodology (µParam transfer, multiple scales, downstream evaluation) is generally sound. The release of code is commendable. The principal weakness is that the quantitative claims are tied to FLOP and parameter formulas that undercount the RMT's cost, and the paper's own timing data show a substantial wall-clock penalty. With corrected accounting, the RMT may still be more efficient than the transformer, but the magnitude is uncertain. The paper is promising but requires major revision.

major comments (5)
  1. [Appendix A.2 / Table 7] The FLOP formula for the RMT in Table 7 is incorrect. Summing the per-operation terms listed in §A.2 gives a Dk-dependent contribution of (6 + 12L) N R Dk Dv, not 6(1 + L) N R Dk Dv as printed. Explicitly, the QKV key-vector contractions contribute 6N Dk Dv R per layer, the O key vectors 2N Dk Dv R, the FF input/output key vectors 4N Dk Dv R per layer, the embedding key vectors 4N Dk Dv R, and the unembedding key vectors 2N Dk Dv R. The printed formula omits 12L N R Dk Dv of the Dk-dependent arithmetic. For the 305M run (L=24, R=16, Dk=1024, Dv=64), this approximately doubles the Dk-related FLOPs. Consequently, the FLOP totals in Tables 3 and 4, the FLOP axes in Figures 2 and 3, and the abstract's '58% fewer FLOPS' claim are not supported until the formula is corrected and all results are recomputed.
  2. [Appendix A.1 / Table 7] The parameter formula for the RMT in Table 7 is also incorrect. The printed expression R(2Dk(3L + 1) + Dv(2L D_FF + V + N)) omits one V Dv term (the unembedding weights) and one Dk term. For the model listed as 305M in Table 9 (L=24, Dk=1024, Dv=64, R=16, D_FF=4096, V=50257, N=512), the formula predicts 255.7M parameters, while the corrected version gives about 305M. This changes the '25% fewer parameters' claim and affects the parameter-efficiency discussion in §4.2.
  3. [Section 4.4 / Table 7] The claim that increasing Dk from 384 to 4096 changes the FLOP count by less than 1% is inconsistent with the paper's own FLOP formula. Plugging the §4.4 configuration (L=12, R=12, Dv=64, N=512, V=50257, D_FF=3072) into Table 7 gives forward FLOPs per sequence of approximately 1.6e11 at Dk=384 and 2.7e11 at Dk=4096, a roughly 72% increase. The '<1%' statement appears to exclude Dk-dependent tensor contractions that are counted elsewhere, and it directly undermines the central contribution that the residual stream can be scaled for nearly free.
  4. [Tables 3 and 4 / Abstract] The wall-clock runtime results show that the RMT is 4% slower to reach the same loss and 43% slower to train on the full 6B tokens, yet the abstract and conclusion emphasize FLOPS, parameters, and tokens while omitting the runtime penalty. Since the reported FLOPS do not translate into actual throughput (the paper attributes the gap to unoptimized tensor contractions in Appendix E), the practical efficiency claim is overstated. The abstract should either report the runtime results or qualify the efficiency claims to avoid misleading readers.
  5. [Section 4.3] The comparison against transformer variants uses untuned hyperparameters for all variants, while the RMT and transformer baselines were tuned via µParam transfer. This risks disadvantaging the baseline architectures and makes the claim that the RMT is the most compute- and parameter-efficient variant difficult to interpret. The paper notes this limitation, but the claim should be tempered or the variants should be given comparable tuning effort.
minor comments (6)
  1. [Abstract] The abstract contains a duplicated word: 'fewer training tokens tokens' should read 'fewer training tokens'.
  2. [Appendix B] The heading 'Moment Propagation Derivation' contains a typo: 'retrival' should be 'retrieval'.
  3. [Appendix C.1] The software name 'Equionx' should be 'Equinox'.
  4. [Appendix C.2.1] 'Nvida' should be 'Nvidia'.
  5. [Appendix C.1] 'Mistral's GPT2 stability tweeks' should be 'Mistral's GPT2 stability tweaks'.
  6. [Section 3.2 / Table 2] Table 2 would benefit from a note specifying which model shapes are used for the RMT entries; for example, the attention storage value of 0.4 for the RMT depends on R and Dk, and the caption's reference to 'GPT2-medium model shapes' is not fully explicit.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the RMT efficiency and moment-propagation claims rest on explicit formulas and independent experiments, not on fitted inputs or load-bearing self-citations.

full rationale

The paper's core claims are supported by (1) measured training curves comparing the RMT to the transformer and other baselines, (2) explicit closed-form parameter and FLOP formulas given in Appendix A that are derived from the architecture rather than fitted to the loss data, and (3) a moment-propagation derivation in Appendix B with stated assumptions (zero-mean, independent weight initialization). The headlined '58% fewer FLOPS' figure is computed from these stated formulas, not from a parameter fitted to the outcome it is said to predict, so it is not circular. The residual-stream scaling experiment in Section 4.4 is an empirical manipulation that holds model size and compute fixed while varying Dk; the claimed <1% FLOP difference follows from the published FLOP equations, and the observed loss improvement is a measured result independent of those equations. The paper contains no load-bearing self-citations: references to Hoffmann et al., Kedia et al., Elhage et al., and the outer-product memory literature are external prior work. Any concern that the FLOP formulas contain arithmetic errors or omit certain tensor-contraction costs is a correctness or reproducibility issue, not a circularity issue, because the formulas are stated explicitly and can be checked and corrected independently without changing the logical structure of the claims. Under the rubric, this is a normal non-circular finding.

Assumptions & free parameters 5 free parameters · 4 assumptions · 1 invented entities

The central claims rest on a small number of architectural hyperparameters (R, Dk, Dv, D_FF) and on domain assumptions about the suitability of outer product memory as a residual stream. The moment propagation theory and FLOP formulas use standard assumptions (i.i.d. zero-mean weights, forward/backward symmetry). No new physical entities are introduced; the residual matrix is a model component with direct experimental support.

free parameters (5)
  • R (number of key vectors per layer) = 12 or 16 in experiments
    Controls the number of stored vectors and the width of the FF input/output (R*Dv). Chosen by hand, not fitted to make the theory work.
  • Dk (residual key dimension) = Varies from 384 to 4096 in scaling experiment; 32 to 1024 in scaling laws
    The scaling variable for residual stream size. Increasing Dk raises parameters and FLOPs only slightly, which is the basis for the independent-scaling claim.
  • Dv (residual value dimension) = 32, 48, or 64 in experiments
    Set to the attention head dimension (transformer Dh). Chosen to keep core computations comparable to the transformer.
  • D_FF (feedforward width) = 1536 to 4096
    Standard transformer feedforward dimension, copied from GPT2 sizes.
  • Chinchilla token multiplier = 20x non-embedding parameters
    Used to set training token budgets for both models following Hoffmann et al. (2024). Not fitted here.
assumptions (4)
  • domain assumption An outer product memory matrix can serve as the residual communication channel without losing expressive power needed for language modeling.
    This is the core architectural premise of the paper. The experiments test it, but it is not proven.
  • domain assumption LayerNorm applied over the residual matrix provides the normalization needed for the outer product store (footnote 1: pre-LayerNorm operations provide the missing normalization from Eq. 12).
    The paper relies on LayerNorm to make the outer product memory well-behaved; the exact behavior of LayerNorm on a Dk×Dv tensor is not analyzed.
  • standard math Moment propagation analysis assumes weights are initialized i.i.d. with zero mean and independent of inputs (Appendix B).
    This is the standard Glorot-style assumption, not verified empirically for the RMT.
  • standard math Backward pass FLOPs are approximated as twice the forward pass FLOPs, following Hoffmann et al. (2024).
    Used in Appendix A to compute the RMT's FLOP count.
invented entities (1)
  • Residual matrix (outer product memory per token) independent evidence
    purpose: Replaces the residual stream vector with a Dk×Dv matrix to store and retrieve features cheaply.
    The paper provides direct experimental evidence (Fig. 6) that scaling this matrix improves loss, and architecture comparisons show it is functional.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Residual Matrix Transformers: Scaling the Size of the Residual Stream." pith.science (2026). https://pith.science/paper/CUHEUNVH

@misc{pith2026250622696,
  author       = {Pith},
  title        = {Pith review of: Residual Matrix Transformers: Scaling the Size of the Residual Stream},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CUHEUNVH}},
  note         = {Machine review of arXiv:2506.22696}
}
read the original abstract

The residual stream acts as a memory bus where transformer layers both store and access features (Elhage et al., 2021). We consider changing the mechanism for retrieving and storing information in the residual stream, and replace the residual stream of the transformer with an outer product memory matrix (Kohonen, 1972, Anderson, 1972). We call this model the Residual Matrix Transformer (RMT). We find that the RMT enjoys a number of attractive properties: 1) the size of the residual stream can be scaled independently of compute and model size, improving performance, 2) the RMT can achieve the same loss as the transformer with 58% fewer FLOPS, 25% fewer parameters, and 41% fewer training tokens tokens, and 3) the RMT outperforms the transformer on downstream evaluations. We theoretically analyze the transformer and the RMT, and show that the RMT allows for more efficient scaling of the residual stream, as well as improved variance propagation properties. Code for this project can be found at https://github.com/bmac3/residual-matrix-transformer.

Figures

Figures reproduced from arXiv: 2506.22696 by the authors.

Figure 1
Figure 1. Model size and per-example FLOPS versus residual stream size for the transformer and RMT. See §3.1 for details. et al., 2023). This presents a need for more data and compute efficient models. When it comes to large scale LLM training, few architectural modifications rival the impact that Mixture of Expert Models (Fedus et al., 2022) have had. Their work inspired many later cutting-edge models (Du et al., 2022, Jiang… view at source ↗
Figure 2
Figure 2. Scaling law curves of RMT vs transformer. Model sizes vary from 46M to 405M. 4.2. RMT vs Transformer In this section we experimentally show that the RMT ex￾hibits better parameter, data, FLOP, and inference time mem￾ory efficiency than the transformer. We verify these trends by examining the scaling properties of both architectures and analyzing the training curves of the largest models. We train a series of four mo… view at source ↗
Figure 4
Figure 4. Train loss curves for the transformer and RMT on a time basis [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figures from the paper (3 more)
Figure 3
Figure 3. Figure 3: Train loss curves for the transformer and RMT on a per￾token and per-flop basis [PITH_FULL_IMAGE:figures/full_fig_p006_3.png]
Figure 5
Figure 5. Figure 5: RMT versus other transformer variants that modify the residual stream (Vaswani et al., 2017, Dou et al., 2018, Chai et al., 2020, Xu et al., 2024). FLOP Efficiency From the top plot of [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Dev loss vs residual stream size. Model size, dataset size, and computational budget are held constant for all runs. Residual stream size is calculated as Dk × Dv. 4.5. Downstream Evaluation In this section we present zero-shot downstream results of the RMT and transfo…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. KromHC: Manifold-Constrained Hyper-Connections with Kronecker-Product Residual Matrices

    cs.CL 2026-01 conditional novelty 6.0 of 10

    KromHC uses Kronecker products of small doubly stochastic matrices to make Hyper-Connection residual matrices exactly balanced with O(n^2C) parameters, and matches or beats prior variants on small LLM pretraining runs.

Reference graph

Works this paper leans on

57 extracted references · 26 canonical work pages · cited by 1 Pith paper

  1. [1]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Data on notable ai models, 2024

    AI, E. Data on notable ai models, 2024. URL https://epoch.ai/data/notable-ai-models. Accessed: 2024-12-14

    work page 2024

  3. [3]

    Anderson, J. A. A simple neural network generating an interactive memory. Mathematical Biosciences, 14 0 (3): 0 197--220, 1972. ISSN 0025-5564. doi:https://doi.org/10.1016/0025-5564(72)90075-2. URL https://www.sciencedirect.com/science/article/pii/0025556472900752

    arXiv 1972

  4. [4]

    P., Mao, H

    Bachlechner, T., Majumder, B. P., Mao, H. H., Cottrell, G. W., and McAuley, J. J. Rezero is all you need: Fast convergence at large depth. CoRR, abs/2003.04887, 2020. URL https://arxiv.org/abs/2003.04887

    arXiv 2003

  5. [5]

    L., Gao, J., and Choi, Y

    Bisk, Y., Zellers, R., Bras, R. L., Gao, J., and Choi, Y. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020

    2020

  6. [6]

    J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., Vander P las, J., Wanderman- M ilne, S., and Zhang, Q

    Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., Vander P las, J., Wanderman- M ilne, S., and Zhang, Q. JAX : composable transformations of P ython+ N um P y programs, 2018. URL http://github.com/jax-ml/jax

    2018

  7. [7]

    Highway transformer: Self-gating enhanced self-attentive networks

    Chai, Y., Jin, S., and Hou, X. Highway transformer: Self-gating enhanced self-attentive networks. In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J. (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp.\ 6887--6900, Online, July 2020. Association for Computational Linguistics. doi:10.18653/v1/2020.acl...

    doi:10.18653/v1/2020.acl-main.616 2020

  8. [8]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, abs/1803.05457, 2018

    arXiv 2018

Show all 57 references
  1. [9]

    and Gu, A

    Dao, T. and Gu, A. Transformers are SSM s: Generalized models and efficient algorithms through structured state space duality. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Proceedings of the 41st International C...

    2024

  2. [10]

    The practitioner’s guide to the maximal update parameterization

    Dey, N., Anthony, Q., and Hestness, J. The practitioner’s guide to the maximal update parameterization . https://www.cerebras.ai/blog/the-practitioners-guide-to-the-maximal-update-parameterization, September 2024

    2024

  3. [11]

    Exploiting deep representations for neural machine translation

    Dou, Z.-Y., Tu, Z., Wang, X., Shi, S., and Zhang, T. Exploiting deep representations for neural machine translation. In Riloff, E., Chiang, D., Hockenmaier, J., and Tsujii, J. (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp.\ ...

    2018 doi

  4. [12]

    Dynamic layer aggregation for neural machine translation with routing-by-agreement

    Dou, Z.-Y., Tu, Z., Wang, X., Wang, L., Shi, S., and Zhang, T. Dynamic layer aggregation for neural machine translation with routing-by-agreement. Proceedings of the AAAI Conference on Artificial Intelligence, 33 0 (01): 0 86--93, Jul. 2019. doi:10.1609/aaai.v33i01.330186. URL...

    2019 doi

  5. [13]

    M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A

    Du, N., Huang, Y., Dai, A. M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A. W., Firat, O., Zoph, B., Fedus, L., Bosma, M. P., Zhou, Z., Wang, T., Wang, E., Webster, K., Pellat, M., Robinson, K., Meier-Hellstern, K., Duke, T., Dixon, L., Zhang, K., Le, Q., Wu, Y...

    2022

  6. [14]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  7. [15]

    A mathematical framework for transformer circuits

    Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., DasSarma, N., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan...

    2021

  8. [16]

    Depth-wise attention ( DWA tt): A layer fusion method for data-efficient classification

    ElNokrashy, M., AlKhamissi, B., and Diab, M. Depth-wise attention ( DWA tt): A layer fusion method for data-efficient classification. In Calzolari, N., Kan, M.-Y., Hoste, V., Lenci, A., Sakti, S., and Xue, N. (eds.), Proceedings of the 2024 Joint International Conference on Co...

    2024

  9. [17]

    Switch transformers: scaling to trillion parameter models with simple and efficient sparsity

    Fedus, W., Zoph, B., and Shazeer, N. Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. J. Mach. Learn. Res., 23 0 (1), January 2022. ISSN 1532-4435

    2022

  10. [18]

    The P ile: An 800gb dataset of diverse text for language modeling

    Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S., and Leahy, C. The P ile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020

    2020 arXiv

  11. [19]

    A framework for few-shot language model evaluation, 12 2023

    Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac'h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., a...

    2023

  12. [20]

    Transformer feed-forward layers are key-value memories

    Geva, M., Schuster, R., Berant, J., and Levy, O. Transformer feed-forward layers are key-value memories. In Moens, M.-F., Huang, X., Specia, L., and Yih, S. W.-t. (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp.\ 5484--5495, O...

    2021 doi

  13. [21]

    and Bengio, Y

    Glorot, X. and Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. In Teh, Y. W. and Titterington, M. (eds.), Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine ...

    2010

  14. [22]

    F., Keller, P

    Gmitro, A. F., Keller, P. E., and Gindi, G. R. Statistical performance of outer-product associative memory models. Appl. Opt., 28 0 (10): 0 1940--1948, May 1989. doi:10.1364/AO.28.001940. URL https://opg.optica.org/ao/abstract.cfm?URI=ao-28-10-1940

    1940 doi

  15. [23]

    Improving language modeling using densely connected recurrent neural networks

    Godin, F., Dambre, J., and De Neve, W. Improving language modeling using densely connected recurrent neural networks. In Blunsom, P., Bordes, A., Cho, K., Cohen, S., Dyer, C., Grefenstette, E., Hermann, K. M., Rimell, L., Weston, J., and Yih, S. (eds.), Proceedings of the 2nd ...

    2017 doi

  16. [24]

    Openwebtext corpus

    Gokaslan, A., Cohen, V., Pavlick, E., and Tellex, S. Openwebtext corpus. http://Skylion007.github.io/OpenWebTextCorpus, 2019

    2019

  17. [25]

    Levanter , 2024

    Hall, D., Zhou, I., Adams, V., Wang, J., and Liang, P. Levanter , 2024. URL https://github.com/stanford-crfm/levanter

    2024

  18. [26]

    A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Vinyals, O., Rae, J

    Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Vinyals, O., Ra...

    2024

  19. [27]

    Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q. Densely connected convolutional networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 2261--2269, 2017. doi:10.1109/CVPR.2017.243

    2017 doi

  20. [28]

    S., Perez, F., Ba, J., and Volkovs, M

    Huang, X. S., Perez, F., Ba, J., and Volkovs, M. Improving transformer optimization through better initialization. In III, H. D. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, ...

    2020

  21. [29]

    Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., de las Casas, D., Hanna, E. B., Bressand, F., Lengyel, G., Bour, G., Lample, G., Lavaud, L. R., Saulnier, L., Lachaux, M.-A., Stock, P., Subramanian, S., Yang, S., Antoniak, S., Scao...

    2024 arXiv

  22. [30]

    B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  23. [31]

    A., Khyalia, S., Jung, J., Goka, H., and Lee, H

    Kedia, A., Zaidi, M. A., Khyalia, S., Jung, J., Goka, H., and Lee, H. Transformers get stable: an end-to-end signal propagation theory for language models. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2024

    2024

  24. [32]

    and Garcia, C

    Kidger, P. and Garcia, C. E quinox: neural networks in JAX via callable P y T rees and filtered transformations. Differentiable Programming workshop at Neural Information Processing Systems 2021, 2021

    2021

  25. [33]

    Correlation matrix memories

    Kohonen, T. Correlation matrix memories. IEEE Transactions on Computers, C-21 0 (4): 0 353--359, 1972. doi:10.1109/TC.1972.5008975

    1972

  26. [34]

    S., Viguier, S., and Ligozat, A.-L

    Luccioni, A. S., Viguier, S., and Ligozat, A.-L. Estimating the carbon footprint of bloom, a 176b parameter language model. Journal of Machine Learning Research, 24 0 (253): 0 1--15, 2023. URL http://jmlr.org/papers/v24/23-0069.html

    2023

  27. [35]

    Locating and editing factual associations in gpt

    Meng, K., Bau, D., Andonian, A., and Belinkov, Y. Locating and editing factual associations in gpt. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp.\ 17359--17372. Curran Associ...

    2022

  28. [36]

    Pointer sentinel mixture models, 2016

    Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models, 2016

    2016

  29. [37]

    Muennighoff, N., Rush, A., Barak, B., Le Scao, T., Tazi, N., Piktus, A., Pyysalo, S., Wolf, T., and Raffel, C. A. Scaling data-constrained language models. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Proce...

    2023

  30. [38]

    N., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernández, R

    Paperno, D., Kruszewski, G., Lazaridou, A., Pham, Q. N., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernández, R. The lambada dataset, Aug 2016

    2016

  31. [39]

    Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Biderman, S., Cao, H., Cheng, X., Chung, M., Derczynski, L., Du, X., Grella, M., Gv, K., He, X., Hou, H., Kazienko, P., Kocon, J., Kong, J., Koptyra, B., Lau, H., Lin, J., Mantri, K. S. I., Mom, F., Saito, A., Son...

    2023

  32. [40]

    Language models are unsupervised multitask learners

    Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. 2019

    2019

  33. [41]

    L., Bhagavatula, C., and Choi, Y

    Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. arXiv preprint arXiv:1907.10641, 2019

    1907 arXiv

  34. [42]

    Dense information flow for neural machine translation

    Shen, Y., Tan, X., He, D., Qin, T., and Liu, T.-Y. Dense information flow for neural machine translation. In Walker, M., Ji, H., and Stent, A. (eds.), Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Lang...

    2018 doi

  35. [43]

    Normformer: Improved transformer pretraining with extra normalization

    Shleifer, S., Weston, J., and Ott, M. Normformer: Improved transformer pretraining with extra normalization. CoRR, abs/2110.09456, 2021. URL https://arxiv.org/abs/2110.09456

    2021 arXiv

  36. [44]

    K., Greff, K., and Schmidhuber, J

    Srivastava, R. K., Greff, K., and Schmidhuber, J. Highway networks, 2015. URL https://arxiv.org/abs/1505.00387

    2015 arXiv

  37. [45]

    Energy and policy considerations for deep learning in NLP

    Strubell, E., Ganesh, A., and McCallum, A. Energy and policy considerations for deep learning in NLP . In Korhonen, A., Traum, D., and M \`a rquez, L. (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.\ 3645--3650, Florence, It...

    2019 doi

  38. [46]

    Retentive network: A successor to transformer for large language models, 2023

    Sun, Y., Dong, L., Huang, S., Ma, S., Xia, Y., Xue, J., Wang, J., and Wei, F. Retentive network: A successor to transformer for large language models, 2023. URL https://arxiv.org/abs/2307.08621

    2023 arXiv

  39. [47]

    N., Kaiser, L., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, pp.\ 6000–6010, Red Hook, NY, USA, 201...

    2017

  40. [48]

    Will we run out of data? limits of llm scaling based on human-generated data, 2024

    Villalobos, P., Ho, A., Sevilla, J., Besiroglu, T., Heim, L., and Hobbhahn, M. Will we run out of data? limits of llm scaling based on human-generated data, 2024. URL https://arxiv.org/abs/2211.04325

    2024 arXiv

  41. [49]

    DeepNet: Scaling Transformers to 1,000 Layers

    Wang, H., Ma, S., Dong, L., Huang, S., Zhang, D., and Wei, F. DeepNet: Scaling Transformers to 1,000 Layers . IEEE Transactions on Pattern Analysis & Machine Intelligence, 46 0 (10): 0 6761--6774, October 2024. ISSN 1939-3539. doi:10.1109/TPAMI.2024.3386927. URL https://doi.ie...

    2024

  42. [50]

    Sustainable ai: Environmental implications, challenges and opportunities

    Wu, C.-J., Raghavendra, R., Gupta, U., Acun, B., Ardalani, N., Maeng, K., Chang, G., Aga, F., Huang, J., Bai, C., Gschwind, M., Gupta, A., Ott, M., Melnikov, A., Candido, S., Brooks, D., Chauhan, G., Lee, B., Lee, H.-H., Akyildiz, B., Balandat, M., Spisak, J., Jain, R., Rabbat...

    2022

  43. [51]

    On layer normalization in the transformer architecture

    Xiong, R., Yang, Y., He, D., Zheng, K., Zheng, S., Xing, C., Zhang, H., Lan, Y., Wang, L., and Liu, T.-Y. On layer normalization in the transformer architecture. In Proceedings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org, 2020

    2020

  44. [52]

    Rewiring the transformer with depth-wise LSTM s

    Xu, H., Song, Y., Liu, Q., van Genabith, J., and Xiong, D. Rewiring the transformer with depth-wise LSTM s. In Calzolari, N., Kan, M.-Y., Hoste, V., Lenci, A., Sakti, S., and Xue, N. (eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, L...

    2024

  45. [53]

    Understanding and improving layer normalization

    Xu, J., Sun, X., Zhang, Z., Zhao, G., and Lin, J. Understanding and improving layer normalization. In Wallach, H., Larochelle, H., Beygelzimer, A., d Alch\' e -Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associ...

    2019

  46. [54]

    J., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W., and Gao, J

    Yang, G., Hu, E. J., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W., and Gao, J. Tensor programs v: tuning large neural networks via zero-shot hyperparameter transfer. In Proceedings of the 35th International Conference on Neural Information P...

    2024

  47. [55]

    Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019

    Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019

    2019

  48. [56]

    Ready-to-go transmission projects 2023

    Zimmerman, Z., Goggin, M., and Gramlich, R. Ready-to-go transmission projects 2023. Technical report, Grid Strategies, 2023

    2023

  49. [57]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.