Pith. sign in

REVIEW 5 major objections 8 minor 46 references

Fast and Simplex: 2-Simplicial Attention in Triton

T0 review · 5 major / 8 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Replacing dot-product attention with trilinear 2-simplicial attention is claimed to change the exponent of LLM scaling laws, improving token efficiency on reasoning and coding benchmarks.

desk verdict The kernel and the determinant-based trilinear form are real contributions, but the scaling-exponent claim does not survive a leave-one-out re-analysis and appears to confuse active with total parameters. read the letter →

arxiv 2507.02754 v1 pith:C6G7IXJW submitted 2025-07-03 cs.LG cs.AI

Aurko Roy , Timothy Chou , Sai Surya Duvvuri , Sijia Chen , Jiecao Yu , Xiaodong Wang , Manzil Zaheer , Rohan Anil This is my paper
classification cs.LGcs.AI
keywords 2-simplicialattentiontrilinearscalinglawstokenefficiencyTritonkernelmixture-of-expertsMatch3rotation-invariantforms
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that replacing dot-product attention with 2-simplicial attention—a trilinear generalization in which each query attends to pairs of keys—changes the exponent of the neural scaling law, not just the offset. On a fixed token budget, similarly sized models with 2-simplicial attention achieve lower negative log-likelihood than standard Transformers on GSM8k, MMLU, MMLU-pro, and MBPP, and the gap grows with active parameter count. The paper reports relative scaling-exponent increases of 18.5%, 8.5%, 20.2%, and 6.8% on those four benchmarks, and attributes the gain to token efficiency rather than raw compute. It also contributes a Triton kernel that reaches about 520 TFLOPS, comparable to the fastest published attention kernels, and a 7-dimensional determinant construction that provably solves a 3SUM-style matching problem with a single attention head.

What carries the argument

The central object is the 2-simplicial attention layer introduced by Clift et al. (2019): a trilinear logit tensor $A_{ijk} = \langle q_i, k_j, k'_k \rangle / \sqrt{d}$, a softmax over the pair $(j,k)$, and an output formed by summing the attention weights times Hadamard products $v_j \circ v'_k$. To make this practical, the paper adds a sliding-window parametrization with complexity $O(n w_1 w_2)$, a rotation-invariant trilinear form based on sums of $3\times 3$ determinants (Sarrus rule) that generalizes RoPE, and a Triton kernel that tiles the trilinear einsum as an elementwise product followed by tensor-core matrix multiplications, reaching roughly 520 TFLOPS. The theoretical construction uses a 7-dimensional embedding in which the determinant logits equal $c \cdot \cos(2\pi(x_i + x_{j_1} + x_{j_2})/M)$, with a seventh 'blank pair' dimension of value $c$, so a single head detects whether any pair sums to a target modulo $M$.

What would settle it

Train both architectures at 1B, 2B, 3.5B, and 5B active parameters with at least three seeds each, and fit the exponent from pretraining loss at identical token counts with confidence intervals; if the 2-simplicial exponent is not steeper on the four benchmarks, or the intervals overlap, the central scaling claim is refuted.

Watch

Extended reading notes

Core claim

The paper establishes that the 2-simplicial Transformer, whose attention logits are the trilinear form $A_{ijk}^{(2s)} = \frac{1}{\sqrt{d}} \sum_l Q_{il} K_{jl} K'_{kl}$ with softmax over the pair $(j,k)$ and outputs weighted Hadamard products $v_j \circ v'_k$, scales more favorably than dot-product attention under a fixed token budget. Across mixture-of-experts models with 1B, 2B, and 3.5B active parameters, the 2-simplicial model's evaluation negative log-likelihood decreases faster with active parameter count on math, knowledge, and coding benchmarks, giving the exponent increases listed in Table 3. The authors interpret this as a genuine change in the exponent $\alpha$ in $L(N) = E' + A/N^\alpha$, in contrast to most architectural modifications which only shift the offset. Supporting this, the paper proves that a single 7-dimensional 2-simplicial attention head can implement the Match3 (3SUM) function, a task that dot-product attention cannot solve with polynomially many layers.

Load-bearing premise

The fitted exponent advantage is computed from three single-run negative-log-likelihood values per model (1B, 2B, and 3.5B active parameters) with no error bars, using evaluation-set NLL rather than pretraining loss, so the steeper 2-simplicial slope could disappear if the 3.5B point carries training noise or a slightly different token budget.

Editorial extensions

If this is right

  • Under token constraints, 2-simplicial models can scale parameters faster than the Chinchilla-optimal tokens-per-parameter ratio, because the parameter exponent $\alpha$ is larger.
  • The exponent gains are largest on the least saturated benchmarks (MMLU-pro and GSM8k), so reasoning-heavy tasks are where the architecture's advantage should show up most.
  • The kernel throughput makes interleaved 2-simplicial attention computationally comparable to standard attention at 48k context length, removing the main practical obstacle.
  • The single-head 7-dimensional Match3 construction demonstrates that 2-simplicial attention can express compositional matching in one layer, which dot-product attention cannot do in polynomially many layers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The exponent comparison is computed from only three single-run benchmark points per architecture; a multi-seed, multi-size fit would determine whether the reported steepening is intrinsic or partly a training-run artifact.
  • If the exponent change is real, it could combine with data-pruning results that also steepen scaling exponents, but this paper does not test such a combination.
  • The rotation-invariant determinant construction suggests a natural path to higher-order (3-simplicial or beyond) attention, though the paper does not explore whether the gains continue or saturate.
  • Because the implemented kernel is a prototype in a higher-level language, production throughput could differ from the reported 520 TFLOPS; a hand-tuned kernel might shift the practical cost-benefit.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 8 minor

Summary. The paper proposes a Triton-based implementation of 2-simplicial attention, a trilinear generalization of dot-product attention, and claims that this architecture improves token efficiency and changes the exponent in scaling laws for knowledge and reasoning tasks. The authors introduce a rotation-invariant determinant-based trilinear form, prove that a 7-dimensional construction of this form can solve a 3SUM-style matching problem, and report kernel throughput of about 520 TFLOPS. The empirical section trains MoE models at 1B, 2B, and 3.5B active parameters and compares negative log-likelihood on GSM8k, MMLU, MMLU-pro, and MBPP, estimating scaling exponents from these three points.

Significance. If the central claim were correct, it would be a notable result: most architectural changes only shift the loss offset rather than the scaling exponent, and a higher exponent would imply better scaling under token constraints. The paper also offers a useful engineering contribution in the Triton kernel and a clear theoretical expressivity theorem for determinant-based trilinear attention. However, the empirical evidence for the exponent change is not robust: it rests on three single-run points, the leave-one-out analysis reverses the claimed ordering on two benchmarks, and the reported exponents do not match the stated use of active parameter counts. These issues undermine the paper's headline conclusion.

major comments (5)
  1. [Section 8, Tables 2 and 3] The exponent comparison is not robust. Re-fitting -log NLL against log N from the three points in Table 2 and dropping the 1B point reverses the ordering on MMLU (Transformer about 0.120 vs 2-simplicial about 0.118) and on MBPP (Transformer about 0.178 vs 2-simplicial about 0.170). The 1B point is the only size at which the 2-simplicial model is worse on these benchmarks, so the reported alpha increase is driven by that single run. With three single-run points and no error bars, the high R^2 values in Table 4 carry no information because any three points fit a line with high R^2. The headline claim requires many more model sizes and repeated runs with uncertainty estimates.
  2. [Section 8, Table 3] The text states 'we use for N the active parameters in each model,' but the reported alpha values are not reproduced by a least-squares fit of -log NLL against log(active parameters) using Table 2. The reported values are instead close to a fit against total parameters (57B, 100B, 176B); for example, on MBPP the Transformer total-parameter slope is about 0.176 versus the reported 0.172, while the active-parameter slope is about 0.158. Since these are MoE models with a large total-to-active ratio, the choice is material to the scientific claim, and the exponents as reported are not reproducible from the table.
  3. [Section 8, Eqs. (19)-(20)] The derivation of the linear relation hides the irreducible-loss term E' inside an N-dependent quantity. Equation (19), log L(N) approximately equals log E'' + log A - alpha log N, is only a valid power law if E' is negligible over the fitted range. The paper asserts E' is small but gives no supporting evidence. The observed pattern in Table 2 (2-simplicial slightly worse or equal at 1B and improving at larger sizes) is exactly what a difference in the constant A or the offset E' would produce even with identical exponents. Thus the fitted slopes cannot distinguish an exponent change from an offset change.
  4. [Section 8] The scaling-law analysis uses negative log-likelihood on four evaluation benchmarks rather than the pretraining loss for which Eq. (1) is posited. No argument is given that these benchmark NLLs obey the same power law with the same exponent, and the benchmarks differ in formatting, shot count, and label space. The claim that the exponent is an intrinsic property of the architecture is therefore not established by the present experiments.
  5. [Section 8] The experiments are not described sufficiently to reproduce the scaling comparison: the total token budget D is never stated, nor are the number of layers, hidden dimension, number of experts, per-layer window configuration, or the data mixture. Since Eq. (17) includes D and the comparison relies on D being fixed across models, the reader cannot verify that the token budgets are actually matched across the two architectures and across the three model sizes.
minor comments (8)
  1. [Section 4, Eq. (4)] Equation (4) uses the pre-softmax logits A_ij in the output summation; it should use the softmax weights S_ij as in Eq. (7).
  2. [Section 7] The section header contains a typo: 'optimizatins' should be 'optimizations.'
  3. [Section 2] The text cites 'Allen (2025)' but no such entry appears in the references.
  4. [Section 5, after Eq. (9)] The sentence beginning 'For any vector q, let q(l) = q = q[3(l-1):3l]' is garbled and should be rewritten with proper superscript notation.
  5. [Table 1] Entries such as '104 .1 ms' suggest a formatting error; the table would benefit from alignment and units.
  6. [Appendix A, proof of Theorem 5.1] The proof does not quantify how large the scaling constant c must be relative to M; a choice such as c much larger than M^2 is needed to make non-matching triples contribute negligibly after softmax.
  7. [Section 7 and Figure 3] The 520 TFLOPS claim and Figure 3 lack hardware, precision, sequence length, and head-dimension specifications, making the performance comparison to FAv3 difficult to evaluate.
  8. [Appendix B, Algorithm 2] The pseudocode contains placeholders ('...') and is not self-contained; the masking conditions and tiling are clearer in the Triton code but should be described in the pseudocode as well.

Circularity Check

1 steps flagged · score 6.0 of 10

The headline exponent change is a log-log fit of the same three NLL points being compared, so it reduces by construction to a fitted restatement; direct NLL gains and kernel results are independent.

  1. fitted input called prediction [Section 8 ('Experiments & Results'), Eqs. (18)-(20) and Tables 2-3]
    "Since we train both the models on the same fixed number of tokens, we may ignore the third term and simply write the loss as: L(N ) = E′ + A/N^α ... Therefore we can estimate α, β for both sets of models from the losses in Table 2 where we use for N the active parameters in each model. We estimate the slope α and the intercept β for both the Transformer as well as the 2-simplicial Transformer in Table 3. We see that 2-simplicial attention has a steeper slope α, i.e. a higher exponent in its scaling law compared to dot product attention Transformer."

    The 'exponent' in Table 3 is not an independent measurement or an out-of-sample prediction; it is the slope of -log(NLL) vs. log(N) computed from the very same three benchmark NLLs in Table 2 that define the comparison. With only three points and one residual degree of freedom, the steeper 2-simplicial slope is algebraically forced by the observed widening of the NLL gap between 1B and 3.5B active parameters. Equation (20) further hides the N-dependent log(1 + (A/N^α)/E') term inside E'', so the fitted slope is not necessarily the α of Eq. (18). The high R² values in Table 4 are tautological for a three-point line fit.

full rationale

Most of the paper is self-contained: the 7-dimensional determinant construction is proved in Appendix A, the Triton kernel is benchmarked against FAv3, and Table 2 gives direct per-size NLL comparisons that do not depend on any regression. The abstract's headline claim, however, is that 2-simplicial attention changes the scaling-law exponent, and that claim is produced by Eq. (20), a log-log regression on the same three NLL values that are being compared. The fitted α difference is therefore a re-description of the widening gap in Table 2 rather than a prediction of scaling behavior at unseen sizes or token budgets. The fragility noted by the skeptic follows from the same structure: removing the 1B point leaves only one residual degree of freedom, so the ordering of slopes is highly sensitive to a single run, and the high R² values are uninformative. There are no load-bearing self-citations at issue; Clift et al. supplies the architecture, and the Match3 construction is proven here. Because the direct token-efficiency improvements and the kernel results are independent of the fit, the appropriate finding is partial circularity rather than full collapse: the exponent result reduces by construction to its own fitted inputs, while other contributions stand on their own.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the assumed power-law form (Eq. 1 and Eq. 18), the approximation in Eq. 19-20 that absorbs the irreducible loss, and the treatment of benchmark NLL as a scaling-law observable. No new physical or mathematical entities are introduced; the 'blank pair' is a proof device in Appendix A. The fitted exponents alpha are the paper's measured quantities, but they are fit to only three model sizes.

free parameters (3)
  • scaling-law exponent alpha (Transformer) = 0.1420 (GSM8k), 0.1256 (MMLU), 0.0901 (MMLU-pro), 0.1720 (MBPP)
    Fitted via Eq. 20 to the three Transformer NLL points in Table 2. The paper's conclusion depends on these slopes being accurate and stable.
  • scaling-law exponent alpha (2-simplicial) = 0.1683 (GSM8k), 0.1364 (MMLU), 0.1083 (MMLU-pro), 0.1837 (MBPP)
    Same fit for the 2-simplicial runs; the claimed increases over the Transformer alphas are 18.5%, 8.5%, 20.2%, and 6.8% (Table 3).
  • scaling-law intercept beta (both models) = Transformer: -1.8280 to -2.2569; 2-simplicial: -2.1181 to -2.5201 (Table 3)
    Absorbs the constants E'' and A in Eq. 20; not the focus of the claim but part of the same regression.
assumptions (4)
  • domain assumption The loss as a function of parameters and tokens follows L(N,D) = E + A/N^alpha + B/D^beta (Eq. 1), and at a fixed token budget the token term is dropped to get Eq. 18.
    This functional form is adopted from Hoffmann et al. (2022) without testing it on the actual training runs; the exponents are then estimated from this assumed form.
  • ad hoc to paper For the model sizes used (1B-3.5B active parameters), log L(N) can be approximated by a pure power law with the irreducible loss E' absorbed into a new constant E'' (Eq. 19-20).
    The paper writes log L(N) approx= log E'' + log A - alpha log N, hiding the additive E' term; the accuracy of this approximation at these sizes is not verified.
  • ad hoc to paper Negative log-likelihood on the four evaluation benchmarks (GSM8k, MMLU, MMLU-pro, MBPP) follows the same power law as pretraining validation loss.
    Eq. 18-20 are fit to benchmark NLLs, not training-set loss; using downstream task NLL as a scaling-law observable is an assumption unique to this paper's protocol.
  • domain assumption The Transformer and 2-simplicial models differ only in attention type, with matched size, data, and token budget.
    The paper reports active and total parameters but omits tokens, context length, data mix, and model width/depth, so the attribution of benchmark differences to the attention mechanism is unverified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Fast and Simplex: 2-Simplicial Attention in Triton." pith.science (2026). https://pith.science/paper/C6G7IXJW

@misc{pith2026250702754,
  author       = {Pith},
  title        = {Pith review of: Fast and Simplex: 2-Simplicial Attention in Triton},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C6G7IXJW}},
  note         = {Machine review of arXiv:2507.02754}
}
abstract

Recent work has shown that training loss scales as a power law with both model size and the number of tokens, and that achieving compute-optimal models requires scaling model size and token count together. However, these scaling laws assume an infinite supply of data and apply primarily in compute-bound settings. As modern large language models increasingly rely on massive internet-scale datasets, the assumption that they are compute-bound is becoming less valid. This shift highlights the need for architectures that prioritize token efficiency. In this work, we investigate the use of the 2-simplicial Transformer, an architecture that generalizes standard dot-product attention to trilinear functions through an efficient Triton kernel implementation. We demonstrate that the 2-simplicial Transformer achieves better token efficiency than standard Transformers: for a fixed token budget, similarly sized models outperform their dot-product counterparts on tasks involving mathematics, coding, reasoning, and logic. We quantify these gains by demonstrating that $2$-simplicial attention changes the exponent in the scaling laws for knowledge and reasoning tasks compared to dot product attention.

Figures

Figures reproduced from arXiv: 2507.02754 by the authors.

Figure 1
Figure 1. Geometry of dot product attention and 2-simplical attention. The final output of the attention layer is then a linear combination of the values according to these attention scores: v˜i = Xn j=1 Aijvj (4) The 2-simplicial Transformer paper Clift et al. (2019) generalizes this to trilinear products where we have two additional key and value projection matrices WK′ and WV ′ , which give us K′ = XWK′ and V ′ = XWV ′ . T… view at source ↗
Figure 2
Figure 2. Left: Visualization of sliding window 2-simplical attention. Each Qi attends to a [w1, w2] shaped rectangle of K, K′ . Right: Tiling to reduce 2-simplicial einsum QKK′ to elementwise mul QK′ on CUDA core and tiled matmul (QK′ )@K on tensor core [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. FLOPs and Latencies of FAv3 vs 2-simplical attention [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

46 extracted references · 11 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints

    Joshua Ainslie, James Lee-Thorp, Michiel De Jong, Yury Zemlyanskiy, Federico Lebr \'o n, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245, 2023

    arXiv 2023

  3. [3]

    Program synthesis with large language models

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021

    arXiv 2021

  4. [4]

    Explaining neural scaling laws

    Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. Proceedings of the National Academy of Sciences, 121 0 (27): 0 e2311878121, 2024

    work page 2024

  5. [5]

    Systematic generalization with edge transformers

    Leon Bergen, Timothy O'Donnell, and Dzmitry Bahdanau. Systematic generalization with edge transformers. Advances in Neural Information Processing Systems, 34: 0 1390--1402, 2021

    work page 2021

  6. [6]

    Loss-to-loss prediction: Scaling laws for all datasets

    David Brandfonbrener, Nikhil Anand, Nikhil Vyas, Eran Malach, and Sham Kakade. Loss-to-loss prediction: Scaling laws for all datasets. arXiv preprint arXiv:2411.12925, 2024

    arXiv 2024

  7. [7]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

  8. [8]

    Logic and the 2 -simplicial transformer

    James Clift, Dmitry Doryn, Daniel Murfet, and James Wallbridge. Logic and the 2 -simplicial transformer. arXiv preprint arXiv:1909.00668, 2019

    arXiv 1909

Show all 46 references
  1. [9]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  2. [10]

    Flashattention: Fast and memory-efficient exact attention with io-awareness

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher R \'e . Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in neural information processing systems, 35: 0 16344--16359, 2022

    2022

  3. [11]

    Universal transformers

    Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and ukasz Kaiser. Universal transformers. arXiv preprint arXiv:1807.03819, 2018

    2018 arXiv

  4. [12]

    Observation on scaling laws, May 2025

    Katie Everett. Observation on scaling laws, May 2025. URL https://x.com/_katieeverett/status/1925665335727808651. [Tweet]

    2025

  5. [13]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023

    2023 arXiv

  6. [14]

    Array programming with numpy

    Charles R Harris, K Jarrod Millman, St \'e fan J Van Der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J Smith, et al. Array programming with numpy. Nature, 585 0 (7825): 0 357--362, 2020

    2020

  7. [15]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  8. [16]

    Deep learning scaling is predictable, empirically

    Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409, 2017

    2017 arXiv

  9. [17]

    Training compute-optimal large language models

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022

    2022 arXiv

  10. [18]

    Gpipe: Efficient training of giant neural networks using pipeline parallelism

    Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32, 2019

    2019

  11. [19]

    Hierarchical mixtures of experts and the em algorithm

    Michael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6 0 (2): 0 181--214, 1994

    1994

  12. [20]

    Highly accurate protein structure prediction with alphafold

    John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Z \' dek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. nature, 596 0 (7873): 0 583--589, 2021

    2021

  13. [21]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  14. [22]

    Transformers are rnns: fast autoregressive transformers with linear attention

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Francois Fleuret. Transformers are rnns: fast autoregressive transformers with linear attention. In Proceedings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org, 2020

    2020

  15. [23]

    Strassen attention: Unlocking compositional abilities in transformers based on a new lower bound method

    Alexander Kozachinskiy, Felipe Urrutia, Hector Jimenez, Tomasz Steifer, Germ \'a n Pizarro, Mat \' as Fuentes, Francisco Meza, Cristian B Calderon, and Crist \'o bal Rojas. Strassen attention: Unlocking compositional abilities in transformers based on a new lower bound method....

    2025

  16. [24]

    Fixing weight decay regularization in adam

    Ilya Loshchilov, Frank Hutter, et al. Fixing weight decay regularization in adam. arXiv preprint arXiv:1711.05101, 5: 0 5, 2017

    2017 arXiv

  17. [25]

    Devanur, Gregory R

    Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia. Pipedream: generalized pipeline parallelism for dnn training. In Proceedings of the 27th ACM Symposium on Operating Systems Principles...

    2019

  18. [26]

    Image transformer

    Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International conference on machine learning, pp.\ 4055--4064. PMLR, 2018

    2018

  19. [27]

    Efficient content-based sparse attention with routing transformers

    Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-based sparse attention with routing transformers. Transactions of the Association for Computational Linguistics, 9: 0 53--68, 2021

    2021

  20. [28]

    N-grammer: Augmenting transformers with latent n-grams

    Aurko Roy, Rohan Anil, Guangda Lai, Benjamin Lee, Jeffrey Zhao, Shuyuan Zhang, Shibo Wang, Ye Zhang, Shen Wu, Rigel Swavely, et al. N-grammer: Augmenting transformers with latent n-grams. arXiv preprint arXiv:2207.06366, 2022

    2022 arXiv

  21. [29]

    Representational strengths and limitations of transformers

    Clayton Sanford, Daniel J Hsu, and Matus Telgarsky. Representational strengths and limitations of transformers. Advances in Neural Information Processing Systems, 36: 0 36677--36707, 2023

    2023

  22. [30]

    Reasoning with latent thoughts: On the power of looped transformers

    Nikunj Saunshi, Nishanth Dikkala, Zhiyuan Li, Sanjiv Kumar, and Sashank J Reddi. Reasoning with latent thoughts: On the power of looped transformers. arXiv preprint arXiv:2502.17416, 2025

    2025 arXiv

  23. [31]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017

    2017 arXiv

  24. [32]

    Scaling laws for linear complexity language models

    Xuyang Shen, Dong Li, Ruitao Leng, Zhen Qin, Weigao Sun, and Yiran Zhong. Scaling laws for linear complexity language models. arXiv preprint arXiv:2406.16690, 2024

    2024 arXiv

  25. [33]

    Searching for efficient transformers for language modeling

    David So, Wojciech Ma \'n ke, Hanxiao Liu, Zihang Dai, Noam Shazeer, and Quoc V Le. Searching for efficient transformers for language modeling. Advances in neural information processing systems, 34: 0 6010--6022, 2021

    2021

  26. [34]

    Beyond neural scaling laws: beating power law scaling via data pruning

    Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 35: 0 19523--19536, 2022

    2022

  27. [35]

    Introduction to linear algebra

    Gilbert Strang. Introduction to linear algebra. SIAM, 2022

    2022

  28. [36]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 0 127063, 2024

    2024

  29. [37]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  30. [38]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  31. [39]

    On the uniform convergence of relative frequencies of events to their probabilities

    Vladimir Vapnik. On the uniform convergence of relative frequencies of events to their probabilities. In Doklady Akademii Nauk USSR, volume 181, pp.\ 781--787, 1968

    1968

  32. [40]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  33. [41]

    Dcn v2: Improved deep & cross network and practical lessons for web-scale learning to rank systems

    Ruoxi Wang, Rakesh Shivanna, Derek Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed Chi. Dcn v2: Improved deep & cross network and practical lessons for web-scale learning to rank systems. In Proceedings of the web conference 2021, pp.\ 1785--1797, 2021

    2021

  34. [42]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In The Thirty-eight Conference on Neural Information Pr...

    2024

  35. [43]

    Looped transformers are better at learning learning algorithms

    Liu Yang, Kangwook Lee, Robert Nowak, and Dimitris Papailiopoulos. Looped transformers are better at learning learning algorithms. arXiv preprint arXiv:2311.12424, 2023

    2023 arXiv

  36. [44]

    Native sparse attention: Hardware-aligned and natively trainable sparse attention

    Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, YX Wei, Lean Wang, Zhiping Xiao, et al. Native sparse attention: Hardware-aligned and natively trainable sparse attention. arXiv preprint arXiv:2502.11089, 2025

    2025 arXiv

  37. [45]

    Big bird: Transformers for longer sequences

    Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33: 0 17283--17297, 2020

    2020

  38. [46]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.