Pith. sign in

REVIEW 1 major objections 6 minor 3 cited by

Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routing

T0 review · 1 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MoSA, a content-based sparse attention built on expert-choice routing, is the only sparse attention variant tested that beats dense transformers at equal compute, improving perplexity by up to 27 percent.

desk verdict The iso-FLOP perplexity result is worth taking seriously, but the KV-cache and inference claims are not supported by the experiments as reported. read the letter →

arxiv 2505.00315 v1 pith:AA43VHFB submitted 2025-05-01 cs.LG cs.CL

Piotr Piękos , Róbert Csordás , Jürgen Schmidhuber This is my paper · ORCID
classification cs.LGcs.CL
keywords sparseattentionmixtureofexpertsexpert-choiceroutinglanguagemodelingefficienttransformersKVcachecontent-basedsparsityperplexity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a learnable, content-based sparse attention can beat full dense attention in language modeling, provided each attention head is allowed to choose its own small set of tokens to attend to. The proposal, Mixture of Sparse Attention (MoSA), treats each head as an expert that picks its top-$k$ tokens from the sequence, cutting each head's cost from $O(T^2)$ to $O(k^2 + T)$ and freeing compute for many more, more specialized heads. In FLOP-matched experiments at four scales (28M to 516M parameters), MoSA improves perplexity — a language-modeling error measure, lower is better — by up to 27% over the dense baseline, and it is the only sparse method among those tested to do so; fixed sparse attention and the Routing Transformer both fall short. At matched perplexity, MoSA simultaneously improves wall-clock time and training memory, and cuts the number of cached key-value pairs by more than half. The practical upshot is a route to transformers that are both better and cheaper to run.

What carries the argument

The carrying mechanism is expert-choice routing transplanted into attention: each head acts as an expert that selects its own top-$k$ tokens through a learned per-token score $r = \sigma(XW_r)$, which gives perfect load balance by construction and makes the sparsity pattern content-dependent. Queries, keys, values, and the attention matrix are computed only on the selected tokens, with the causal mask and rotary positions indexed by each token's original position in the sequence, and the outputs are scaled by their router scores before being scattered back, keeping the whole selection trainable end-to-end by the language-modeling objective. The per-head cost drops to $O(k^2 + T)$, and the paper's FLOP accounting shows that the routing overhead ($2hT + h'k$) is small relative to the projection and attention savings, so the freed budget can be spent on many more heads, up to hundreds of sparse heads in the hybrid models. A second load-bearing design choice is hybridization: keeping four dense heads alongside the sparse heads proves necessary, since pure-MoSA models without dense heads fail to beat the dense baseline in these experiments.

What would settle it

Train the same FLOP-matched hybrid MoSA model with an autoregressive router that may only select already-produced positions during both training and decoding, for instance by post-training a classifier to predict the non-autoregressive top-$k$ selections as the paper's discussion of Mixture-of-Depths suggests, and re-measure perplexity, wall-clock time, and decoding-time KV-cache size. The central claim stands only if a substantial share of the reported gains survives that change; if the gains evaporate, the results depend on training-time access to future tokens.

Watch

Extended reading notes

Core claim

The central claim is that dynamic, learned content-based sparsity makes sparse attention a strict improvement over dense attention rather than a compromise. MoSA implements this by giving every attention head a router that scores each token with a sigmoid and keeps the top-$k$ tokens, always including the first token; the head then computes queries, keys, values, and attention only on those $k$ tokens, re-inserts each output at its original position using the original positions for the causal mask and rotary embeddings, and scales it by the router score. Because each head now costs $O(k^2 + T)$ instead of $O(T^2)$, the saved compute is reinvested as many additional heads, and the paper finds that keeping four dense heads alongside the sparse heads performs best: in a fixed-FLOP comparison, perplexity falls by up to 27% (Tiny: 22.46 to 16.39) and improves at every model scale from 28M to 516M parameters, while perplexity-matched MoSA models run faster, use less memory, and cut the key-value cache by 51–70%. The same mechanism keeps its advantage on sequences up to 8192 tokens when combined with local attention, and a parameter-matched MoSA model (442M parameters, sparsity 8) beats a larger dense baseline (516M parameters) on perplexity, 12.16 versus 12.20.

Load-bearing premise

The load-bearing premise is that the router inspects the entire input sequence, including future tokens, when it picks each head's top-$k$ tokens; at generation time those future tokens do not exist, so the paper has not shown that the same selection, or the reported speed, memory, and KV-cache benefits, can be reproduced during one-token-at-a-time inference without an extra adaptation.

Editorial extensions

If this is right

  • Within a fixed FLOP budget, dense heads can be traded for many sparse MoSA heads: perplexity keeps improving up to sparsity $\rho \approx 32$–$64$, turning head count and specialization into a free knob.
  • At matched perplexity, MoSA cuts the total number of key-value pairs by 51–70% relative to dense attention, which directly shrinks the KV cache that dominates inference memory.
  • MoSA composes with optimized attention kernels such as Flash Attention and with local attention for long sequences, where it beats fixed sparse attention and the Routing Transformer at sequence lengths up to 8192 while using fewer FLOPs.
  • The gains do not rest on FLOP accounting alone: a 442M-parameter MoSA model at sparsity 8 reaches 12.16 perplexity, below the 12.20 of the 516M-parameter dense baseline.
  • Ablations in the paper show sparse heads need a few dense heads to train stably, which fixes the architecture choice for anyone adopting the method.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the router sees the full sequence at training time, the reported KV-cache reductions describe training-time key-value materialization, not the cache maintained during greedy generation; how much the decoding-time cache shrinks depends on how faithfully an autoregressive adaptation reproduces the non-causal top-$k$ choices.
  • The comparison isolates the causal factor: fixed sparse attention enjoys the same projection savings as MoSA yet still loses to dense attention, so the design implies that content-based selection itself, not merely the smaller number of computed tokens, is what buys the perplexity gain.
  • A direct test of the mechanism is to restrict the router to causal information during training; if most of the perplexity gain persists, MoSA-style head specialization transfers cleanly to generative decoding, and if it disappears, the method's value is mostly in non-autoregressive settings.
  • The head-specialization principle should transfer to other modalities: in vision transformers or bidirectional encoders, where the router's non-causal selection is unproblematic, MoSA-style heads could deliver the same FLOP-matched gains without needing any autoregressive adaptation.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 6 minor

Summary. The paper introduces Mixture of Sparse Attention (MoSA), a sparse attention mechanism in which each attention head selects its own top-k tokens via expert-choice routing, reducing per-head cost from O(T^2) to O(k^2+T). The saved compute is used to add more heads, and the main models are hybrids that keep four dense heads alongside many sparse heads. Experiments on C4 language modeling at four scales (28M to 516M dense baselines) report iso-FLOP perplexity improvements over the dense baseline of up to 27%, claim that MoSA outperforms fixed sparse attention and Routing Transformer, report perplexity-matched wall-clock, memory, and KV-pair reductions, and include long-sequence and zero-shot downstream evaluations. The central empirical claim is that MoSA is the only tested sparse attention variant that beats the dense baseline under a fixed FLOP budget.

Significance. If the iso-FLOP result is robust, it is a valuable finding: it suggests that learned content-based sparsity with perfect load balancing can improve language modeling at fixed compute, contrary to the static-sparsity baselines tested. The paper's strengths include full per-sparsity perplexity curves across scales, explicit FLOP derivations, an ablation of hybrid versus pure MoSA, and a public code repository. However, the headline resource-optimization claims are not established: the top-k selection is non-autoregressive, so the reported KV-cache reduction does not translate directly to autoregressive inference, and the empirical protocol selects the best sparsity on test perplexity from single runs without seeds. The iso-FLOP central claim is plausible; the resource claims need substantial reframing or additional evidence.

major comments (1)
  1. [Section 3.3, Table 2, Abstract, and Conclusions] The resource measurements are training-step numbers: 'Wall-time/step' is measured per training step and 'Memory (GB)' is training memory, not generation latency or inference-time memory. The Abstract and Conclusions nonetheless imply inference benefits ('faster in wall-clock time', 'reduce the resource usage', 'potentially enabling more economical scaling'). Please separate training-time efficiency from inference-time efficiency and state clearly which claim is supported by which measurement.
minor comments (6)
  1. [Section 3.2, 'Appendix 5'] The reference 'the Appendix 5' is unclear; it should point to Table 5 or a named appendix section.
  2. [Section 3.4, Figure 4] The x-axis label '8196' should be '8192'.
  3. [Section 3.5 / Introduction] The Introduction says 'In Section 3 we show the performance of different models in downstream zero-shot tasks', but this content is in Section 3.5; the cross-reference should be corrected.
  4. [Section 3, Implementation details] The sentence 'we train on the 105SB≈ 6.5B tokens from the dataset' is garbled; it should read '100k batches × batch size 64 × sequence length 1024 ≈ 6.5B tokens'.
  5. [Related Work] There is a typo: 'pretaining phase' should be 'pretraining phase'.
  6. [Table 1 / Appendix B] The main claim refers to hybrid MoSA models with four dense heads; pure MoSA without dense heads underperforms the dense baseline (Figure 5). The paper does disclose this, but contribution statements such as 'MoSA improves perplexity' should consistently say 'hybrid MoSA with four dense heads' to avoid overgeneralization.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the iso-FLOP perplexity comparison is an empirical benchmark against an external dense baseline, and the autoregressive KV-cache limitation is disclosed as a scope issue rather than a circular reduction.

full rationale

The central IsoFLOP claim (Table 1, Fig. 3) is an empirical measurement: fixed-FLOP MoSA hybrids are compared with dense, Fixed, and Routing baselines under an explicit FLOP-counting scheme (Sec. 3.2, App. A). The FLOP formulas are layer-level arithmetic derived from the definition of MoSA (Sec. 2.2) and do not assume the result they are used to demonstrate. Selecting the best sparsity from the test curves is a model-selection protocol applied equally to all methods; it is not a fitted parameter being relabeled as a prediction. The choice of four dense heads is justified by an ablation in App. B, not by the main result. The sigmoid router activation is adopted by citation to σ-MoE [38], and SwitchHead [31] is cited in related work; these self-citations are not load-bearing. The paper's Limitations section explicitly states that MoSA 'requires adaptations to be directly applicable to the autoregressive scenario,' which undercuts the unqualified Sec. 3.3 statement that KV = TH_dense + kH_mosa 'directly corresponds to the size of the costly KV-Cache in the autoregressive setting.' That is a substantive correctness/scope concern, but it is not a circular derivation: the KV formula counts materialized key-value pairs by construction, and whether the same set is available during autoregressive decoding is an empirical bridge the paper acknowledges it has not built. No equation-level reduction, fitted-input-as-prediction, or load-bearing self-citation chain is present.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The main empirical claim rests on two tuned hyperparameters (the sparsity grid and the 4-dense-head hybrid choice), the first-token inclusion rule, and the FLOP-accounting scheme. No hidden free parameters appear in the architecture beyond standard transformer hyperparameters. No new physical or theoretical entities are introduced.

free parameters (2)
  • Sparsity ratio rho = T/k (selected tokens per head) = Best values on test curves: Tiny=64, Small=64, Medium=32, Large=4
    The headline result is the best test perplexity selected across a grid of sparsity values in Section 3.2; rho is a tuned hyperparameter, not derived from theory.
  • Number of dense heads in hybrid models = 4
    Appendix B selects 4 dense heads from an ablation on the Small model and fixes it for all scales; this choice is based on the same evaluation metric used for the main claim.
assumptions (4)
  • domain assumption The first token is always included in every MoSA head.
    Section 3.2 states that the first token is always included in all MoSA heads, importing the attention-sink result from StreamingLLM; the method's gains depend on this external empirical finding.
  • domain assumption The FLOP formulas in Section 3.2 and Appendix A accurately capture compute equality between dense and sparse heads.
    All iso-FLOP comparisons rely on these counts; top-k selection cost, memory movement, and kernel efficiency are not fully captured, though the authors include a routing overhead term.
  • domain assumption Perplexity on C4 is a sufficient proxy for the value of a sparse-attention architecture at the tested scale.
    The central comparisons use C4 language modeling; Section 3.5 shows downstream accuracy does not consistently improve, so the proxy is not universally valid.
  • domain assumption A hybrid with dense heads is required for the main result; pure MoSA fails to beat dense.
    Appendix B reports that pure MoSA underperforms dense baselines, so the claimed gains apply only to the hybrid configuration, not to MoSA alone.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routing." pith.science (2026). https://pith.science/paper/AA43VHFB

@misc{pith2026250500315,
  author       = {Pith},
  title        = {Pith review of: Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AA43VHFB}},
  note         = {Machine review of arXiv:2505.00315}
}
abstract

Recent advances in large language models highlighted the excessive quadratic cost of self-attention. Despite the significant research efforts, subquadratic attention methods still suffer from inferior performance in practice. We hypothesize that dynamic, learned content-based sparsity can lead to more efficient attention mechanisms. We present Mixture of Sparse Attention (MoSA), a novel approach inspired by Mixture of Experts (MoE) with expert choice routing. MoSA dynamically selects tokens for each attention head, allowing arbitrary sparse attention patterns. By selecting $k$ tokens from a sequence of length $T$, MoSA reduces the computational complexity of each attention head from $O(T^2)$ to $O(k^2 + T)$. This enables using more heads within the same computational budget, allowing higher specialization. We show that among the tested sparse attention variants, MoSA is the only one that can outperform the dense baseline, sometimes with up to 27% better perplexity for an identical compute budget. MoSA can also reduce the resource usage compared to dense self-attention. Despite using torch implementation without an optimized kernel, perplexity-matched MoSA models are simultaneously faster in wall-clock time, require less memory for training, and drastically reduce the size of the KV-cache compared to the dense transformer baselines.

Figures

Figures reproduced from arXiv: 2505.00315 by the authors.

Figure 1
Figure 1. MoSA layer compared to the dense attention layer. MoSA replaces each dense head with [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Attention variants visualized. In the plot, the colors indicate different heads. Sparse [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Perplexity (↓) of FLOP matched models under different sparsities. Each plot corresponds to a specified FLOP budget per step. The number in parenthesis is the number of parameters of the dense baseline. Sparsity 1 represents the dense baseline. As sparsity increases, MoSA’s perplexity improves monotonically until reaching a saturation point around sparsity 32-64, beyond which performance deteriorates. This is likely … view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Perplexity of sparse-attention methods (MoSA, Fixed, and Routing) as sequence length increases. Each method has a fixed size window size (cluster size for the Routing Transformer, number of tokens selected for each head in MoSA and Fixed) regardless of total sequence l…
Figure 5
Figure 5. Figure 5: Perplexity of IsoFLOP matching models under pure MoSA setting. Each curve corresponds [PITH_FULL_IMAGE:figures/full_fig_p020_5.png]
Figure 6
Figure 6. Figure 6: Training losses of the Tiny models comparing the baseline, pure MoSA, and hybrid [PITH_FULL_IMAGE:figures/full_fig_p021_6.png]
Figure 7
Figure 7. Figure 7: Perplexity of the FLOP matched models with a different number of dense heads for [PITH_FULL_IMAGE:figures/full_fig_p021_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Compressed Sensing for Capability Localization in Large Language Models

    cs.CL 2026-02 conditional novelty 6.0 of 10

    LLM capabilities are concentrated in small sets of attention heads, and a compressed-sensing method can find those heads efficiently.

  2. Neural Attention Search Linear: Towards Adaptive Token-Level Hybrid Attention Models

    cs.CL 2026-02 conditional novelty 6.0 of 10

    NAtS-L learns per-chunk routing between Gated DeltaNet and softmax attention, improving long-context retrieval and length extrapolation over fixed hybrids.

  3. Direct3D-S2: Gigascale 3D Generation Made Easy with Spatial Sparse Attention

    cs.CV 2025-05 conditional novelty 6.0 of 10

    Direct3D-S2 uses a new Spatial Sparse Attention mechanism to train a sparse-volume diffusion transformer at 1024^3 resolution on 8 GPUs.

Reference graph

Works this paper leans on

70 extracted references · 47 canonical work pages · cited by 3 Pith papers

  1. [1]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proc. Advances in Neural Information Processing Systems (NIPS), pages 5998–6008, Long Beach, CA, USA, December 2017

    work page 2017

  2. [2]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Proc. Advances in Neural Information Processing Systems (NeurIPS), pages 1877–1901, 2020

    work page 1901

  3. [3]

    LLaMA: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient foundation language models. Preprint arXiv:2302.13971, 2023

    arXiv 2023

  4. [4]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024

    arXiv 2024

  5. [5]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

  6. [6]

    Hippo: Recurrent memory with optimal polynomial projections

    Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. In Proc. Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 1474–1487, 2020

    work page 2020

  7. [7]

    Efficiently modeling long sequences with structured state spaces

    Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. In Int. Conf. on Learning Representations (ICLR), 2022

    work page 2022

  8. [8]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023

    arXiv 2023

Show all 70 references
  1. [9]

    State space model for new-generation network alternative to transformers: A survey

    Xiao Wang, Shiao Wang, Yuhe Ding, Yuehang Li, Wentao Wu, Yao Rong, Weizhe Kong, Ju Huang, Shihao Li, Haoxiang Yang, et al. State space model for new-generation network alternative to transformers: A survey. arXiv preprint arXiv:2404.09516, 2024

    2024 arXiv

  2. [10]

    Gated delta networks: Improving mamba2 with delta rule

    Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule. In Int. Conf. on Learning Representations (ICLR), 2025

    2025

  3. [11]

    Can mamba learn how to learn? a comparative study on in-context learning tasks

    Jongho Park, Jaeseung Park, Zheyang Xiong, Nayoung Lee, Jaewoong Cho, Samet Oymak, Kangwook Lee, and Dimitris Papailiopoulos. Can mamba learn how to learn? a comparative study on in-context learning tasks. In Proc. Int. Conf. on Machine Learning (ICML), 2024

    2024

  4. [12]

    Efficient long sequence modeling via state space augmented transformer

    Simiao Zuo, Xiaodong Liu, Jian Jiao, Denis Charles, Eren Manavoglu, Tuo Zhao, and Jianfeng Gao. Efficient long sequence modeling via state space augmented transformer. arXiv preprint arXiv:2212.08136, 2022

    2022 arXiv

  5. [13]

    Jamba: A hybrid transformer-mamba language model

    Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, et al. Jamba: A hybrid transformer-mamba language model. arXiv preprint arXiv:2403.19887, 2024

    2024 arXiv

  6. [14]

    Transformers are RNNs: Fast autoregressive transformers with linear attention

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are RNNs: Fast autoregressive transformers with linear attention. In Proc. Int. Conf. on Machine Learning (ICML), volume 119, pages 5156–5165, Virtual Only, 2020

    2020

  7. [15]

    Linear transformers are secretly fast weight programmers

    Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear transformers are secretly fast weight programmers. In Proc. Int. Conf. on Machine Learning (ICML), volume 139, pages 9355–9366, Virtual only, 2021

    2021

  8. [16]

    Learning to control fast-weight memories: An alternative to recurrent nets

    Jürgen Schmidhuber. Learning to control fast-weight memories: An alternative to recurrent nets. Neural Computation, 4(1):131–139, 1992

    1992

  9. [17]

    The devil in linear transformer

    Zhen Qin, Xiaodong Han, Weixuan Sun, Dongxu Li, Lingpeng Kong, Nick Barnes, and Yiran Zhong. The devil in linear transformer. In Proc. Conf. on Empirical Methods in Natural Language Processing (EMNLP), pages 7025–7041, 2022

    2022

  10. [18]

    Generating long sequences with sparse transformers

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019. 15

    1904 arXiv

  11. [19]

    Big bird: Transformers for longer sequences

    Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. In Proc. Advances in Neural Information Processing Systems (NeurIPS), vol...

    2020

  12. [20]

    Peters, and Arman Cohan

    Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv:2004.05150, 2020

    2004 arXiv

  13. [21]

    Zoology: Measuring and improving recall in efficient language models

    Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré. Zoology: Measuring and improving recall in efficient language models. Int. Conf. on Learning Representations (ICLR), 2024

    2024

  14. [22]

    Repeat after me: Transformers are better than state space models at copying

    Samy Jelassi, David Brandfonbrener, Sham M Kakade, and Eran Malach. Repeat after me: Transformers are better than state space models at copying. In Proc. Int. Conf. on Machine Learning (ICML), 2024

    2024

  15. [23]

    Synthesizer: Rethinking self-attention for transformer models

    Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng. Synthesizer: Rethinking self-attention for transformer models. In Proc. Int. Conf. on Machine Learning (ICML), pages 10183–10192, 2021

    2021

  16. [24]

    Fast transformers with clustered attention

    Apoorv Vyas, Angelos Katharopoulos, and François Fleuret. Fast transformers with clustered attention. In Proc. Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 21665–21674, 2020

    2020

  17. [25]

    Efficient content-based sparse attention with routing transformers

    Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-based sparse attention with routing transformers. Transactions of the Association for Computational Linguistics (TACL), 9:53–68, 2021

    2021

  18. [26]

    Convergence properties of the k-means algorithms

    Leon Bottou and Yoshua Bengio. Convergence properties of the k-means algorithms. In Proc. Advances in Neural Information Processing Systems (NIPS), volume 7, 1994

    1994

  19. [27]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In Int. Conf. on Learning Representations (ICLR), Toulon, France, April 2017

    2017

  20. [28]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research (JMLR), 23(1):5232–5270, 2022

    2022

  21. [29]

    Mixture-of-experts with expert choice routing

    Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, et al. Mixture-of-experts with expert choice routing. In Proc. Advances in Neural Information Processing Systems (NeurIPS), volume 35, pages 7103–7114, 2022

    2022

  22. [30]

    Mixture of attention heads: Selecting attention heads per token

    Xiaofeng Zhang, Yikang Shen, Zeyu Huang, Jie Zhou, Wenge Rong, and Zhang Xiong. Mixture of attention heads: Selecting attention heads per token. In Proc. Conf. on Empirical Methods in Natural Language Processing (EMNLP), pages 4150–4162, Abu Dhabi, United Arab Emirates, December 2022

    2022

  23. [31]

    Switchhead: Accelerating transformers with mixture-of-experts attention

    Róbert Csordás, Piotr Pi˛ ekos, Kazuki Irie, and Jürgen Schmidhuber. Switchhead: Accelerating transformers with mixture-of-experts attention. In Proc. Advances in Neural Information Processing Systems (NeurIPS), Vancouver, Canada, December 2024

    2024

  24. [32]

    Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time

    Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time. In Proc. Advances in Neural Informatio...

    2023

  25. [33]

    Snapkv: Llm knows what you are looking for before generation

    Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Llm knows what you are looking for before generation. In Proc. Advances in Neural Information Processing Systems (NeurIPS), volume 37, pages...

    2025

  26. [34]

    Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling

    Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, and Xiao Wen. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling. arXiv preprint arXiv:2406.02069, 2024

    2024 arXiv

  27. [35]

    Gshard: Scaling giant models with condi- tional computation and automatic sharding

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with condi- tional computation and automatic sharding. In Int. Conf. on Learning Representations (ICLR), 2021. 16

    2021

  28. [36]

    Fast transformer decoding: One write-head is all you need

    Noam Shazeer. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150, 2019

    1911 arXiv

  29. [37]

    Native sparse attention: Hardware-aligned and natively trainable sparse attention

    Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, YX Wei, Lean Wang, Zhiping Xiao, et al. Native sparse attention: Hardware-aligned and natively trainable sparse attention. arXiv preprint arXiv:2502.11089, 2025

    2025 arXiv

  30. [38]

    Approximating two-layer feedforward networks for efficient transformers

    Róbert Csordás, Kazuki Irie, and Jürgen Schmidhuber. Approximating two-layer feedforward networks for efficient transformers. InFindings of the Association for Computational Linguistics: EMNLP 2023, November 2023

    2023

  31. [39]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y . Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Proc. Advances in Neural Information Processing Systems (NeurIPS), New Orleans, Louisiana, USA, December 2022

    2022

  32. [40]

    PyTorch: An imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu ...

    2019

  33. [41]

    RoFormer: Enhanced transformer with rotary position embedding

    Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. RoFormer: Enhanced transformer with rotary position embedding. Preprint arXiv:2104.09864, 2021

    2021 arXiv

  34. [42]

    Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing

    Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proc. Conf. on Empirical Methods in Natural Language Processing (EMNLP), pages 66–71, Brussels, Belgium, October 2018

    2018

  35. [43]

    Neural machine translation of rare words with subword units

    Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proc. Association for Computational Linguistics (ACL), pages 1715–1725, Berlin, Germany, August 2016

    2016

  36. [44]

    Japanese and korean voice search

    Mike Schuster and Kaisuke Nakajima. Japanese and korean voice search. In Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), pages 5149–5152, Kyoto, Japan, March 2012

    2012

  37. [45]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research (JMLR) , 21:140:1–140:67, 2020

    2020

  38. [46]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, Int. Conf. on Learning Representations (ICLR), San Diego, CA, USA, May 2015

    2015

  39. [47]

    Efficient long-range transformers: You need to attend more, but not necessarily at every layer

    Qingru Zhang, Dhananjay Ram, Cole Hawkins, Sheng Zha, and Tuo Zhao. Efficient long-range transformers: You need to attend more, but not necessarily at every layer. In Proc. Conf. on Empirical Methods in Natural Language Processing (EMNLP), 2023

    2023

  40. [48]

    Efficient streaming language models with attention sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In Int. Conf. on Learning Representations (ICLR), 2024

    2024

  41. [49]

    Reformer: The efficient transformer

    Nikita Kitaev, Kaiser Łukasz, and Anselm Levskaya. Reformer: The efficient transformer. In Int. Conf. on Learning Representations (ICLR), 2020

    2020

  42. [50]

    From 128k to 4m: Efficient training of ultra-long context large language models

    Chejian Xu, Wei Ping, Peng Xu, Zihan Liu, Boxin Wang, Mohammad Shoeybi, Bo Li, and Bryan Catanzaro. From 128k to 4m: Efficient training of ultra-long context large language models. arXiv preprint arXiv:2504.06214, 2025

    2025 arXiv

  43. [51]

    The LAMBADA dataset: Word prediction requiring a broad discourse context

    Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proc. Association for Compu- tational Lingu...

    2016

  44. [52]

    Winogrande: An adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In Proc. AAAI Conf. on Artificial Intelligence, pages 8732–8740, New York, NY , USA, February 2020. 17

    2020

  45. [53]

    Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. BLiMP: The benchmark of linguistic minimal pairs for English. Transactions of the Association for Computational Linguistics (TACL), 8:377–392, 2020

    2020

  46. [54]

    Hellaswag: Can a machine really finish your sentence? In Proc

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proc. Association for Computational Linguistics (ACL), pages 4791–4800, Florence, Italy, August 2019

    2019

  47. [55]

    PIQA: reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: reasoning about physical commonsense in natural language. In Proc. AAAI Conf. on Artificial Intelligence, pages 7432–7439, New York, NY , USA, February 2020. AAAI Press

    2020

  48. [56]

    Think you have solved question answering? try ARC, the AI2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try ARC, the AI2 reasoning challenge. Preprint arXiv:1803.05457, 2018

    2018 arXiv

  49. [57]

    St-moe: Designing stable and transferable sparse expert models

    Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. St-moe: Designing stable and transferable sparse expert models. arXiv preprint arXiv:2202.08906, 2022

    2022 arXiv

  50. [58]

    Mixture-of-experts meets instruction tuning: A winning combination for large language models

    Sheng Shen, Le Hou, Yanqi Zhou, Nan Du, Shayne Longpre, Jason Wei, Hyung Won Chung, Barret Zoph, William Fedus, Xinyun Chen, et al. Mixture-of-experts meets instruction tuning: A winning combination for large language models. In Int. Conf. on Learning Representations (ICLR), 2024

    2024

  51. [59]

    Colwell, and Adrian Weller

    Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamás Sarlós, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J. Colwell, and Adrian Weller. Rethinking attention with performers. In ...

    2021

  52. [60]

    Sepllm: Accelerate large language models by compressing one segment into one separator

    Guoxuan Chen, Han Shi, Jiawei Li, Yihang Gao, Xiaozhe Ren, Yimeng Chen, Xin Jiang, Zhenguo Li, Weiyang Liu, and Chao Huang. Sepllm: Accelerate large language models by compressing one segment into one separator. arXiv preprint arXiv:2412.12094, 2024

    2024 arXiv

  53. [61]

    Hashattention: Semantic sparsity for faster inference

    Aditya Desai, Shuo Yang, Alejandro Cuadron, Ana Klimovic, Matei Zaharia, Joseph E Gon- zalez, and Ion Stoica. Hashattention: Semantic sparsity for faster inference. arXiv preprint arXiv:2412.14468, 2024

    2024 arXiv

  54. [62]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  55. [63]

    Mixtral of experts

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024

    2024 arXiv

  56. [64]

    Jetmoe: Reaching llama2 performance with 0.1 m dollars

    Yikang Shen, Zhen Guo, Tianle Cai, and Zengyi Qin. Jetmoe: Reaching llama2 performance with 0.1 m dollars. arXiv preprint arXiv:2404.07413, 2024

    2024 arXiv

  57. [65]

    BASE layers: Simplifying training of large, sparse models

    Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. BASE layers: Simplifying training of large, sparse models. In Marina Meila and Tong Zhang, editors, Proc. Int. Conf. on Machine Learning (ICML), volume 139, pages 6265–6274, Virtual only, July 2021

    2021

  58. [66]

    Hash layers for large sparse models

    Stephen Roller, Sainbayar Sukhbaatar, Jason Weston, et al. Hash layers for large sparse models. In Proc. Advances in Neural Information Processing Systems (NeurIPS) , volume 34, pages 17555–17566, 2021

    2021

  59. [67]

    Mixture-of-depths: Dynamically allocating compute in transformer-based language models

    David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. Mixture-of-depths: Dynamically allocating compute in transformer-based language models. arXiv preprint arXiv:2404.02258, 2024

    2024 arXiv

  60. [68]

    Moh: Multi-head attention as mixture-of-head attention

    Peng Jin, Bo Zhu, Li Yuan, and Shuicheng Yan. Moh: Multi-head attention as mixture-of-head attention. arXiv preprint arXiv:2410.11842, 2024

    2024

  61. [69]

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. In Proc. Conf. on Empirical Methods in Natural Language Processing (EMNLP), pages 4895–...

    2023

  62. [70]

    On layer normalization in the transformer architecture

    Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tie-Yan Liu. On layer normalization in the transformer architecture. In Proc. Int. Conf. on Machine Learning (ICML), volume 119, pages 10524–10533, Virtual Only,...

    2020

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.