Pith. sign in

REVIEW 4 major objections 4 minor 54 references

Selective Attention: Enhancing Transformer through Principled Context Control

T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper proposes a Selective Self-Attention layer that gives each query its own softmax temperature, letting a transformer control how sharply it focuses without growing its weight matrices.

desk verdict A plausible parameter-efficient attention tweak with broad but thin experiments; the theory overclaims because the implemented temperature module cannot express the optimal temperatures the proofs construct. read the letter →

arxiv 2411.12892 v1 pith:ZUMZPCVG submitted 2024-11-19 cs.LG cs.CL

Xuechen Zhang , Xiangyu Chang , Mingchen Li , Amit Roy-Chowdhury , Jiasi Chen , Samet Oymak This is my paper · ORCID
classification cs.LGcs.CL
keywords selectiveattentiontemperaturescalingsoftmaxspikinessdilutioncontextualsparsitytransformerlanguagemodelinglong-contextretrieval
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Standard self-attention applies the same softmax map to every query, so a transformer cannot independently decide how spiky its attention should be once the key-query weights are fixed. This paper argues that this uniformity forces weight matrices to grow large when different queries need different levels of focus, which slows optimization and dilutes attention over long contexts. As a remedy, it introduces Selective Self-Attention (SSA): a learnable inverse-temperature per query and value, plus a positional term that grows logarithmically with context length, applied as elementwise scaling of the embeddings. The paper claims this decouples semantic similarity from contextual sparsity, and it backs the claim with lower bounds showing when temperature helps, a synthetic denoising task, and experiments where SSA improves perplexity and accuracy on several transformer families with less than 0.5% added parameters under weight sharing. The payoff, if correct, is a general, nearly free modification that makes existing transformers focus better and retrieve better in long contexts.

What carries the argument

The load-bearing object is the learned inverse-temperature field $\tau(x)=\tanh(f(x))+1+\sigma(\alpha)\log n$, applied elementwise to query and value embeddings, with $f(x)=W_{\mathrm{tmp}}\,\mathrm{GeLU}(W_{k/q/v}x)$ reusing existing attention projections plus one small output vector per head. It acts as a scalar gate on each token: positive values sharpen the softmax, negative values can de-emphasize or suppress a token, and the $\log n$ term compensates for attention flattening as context grows. The mechanism's work is to decouple the two roles of $W_qW_k^\top$: semantic similarity stays in the projection matrices while $\tau$ carries contextual sparsity, so the required spectral norm no longer grows with specificity differences or with $\log L$.

What would settle it

Train an SSA-equipped language model while clamping every learned temperature to a narrow interval such as [0.9, 1.1] or freezing the temperature module to a constant; if perplexity and passkey accuracy stay at the SSA level instead of falling back to the vanilla baseline, then the reported gains do not come from query-dependent temperature control.

Watch

Extended reading notes

Core claim

The central claim is that softmax temperature is not a nuisance hyperparameter but a missing degree of freedom in attention. In canonical self-attention $V^\top\mathrm{softmax}(Kq)$, the same weights $W=W_qW_k^\top$ must simultaneously encode which tokens are semantically similar and how concentrated the attention map should be; the paper proves (Lemma 1 and Proposition 1) that when two queries are semantically close yet need very different spikiness, standard attention needs a weight norm that blows up as the inverse of their embedding distance, whereas query-temperature $\tau(q)$ lets the same weights serve both queries with bounded norm. For position, the paper derives (Proposition 2) that a flat temperature cannot solve even a two-token imbalanced-context retrieval problem, while $\tau_n=\log\kappa_n+\log(\alpha/(1-\alpha))$ solves it exactly, motivating a positional term of the form $1+\sigma(\alpha)\log n$. For values, a linear value projection provably fails a synthetic denoising task that nonlinear value scaling nearly matches at the Bayes-optimal level. Empirically, SSA-equipped models achieve consistent accuracy gains over baselines on standard language modeling benchmarks and a large jump on passkey retrieval (for example, Pythia-160m from 56.9% to 74.4%).

Load-bearing premise

The benefits hinge on the small learned temperature module being able to produce the very different per-token and per-position temperature values the theory requires, which the paper does not directly verify.

Editorial extensions

If this is right

  • Transformer weights no longer have to grow like $1/\|a-b\|_2$ to give similar tokens different focus; SSA keeps effective weights bounded, which should reduce vanishing-softmax slowdowns and sensitivity to small context perturbations.
  • The positional term $1+\sigma(\alpha)\log n$ is a differentiable, principled replacement for hand-tuned length-scaling rules, and it provably solves the imbalanced-context task that flat temperature cannot.
  • Value-temperature gives attention a nonlinear denoising channel, so irrelevant or noisy tokens can be suppressed even when a linear value projection cannot separate them.
  • Because temperature adapts per query and per position, long-context retrieval improves, with the paper reporting passkey retrieval rising from 56.9% to 74.4% for Pythia-160m under SSA.
  • Weight sharing makes the module nearly parameter-free and compatible with fast attention implementations, so the gains transfer to existing large language models by fine-tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same temperature field can be read as an inductive bias toward input-dependent gating inside attention; if correct, comparisons with other selective mechanisms (gates, sparse top-k attention) should be made at the level of the effective temperature they induce, not just parameter count.
  • Proposition 2's logarithmic rule is likely a low-order correction that matters most for very long contexts; a testable extension is whether SSA reduces the need for additional position-embedding scaling when the context window is extended during fine-tuning.
  • Because the theory constructs temperatures that the implemented $\tanh$ MLP may not reach, the empirical gains could be improved or matched by a more expressive closed-form temperature rule; a direct test would clamp or span-check the learned $\tau$ values.
  • The decoupling story suggests a diagnostic: attention-head entropy should vary more across queries with SSA than without, which can be measured on real corpora to verify the mechanism is actually being used.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a Selective Self-Attention (SSA) layer that applies input-dependent inverse temperatures to query and value embeddings, together with a position-dependent term tau_pos(x) = 1 + sigma(alpha) log n. Theoretical arguments (Lemma 1, Propositions 1 and 2, Lemma 2) claim that such scaling controls attention spikiness, mitigates attention dilution, and improves expressivity and denoising. The method is evaluated on language modeling benchmarks with GPT-2, Pythia, Llama, and Llama-3, under both fine-tuning and pre-training, reporting perplexity/accuracy gains and improved passkey retrieval, with weight-sharing and feature-based variants that add less than 0.5% parameters.

Significance. If the link between the theory and the implemented temperature module were established, SSA would be an attractive lightweight, architecture-agnostic improvement: it is simple, parameter-efficient, and shows broad gains across model families and training regimes. The paper also offers a formal derivation of a YaRN-like logarithmic temperature rule, which is a useful conceptual contribution. However, the theory is an existence argument that sets temperatures by construction (tau(a)=La, tau(b)=Lb, and tau_n=log kappa_n + log(alpha/(1-alpha))), while the implemented parameterization is a bounded tanh-plus-log function whose expressivity is never characterized. Several synthetic experiments replace the learned module with hand-set temperatures or hard thresholding proxies. The empirical results are broad but all single-run, so the 'consistent improvement' claim is not yet statistically supported.

major comments (4)
  1. [Section 3 and Section 4.2 (Proposition 2)] The implemented temperature function tau(x) = tanh(f(x)) + 1 + sigma(alpha) log n is, for every n > 1, bounded below by sigma(alpha) log n > 0 and above by 2 + sigma(alpha) log n. It therefore cannot realize the optimal inverse temperatures of Proposition 2, tau_n = log kappa_n + log(alpha/(1-alpha)), which are zero or negative for natural parameter choices (e.g., alpha = 1/2, kappa_n = 1 gives tau_n = 0; alpha < 1/2 and kappa_n < (1-alpha)/alpha gives negative tau_n). The same issue applies to the constructions in Lemma 1 and Proposition 1 when the desired temperatures La, Lb lie outside the attainable interval. This directly contradicts the statement in Section 3 that 'we don't restrict tau_{k/q/v} to be non-negative.' Since no expressivity analysis of the learned module is provided, the theoretical advantages derived in Section 4 do not carry over to the actual SSA layer as implemented, and the claimed 'principled' bridge is not established.
  2. [Section 5.1, Table 3] All empirical results are reported from single runs with no seeds, confidence intervals, or significance tests. Several reported gains are small (e.g., fine-tuned Pythia-160m Wikitext perplexity 26.681 vs. 26.514, and Lambada_std perplexity 47.996 vs. 47.945), and at least one weight-sharing entry is worse than the vanilla baseline (Pythia-160m fine-tune Wikitext perplexity 26.780 vs. 26.681). Given the abstract's claim of a 'noticeable and consistent accuracy improvement,' the absence of repeated runs is a load-bearing gap: the aggregate pattern is suggestive, but the consistency claim cannot be evaluated statistically.
  3. [Appendix C.1 (proof of Proposition 1)] The proof of Proposition 1 is not rigorous. It writes ||P* - S(EWE^T)||_infty as ||1/(1+e^{-EWE^T}) - P*||_infty, which incorrectly treats the row-wise softmax as an elementwise sigmoid. The subsequent line '||EWE^T||_infty >= 1/(4 epsilon) - Gamma' does not follow from the preceding expression, and the final lower bound on ||W|| is left unexplained. As written, this proof does not establish the proposition, which is one of the paper's theoretical contributions.
  4. [Section 4.3 and Table 2] The denoising experiment does not evaluate the proposed SSA value-temperature module. The paper states that, as a proxy for token-aware temperature scaling, it uses the hard thresholding function max_j x_{ij} >= 1/2, rather than the learned tanh-based temperature function. Similarly, the token-generation experiment in Section 4.1 (Table 1) assigns temperatures by hand to groups of nodes with equal degree. These experiments therefore demonstrate the utility of input-dependent scaling in principle, but they do not provide evidence that the actual SSA temperature module realizes these behaviors.
minor comments (4)
  1. [Section 1 (Introduction)] The bullet list says SSA 'introduces 5% additional parameters to the model,' which contradicts the Abstract's 'less than 0.5% new parameters' and the weight-sharing discussion in Section 3. This should be corrected.
  2. [Figure 2 caption] The caption states 'the norm of the vanilla attention layer is approximately three times larger than that of SSA (dashed red line compare to green line),' but the figure legend and color references are unclear; the dashed-line convention should be explained precisely.
  3. [Appendix A] The implementation details state that pre-training takes about 2 hours on 4 A40 GPUs for 15B tokens, while fine-tuning takes about 2 days. This appears to be a typographical inconsistency and should be clarified.
  4. [Throughout] There are numerous typographical errors, including 'perfromance,' 'di fferent,' 'trainig,' and 'funtion'; a careful proofreading pass is needed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the theoretical statements are explicit constructions, and the empirical results rest on external benchmarks.

full rationale

The paper's theoretical contributions are explicit existence and converse arguments, not fitted predictions renamed as results. In Section 4.1, the authors set tau(a)=La and tau(b)=Lb with norm W = 1 to show that query-temperature decouples specificity from semantic distance; this is a construction compared against Lemma 1, not a parameter fitted to data. In Section 4.2, Proposition 2 derives tau_n = log kappa_n + log(alpha/(1-alpha)) algebraically from the requirement that the softmax mixture equal the target alpha, then proves a separate lower bound showing flat temperature fails; the paper explicitly says this result inspired the implemented position-aware rule. The implemented tanh-plus-log temperature module is a heuristic whose expressivity is not analyzed against the theoretical optimal temperatures; that is a genuine theory-to-implementation gap and a correctness risk, but it is not circularity because the empirical claims are validated on external benchmarks (Wikitext, Lambada, passkey retrieval) that do not reduce to the theoretical construction. Self-citations appear only for related work, experimental inspiration, and baselines, and none serves as an external uniqueness theorem that forces the central claim. Appendix E contains unproved claims with promised but missing assumptions; that is missing support rather than circular derivation.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The theoretical contributions are existence proofs that choose temperatures based on the target statistics. They do not predict temperatures from first principles, and the implemented module is a heuristic approximation. The empirical results are the main independent evidence that the temperature module learns useful scalings.

free parameters (2)
  • Ideal query temperatures τ(a)=La, τ(b)=Lb (Lemma 1) = Set to target specificities La, Lb
    The lemma's comparison assumes the temperature can be set exactly to the desired specificity values; this is a construction, not a learned or derived quantity, and it creates the claimed norm-decoupling effect.
  • Ideal positional temperature τ_n=log κ_n+log(α/(1-α)) (Prop 2) = Uses true mixture α and per-position imbalance ratio κ_n
    The success direction of the theorem requires knowing the target mixture and the data imbalance at each position; the implemented τ_pos=1+σ(α)log n is a simplification with a trainable α, so the theoretical optimum is not achieved automatically.
assumptions (3)
  • domain assumption Token embeddings are unit Euclidean norm and linearly independent in the theoretical setups (Lemma 1, Prop 2).
    Used to construct W and to bound norms; restricts the theory to idealized embeddings rather than real learned embeddings.
  • domain assumption The top-probability bound ∥s_L∥∞ ≤ 1/(1+(L-1)e^{-2∥W∥}) in Sec 4.2 holds for the worst-case score alignment.
    Stated without derivation; it motivates the logarithmic positional temperature but is not a property of all attention score matrices.
  • ad hoc to paper The implemented temperature module can express the ideal temperatures from the theory.
    The paper assumes the tanh-MLP plus positional term can produce the required temperature values; no expressivity guarantee is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Selective Attention: Enhancing Transformer through Principled Context Control." pith.science (2026). https://pith.science/paper/ZUMZPCVG

@misc{pith2026241112892,
  author       = {Pith},
  title        = {Pith review of: Selective Attention: Enhancing Transformer through Principled Context Control},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZUMZPCVG}},
  note         = {Machine review of arXiv:2411.12892}
}
abstract

The attention mechanism within the transformer architecture enables the model to weigh and combine tokens based on their relevance to the query. While self-attention has enjoyed major success, it notably treats all queries $q$ in the same way by applying the mapping $V^\top\text{softmax}(Kq)$, where $V,K$ are the value and key embeddings respectively. In this work, we argue that this uniform treatment hinders the ability to control contextual sparsity and relevance. As a solution, we introduce the $\textit{Selective Self-Attention}$ (SSA) layer that augments the softmax nonlinearity with a principled temperature scaling strategy. By controlling temperature, SSA adapts the contextual sparsity of the attention map to the query embedding and its position in the context window. Through theory and experiments, we demonstrate that this alleviates attention dilution, aids the optimization process, and enhances the model's ability to control softmax spikiness of individual queries. We also incorporate temperature scaling for value embeddings and show that it boosts the model's ability to suppress irrelevant/noisy tokens. Notably, SSA is a lightweight method which introduces less than 0.5% new parameters through a weight-sharing strategy and can be fine-tuned on existing LLMs. Extensive empirical evaluations demonstrate that SSA-equipped models achieve a noticeable and consistent accuracy improvement on language modeling benchmarks.

Figures

Figures reproduced from arXiv: 2411.12892 by the authors.

Figure 1
Figure 1. A quotation by Steve Jobs. We highlight tokens according to their temperatures learned by [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The operator norm of W with and without Query-temperature scaling, scaled by ×103 . The fig￾ure depicts the distribution across 1000 tokens. The dashed line is the average norm. Notably, the norm of the vanilla attention layer is approximately three times larger than that of SSA(dashed red line compare to green line). Furthermore, the vanilla attention layer exhibits a lower spikiness score (0.39) compared to SSA (0… view at source ↗
Figure 3
Figure 3. We compare 1-layer SSA and 1-layer attention when solving next-token prediction on a [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Comparison of training curves. SSA provides reasonable benefits in terms of training speedup. Additionally, SSA can accelerate the training process by achieving comparable performance with fewer tokens. This efficiency not only reduces the demand on compu￾tational reso…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

54 extracted references · 15 canonical work pages

  1. [1]

    Generalization on the unseen, logic reasoning and degree curriculum

    Emmanuel Abbe, Samy Bengio, Aryo Lotfi, and Kevin Rizk. Generalization on the unseen, logic reasoning and degree curriculum. In International Conference on Machine Learning , pages 31–60. PMLR, 2023

    work page 2023

  2. [2]

    Simple linear attention language models balance the recall-throughput tradeoff

    Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, Dylan Zinsley, James Zou, Atri Rudra, and Christopher Ré. Simple linear attention language models balance the recall-throughput tradeoff. arXiv preprint arXiv:2402.18668, 2024

    arXiv 2024

  3. [3]

    Pythia: A suite for analyzing large language models across training and scaling

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397–2430. PMLR, 2023

    2023

  4. [4]

    Piqa: Reasoning about phys- ical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about phys- ical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439, 2020

    2020

  5. [5]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  6. [6]

    Scatterbrain: Unifying sparse and low-rank attention

    Beidi Chen, Tri Dao, Eric Winsor, Zhao Song, Atri Rudra, and Christopher Ré. Scatterbrain: Unifying sparse and low-rank attention. Advances in Neural Information Processing Systems, 34:17413–17426, 2021

    work page 2021

  7. [7]

    Attention Alignment and Flexible Positional Embeddings Improve Transformer Length Extrapolation

    Ta-Chung Chi, Ting-Han Fan, and Alexander I Rudnicky. Attention alignment and flexible posi- tional embeddings improve transformer length extrapolation. arXiv preprint arXiv:2311.00684, 2023

    work page Pith review arXiv 2023

  8. [8]

    Generating long sequences with sparse transformers

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019

    arXiv 1904

Show all 54 references
  1. [9]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1– 113, 2023

    2023

  2. [10]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  3. [11]

    Flashattention-2: Faster attention with better parallelism and work partitioning

    Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691, 2023

    2023 arXiv

  4. [12]

    Flashattention: Fast and memory-efficient exact attention with io-awareness

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344–16359, 2022

    2022

  5. [13]

    Language modeling with gated convolutional networks

    Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In International conference on machine learning, pages 933–941. PMLR, 2017

    2017

  6. [14]

    Griffin: Mixing gated linear recurrences with local attention for e fficient language models

    Soham De, Samuel L Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, et al. Griffin: Mixing gated linear recurrences with local attention for e fficient language models. arXiv preprint...

    2024 arXiv

  7. [15]

    Attention is not all you need: Pure attention loses rank doubly exponentially with depth

    Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In International Conference on Machine Learning, pages 2793–2803. PMLR, 2021

    2021

  8. [16]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  9. [17]

    The pile: An 800gb dataset of diverse text for language modeling

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020

    2020 arXiv

  10. [18]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023

    2023 arXiv

  11. [19]

    Long short-term memory

    Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997

    1997

  12. [20]

    From self-attention to markov models: Unveiling the dynamics of generative transformers

    M Emrullah Ildiz, Yixiao Huang, Yingcong Li, Ankit Singh Rawat, and Samet Oymak. From self-attention to markov models: Unveiling the dynamics of generative transformers. Interna- tional Conference on Machine Learning, 2024

    2024

  13. [21]

    Gateloop: Fully data-controlled linear recurrence for sequence modeling

    Tobias Katsch. Gateloop: Fully data-controlled linear recurrence for sequence modeling. arXiv preprint arXiv:2311.01927, 2023

    2023 arXiv

  14. [22]

    Au- tobalance: Optimized loss functions for imbalanced data

    Mingchen Li, Xuechen Zhang, Christos Thrampoulidis, Jiasi Chen, and Samet Oymak. Au- tobalance: Optimized loss functions for imbalanced data. Advances in Neural Information Processing Systems, 34:3163–3177, 2021

    2021

  15. [23]

    Exposing attention glitches with flip-flop language modeling

    Bingbin Liu, Jordan Ash, Surbhi Goel, Akshay Krishnamurthy, and Cyril Zhang. Exposing attention glitches with flip-flop language modeling. Advances in Neural Information Processing Systems, 36, 2024

    2024

  16. [24]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  17. [25]

    Long range language modeling via gated state spaces

    Harsh Mehta, Ankit Gupta, Ashok Cutkosky, and Behnam Neyshabur. Long range language modeling via gated state spaces. In International Conference on Learning Representations, 2023

    2023

  18. [26]

    Long-tail learning via logit adjustment

    Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. ICLR, 2021

    2021

  19. [27]

    Pointer sentinel mixture models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016

    2016 arXiv

  20. [28]

    Efficient estimation of word representations in vector space

    Tomas Mikolov. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013

    2013 arXiv

  21. [29]

    Landmark attention: Random-access infinite context length for transformers

    Amirkeivan Mohtashami and Martin Jaggi. Landmark attention: Random-access infinite context length for transformers. arXiv preprint arXiv:2305.16300, 2023

    2023 arXiv

  22. [30]

    In-context learning and induction heads

    Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11895, 2022

    2022 arXiv

  23. [31]

    Fast attention over long sequences with dynamic sparse flash attention

    Matteo Pagliardini, Daniele Paliotta, Martin Jaggi, and François Fleuret. Fast attention over long sequences with dynamic sparse flash attention. In Thirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  24. [32]

    The lambada dataset: Word prediction requiring a broad discourse context

    Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint arXiv:1606.06031, 2016. 12

    2016 arXiv

  25. [33]

    Yarn: E fficient context window extension of large language models

    Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: E fficient context window extension of large language models. arXiv preprint arXiv:2309.00071, 2023

    2023 arXiv

  26. [34]

    Language models are unsupervised multitask learners

    Alec Radford, Je ffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019

    2019

  27. [35]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020

    2020

  28. [36]

    Sparse modular activation for efficient sequence modeling

    Liliang Ren, Yang Liu, Shuohang Wang, Yichong Xu, Chenguang Zhu, and ChengXiang Zhai. Sparse modular activation for efficient sequence modeling. In Thirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  29. [37]

    Unraveling attention via convex duality: Analysis and interpretations of vision transformers

    Arda Sahiner, Tolga Ergen, Batu Ozturkler, John Pauly, Morteza Mardani, and Mert Pilanci. Unraveling attention via convex duality: Analysis and interpretations of vision transformers. In International Conference on Machine Learning, pages 19050–19088. PMLR, 2022

    2022

  30. [38]

    Winogrande: An adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021

    2021

  31. [39]

    Multitask prompted training enables zero-shot task generalization

    Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207, 2021

    arXiv

  32. [40]

    Glu variants improve transformer

    Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020

    2002 arXiv

  33. [41]

    Slimpajama-dc: Understanding data combinations for llm training

    Zhiqiang Shen, Tianhua Tao, Liqun Ma, Willie Neiswanger, Joel Hestness, Natalia Vassilieva, Daria Soboleva, and Eric Xing. Slimpajama-dc: Understanding data combinations for llm training. arXiv preprint arXiv:2309.10818, 2023

    2023 arXiv

  34. [42]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024

    2024

  35. [43]

    Max-margin token selection in attention mechanism

    Davoud Ataee Tarzanagh, Yingcong Li, Xuechen Zhang, and Samet Oymak. Max-margin token selection in attention mechanism. In Thirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  36. [44]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  37. [45]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  38. [46]

    Mambabyte: Token-free selective state space model

    Junxiong Wang, Tushaar Gangavarapu, Jing Nathan Yan, and Alexander M Rush. Mambabyte: Token-free selective state space model. arXiv preprint arXiv:2401.13660, 2024

    2024 arXiv

  39. [47]

    An explanation of in-context learning as implicit bayesian inference

    Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. arXiv preprint arXiv:2111.02080, 2021

    2021 arXiv

  40. [48]

    Gated linear attention transformers with hardware-efficient training

    Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training. arXiv preprint arXiv:2312.06635, 2023

    2023 arXiv

  41. [49]

    Self-attention networks can process bounded hierarchical languages

    Shunyu Yao, Binghui Peng, Christos Papadimitriou, and Karthik Narasimhan. Self-attention networks can process bounded hierarchical languages. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Comp...

    2021

  42. [50]

    Differential transformer

    Tianzhu Ye, Li Dong, Yuqing Xia, Yutao Sun, Yi Zhu, Gao Huang, and Furu Wei. Differential transformer. arXiv preprint arXiv:2410.05258, 2024. 13

    2024 arXiv

  43. [51]

    Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    1905 arXiv

  44. [52]

    Class- attribute priors: Adapting optimization to heterogeneity and fairness objective

    Xuechen Zhang, Mingchen Li, Jiasi Chen, Christos Thrampoulidis, and Samet Oymak. Class- attribute priors: Adapting optimization to heterogeneity and fairness objective. to appear at AAAI, 2024

    2024

  45. [53]

    What algorithms can transformers learn? a study in length generalization

    Hattie Zhou, Arwen Bradley, Etai Littwin, Noam Razin, Omid Saremi, Josh Susskind, Samy Bengio, and Preetum Nakkiran. What algorithms can transformers learn? a study in length generalization. arXiv preprint arXiv:2310.16028, 2023

    arXiv

  46. [54]

    spikiness of attention

    Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint arXiv:2401.09417, 2024. 14 Table 5: Fine-tuning experiment results for language mode...

    2024 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.