Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Faster Query-Key Learning Sharpens Attention in Self-Attention Models

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Raising the query-key learning rate relative to the output-value rate sharpens self-attention even when predictive loss stays flat.

desk verdict A clean collapsed-circuit result with an overreaching factorization story: the log^2 sharpening is real under strong assumptions, but the bridge to factorized transformers is asserted, not proved. read the letter →

arxiv 2608.06776 v1 pith:23BX4TTJ submitted 2026-08-07 cs.LG

Rahul Vashisht , Harish G. Ramaswamy This is my paper · ORCID
classification cs.LG MSC 68T07
keywords self-attentionattentionsharpeningtrainingdynamicsgradientflowquery-keycircuitoutput-valuelearning-raterationext-tokenprediction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that attention sharpening in a self-attention model is governed by the relative learning speeds of its two circuits, not by the final loss alone. For a single-layer transformer trained for next-token prediction, the query-key circuit (which allocates attention) and the output-value circuit (which maps attended tokens to predictions) evolve at different effective rates, and the ratio of their learning rates, $r = \eta_{QK}/\eta_{OV}$, controls how much attention mass lands on task-relevant tokens. Under population gradient flow in an orthogonal synthetic setting, the paper derives closed-form scalar dynamics in which the output-value scale grows like log of time while the query-key scale grows like $(\eta_{QK}/\eta_{OV}) \log^2$ of time; a larger ratio pushes attention toward relevant tokens even as training loss decays at the same rate. The paper argues this mechanism explains why factorized versus collapsed parameterizations of the two circuits produce qualitatively different attention patterns at similar predictive performance, and shows experimentally that raising the query-key learning rate sharpens attention on real datasets.

What carries the argument

The argument is carried by decomposing the self-attention layer into two linear circuits — the query-key circuit $W_K W_Q^\top$, which sets the attention pattern, and the output-value circuit $W_O W_V$, which maps attended representations to logits — and tracking their population gradient-flow trajectories under factorization. Lemma 2 shows that factorized training induces a preconditioned gradient flow on collapsed parameters with the state-dependent, degree-1 homogeneous operator $\Omega(C) = (C^\top C)^{1/2} \oplus (C C^\top)^{1/2}$, so factorization behaves like a learning-rate rescale that grows over time. Under orthogonality assumptions on the token vectors and zero initialization, Theorem 1 collapses the full training dynamics to two coupled scalar ordinary differential equations for the output-value scale $\mu_{OV}$ and the query-key scale $\mu_{QK}$, and Lemma 3 derives their relative growth rates — log versus log-squared in time — which drive the attention mass $\alpha(t) = m \exp(\mu_{QK}(t)) / (m \exp(\mu_{QK}(t)) + n)$ through the softmax nonlinearity.

What would settle it

Under the paper's orthogonal synthetic setting, train with $r = 100$ and with $r = 0.01$, stopping each run at the same loss threshold: the relevant-token attention mass $\alpha(t)$ should be near one in the first case and near uniform in the second. If attention at matched loss is independent of $r$, the claimed decoupling between loss decay and attention sharpening is contradicted. A second check: with fixed large $r$, $\alpha(t)$ should keep growing roughly as $\log^2$ of time after the loss is already near zero, so extending training should show attention still concentrating while loss stays flat.

Watch

Extended reading notes

Core claim

The central claim is that faster query-key learning relative to output-value learning produces sharper attention, because the model compensates for slower output-value learning by concentrating attention mass on relevant tokens. Concretely, Lemma 3 states that in the pre-saturation regime the output-value scale obeys $\mu_{OV}(t) = \Theta(\ln(1+t))$ while the query-key scale obeys $\mu_{QK}(t) = \Theta((\eta_{QK}/\eta_{OV})(\ln(1+t))^2)$, and since the attention mass on relevant tokens is $\alpha(t) = m \exp(\mu_{QK}(t)) / (m \exp(\mu_{QK}(t)) + n)$, that mass grows faster when the learning-rate ratio $r = \eta_{QK}/\eta_{OV}$ is large. Because $\mu_{OV}$ drives loss decay and $\mu_{QK}$ drives attention concentration, the two quantities decouple: training loss can reach zero while attention remains near uniform (small $r$) or concentrates to near one (large $r$). The paper further proves that factorized training of either circuit induces a state-dependent preconditioned gradient flow on the collapsed parameters, so factorization acts as an implicit, time-varying rescale of that circuit's learning rate, and this is why collapsed and factorized models can achieve comparable loss with different attention trajectories.

Load-bearing premise

The quantitative theory depends on a synthetic idealization — class tokens mutually orthogonal, zero-mean background tokens orthogonal to all class tokens, attention scores forced to exactly two values, and both parameter matrices started at zero — conditions that real embeddings, layer normalization, and random initialization violate, so the exact log-squared law is guaranteed only in this simplified setting.

Editorial extensions

If this is right

  • With a large ratio $r = \eta_{QK}/\eta_{OV}$, attention mass on relevant tokens approaches one even though training loss decays at essentially the same rate as for small $r$, so final loss alone cannot reveal whether attention is concentrated.
  • Factorized and collapsed parameterizations of the same circuit are not equivalent under optimization: factorization acts as a state-dependent learning-rate rescale, so models with equal predictive performance can follow different attention trajectories.
  • Raising the query-key learning rate improves attention-based interpretability proxies — relevant-token attention, sufficiency, comprehensiveness, DTAP summary metrics — without architectural change and without degrading test performance.
  • The qualitative effect persists in deeper, multi-layer transformers, even though the closed-form dynamics are derived for a single layer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If relative circuit speed is the true driver, then any intervention that slows output-value learning — weight decay on $W_O$ and $W_V$, a smaller OV learning rate, or unbalanced initialization — should sharpen attention in the same way; this is a directly testable corollary the paper does not run.
  • The log versus log-squared growth asymmetry implies sharpening compounds with training time: attention should keep concentrating long after the loss has visually converged, so early stopping that watches only loss would silently discard attention structure.
  • Adaptive optimizers partially normalize away raw per-group learning-rate differences, which may be why the experiments need large QK multipliers (20x-100x); a principled alternative would be to scale the QK learning rate relative to the OV gradient scale rather than to a shared base rate.
  • The compensation story suggests a possible origin for the familiar diversity of attention patterns among equally performant transformers: not architecture alone, but the accidental effective ratio of query-key to output-value learning speeds during training.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper studies how the relative optimization speeds of the query-key and output-value circuits in a single-layer self-attention transformer shape attention concentration during next-token prediction. It compares collapsed (W_QK = W_K W_Q^T, W_OV = W_O W_V) and factorized parameterizations and proposes that factorization induces a state-dependent preconditioning that rescales the circuits' effective learning rates. In a synthetic orthogonal data model, the authors derive closed-form population gradient-flow ODEs (Theorem 1) and prove growth bounds (Lemma 3): mu_OV(t) = Theta(ln(1+t)) and mu_QK(t) = Theta((eta_QK/eta_OV)(ln(1+t))^2) in a pre-saturation regime, yielding attention mass on relevant tokens that increases with r = eta_QK/eta_OV. Experiments on synthetic data, SQuAD, HateXplain, and subject-verb agreement show that increasing the query-key learning-rate multiplier sharpens attention-based interpretability proxies while preserving predictive performance.

Significance. If the mechanism holds as stated, the paper offers a clean, falsifiable account of how training dynamics, rather than only the loss landscape, determine attention sharpening, and it gives practitioners a simple intervention (the QK/OV learning-rate ratio) for controlling attention concentration. The closed-form bounds in Lemma 3 are a useful addition to the dynamics-of-attention literature, and the paper ships code and reports confidence intervals. However, the quantitative theory is established for collapsed parameterizations only; the bridge to factorized parameterizations, which is a headline contribution, is asserted via Lemma 2 but never carried through the ODE analysis. This gap tempers the significance of the main theoretical claim.

major comments (3)
  1. [Section 6.1, Theorem 1; Appendix B] The proof that attention scores take exactly two values, u_j = mu_QK(t) for distinct tokens and u_j = 0 for common tokens, requires the query token x_T to satisfy <sum_tau s^tau_y, x_T> = 1 for the true class y and to be orthogonal to the other class sums. Assumptions 1-3 state nothing about the query-token distribution or normalization. Without such a condition, the formula alpha(t) = m exp(mu_QK(t)) / (m exp(mu_QK(t)) + n), which is used throughout Lemma 3, is not justified. Please state the required assumption on x_T explicitly or carry the factor <sum_tau s^tau_y, x_T> through the derivation.
  2. [Section 6, Lemmas 1-2 vs. Theorem 1 and Lemma 3] Theorem 1 and Lemma 3 analyze plain gradient flow on the collapsed parameters W_OV and W_QK. Lemma 2 (and Lemma 5 for QK) shows that factorized training induces the preconditioned flow d/dt vec(W_OV) = -Omega(W_OV) vec(nabla L_1(W_OV)) with a degree-1 homogeneous Omega. For the rank-one trajectories of Theorem 1, Omega(mu C_0) scales linearly in mu, so the factorized scalar ODE becomes d mu/dt = eta_F c mu G(mu), not d mu/dt = eta G(mu). The paper never integrates this mu-dependent rescaling into the coupled ODEs, so the explicit Theta((eta_QK/eta_OV) (ln(1+t))^2) growth and the ratio-r mechanism are not proven for the factorized parameterizations emphasized in the abstract and used in the Section 7 experiments. Either extend the derivation to the preconditioned flow or restrict the theoretical claims to collapsed circuits and present the factorized behavior as empirical.
  3. [Section 7, Table 1; Appendix A.2.2] The real-data experiments vary only the explicit query-key learning-rate multiplier (20x/30x/100x) in factorized models trained with AdamW, whereas Lemma 3 is a gradient-flow statement about the ratio eta_QK/eta_OV for collapsed parameters. No experiment isolates the parameterization-induced rescaling (for example, factorized versus collapsed with matched effective preconditioning), and no SGD comparison is reported for the real data. In addition, Appendix A.2.2 states that the SVA model is trained with SGD, while Section 7 states that all models are trained with AdamW. The paper should either add explicit validation of the factorization-induced rescaling or label the real-data results as qualitative evidence for the collapsed-circuit mechanism only.
minor comments (5)
  1. [Section 6.1, Assumption 1] Assumption 1 should be strengthened to state that all bM class-token vectors {s^tau_y} are mutually orthogonal and unit-norm; the proof of Theorem 1 uses orthogonality across both tau and classes simultaneously (Appendix B, Eq. (7)).
  2. [Lemma 2 / Appendix D.3] The notation "direct sum" (Kroneycker sum) for Omega(C) is used without a definition in the main text; the explicit definition (C^T C)^{1/2} \otimes I_M + I_d \otimes (C C^T)^{1/2} should appear where Omega is first introduced.
  3. [Figure 3 and Figure 4 captions] The Figure 3 captions contain apparent missing symbols (for example, "Lt( )"), and the Figure 4(b) caption contains a typo in "mu_OV (t)evolution".
  4. [Section 7] The statement that adaptive optimizers "treat all parameter groups equally" is imprecise: Adam's per-parameter normalization interacts with a learning-rate multiplier, so the effective ratio r under AdamW is not guaranteed to equal the nominal multiplier.
  5. [Section 4 / Appendix A.2.2] The stated vocabulary size of 50 tokens appears inconsistent with 4 classes x 10 distinct tokens + 10 common tokens + 4 query/next tokens = 54 tokens; please clarify whether query and next tokens are drawn from the distinct or common sets.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the sharpening mechanism is derived from the stated ODEs, not assumed or fitted.

full rationale

The central derivation is self-contained. Theorem 1 obtains the scalar ODEs for mu_OV and mu_QK from the population gradient of the cross-entropy loss under Assumptions 1-3; the fixed-direction trajectory ansatz is verified by substitution using orthogonality and the zero-mean background, rather than being an equivalent restatement of the conclusion. Lemma 3 then proves the Theta(log(1+t)) and Theta((eta_QK/eta_OV) log^2(1+t)) growth rates from these ODEs via explicit comparison inequalities and integration, and the attention mass alpha(t)=m exp(mu_QK)/(m exp(mu_QK)+n) is a monotone transform of the solved variable, not an additional fitted input. The factorized-circuit results (Proposition 1, Lemmas 1-2) are proved from the chain rule and balancedness, and the paper does not use those lemmas to define the collapsed ODEs; whether the log^2 mechanism persists under the factorized preconditioner is a correctness/generalization concern, not a circular one. The real-data experiments intervene on the explicit query-key learning rate and report attention metrics that are not fed back into the theory, so no fitted parameter is renamed as a prediction. The only self-citations (Vashisht & Ramaswamy 2023; Pandey et al. 2023) appear in related-work context and carry no load-bearing weight for the derivation. No circular step is identifiable in the manuscript.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The mathematical derivation adds no fitted constants and makes no use of the experimental QK multipliers. It depends on an idealized orthogonal data model and balanced zero initialization, which are strong structural assumptions rather than fitted numbers. The only hand-selected experimental quantity relevant to the empirical claim is the per-dataset QK multiplier, and the DTAP binning thresholds are author-defined metric choices that do not enter the theory.

free parameters (1)
  • per-dataset query-key learning rate multiplier (r = η_QK / η_OV) = 20x (SQuAD), 30x (HateXplain), 100x (SVA)
    Chosen by validation-based hyperparameter tuning from {5x, 10x, 20x, 30x, 100x} according to Appendix A.2. The theory predicts the qualitative effect for any r greater than 1, but not the optimal magnitude, so this is a hand-chosen experimental quantity rather than a fitted constant in the derivation.
assumptions (5)
  • standard math Deep linear network balancedness and preconditioning theorems (Arora et al. 2018; Du et al. 2018) carry over to the factorized OV and QK circuits.
    Used in Lemmas 1, 2, 4, and 5 to relate factorized gradient flow to collapsed parameters through the degree-1 homogeneous operator Omega.
  • domain assumption Assumptions 1-3 (Section 6.1): class-specific token vectors are mutually orthogonal, background tokens are zero-mean, and their support is orthogonal to all class vectors.
    These assumptions eliminate cross-token interactions and make the population gradient reduce to the scalar ODEs in Theorem 1. Real embeddings and data violate them.
  • ad hoc to paper Zero initialization W_OV(0)=W_QK(0)=0 in Theorem 1.
    The closed-form trajectories start from the balanced, symmetric regime; standard training uses random initialization, so the theorem applies only to this controlled setting.
  • domain assumption Population full-batch gradient flow rather than stochastic or adaptive training.
    The theorem and Lemma 3 analyze population gradient flow; experiments use SGD and AdamW, so the quantitative growth rates are not directly testable in real experiments.
  • ad hoc to paper The synthetic data generation with distinct and common tokens yields attention scores taking only two values under the trajectory ansatz.
    This is the key modeling shortcut that closes the ODE system; it is purpose-built for the analysis.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Faster Query-Key Learning Sharpens Attention in Self-Attention Models." pith.science (2026). https://pith.science/paper/23BX4TTJ

@misc{pith2026260806776,
  author       = {Pith},
  title        = {Pith review of: Faster Query-Key Learning Sharpens Attention in Self-Attention Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/23BX4TTJ}},
  note         = {Machine review of arXiv:2608.06776}
}
read the original abstract

A standard self-attention layer consists of two interacting circuits: the query-key circuit that governs attention allocation, and the output-value circuit that maps attended representations to predictions. Collapsed and factorized parameterizations of the query-key and output-value circuits lead to qualitatively different attention patterns. In particular, some parameterizations give sharper attention to task-relevant tokens, at a similar training loss. We analyze how the parameterizations of these circuits shape the parameter trajectories in single-layer self-attention models trained for next-token prediction. Through gradient-flow analysis, we show that factorization induces implicit rescaling of the two circuits' learning rates. We derive closed-form dynamics showing that output-value and query-key parameters move along a line, with relative speeds determined by their learning rates. Faster query-key learning relative to output-value learning thus produces sharper attention, as the model compensates for slower output-value learning by increasing attention mass on relevant tokens. Experiments show that differences in the relative learning rates of the two circuits govern attention concentration. This improves attention interpretability proxies while maintaining comparable predictive performance.

Figures

Figures reproduced from arXiv: 2608.06776 by the authors.

Figure 1
Figure 1. Overall Setting: (a) A sequence X with contextual tokens and query token{x1, x2, . . . , xT −1, xT } is fed into 1− layer transformer to predict the next token y = xT +1 (b) Synthetic data illustration, For each sequence class there exists a set of tokens specific to that class, called distinct tokens and a set of tokens common to all sequence classes called common tokens. For example ‘the’ ‘sat’ ‘ate’ and ‘on’ are … view at source ↗
Figure 2
Figure 2. Distinct Token Attention-Prediction heat map of test data for different parameterizations under SGD with same learning rates (Top-Row) and Faster Learning for Query-Key (Bottom-Row). All models are trained until the negative log-likelihood is close to zero (≤ 0.001). Here, we train the model for 5 random seeds and report the average, refer to appendix for standard deviation in the heatmaps. an input sequence, where … view at source ↗
Figure 3
Figure 3. (a) and (b) Evolution of the training loss for fixed values of α under gradient descent. (c) and (d) show improvement incentive curves, measuring the change in loss resulting from a small increase δ = 0.005 in α. output–value circuit. We define the fixed-attention loss curve as the training loss evaluated under gradient descent while holding the attention mass α fixed for distinct to￾kens. To quantify the incentive … view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Population gradient-flow simulations for the orthogonal synthetic setting of Theorem 1 with m = 5, n = 50, b = 50, M = 5, varying the ratio r = ηQK/ηOV . (a) Training loss vs. normalized time, (b) µOV (t) at matched stopping loss, (d) µQK(t) at matched stopping loss, a…
Figure 5
Figure 5. Figure 5: 6 layer model Distinct Token Attention-Prediction heat map of SQuAD QA Data (a)(b) Baseline Model Average and Standard Deviation (c)(d) FasterQK Model Average and Standard Deviation (5 runs) 0 0.1 0.5 1.0 Distinct token attention 1 0.5 0.1 0 True Token Probability 52.0…
Figure 6
Figure 6. Figure 6: 1 layer model Distinct Token Attention-Prediction heat map of SVA data (a)(b) Baseline Model Average and Standard Deviation (c)(d) Faster QK Model Average and Standard Deviation (5 runs) 0 0.33 0.66 1 Distinct token attention 1 0.66 0.33 0 True Token Probability 49.04 …
Figure 7
Figure 7. Figure 7: 1 layer model Distinct Token Attention-Prediction heat map of HateXplain data (a)(b) Baseline Model Average and Standard Deviation (c)(d) Faster QK Model Average and Standard Deviation (5 runs) Comprehensiveness = f(x) − f(x/r:k%) where r:k% refers to top-k% most impor…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Power law graph attention: exact generalization of scaled dot-product attention, empirical collapse at inference

    cs.LG 2026-08 conditional novelty 6.0 of 10 full

    After training, a 110M-parameter power-law attention model's learned scoring operator becomes nearly input-invariant, so inference can cache it; the paper proves this collapse conditionally, measures it at 1e-6 and be...

Reference graph

Works this paper leans on

68 extracted references · 42 canonical work pages · cited by 1 Pith paper

  1. [2]

    On the optimization of deep networks: Implicit acceleration by overparameterization

    Arora, S., Cohen, N., and Hazan, E. On the optimization of deep networks: Implicit acceleration by overparameterization. ArXiv, abs/1802.06509, 2018. URL https://api.semanticscholar.org/CorpusID:3331630

    arXiv 2018

  2. [3]

    Self-attention networks localize when qk-eigenspectrum concentrates

    Bao, H., Hataya, R., and Karakida, R. Self-attention networks localize when qk-eigenspectrum concentrates. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2024

    work page 2024

  3. [6]

    Birth of a transformer: a memory viewpoint

    Bietti, A., Cabannes, V., Bouchacourt, D., J\' e gou, H., and Bottou, L. Birth of a transformer: a memory viewpoint. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc

    work page 2023

  4. [7]

    Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, ...

    arXiv 2020

  5. [8]

    Universal transformers

    Dehghani, M., Gouws, S., Vinyals, O., Uszkoreit, J., and Kaiser, L. Universal transformers. ArXiv, abs/1807.03819, 2018. URL https://api.semanticscholar.org/CorpusID:49667762

    arXiv 2018

  6. [9]

    On the optimization and generalization of multi-head attention

    Deora, P., Ghaderi, R., Taheri, H., and Thrampoulidis, C. On the optimization and generalization of multi-head attention. Transactions on Machine Learning Research, 2024 a . ISSN 2835-8856. URL https://openreview.net/forum?id=wTGjn7JvYK

    work page 2024

  7. [10]

    On the optimization and generalization of multi-head attention, 2024 b

    Deora, P., Ghaderi, R., Taheri, H., and Thrampoulidis, C. On the optimization and generalization of multi-head attention, 2024 b . URL https://arxiv.org/abs/2310.12680

    arXiv 2024

  8. [12]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. ArXiv, abs/2010.11929, 2020. URL https://api.semanticscholar.org/CorpusID:225039882

    arXiv 2010

Show all 68 references
  1. [13]

    S., Hu, W., and Lee, J

    Du, S. S., Hu, W., and Lee, J. D. Algorithmic regularization in learning deep homogeneous models: layers are automatically balanced. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, pp.\ 382–393, Red Hook, NY, USA, 2018. Cu...

    2018

  2. [14]

    A mathematical framework for transformer circuits

    Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., DasSarma, N., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan...

    2021

  3. [15]

    The emergence of clusters in self-attention dynamics

    Geshkovski, B., Letrouit, C., Polyanskiy, Y., and Rigollet, P. The emergence of clusters in self-attention dynamics. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc

    2023

  4. [16]

    and Wallace, B

    Jain, S. and Wallace, B. C. Attention is not explanation. In North American Chapter of the Association for Computational Linguistics, 2019. URL https://api.semanticscholar.org/CorpusID:67855860

    2019

  5. [17]

    Clustering in causal attention masking

    Karagodin, N., Polyanskiy, Y., and Rigollet, P. Clustering in causal attention masking. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS '24, Red Hook, NY, USA, 2024. Curran Associates Inc. ISBN 9798331314385

    2024

  6. [18]

    Transformers in speech processing: A survey

    Latif, S., Zaidi, A., Cuay \'a huitl, H., Shamshad, F., Shoukat, M., and Qadir, J. Transformers in speech processing: A survey. ArXiv, abs/2303.11607, 2023. URL https://api.semanticscholar.org/CorpusID:257636830

    2023 arXiv

  7. [19]

    How do transformers learn topic structure: towards a mechanistic understanding

    Li, Y., Li, Y., and Risteski, A. How do transformers learn topic structure: towards a mechanistic understanding. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023

    2023

  8. [20]

    E., Rawat, A

    Li, Y., Huang, Y., Ildiz, M. E., Rawat, A. S., and Oymak, S. Mechanics of next token prediction with self-attention. ArXiv, abs/2403.08081, 2024. URL https://api.semanticscholar.org/CorpusID:268379753

    2024 arXiv

  9. [22]

    On the dynamics of training attention models

    Lu, H., Mao, Y., and Nayak, A. On the dynamics of training attention models. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=1OCTOShAmqB

    2021

  10. [23]

    M., Biemann, C., Goyal, P., and Mukherjee, A

    Mathew, B., Saha, P., Yimam, S. M., Biemann, C., Goyal, P., and Mukherjee, A. Hatexplain: A benchmark dataset for explainable hate speech detection. In AAAI Conference on Artificial Intelligence, 2020. URL https://api.semanticscholar.org/CorpusID:229332119

    2020

  11. [24]

    In-context learning and induction heads

    Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Johnston, S., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Cla...

    2022

  12. [25]

    N., Vashisht, R., and Ramaswamy, H

    Pandey, L. N., Vashisht, R., and Ramaswamy, H. G. On the interpretability of attention networks. In Khan, E. and Gonen, M. (eds.), Proceedings of The 14th Asian Conference on Machine Learning, volume 189 of Proceedings of Machine Learning Research, pp.\ 832--847. PMLR, 12--14 ...

    2023

  13. [26]

    Attention is turing complete

    P\' e rez, J., Barcel\' o , P., and Marinkovic, J. Attention is turing complete. J. Mach. Learn. Res., 22 0 (1), January 2021. ISSN 1532-4435

    2021

  14. [27]

    Improving language understanding by generative pre-training

    Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. Improving language understanding by generative pre-training. 2018

    2018

  15. [30]

    Scan and snap: understanding training dynamics and token composition in 1-layer transformer

    Tian, Y., Wang, Y., Chen, B., and Du, S. Scan and snap: understanding training dynamics and token composition in 1-layer transformer. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc

    2023

  16. [31]

    and Ramaswamy, H

    Vashisht, R. and Ramaswamy, H. G. On the learning dynamics of attention networks. In European Conference on Artificial Intelligence, 2023. URL https://api.semanticscholar.org/CorpusID:260155204

    2023

  17. [32]

    N., Kaiser, L

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. Attention is all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Informatio...

    2017

  18. [34]

    S., Reddi, S

    Yun, C., Bhojanapalli, S., Rawat, A. S., Reddi, S. J., and Kumar, S. Are transformers universal approximators of sequence-to-sequence functions? ArXiv, abs/1912.10077, 2019. URL https://api.semanticscholar.org/CorpusID:209444410

    1912 arXiv

  19. [35]

    Attention is All you Need , url =

    Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N and Kaiser, ukasz and Polosukhin, Illia , biburl =. Attention is All you Need , url =. Advances in Neural Information Processing Systems , editor =

  20. [36]

    ArXiv , year=

    Transformers in Speech Processing: A Survey , author=. ArXiv , year=

  21. [37]

    and Kumar, Sanjiv , biburl =

    Yun, Chulhee and Bhojanapalli, Srinadh and Rawat, Ankit Singh and Reddi, Sashank J. and Kumar, Sanjiv , biburl =. Are Transformers universal approximators of sequence-to-sequence functions? , url =. ICLR , crossref =

  22. [38]

    On the Computational Power of Transformers and Its Implications in Sequence Modeling

    Bhattamishra, Satwik and Patel, Arkil and Goyal, Navin. On the Computational Power of Transformers and Its Implications in Sequence Modeling. Proceedings of the 24th Conference on Computational Natural Language Learning. 2020. doi:10.18653/v1/2020.conll-1.37

    2020 doi

  23. [39]

    On the A bility and L imitations of T ransformers to R ecognize F ormal L anguages

    Bhattamishra, Satwik and Ahuja, Kabir and Goyal, Navin. On the A bility and L imitations of T ransformers to R ecognize F ormal L anguages. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020. doi:10.18653/v1/2020.emnlp-main.576

    2020 doi

  24. [40]

    ArXiv , year=

    Universal Transformers , author=. ArXiv , year=

  25. [41]

    Attention is turing complete , year =

    P\'. Attention is turing complete , year =. J. Mach. Learn. Res. , month = jan, articleno =

  26. [42]

    and Ba, Jimmy , biburl =

    Kingma, Diederik P. and Ba, Jimmy , biburl =. Adam: A Method for Stochastic Optimization. , url =. ICLR (Poster) , crossref =

  27. [43]

    Radford, Alec and Narasimhan, Karthik and Salimans, Tim and Sutskever, Ilya , biburl =

  28. [44]

    2021 , journal=

    A Mathematical Framework for Transformer Circuits , author=. 2021 , journal=

    2021

  29. [45]

    In-context Learning and Induction Heads

    Olsson, Catherine and Elhage, Nelson and Nanda, Neel and Joseph, Nicholas and DasSarma, Nova and Henighan, Tom and Mann, Ben and Askell, Amanda and Bai, Yuntao and Chen, Anna and Conerly, Tom and Drain, Dawn and Ganguli, Deep and Hatfield-Dodds, Zac and Hernandez, Danny and Jo...

  30. [46]

    Birth of a transformer: a memory viewpoint , year =

    Bietti, Alberto and Cabannes, Vivien and Bouchacourt, Diane and J\'. Birth of a transformer: a memory viewpoint , year =. Proceedings of the 37th International Conference on Neural Information Processing Systems , articleno =

  31. [47]

    SQ u AD : 100,000+ Questions for Machine Comprehension of Text

    Rajpurkar, Pranav and Zhang, Jian and Lopyrev, Konstantin and Liang, Percy. SQ u AD : 100,000+ Questions for Machine Comprehension of Text. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. 2016. doi:10.18653/v1/D16-1264

    2016 doi

  32. [48]

    Assessing the Ability of LSTM s to Learn Syntax-Sensitive Dependencies

    Linzen, Tal and Dupoux, Emmanuel and Goldberg, Yoav. Assessing the Ability of LSTM s to Learn Syntax-Sensitive Dependencies. Transactions of the Association for Computational Linguistics. 2016. doi:10.1162/tacl_a_00115

    2016 doi

  33. [49]

    AAAI Conference on Artificial Intelligence , year=

    HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection , author=. AAAI Conference on Artificial Intelligence , year=

  34. [50]

    Proceedings of the 37th International Conference on Neural Information Processing Systems , articleno =

    Tian, Yuandong and Wang, Yiping and Chen, Beidi and Du, Simon , title =. Proceedings of the 37th International Conference on Neural Information Processing Systems , articleno =. 2023 , publisher =

    2023

  35. [51]

    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , url =

    Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina , biburl =. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , url =

  36. [52]

    Brown, Tom B. and Mann, Benjamin and Ryder, Nick and Subbiah, Melanie and Kaplan, Jared and Dhariwal, Prafulla and Neelakantan, Arvind and Shyam, Pranav and Sastry, Girish and Askell, Amanda and Agarwal, Sandhini and Herbert-Voss, Ariel and Krueger, Gretchen and Henighan, Tom ...

    doi arXiv

  37. [53]

    ArXiv , year=

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale , author=. ArXiv , year=

  38. [54]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , month =

    Liu, Ze and Hu, Han and Lin, Yutong and Yao, Zhuliang and Xie, Zhenda and Wei, Yixuan and Ning, Jia and Cao, Yue and Zhang, Zheng and Dong, Li and Wei, Furu and Guo, Baining , title =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , m...

    2022

  39. [55]

    ArXiv , year=

    Are Transformers universal approximators of sequence-to-sequence functions? , author=. ArXiv , year=

  40. [56]

    and Hu, Wei and Lee, Jason D

    Du, Simon S. and Hu, Wei and Lee, Jason D. , title =. Proceedings of the 32nd International Conference on Neural Information Processing Systems , pages =. 2018 , publisher =

    2018

  41. [57]

    2022 , journal=

    In-context Learning and Induction Heads , author=. 2022 , journal=

    2022

  42. [58]

    ArXiv , year=

    On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization , author=. ArXiv , year=

  43. [59]

    2025 , eprint=

    From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks , author=. 2025 , eprint=

    2025

  44. [60]

    Attention is not not Explanation

    Wiegreffe, Sarah and Pinter, Yuval. Attention is not not Explanation. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019. doi:10.18653/v1/D19-1002

    2019 doi

  45. [61]

    North American Chapter of the Association for Computational Linguistics , year=

    Attention is not Explanation , author=. North American Chapter of the Association for Computational Linguistics , year=

  46. [62]

    Is Attention Interpretable?

    Serrano, Sofia and Smith, Noah A. Is Attention Interpretable?. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019. doi:10.18653/v1/P19-1282

    2019 doi

  47. [63]

    2024 , eprint=

    On the Optimization and Generalization of Multi-head Attention , author=. 2024 , eprint=

    2024

  48. [64]

    ArXiv , year=

    Mechanics of Next Token Prediction with Self-Attention , author=. ArXiv , year=

  49. [65]

    Proceedings of the 41st International Conference on Machine Learning , articleno =

    Bao, Han and Hataya, Ryuichiro and Karakida, Ryo , title =. Proceedings of the 41st International Conference on Machine Learning , articleno =. 2024 , publisher =

    2024

  50. [66]

    Transactions on Machine Learning Research , issn=

    On the Optimization and Generalization of Multi-head Attention , author=. Transactions on Machine Learning Research , issn=. 2024 , url=

    2024

  51. [67]

    Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence , articleno =

    Yao, Xinhao and Qian, Hongjin and Hu, Xiaolin and Xu, Gengze and Liu, Wei and Luan, Jian and Wang, Bin and Liu, Yong , title =. Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence , articleno =. 2025 , isbn =. doi:10.24963/ijcai.2025/760 ...

    2025 doi

  52. [68]

    International Conference on Learning Representations , year=

    On the Dynamics of Training Attention Models , author=. International Conference on Learning Representations , year=

  53. [69]

    Vashisht, Rahul and Ramaswamy, Harish G. , year=. On the Learning Dynamics of Attention Networks , ISBN=. doi:10.3233/faia230541 , booktitle=

    doi

  54. [70]

    Proceedings of the 37th International Conference on Neural Information Processing Systems , articleno =

    Geshkovski, Borjan and Letrouit, Cyril and Polyanskiy, Yury and Rigollet, Philippe , title =. Proceedings of the 37th International Conference on Neural Information Processing Systems , articleno =. 2023 , publisher =

    2023

  55. [71]

    Proceedings of the 38th International Conference on Neural Information Processing Systems , articleno =

    Karagodin, Nikita and Polyanskiy, Yury and Rigollet, Philippe , title =. Proceedings of the 38th International Conference on Neural Information Processing Systems , articleno =. 2024 , isbn =

    2024

  56. [72]

    Proceedings of the 40th International Conference on Machine Learning , articleno =

    Li, Yuchen and Li, Yuanzhi and Risteski, Andrej , title =. Proceedings of the 40th International Conference on Machine Learning , articleno =. 2023 , publisher =

    2023

  57. [73]

    Quantifying Attention Flow in Transformers

    Abnar, Samira and Zuidema, Willem. Quantifying Attention Flow in Transformers. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020. doi:10.18653/v1/2020.acl-main.385

    2020 doi

  58. [74]

    ERASER : A Benchmark to Evaluate Rationalized NLP Models

    DeYoung, Jay and Jain, Sarthak and Rajani, Nazneen Fatema and Lehman, Eric and Xiong, Caiming and Socher, Richard and Wallace, Byron C. ERASER : A Benchmark to Evaluate Rationalized NLP Models. Proceedings of the 58th Annual Meeting of the Association for Computational Linguis...

    2020 doi

  59. [75]

    Proceedings of The 14th Asian Conference on Machine Learning , pages =

    On the Interpretability of Attention Networks , author =. Proceedings of The 14th Asian Conference on Machine Learning , pages =. 2023 , editor =

    2023

  60. [76]

    European Conference on Artificial Intelligence , year=

    On the Learning Dynamics of Attention Networks , author=. European Conference on Artificial Intelligence , year=

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.