Pith. sign in

REVIEW 3 major objections 4 minor 5 cited by

Training Dynamics of In-Context Learning in Linear Attention

T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Training a linear attention model with separate key and query matrices acquires in-context linear regression progressively, one principal component at a time, while the merged parametrization acquires it in a single abrupt drop.

desk verdict The merged key/query case is a clean, rigorous result with parameter-free predictions; the separate key/query headline claim about progressive PCR rests on an empirically supported but unproven single-active-head ansatz, and the paper's own text is honest about this. read the letter →

arxiv 2501.16265 v2 pith:VLFIUOWS submitted 2025-01-27 cs.LG

Yedi Zhang , Aaditya K. Singh , Peter E. Latham , Andrew Saxe This is my paper · ORCID
classification cs.LG MSC 68T07
keywords in-contextlearninglinearattentiongradientflowtrainingdynamicsprincipalcomponentregressionsaddle-to-saddlekey-queryparametrizationlosslandscape
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Gradient descent training of a multi-head linear self-attention layer on in-context linear regression has a tractable full time course, and the paper's thesis is that the shape of that time course is set by parametrization. With the key and query matrices merged into one matrix, the loss landscape has exactly two fixed-point manifolds and the training loss drops once, abruptly, with an explicit sigmoidal time-course when the token covariance is white. With separate key and query matrices, the parametrization closer to real transformers, the landscape has $2^D$ fixed points and the loss undergoes $D$ staged drops, each corresponding to the model learning one more principal component of the input covariance. The load-bearing interpretation is that during the $(m+1)$-th plateau the model is approximately performing in-context principal component regression with $m$ components, so early stopping yields a concrete, interpretable algorithm rather than an arbitrary partially trained network. If correct, this gives a theoretical reason why in-context learning can appear either abruptly or gradually depending on architectural choices, and connects ICL emergence to linear-network saddle-to-saddle dynamics.

What carries the argument

The object that carries the argument is the cubic feature $z(X)=\mathrm{vec}(\beta x_q^\top)$, built from the in-context input-output correlation $\beta=\frac{1}{N}\sum_n y_n x_n$ and the query token $x_q$. Rewriting each attention head in terms of $z$ turns merged key/query attention into a two-layer fully-connected linear network with weights $w_2$, $W_1$, and separate key/query attention into a sum of three-layer convolutional linear networks, so existing linear-network tools, including conservation laws, saddle escape times, and alignment behavior, transfer to attention. The separate case is then closed by the alignment ansatz: during the $(m+1)$-th plateau, already-learned heads satisfy $k_i=q_i=v_i e_i$, the next head satisfies $k_{m+1}=q_{m+1}=v_{m+1}(t) e_{m+1}$, and all other heads are frozen. Substituting this ansatz collapses the high-dimensional gradient flow to the scalar ODE $\tau \dot v = \lambda_{m+1}^2 v^2 - \lambda_{m+1} a_{m+1} v^5$, whose solution time gives the plateau durations and the order of principal-component acquisition.

What would settle it

Train separate rank-one key/query linear attention on in-context linear regression with distinct covariance eigenvalues and small initialization, then measure during each loss plateau the principal angle between the active head's key vector $k_i$ and the predicted eigenvector $e_{m+1}$, and the norm ratio $\|k_i\|/v_i$; if the angle does not approach zero before the loss drop or the norm ratio moves away from 1, the ansatz and the scalar ODE do not describe the true gradient flow.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that gradient flow on multi-head linear self-attention for in-context linear regression is solvable in the rich, small-initialization regime, and the solution differs qualitatively by parametrization. For merged key and query (ATTNM), the prediction is $\sum_i v_i \beta^\top U_i x_q$, which is exactly a two-layer fully-connected linear network acting on the cubic feature $z = \mathrm{vec}(\beta x_q^\top)$; this system has exactly two fixed-point manifolds, the zero point and the global-minimum manifold, and the loss trajectory is a single plateau followed by one abrupt drop, with an analytic solution in the white-covariance case. For separate rank-one key and query (ATTNS), the prediction is a sum of three-layer convolutional linear networks on the same feature, the function space contains $2^D$ fixed points, and small-initialization training visits $D+1$ of them, moving from $M_0$ to $M_1$ to ... to $M_D$ as heads align with successive eigenvectors of the input covariance $\Lambda$. The paper reduces this high-dimensional dynamics to scalar ordinary differential equations, and shows the loss at each plateau is $\mathrm{tr}(\Lambda) - \sum_{d=1}^m \lambda_d(1+(1+\mathrm{tr}(\Lambda)/\lambda_d)/N)^{-1}$, so each drop removes roughly the next eigenvalue. At plateau $m$, the implemented map is $\beta^\top \sum_{d=1}^m \lambda_d^{-1}(1+(1+\mathrm{tr}(\Lambda)/\lambda_d)/N)^{-1} e_d e_d^\top x_q$, which is in-context principal component regression with $m$ components; at convergence this becomes least squares. The paper also reports that softmax attention shows the same single-drop versus multi-drop qualitative difference.

Load-bearing premise

Everything about the staged, one-principal-component-at-a-time dynamics for separate key/query rests on the posited alignment ansatz, that at each plateau the active head's key and query vectors exactly match the next eigenvector with equal norms while all other heads stay frozen, which the paper validates by simulation rather than deriving from the gradient flow.

Editorial extensions

If this is right

  • With separate key/query, an early-stopped model is not an arbitrary partial solution: stopping during the $(m+1)$-th plateau yields a network whose in-context map is approximately principal component regression with the first $m$ components, converging exactly as $N \to \infty$.
  • Parametrization, not the task, controls the emergence shape: the same in-context linear regression task shows one abrupt drop under merged key/query and $D$ staged drops under separate key/query, so experiments using merged key/query may overstate abrupt emergence.
  • The key/query rank controls the speed of intermediate stages: with rank-$R$ key and query, conspicuous plateaus appear only when $m$ divides $R$, so increasing the rank shortens or removes intermediate plateaus.
  • The equivalence to linear networks imports a toolbox: conservation laws and convergence guarantees for deep linear networks apply directly to linear attention on this task, predicting weight alignment such as $\|k_i\| \approx \|q_i\| \approx v_i$ throughout training.
  • Softmax attention shows the same qualitative contrast in simulations, suggesting the single-drop versus multi-drop distinction extends beyond the linearized setting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Testable extension: measure the number of sharp loss drops in a real transformer as a function of key/query rank on a controlled regression task; the theory predicts the number of conspicuous drops shrinks as the rank grows.
  • The progressive-PCR picture offers a mechanistic hypothesis for natural language: if separate key/query attention is the realistic parametrization, then emergent in-context learning in language models may be a sequence of component-wise algorithm refinements rather than a single phase transition, and the timing of each stage should track the eigenvalues of the effective input covariance.
  • If the alignment ansatz holds only approximately for softmax attention, the scalar-ODE prediction for plateau durations still gives a quantitative target: one could compare measured loss-drop times against the predicted scaling with the inverse square of the next eigenvalue.
  • The equivalence-based method suggests that other polynomial features beyond the cubic one generate analogous solvable attention dynamics, so in-context algorithms other than linear regression may admit similar staged PCR-like descriptions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper studies the gradient-flow training dynamics of multi-head linear self-attention on in-context linear regression, comparing two parametrizations: merged key/query (ATTNM) and separate key/query (ATTNS). For ATTNM, the authors prove an exact equivalence to a two-layer fully-connected linear network acting on a cubic feature, characterize two fixed-point manifolds, derive an analytic time-course solution for white input covariance, and show that the converged model approximates in-context least-squares regression. For ATTNS, they establish an equivalence to a sum of three-layer convolutional linear networks, identify 2^D fixed points in function space (proving validity but not completeness), and use an alignment ansatz to reduce the high-dimensional dynamics to a scalar ODE that predicts D plateau-and-drop stages; early-stopped models are interpreted as implementing in-context principal component regression with an increasing number of components. The paper also reports analogous qualitative behavior in softmax attention and in low-rank extensions, and provides open-source code for the main experiments.

Significance. If the main claims hold, this is a valuable contribution to the theory of in-context learning dynamics. The merged key/query analysis is rigorous and connects linear attention to well-understood linear-network dynamics, yielding an explicit time-course (Eq. 10). For the separate key/query model, the plateau loss values (Eq. 19) and the scalar ODE (Eq. 21) are parameter-free predictions from the data statistics (Λ, N) and are compared against simulations without fitted constants, which is a real strength. The paper is also transparent about its main assumption, explicitly labeling the alignment structure as an ansatz. However, the central claim that ATTNS exhibits saddle-to-saddle dynamics with exactly one new principal component per drop is not fully established, because the ansatz is posited rather than derived and the unstable manifold around each saddle is degenerate. The theoretical characterization is therefore only partially supported.

major comments (3)
  1. [§4.3, Eq. (20) and Appendix E.6.2] The reduction of the separate key/query dynamics to the scalar ODE (21) rests on the alignment ansatz that during the (m+1)-th plateau exactly one new head grows along e_{m+1} while all other silent heads remain frozen. The paper itself labels Eq. (20) as an ansatz, and the validation is empirical. This gap is load-bearing: Eq. (74) shows that the unstable manifold around M_m is degenerate, in the sense that every silent head i ≥ m+1 has the same leading eigenvalue λ_{m+1}^2 with eigenvector e_{m+1}, so small random initialization gives all H−m silent heads comparable overlap with the fastest direction. Nothing in the gradient flow or the conservation law (83) obviously selects a single head. If several heads grow together, Eq. (21) is not the correct reduced dynamics and the claims about plateau-escape times, loss drops, and the progressive-PCR interpretation need modification. Please derive the single-head selection mechanism (e.g., a generic perturbation argument showing that relative amplitudes along the degenerate unstable manifold separate and one head dominates), or explicitly state the saddle-to-saddle description as a conjecture supported by simulations rather than a theorem.
  2. [§4.2 and Appendix E.3] The text states that the gradient-flow dynamics in Eq. (14) has 2^D fixed points in function space, but the proof in Appendix E.3 only establishes that the listed configurations are fixed points; it does not rule out other fixed points that do not satisfy conditions (C1)-(C3). Since the sharp contrast between 'two fixed points' for ATTNM and 'exponentially many fixed points' for ATTNS is a stated contribution, the count needs a completeness argument. If the intended claim is only that at least 2^D such fixed points exist, the wording should be changed accordingly; otherwise a proof of exhaustiveness is required.
  3. [§5 and Appendix F.4] The rank-R result that conspicuous plateaus occur exactly when m divides R (with brief or no plateau otherwise) is supported by simulations in Figures 4, 11, and 12 and by a heuristic explanation in Appendix F.4, but no formal derivation or theorem statement is given. Because this is presented as a crisp property of the parametrization, the claim should either be proved (or at least stated as a conjecture with the precise hypotheses under which it is expected to hold). The current text blends heuristic and empirical evidence into a categorical statement, which overstates the theoretical status of this part of the paper.
minor comments (4)
  1. [Figure 3 caption] The caption ends with 'A video of the dynamics is provided at URL', which appears to be a placeholder that should be replaced with the actual link or removed.
  2. [References] The reference to 'V on Oswald' appears in several places (e.g., Section 1 and the bibliography) and should read 'von Oswald'.
  3. [Eq. (20)] The head index in Eq. (20) is used both as an eigenvector label and as a permuted head index; the text notes the permutation in a footnote, but this should be stated more prominently to avoid confusion when reading Eq. (21) and the subsequent discussion.
  4. [Section 4.2] The phrase '2D fixed points' in the running text is potentially ambiguous because it can be read as '2 times D'; using '2^D' consistently in all prose (as done in the displayed definitions) would improve clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: plateau losses and PCR algorithms follow from the gradient-flow equations and data statistics, and the single-active-head ansatz is transparently stated as an ansatz rather than a re-imported conclusion.

full rationale

The paper's load-bearing quantitative claims do not reduce to their inputs by construction. For the merged parametrization, the fixed points (Eq. 9) and time-course (Eq. 10) follow from the algebraic equivalence to a two-layer linear network (Eqs. 6-7) plus the white-covariance dynamics solved in Appendix D.5; the only externally invoked ingredient is the standard deep-linear alignment theorem of Saxe et al. (2014) and Atanasov et al. (2022), a parameter-free mathematical result whose assumptions do not include the paper's target conclusion, so this self-citation is not circular under the stated rules. For the separate parametrization, the plateau losses (Eq. 19), fixed-point count (Eq. 16), and in-context PCR interpretation (Eq. 22) are computed from the stationary conditions (C1-C3) and the data statistic E[Lambda-hat^2] (Eqs. 15, 31-32), then compared with simulations; no fitted constants are renamed as predictions. The scalar reduction to Eq. (21) does rest on the single-active-head alignment ansatz (Eq. 20), which the paper explicitly proposes "based on simulations" rather than deriving, and the linearized dynamics (Eq. 74) show that all silent heads share the same leading eigenvalue/eigenvector, so the ansatz is a genuine gap. However, an unproved but explicitly labeled ansatz is an assumption, not a circular derivation: the paper does not define the fixed point in terms of the plateau loss or fit the ODE to the loss curve. Self-citations to Singh et al. and Saxe et al. are contextual or mathematical support, not uniqueness constraints forcing the paper's choices. Accordingly, no circular step is exhibited, and the correct circularity finding is no significant circularity (score 0).

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

No fitted constants appear in the central predictions: plateau losses (Eq. 19) and the time-course (Eq. 10) are functions of the known Λ, N, and initialization scale. The main load-bearing input is the alignment ansatz, which is inferred from simulations. The data distribution assumptions set the regime of validity.

assumptions (6)
  • domain assumption Task vectors w are sampled i.i.d. from N(0, I) for each training sequence, independently of x tokens.
    Used in Appendix D.1 to show that value and key-query offset blocks remain zero. If w were not zero-mean, in-weight learning would mix into the dynamics (Appendix G).
  • domain assumption Input tokens x_n and query x_q are i.i.d. Gaussian N(0, Λ) with fixed covariance Λ.
    Used to compute E[z z^T], E[Λ̂^2], and plateau losses (Appendix C.1, E.4).
  • domain assumption Λ has distinct eigenvalues λ_1 > ... > λ_D.
    The deterministic ordering M_0 to M_1 to ... to M_D relies on distinct eigenvalues; repeated eigenvalues are handled only qualitatively in Appendix F.5.
  • ad hoc to paper The alignment ansatz (Eq. 20): during the (m+1)-th plateau, k_i = q_i = v_i e_i for learned heads, k_{m+1} = q_{m+1} = v_{m+1}(t) e_{m+1}, and all other weights are zero.
    This is posited, not derived. It reduces the dynamics to scalar ODEs (Eq. 21) and underpins the claim that heads align with eigenvectors in descending order. Simulations support it, but no proof is given.
  • domain assumption Small initialization (rich regime), with vanishing w_init so the conserved quantities k_i^T k_i - q_i^T q_i and k_i^T k_i - v_i^2 vanish.
    Used to justify the balanced norm form in the ansatz and the escape-time estimates (Appendices E.5, E.8).
  • standard math Gradient flow with infinitesimal learning rate is a faithful model of gradient descent.
    Equation (5) defines gradient flow; standard in optimization theory.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Training Dynamics of In-Context Learning in Linear Attention." pith.science (2026). https://pith.science/paper/VLFIUOWS

@misc{pith2026250116265,
  author       = {Pith},
  title        = {Pith review of: Training Dynamics of In-Context Learning in Linear Attention},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VLFIUOWS}},
  note         = {Machine review of arXiv:2501.16265}
}
read the original abstract

While attention-based models have demonstrated the remarkable ability of in-context learning (ICL), the theoretical understanding of how these models acquired this ability through gradient descent training is still preliminary. Towards answering this question, we study the gradient descent dynamics of multi-head linear self-attention trained for in-context linear regression. We examine two parametrizations of linear self-attention: one with the key and query weights merged as a single matrix (common in theoretical studies), and one with separate key and query matrices (closer to practical settings). For the merged parametrization, we show that the training dynamics has two fixed points and the loss trajectory exhibits a single, abrupt drop. We derive an analytical time-course solution for a certain class of datasets and initialization. For the separate parametrization, we show that the training dynamics has exponentially many fixed points and the loss exhibits saddle-to-saddle dynamics, which we reduce to scalar ordinary differential equations. During training, the model implements principal component regression in context with the number of principal components increasing over training time. Overall, we provide a theoretical description of how ICL abilities evolve during gradient descent training of linear attention, revealing abrupt acquisition or progressive improvements depending on how the key and query are parametrized.

Figures

Figures reproduced from arXiv: 2501.16265 by the authors.

Figure 1
Figure 1. Multi-head linear attention with merged key and query, ATTNM(X)D+1,N+1, is equivalent to a two-layer fully-connected linear network with cubic feature input, MLP(z). Left: Schematic of the equivalence. Right: Loss trajectories of linear attention and the fully-connected linear network match well. The two models are trained with the same data and initialization. Both exhibit the characteristic abrupt loss drop docume… view at source ↗
Figure 2
Figure 2. Multi-head linear attention with separate rank-one key and query ATTNS(X)D+1,N+1 is a sum of H (number of heads) three-layer convolutional linear networks with the cubic feature z as input. Here we take D = 3 to avoid clutter. Entries in the vectors are denoted as xq = [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Multi-head linear attention with separate rank-one key and query exhibits saddle-to-saddle dynamics. (a) The loss curve has D abrupt drops, separated by plateaus (six runs from different random initialization are plotted). The loss at each plateau matches our theoretical prediction in Equation (19) (dashed gray lines). (b) The value weight vi in each head for one of the runs in (a) is plotted in solid blue curves. T… view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Multi-head linear attention with separate low-rank key and query exhibits saddle-to-saddle dynamics, with the duration of plateaus depending on the rank R. Solid black curves are loss trajectories from six random initializations. Dashed gray lines mark the loss values …
Figure 5
Figure 5. Figure 5: Loss trajectories of softmax attention with merged or separate key and query. Six runs from different random initial￾ization are plotted. Similar to the linear attention case, softmax ATTNM exhibits one abrupt loss drop, while softmax ATTNS ex￾hibits multiple loss drop…
Figure 7
Figure 7. Figure 7: Loss trajectories of linear ATTNM and ATTNS trained with the next token prediction loss defined in Equation (23) with Nmax = 31. In this case, the models are trained on sequences of varying lengths, which they can handle due to the 1/N scaling factor in Equations (4), …
Figure 8
Figure 8. Figure 8: Loss trajectories of linear ATTNM and ATTNS with high-dimensional data. Here the sequence length is N = 127, Λ has trace 1 and eigenvalues λd ∝ d −1 . Other hyperparameters are labeled at the top of each panel. B. Additional Related Work A concurrent work by Geshkovski…
Figure 9
Figure 9. Figure 9: The dynamics of weights in multi-head linear attention with merged key and query can be predicted with statistics of the training dataset. We plot the weights at different times in training, corresponding to the loss trajectories in [PITH_FULL_IMAGE:figures/full_fig_p…
Figure 10
Figure 10. Figure 10: Illustration of t1, · · · , tD. The loss trajectory plotted is one of the trajectories of linear attention with separate rank-one key and query in Figure 3a. The time td (d = 1, . . . , D) denotes the time when the loss has just undergone the d-th abrupt drop. E.5.1. …
Figure 11
Figure 11. Figure 11: Loss and value weights trajectories. The setting is the same as Figure 3b except different ranks R = 2, 3, 4. In the rank-one case in Figure 3b, value weights in four heads grow, each corresponding to an abrupt loss drop from L(Mm) to L(Mm+1) (m = 0, 1, 2, 3). In the …
Figure 12
Figure 12. Figure 12: Same as [PITH_FULL_IMAGE:figures/full_fig_p038_12.png]
Figure 13
Figure 13. Figure 13: Loss trajectories of multi-head linear attention with separate key and query. The setup is the same as in Figure 3a except that Λ has eigenvalues 0.35, 0.35, 0.15, 0.15. The four panels differ only in the rank of the key and query weights. Although some eigenvalues ar…
Figure 14
Figure 14. Figure 14: Dynamics of in-context and in-weight learning in linear attention with merged key and query. The training set is the same as the in-context linear regression task described in Section 2.1 except that a portion of the task vectors w are fixed. The portion of fixed task…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Specialization of softmax attention heads: insights from the high-dimensional single-location model

    cs.LG 2026-03 conditional novelty 7.0 of 10

    In a high-dimensional toy task, multi-head softmax attention first aligns all heads with the mean signal, then sequentially specializes to latent directions; the paper introduces Bayes-softmax, which attains the Bayes...

  2. Understanding LoRA as Knowledge Memory: An Empirical Analysis

    cs.LG 2026-03 conditional novelty 7.0 of 10

    LoRA modules function as composable knowledge memories for LLMs with measurable storage capacity, internalization efficiency, and advantages in multi-module long-context reasoning.

  3. Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatch

    stat.ML 2026-07 conditional novelty 6.0 of 10

    Within-context token correlations reduce ICL to an effective shorter i.i.d. context length, while query–context correlations lower error and favor softmax over linear attention.

  4. Breaking the Reversal Curse in Autoregressive Language Models via Identity Bridge

    cs.AI 2026-02 conditional novelty 6.0 of 10

    Identity-bridge regularization, rephrased into an out-of-context reasoning form, yields ~40% reversal accuracy in a 1B LLM and provably fixes reversal in an idealized one-layer transformer.

  5. Unveiling the Mechanisms of Multi-Hop Reasoning in Transformers via Identity Bridge

    cs.LG 2025-09 conditional novelty 6.0 of 10

    Adding identity supervision on bridge tokens enables out-of-distribution two-hop reasoning in simple transformers, with a nuclear-norm theory explaining the benefit.

Reference graph

Works this paper leans on

85 extracted references · 57 canonical work pages · cited by 5 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Context-Scaling versus Task-Scaling in In-Context Learning

    Abedsoltan, A., Radhakrishnan, A., Wu, J., and Belkin, M. Context-scaling versus task-scaling in in-context learning, 2024. URL https://arxiv.org/abs/2410.12783

    work page Pith review arXiv 2024

  3. [3]

    Transformers learn to implement preconditioned gradient descent for in-context learning

    Ahn, K., Cheng, X., Daneshmand, H., and Sra, S. Transformers learn to implement preconditioned gradient descent for in-context learning. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, volume 36, pp.\ 45614--45650. Curran Associates, Inc., 2023. URL https://proceedings...

    2023

  4. [4]

    Linear attention is (maybe) all you need (to understand transformer optimization)

    Ahn, K., Cheng, X., Song, M., Yun, C., Jadbabaie, A., and Sra, S. Linear attention is (maybe) all you need (to understand transformer optimization). In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=0uI5415ry7

    2024

  5. [5]

    What learning algorithm is in-context learning? investigations with linear models

    Aky \"u rek, E., Schuurmans, D., Andreas, J., Ma, T., and Zhou, D. What learning algorithm is in-context learning? investigations with linear models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=0g0X4H8yN4I

    2023

  6. [6]

    A., Merullo, J., and Pavlick, E

    Anand, S., Lepori, M. A., Merullo, J., and Pavlick, E. Dual process learning: Controlling use of in-context vs. in-weights strategies with weight forgetting. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=jDsmB4o5S0

    2025

  7. [7]

    V., Kirsch, L., Krueger, D., and Frei, S

    Anwar, U., Oswald, J. V., Kirsch, L., Krueger, D., and Frei, S. Adversarial robustness of in-context learning in transformers for linear regression, 2024. URL https://arxiv.org/abs/2411.05189

    arXiv 2024

  8. [8]

    A convergence analysis of gradient descent for deep linear neural networks

    Arora, S., Cohen, N., Golowich, N., and Hu, W. A convergence analysis of gradient descent for deep linear neural networks. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=SkMQg3C5K7

    work page 2019

Show all 85 references
  1. [9]

    Max-margin token selection in attention mechanism

    Ataee Tarzanagh, D., Li, Y., Zhang, X., and Oymak, S. Max-margin token selection in attention mechanism. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, volume 36, pp.\ 48314--48362. Curran...

    2023

  2. [10]

    Neural networks as kernel learners: The silent alignment effect

    Atanasov, A., Bordelon, B., and Pehlevan, C. Neural networks as kernel learners: The silent alignment effect. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=1NvflqAdoom

    2022

  3. [11]

    Transformers as statisticians: Provable in-context learning with in-context algorithm selection

    Bai, Y., Chen, F., Wang, H., Xiong, C., and Mei, S. Transformers as statisticians: Provable in-context learning with in-context algorithm selection. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing S...

    2023

  4. [12]

    Transformers learn through gradual rank increase

    Boix-Adsera, E., Littwin, E., Abbe, E., Bengio, S., and Susskind, J. Transformers learn through gradual rank increase. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, volume 36, pp.\ 24519-...

    2023

  5. [13]

    Boix-Adser \`a , E., Saremi, O., Abbe, E., Bengio, S., Littwin, E., and Susskind, J. M. When can transformers reason with abstract symbols? In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=STUGfUz8ob

    2024

  6. [14]

    Infinite limits of multi-head transformer dynamics

    Bordelon, B., Chaudhry, H., and Pehlevan, C. Infinite limits of multi-head transformer dynamics. In Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J., and Zhang, C. (eds.), Advances in Neural Information Processing Systems, volume 37, pp.\ 35824--35878....

    2024

  7. [15]

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litw...

    1901

  8. [16]

    Toward understanding in-context vs

    Chan, B., Chen, X., Gy \"o rgy, A., and Schuurmans, D. Toward understanding in-context vs. in-weight learning. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=aKJr5NnN8U

    2025

  9. [17]

    Data distributional properties drive emergent in-context learning in transformers

    Chan, S., Santoro, A., Lampinen, A., Wang, J., Singh, A., Richemond, P., McClelland, J., and Hill, F. Data distributional properties drive emergent in-context learning in transformers. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances ...

    2022

  10. [18]

    L., and Saphra, N

    Chen, A., Shwartz-Ziv, R., Cho, K., Leavitt, M. L., and Saphra, N. Sudden drops in the loss: Syntax acquisition, phase transitions, and simplicity bias in MLM s. In The Twelfth International Conference on Learning Representations, 2024 a . URL https://openreview.net/forum?id=M...

    2024

  11. [19]

    and Li, Y

    Chen, S. and Li, Y. Provably learning a multi-head attention layer, 2024. URL https://arxiv.org/abs/2402.04084

    2024 arXiv

  12. [20]

    Training dynamics of multi-head softmax attention for in-context learning: Emergence, convergence, and optimality

    Chen, S., Sheen, H., Wang, T., and Yang, Z. Training dynamics of multi-head softmax attention for in-context learning: Emergence, convergence, and optimality. In Agrawal, S. and Roth, A. (eds.), Proceedings of Thirty Seventh Conference on Learning Theory, volume 247 of Proceed...

    2024

  13. [21]

    Unveiling induction heads: Provable training dynamics and feature learning in transformers

    Chen, S., Sheen, H., Wang, T., and Yang, Z. Unveiling induction heads: Provable training dynamics and feature learning in transformers. In Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J., and Zhang, C. (eds.), Advances in Neural Information Processing...

    2024

  14. [22]

    On lazy training in differentiable programming

    Chizat, L., Oyallon, E., and Bach, F. On lazy training in differentiable programming. In Wallach, H., Larochelle, H., Beygelzimer, A., d Alch\' e -Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2...

    2019

  15. [23]

    S., Hu, W., and Lee, J

    Du, S. S., Hu, W., and Lee, J. D. Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Sys...

    2018

  16. [24]

    Finite sample analysis and bounds of generalization error of gradient descent in in-context linear regression, 2024

    Duraisamy, K. Finite sample analysis and bounds of generalization error of gradient descent in in-context linear regression, 2024. URL https://arxiv.org/abs/2405.02462

    2024 arXiv

  17. [25]

    The evolution of statistical induction heads: In-context learning markov chains

    Edelman, E., Tsilivis, N., Edelman, B., Malach, E., and Goel, S. The evolution of statistical induction heads: In-context learning markov chains. In Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J., and Zhang, C. (eds.), Advances in Neural Information ...

    2024

  18. [26]

    and Vardi, G

    Frei, S. and Vardi, G. Trained transformer classifiers generalize and exhibit benign overfitting in-context. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=jwsPS8yRe4

    2025

  19. [27]

    Transformers learn to achieve second-order convergence rates for in-context linear regression

    Fu, D., Chen, T.-q., Jia, R., and Sharan, V. Transformers learn to achieve second-order convergence rates for in-context linear regression. In Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J., and Zhang, C. (eds.), Advances in Neural Information Proces...

    2024

  20. [28]

    Effect of batch learning in multilayer neural networks

    Fukumizu, K. Effect of batch learning in multilayer neural networks. Gen, 1 0 (04): 0 1E--03, 1998

    1998

  21. [29]

    S., and Valiant, G

    Garg, S., Tsipras, D., Liang, P. S., and Valiant, G. What can transformers learn in-context? a case study of simple function classes. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35...

    2022

  22. [30]

    J., Jegelka, S., and Kumar, S

    Gatmiry, K., Saunshi, N., Reddi, S. J., Jegelka, S., and Kumar, S. On the role of depth and looping for in-context learning with task diversity, 2024. URL https://arxiv.org/abs/2410.21698

    2024 arXiv

  23. [31]

    Dynamic metastability in the self-attention model, 2024

    Geshkovski, B., Koubbi, H., Polyanskiy, Y., and Rigollet, P. Dynamic metastability in the self-attention model, 2024. URL https://arxiv.org/abs/2410.06833

    2024 arXiv

  24. [32]

    In-context linear regression demystified: Training dynamics and mechanistic interpretability of multi-head softmax attention, 2025

    He, J., Pan, X., Chen, S., and Yang, Z. In-context linear regression demystified: Training dynamics and mechanistic interpretability of multi-head softmax attention, 2025. URL https://arxiv.org/abs/2503.12734

    2025 arXiv

  25. [33]

    Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic tasks

    He, T., Doshi, D., Das, A., and Gromov, A. Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic tasks. In Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J., and Zhang, C. (eds.), Advances in Neural Information P...

    2024

  26. [34]

    T., Schrodi, S., Bratuli\' c , J., Behrmann, N., Fischer, V., and Brox, T

    Hoffmann, D. T., Schrodi, S., Bratuli\' c , J., Behrmann, N., Fischer, V., and Brox, T. Eureka-moments in transformers: Multi-step tasks reveal softmax induced optimization problems. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berken...

    2024

  27. [35]

    Huang, J., Wang, Z., and Lee, J. D. Transformers learn to implement multi-step gradient descent with chain of thought. In The Thirteenth International Conference on Learning Representations, 2025 a . URL https://openreview.net/forum?id=r3DF5sOo5B

    2025

  28. [36]

    Non-asymptotic convergence of training transformers for next-token prediction

    Huang, R., Liang, Y., and Yang, J. Non-asymptotic convergence of training transformers for next-token prediction. In Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J., and Zhang, C. (eds.), Advances in Neural Information Processing Systems, volume 37, p...

    2024

  29. [37]

    In-context convergence of transformers

    Huang, Y., Cheng, Y., and Liang, Y. In-context convergence of transformers. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Procee...

    2024

  30. [38]

    A theoretical analysis of self-supervised learning for vision transformers

    Huang, Y., Wen, Z., Chi, Y., and Liang, Y. A theoretical analysis of self-supervised learning for vision transformers. In The Thirteenth International Conference on Learning Representations, 2025 b . URL https://openreview.net/forum?id=Antib6Uovh

    2025

  31. [39]

    E., Huang, Y., Li, Y., Rawat, A

    Ildiz, M. E., Huang, Y., Li, Y., Rawat, A. S., and Oymak, S. From self-attention to M arkov models: Unveiling the dynamics of generative transformers. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Proceedings of ...

    2024

  32. [40]

    D., and Ryu, E

    Jang, U., Lee, J. D., and Ryu, E. K. L o RA training in the NTK regime has no spurious local minima. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Proceedings of the 41st International Conference on Machine Learn...

    2024

  33. [41]

    Vision transformers provably learn spatial structure

    Jelassi, S., Sander, M., and Li, Y. Vision transformers provably learn spatial structure. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp.\ 37822--37836. Curran Associates, Inc....

    2022

  34. [42]

    and Telgarsky, M

    Ji, Z. and Telgarsky, M. Gradient descent aligns the layers of deep linear networks. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=HJflg30qKX

    2019

  35. [43]

    Unveil benign overfitting for transformer in vision: Training dynamics, convergence, and generalization

    Jiang, J., Huang, W., Zhang, M., Suzuki, T., and Nie, L. Unveil benign overfitting for transformer in vision: Training dynamics, convergence, and generalization. In Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J., and Zhang, C. (eds.), Advances in Neu...

    2024

  36. [44]

    Julistiono, A. A. K., Tarzanagh, D. A., and Azizan, N. Optimizing attention with mirror descent: Generalized max-margin token selection, 2024. URL https://arxiv.org/abs/2410.14581

    2024

  37. [45]

    and Suzuki, T

    Kim, J. and Suzuki, T. Transformers learn nonlinear features in context: Nonconvex mean-field dynamics on the attention landscape. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Proceedings of the 41st Internation...

    2024

  38. [46]

    Geometry of linear convolutional networks

    Kohn, K., Merkh, T., Mont\' u far, G., and Trager, M. Geometry of linear convolutional networks. SIAM Journal on Applied Algebra and Geometry, 6 0 (3): 0 368--406, 2022. doi:10.1137/21M1441183. URL https://doi.org/10.1137/21M1441183

    2022 doi

  39. [47]

    Function space and critical points of linear convolutional networks

    Kohn, K., Mont\' u far, G., Shahverdi, V., and Trager, M. Function space and critical points of linear convolutional networks. SIAM Journal on Applied Algebra and Geometry, 8 0 (2): 0 333--362, 2024. doi:10.1137/23M1565504. URL https://doi.org/10.1137/23M1565504

    2024 doi

  40. [48]

    Is attention required for ICL ? exploring the relationship between model architecture and in-context learning ability

    Lee, I., Jiang, N., and Berg-Kirkpatrick, T. Is attention required for ICL ? exploring the relationship between model architecture and in-context learning ability. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=Qwq4cpLtoX

    2024

  41. [49]

    Fine-grained analysis of in-context linear estimation: Data, architecture, and beyond

    Li, Y., Rawat, A., and Oymak, S. Fine-grained analysis of in-context linear estimation: Data, architecture, and beyond. In Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J., and Zhang, C. (eds.), Advances in Neural Information Processing Systems, volume...

    2024

  42. [50]

    M., Letey, M

    Lu, Y. M., Letey, M. I., Zavatone-Veth, J. A., Maiti, A., and Pehlevan, C. Asymptotic theory of in-context learning by linear attention, 2024. URL https://arxiv.org/abs/2405.11751

    2024

  43. [51]

    V., Hashimoto, T., and Ma, T

    Mahankali, A. V., Hashimoto, T., and Ma, T. One step of gradient descent is provably the optimal in-context learner with one layer of linear self-attention. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=8p3fu56lKc

    2024

  44. [52]

    V., Bondaschi, M., Girish, A., Nagle, A., Kim, H., Gastpar, M., and Ekbote, C

    Makkuva, A. V., Bondaschi, M., Girish, A., Nagle, A., Kim, H., Gastpar, M., and Ekbote, C. Local to global: Learning dynamics and effect of initialization for transformers. In Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J., and Zhang, C. (eds.), Adva...

    2024

  45. [53]

    Progress measures for grokking via mechanistic interpretability

    Nanda, N., Chan, L., Lieberum, T., Smith, J., and Steinhardt, J. Progress measures for grokking via mechanistic interpretability. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=9XFSbDPmdW

    2023

  46. [54]

    and Reddy, G

    Nguyen, A. and Reddy, G. Differential learning kinetics govern the transition from memorization to generalization during in-context learning. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=INyi7qUdjZ

    2025

  47. [55]

    Nichani, E., Damian, A., and Lee, J. D. How transformers learn causal structure with gradient descent. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Proceedings of the 41st International Conference on Machine Lea...

    2024

  48. [56]

    In-context learning and induction heads, 2022

    Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Johnston, S., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Cla...

    2022

  49. [57]

    and Reznikoff, M

    Otto, F. and Reznikoff, M. G. Slow motion of gradient flows. Journal of Differential Equations, 237 0 (2): 0 372--420, 2007. ISSN 0022-0396. doi:https://doi.org/10.1016/j.jde.2007.03.007. URL https://www.sciencedirect.com/science/article/pii/S0022039607000824

    2007 doi

  50. [58]

    F., Lubana, E

    Park, C. F., Lubana, E. S., and Tanaka, H. Algorithmic phases of in-context learning. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=XgH1wfHSX8

    2025

  51. [59]

    The mechanistic basis of data dependence and abrupt learning in an in-context classification task

    Reddy, G. The mechanistic basis of data dependence and abrupt learning in an in-context classification task. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=aN4Jf6Cx69

    2024

  52. [60]

    Ren, Y., Wang, Z., and Lee, J. D. Learning and transferring sparse contextual bigrams with linear transformers. In Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J., and Zhang, C. (eds.), Advances in Neural Information Processing Systems, volume 37, pp....

    2024

  53. [61]

    A distributional simplicity bias in the learning dynamics of transformers

    Rende, R., Gerace, F., Laio, A., and Goldt, S. A distributional simplicity bias in the learning dynamics of transformers. In Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J., and Zhang, C. (eds.), Advances in Neural Information Processing Systems, volu...

    2024

  54. [62]

    M., McClelland, J

    Saxe, A. M., McClelland, J. L., and Ganguli, S. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In The Second International Conference on Learning Representations, 2014. URL https://openreview.net/forum?id=_wzZwKpTDF_9C

    2014

  55. [63]

    M., McClelland, J

    Saxe, A. M., McClelland, J. L., and Ganguli, S. A mathematical theory of semantic development in deep neural networks. Proceedings of the National Academy of Sciences, 116 0 (23): 0 11537--11546, 2019. doi:10.1073/pnas.1820226116. URL https://www.pnas.org/doi/abs/10.1073/pnas....

    2019 doi

  56. [64]

    Linear transformers are secretly fast weight programmers

    Schlag, I., Irie, K., and Schmidhuber, J. Linear transformers are secretly fast weight programmers. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp.\ 9355--9366....

    2021

  57. [65]

    Exponential convergence time of gradient descent for one-dimensional deep linear neural networks

    Shamir, O. Exponential convergence time of gradient descent for one-dimensional deep linear neural networks. In Beygelzimer, A. and Hsu, D. (eds.), Proceedings of the Thirty-Second Conference on Learning Theory, volume 99 of Proceedings of Machine Learning Research, pp.\ 2691-...

    2019

  58. [66]

    Sheen, H., Chen, S., Wang, T., and Zhou, H. H. Implicit regularization of gradient flow on one-layer softmax attention, 2024. URL https://arxiv.org/abs/2403.08699

    2024 arXiv

  59. [67]

    K., Chan, S., Moskovitz, T., Grant, E., Saxe, A., and Hill, F

    Singh, A. K., Chan, S., Moskovitz, T., Grant, E., Saxe, A., and Hill, F. The transient nature of emergent in-context learning in transformers. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems...

    2023

  60. [68]

    K., Moskovitz, T., Hill, F., Chan, S

    Singh, A. K., Moskovitz, T., Hill, F., Chan, S. C., and Saxe, A. M. What needs to go right for an induction head? A mechanistic study of in-context learning circuits and their formation. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Be...

    2024

  61. [69]

    K., Moskovitz, T., Dragutinovic, S., Hill, F., Chan, S

    Singh, A. K., Moskovitz, T., Dragutinovic, S., Hill, F., Chan, S. C. Y., and Saxe, A. M. Strategy coopetition explains the emergence and transience of in-context learning, 2025. URL https://arxiv.org/abs/2503.05631

    2025 arXiv

  62. [70]

    Unraveling the gradient descent dynamics of transformers

    Song, B., Han, B., Zhang, S., Ding, J., and Hong, M. Unraveling the gradient descent dynamics of transformers. In Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J., and Zhang, C. (eds.), Advances in Neural Information Processing Systems, volume 37, pp.\...

    2024

  63. [71]

    A., Li, Y., Thrampoulidis, C., and Oymak, S

    Tarzanagh, D. A., Li, Y., Thrampoulidis, C., and Oymak, S. Transformers as support vector machines, 2024. URL https://arxiv.org/abs/2308.16898

    2024 arXiv

  64. [72]

    Tian, Y., Wang, Y., Chen, B., and Du, S. S. Scan and snap: Understanding training dynamics and token composition in 1-layer transformer. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, volu...

    2023

  65. [73]

    Tong, W. L. and Pehlevan, C. MLP s learn in-context on regression and classification tasks. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=MbX0t1rUlp

    2025

  66. [74]

    Implicit bias and fast convergence rates for self-attention

    Vasudeva, B., Deora, P., and Thrampoulidis, C. Implicit bias and fast convergence rates for self-attention. Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL https://openreview.net/forum?id=pKilnjQsb0

    2025

  67. [75]

    N., Kaiser, L

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. Attention is all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Informatio...

    2017

  68. [76]

    Linear transformers are versatile in-context learners

    Vladymyrov, M., von Oswald, J., Sandler, M., and Ge, R. Linear transformers are versatile in-context learners. In Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J., and Zhang, C. (eds.), Advances in Neural Information Processing Systems, volume 37, pp.\...

    2024

  69. [77]

    Transformers learn in-context by gradient descent

    Von Oswald, J., Niklasson, E., Randazzo, E., Sacramento, J., Mordvintsev, A., Zhmoginov, A., and Vladymyrov, M. Transformers learn in-context by gradient descent. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40t...

    2023

  70. [78]

    How transformers implement induction heads: Approximation and optimization analysis, 2024 a

    Wang, M., Yu, R., E, W., and Wu, L. How transformers implement induction heads: Approximation and optimization analysis, 2024 a . URL https://arxiv.org/abs/2410.11474

    2024 arXiv

  71. [79]

    Wang, Z., Wei, S., Hsu, D., and Lee, J. D. Transformers provably learn sparse token selection while fully-connected nets cannot. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Proceedings of the 41st International...

    2024

  72. [80]

    D., Moroshko, E., Savarese, P., Golan, I., Soudry, D., and Srebro, N

    Woodworth, B., Gunasekar, S., Lee, J. D., Moroshko, E., Savarese, P., Golan, I., Soudry, D., and Srebro, N. Kernel and rich regimes in overparametrized models. In Abernethy, J. and Agarwal, S. (eds.), Proceedings of Thirty Third Conference on Learning Theory, volume 125 of Pro...

    2020

  73. [81]

    How many pretraining tasks are needed for in-context learning of linear regression? In The Twelfth International Conference on Learning Representations, 2024

    Wu, J., Zou, D., Chen, Z., Braverman, V., Gu, Q., and Bartlett, P. How many pretraining tasks are needed for in-context learning of linear regression? In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=vSh5ePa0ph

    2024

  74. [82]

    V., Pasunuru, R., Chen, D., Zettlemoyer, L., and Stoyanov, V

    Xia, M., Artetxe, M., Zhou, C., Lin, X. V., Pasunuru, R., Chen, D., Zettlemoyer, L., and Stoyanov, V. Training trajectories of language models across scales. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for C...

    2023 doi

  75. [83]

    B., Jegelka, S., and Andreas, J

    Yau, M., Akyürek, E., Mao, J., Tenenbaum, J. B., Jegelka, S., and Andreas, J. Learning linear attention in polynomial time, 2024. URL https://arxiv.org/abs/2410.10101

    2024

  76. [84]

    Zhang, R., Frei, S., and Bartlett, P. L. Trained transformers learn linear models in-context. Journal of Machine Learning Research, 25 0 (49): 0 1--55, 2024 a . URL http://jmlr.org/papers/v25/23-1042.html

    2024

  77. [85]

    In-context learning of a linear transformer block: Benefits of the mlp component and one-step gd initialization

    Zhang, R., Wu, J., and Bartlett, P. In-context learning of a linear transformer block: Benefits of the mlp component and one-step gd initialization. In Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J., and Zhang, C. (eds.), Advances in Neural Informati...

    2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.