REVIEW 11 cited by
One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attention
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Recent works have empirically analyzed in-context learning and shown that transformers trained on synthetic linear regression tasks can learn to implement ridge regression, which is the Bayes-optimal predictor, given sufficient capacity [Aky\"urek et al., 2023], while one-layer transformers with linear self-attention and no MLP layer will learn to implement one step of gradient descent (GD) on a least-squares linear regression objective [von Oswald et al., 2022]. However, the theory behind these observations remains poorly understood. We theoretically study transformers with a single layer of linear self-attention, trained on synthetic noisy linear regression data. First, we mathematically show that when the covariates are drawn from a standard Gaussian distribution, the one-layer transformer which minimizes the pre-training loss will implement a single step of GD on the least-squares linear regression objective. Then, we find that changing the distribution of the covariates and weight vector to a non-isotropic Gaussian distribution has a strong impact on the learned algorithm: the global minimizer of the pre-training loss now implements a single step of $\textit{pre-conditioned}$ GD. However, if only the distribution of the responses is changed, then this does not have a large effect on the learned algorithm: even when the response comes from a more general family of $\textit{nonlinear}$ functions, the global minimizer of the pre-training loss still implements a single step of GD on a least-squares linear regression objective.
Forward citations
Cited by 11 Pith papers
-
Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatch
Within-context token correlations reduce ICL to an effective shorter i.i.d. context length, while query–context correlations lower error and favor softmax over linear attention.
-
Breaking the Reversal Curse in Autoregressive Language Models via Identity Bridge
Identity-bridge regularization, rephrased into an out-of-context reasoning form, yields ~40% reversal accuracy in a 1B LLM and provably fixes reversal in an idealized one-layer transformer.
-
Transformers as Multi-task Learners: Decoupling Features in Hidden Markov Models
A hand-constructed O(ln L + T)-layer Transformer is shown to approximate low-rank hidden Markov models in-context, with lower layers extracting local history features and upper layers performing regression-based decoding.
-
A Unified Perspective on the Dynamics of Deep Transformers
Attention-only Transformer stacks are shown to be well-posed as mean-field PDEs for many attention variants, and Gaussian inputs evolve via explicit covariance ODEs that predict clustering or blow-up.
-
Reversed Attention: On The Gradient Descent Of Attention Layers In GPT
A GPT's backward pass through attention computes a "Reversed Attention" matrix, which can rank attention heads and, when injected into the forward pass, improve task accuracy without weight updates.
-
Rethinking Associative Memory Mechanism in Induction Head
A two-layer transformer with relative positional encoding keeps its induction head active across the whole sequence, while absolute positional encoding loses it in the second half.
-
Re-examining learning linear functions in context
Transformer models trained from scratch on in-context linear function prediction learn boundary-limited interpolation, not general linear regression.
-
Data-Efficient Adaptation of LLMs via Attention Head Reweighting
Learning a single scalar per attention head lets LLMs adapt to few-shot text classification better than LoRA, with 200–1000x fewer trainable parameters.
-
Minimalist Softmax Attention Provably Learns Constrained Boolean Functions
With teacher forcing that reveals pairwise products of the relevant bits, one gradient step lets a single-head attention recover the support of a k-bit AND/OR; the paper's claimed end-to-end hardness lower bound is in...
-
Transformers and Their Roles as Time Series Foundation Models
Transformers can implement autoregressive least-squares regression in-context on time series, and pretraining on weakly dependent data gives test error decaying as 1 divided by the square root of the number of pretrai...
-
In-context learning of closed form solution to simple linear regression task using transformer with linear self-attention
A two-layer linear-attention transformer can approximate the closed-form least-squares solution by using layer normalization for division, and one l1-regularized training run mostly reproduces this construction.
Discussion (0). Continue with ORCID to comment.