Pith. sign in

REVIEW 3 major objections 5 minor 20 references

Transformers Don't In-Context Learn Least Squares Regression

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Transformers trained for in-context regression do not implement ordinary least squares; their success is tied to the pretraining distribution.

desk verdict Solid subspace OOD result, but the title overreaches: the paper's own full-space transformer may match OLS, so the universal claim needs retreating. read the letter →

arxiv 2507.09440 v1 pith:TTRPYHQC submitted 2025-07-13 cs.LG cs.AI

Joshua Hill , Benjamin Eyre , Elliot Creager This is my paper
classification cs.LGcs.AI
keywords in-contextlearninglinearregressionout-of-distributiongeneralizationspectralsignatureresidualstreamtransformerinterpretabilityordinaryleastsquares
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks what a transformer actually computes when it performs in-context linear regression, and it argues that the answer is neither ordinary least squares nor gradient descent. Training a GPT-2-style transformer on prompts whose inputs or weights lie in a 10-dimensional subspace of $\mathbb{R}^{20}$, the authors find that evaluation on the orthogonal subspace or the full space causes a large error increase, while OLS, ridge, and other classical regressors stay accurate. Even on in-distribution prompts in the overdetermined regime, the transformer's asymptotic error is orders of magnitude above the OLS baseline. The paper also identifies a spectral signature in the residual stream: in-distribution prompts produce representations whose top two singular vectors are highly stable, and the degree of that stability correlates with low loss. If correct, the work redirects explanations of ICL from "the model learns a regression algorithm" toward "the model relies on prompt similarity to its pretraining distribution."

What carries the argument

The machinery is the singular value decomposition of the residual-stream representation matrix $Z_p \in \mathbb{R}^{(k+1)\times d}$ formed from the token representations just before the final readout head of the transformer. For a prompt $p$, the paper defines canonical right singular vectors $V^*$ from a batch of in-distribution prompts and measures alignment via $C_p = \operatorname{diag}(V^{*\top} V_p)$. The spectral signature is the observation that in-distribution prompts concentrate most of their representation norm and most of the readout weight in the first two singular vectors, which are highly consistent across prompts; out-of-distribution prompts produce flatter singular-value spectra and unstable principal directions. This signature is what the paper uses to distinguish in-distribution from out-of-distribution prompts and to correlate with prediction loss.

What would settle it

A direct test would be to take an in-distribution prompt for $T_\parallel$, alter its residual-stream representation so that the top two singular directions are replaced by the corresponding directions from an out-of-distribution prompt, and check whether the prediction error rises to out-of-distribution levels; if it does not, the spectral signature is a diagnostic correlate rather than the causal pathway.

Watch

Extended reading notes

Core claim

The central claim is that transformers trained for in-context regression do not emulate the optimal least-squares solution, and their generalization is tied to the pretraining distribution. The paper supports this with out-of-distribution experiments: when the training inputs or weights are restricted to a subspace $A$ of dimension 10 in $\mathbb{R}^{20}$, the transformer $T_\parallel$ performs well on prompts from that subspace but degrades sharply on prompts whose inputs lie in the orthogonal subspace $B$ or the full space, whereas OLS and ridge regression generalize across all of these. The same pattern appears when the shift is in the weight vectors rather than the inputs. A convex-combination experiment shows that error grows gradually as the test prompt is blended from the training subspace to the orthogonal one, and a projected-OLS baseline shows the transformer is not simply applying OLS inside the training subspace. The paper further shows that the residual-stream representations of in-distribution prompts have a distinctive spectral signature---two stable dominant right singular vectors---and that this signature carries most of the readout weight and correlates with lower loss, while out-of-distribution prompts lack it.

Load-bearing premise

The load-bearing premise is that the stable top-two singular vectors of the residual stream are the mechanism behind in-distribution ICL success, not merely a correlate of distribution shift, and the paper does not intervene on those representations to test that.

Editorial extensions

If this is right

  • In-context regression models trained on a restricted subspace cannot be safely deployed on inputs or tasks outside that subspace, even when a classical regressor would have no trouble.
  • Claims that transformers implement OLS or gradient descent must be revised: any proposed mechanism has to explain both the distribution dependence and the in-distribution gap to OLS.
  • The spectral signature gives a representation-level signal that could be used to flag out-of-distribution prompts before the final prediction is trusted.
  • Increasing pretraining diversity, for example training on multiple input scales, reduces the out-of-distribution degradation, so the robustness of ICL is shaped by the diversity of the pretraining corpus.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A causal test of the spectral-signature story would intervene on the residual stream: zeroing or rotating the top two singular directions of an in-distribution prompt's representation should make its prediction as poor as an out-of-distribution prompt if the signature is the mechanism; the paper stops at correlation.
  • The same top-two-singular-vector diagnostic may transfer to language models---the appendix's Caesar-cipher experiment shows a similar spectral flattening and alignment drop for rare shifts---so it could become a monitoring signal, but the paper does not establish thresholds for real tasks.
  • One testable consequence of the distribution-coupling view is that fine-tuning on full-space prompts after subspace pretraining should erode the OOD gap and shift the stable singular directions; that prediction is not directly tested here.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies in-context learning (ICL) for synthetic linear regression with a GPT-2-style transformer. It trains transformers on prompts whose inputs or weights are restricted to a random 10-dimensional subspace, then evaluates on the training subspace, the orthogonal subspace, and the full space. The authors report that (i) transformers fail to generalize to out-of-distribution prompts after such shifts, (ii) even in-distribution the transformer's asymptotic error is orders of magnitude worse than OLS, and (iii) residual-stream representations of in-distribution prompts exhibit a spectral signature—two stable dominant singular vectors—that is absent for out-of-distribution prompts and correlates with loss. The paper argues these results contradict claims that transformers implement OLS or gradient descent during ICL.

Significance. The paper's strengths are its controlled synthetic setup, the systematic variation of OOD shifts (input subspace, weight subspace, input scale, label noise), and the honest limitations section. If the central claims were fully supported, the paper would be an important counterpoint to prior work on transformers implementing OLS or gradient descent. The proposed spectral signature is also a potentially useful diagnostic for OOD detection. However, the headline claim is currently broader than the evidence: the in-distribution comparison with OLS is demonstrated only for the subspace-restricted transformer, and the spectral analysis is correlational rather than causal. With a revised framing, the paper would be a valuable empirical contribution about the dependence of ICL on pretraining distribution.

major comments (3)
  1. [Section 4 and Abstract] The claim that 'transformers do not emulate the optimal least-squares solution, even in-distribution' is supported only for the subspace-restricted transformer T_parallel trained on D(P=P_A). Appendix B reports that T_square, which is trained on unrestricted prompts from D_square, is at least competitive with OLS (the text says the noisy-label model T*_square performs 'poorer than OLS and T_square'). This directly undermines the universal phrasing in the title and abstract. The evidence supports a narrower claim: a transformer trained on a restricted subspace can fail to match OLS in-distribution and can fail to generalize OOD. I recommend adding an explicit in-distribution comparison of T_square versus OLS in the main text and revising the title/abstract to avoid implying no transformer implementation of OLS exists.
  2. [Section 5.3, Tables 2 and 3, Figure 6] The phrase 'spectral signatures impact prediction' overstates what the experiments show. The evidence is correlational: top-two singular vectors carry most representation norm and readout weight, and the signature correlates with loss. No intervention is performed (e.g., ablating or rotating the top singular directions and measuring whether OOD behavior changes), so the paper cannot distinguish a causal mechanism from a side effect of distribution shift. The limitations section correctly calls the spectral signature a 'diagnostic and OOD detector'; the main-text discussion should be consistently phrased in that way.
  3. [Section 5.2, Table 1 and Appendix H] The validation of the spectral signature as an OOD detector is circular in an important sense. The canonical singular vectors V* are computed from the training distribution, and the bivariate Gaussian is fit to the first two components of C_p for prompts from S_parallel; Table 1 then tests holdout prompts from the same S_parallel against that fitted model. This measures within-distribution consistency, not the ability of the signature to generalize as a detector across architectures, tasks, or pretraining distributions. A separate validation on a held-out training distribution or an entirely different task would be needed to support the claim that the signature can 'predict' OOD behavior.
minor comments (5)
  1. [Appendix C] There is a typo in the description of the projected-input OLS baseline: 'trainign' should be 'training'.
  2. [Appendix E] The text says 'we specifically analyze P_A beta_hat for the training subspace and P_A beta_hat for the orthogonal subspace,' but the second projection should be P_B beta_hat; as written, both projections are onto the training subspace.
  3. [Section 5.3] The sentence introducing Table 3 says 'we plot' but the results are presented in a table; this should be rephrased.
  4. [Section 5.1] The symbol d is used both for the input dimensionality (d=20) and for the residual-stream dimension in Equation (3). This is confusing; please introduce separate notation for the residual-stream dimension.
  5. [Figures 2, 3, 9, 11, and 14] These main performance figures do not show error bars or multiple-seed variability, even though Table 1 reports standard deviations. Adding variability information would strengthen the quantitative claims.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the main OLS claim is an empirical comparison and the spectral signature is a diagnostic evaluated on held-out data.

full rationale

The paper's central claim is empirical: a subspace-restricted transformer underperforms OLS in-distribution and degrades under OOD shifts, and this is measured directly against classical baselines rather than derived from a fitted parameter. No load-bearing self-citation appears; the paper cites prior ICL work but does not rest its conclusions on a self-cited uniqueness theorem or ansatz. The spectral-signature analysis is also not circular in the prohibited sense: canonical singular vectors are computed from pooled prompts per distribution, and the Gaussian OOD test is fit on a subset of S∥ and then evaluated on held-out S∥ and S⊥ prompts, so the classification is not forced by construction. The projection-dominance result in Table 2 follows from SVD optimality but is presented as a sanity check, not as a prediction, and the readout-weight analysis in Table 3 adds independent evidence. Any concern that the full-space transformer T□ may match OLS in-distribution concerns the breadth of the paper's title claim, which is a correctness or generalization issue rather than circularity.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The main OLS-incompatibility claim requires no fitted parameters; it is a direct comparison of measured errors. The spectral signature, however, depends on a reference SVD computed from in-distribution prompts and a Gaussian threshold fitted to the same class, so these parameters are data-derived. Baseline hyperparameters for ridge, kernel ridge, and gradient descent are not reported, which underdetermines those comparisons.

free parameters (5)
  • Canonical singular vectors V* = not reported
    Computed from a batch of 20 prompts from S|| via pooled SVD; used as reference for cosine similarity in Eq. 5.
  • Bivariate Gaussian parameters (mu, Sigma) = not reported
    Fitted to the first two indices of Cp from a subset of S|| in Appendix H, Eq. 6, and used to classify hold-out prompts.
  • Ridge regularization lambda = not reported
    Used for ridge regression baseline; value not stated in the paper, so the baseline is not fully reproducible.
  • Kernel ridge lambda and sigma = not reported
    RBF kernel parameters for kernel ridge regression baseline; values not specified in Appendix C.
  • Gradient descent learning rate eta = not reported
    Learning rate for gradient descent baseline; value not specified in Appendix C.
assumptions (3)
  • domain assumption The GPT-2-style transformer and training procedure are representative of transformers that exhibit ICL.
    The paper generalizes from this one architecture and synthetic task to claims about transformers in general.
  • domain assumption The subspace distribution shift from D|| to D□ is a meaningful OOD test that OLS should handle well.
    The comparison relies on OLS as a reference point; the shift is constructed to be benign for OLS but the paper does not prove this independently.
  • ad hoc to paper The top two singular vectors of the residual stream capture the mechanism of ICL failure.
    The spectral signature is chosen post hoc after observing the spectra; no theory explains why exactly two directions are special.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Transformers Don't In-Context Learn Least Squares Regression." pith.science (2026). https://pith.science/paper/TTRPYHQC

@misc{pith2026250709440,
  author       = {Pith},
  title        = {Pith review of: Transformers Don't In-Context Learn Least Squares Regression},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TTRPYHQC}},
  note         = {Machine review of arXiv:2507.09440}
}
read the original abstract

In-context learning (ICL) has emerged as a powerful capability of large pretrained transformers, enabling them to solve new tasks implicit in example input-output pairs without any gradient updates. Despite its practical success, the mechanisms underlying ICL remain largely mysterious. In this work we study synthetic linear regression to probe how transformers implement learning at inference time. Previous works have demonstrated that transformers match the performance of learning rules such as Ordinary Least Squares (OLS) regression or gradient descent and have suggested ICL is facilitated in transformers through the learned implementation of one of these techniques. In this work, we demonstrate through a suite of out-of-distribution generalization experiments that transformers trained for ICL fail to generalize after shifts in the prompt distribution, a behaviour that is inconsistent with the notion of transformers implementing algorithms such as OLS. Finally, we highlight the role of the pretraining corpus in shaping ICL behaviour through a spectral analysis of the learned representations in the residual stream. Inputs from the same distribution as the training data produce representations with a unique spectral signature: inputs from this distribution tend to have the same top two singular vectors. This spectral signature is not shared by out-of-distribution inputs, and a metric characterizing the presence of this signature is highly correlated with low loss.

Figures

Figures reproduced from arXiv: 2507.09440 by the authors.

Figure 1
Figure 1. To examine the generalization of in-context learning in transformers, we pre-train on synthetic [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Prediction error of transformers trained on prompts with an input projection, shown for prompts [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Prediction error as we blend an in-distribution inputs with OOD inputs (Equation 2). The “Least Square (proj inputs)” baseline corresponds to running OLS on the test prompt after projecting it onto the training subspace (which incurs error because it ignores some features of the test prompt). Note the the performance discrepancy between the transformer T∥ and the projected OLS baseline reveals that the transformer i… view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Singular-value spectra of residual-stream embeddings for training subspace, orthogonal subspace and full space prompts. 5 Identifying ICL Generalization Through Spectral Signatures 5.1 Measuring Distribution Shift in the Transformer Residual Stream So far we have shown…
Figure 5
Figure 5. Figure 5: Alignment of singular-vector directions across prompts. For each distribution, we compute cosine [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Scatter plot of ||Cp,:2||2 2 vs MSE over all token positions for prompts from D∥ and D⊥. Line of best fit is depicted along with associated Pearson correlation. the spectral signature of other tasks, this geometric cue could inform real-time monitoring or trigger human…
Figure 7
Figure 7. Figure 7: Prediction error of several models (Section C), including transformers trained on prompts with and [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Prediction error of a transformer trained on a noisy variant of [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Comparing last-token prediction error versus input scale. [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Prediction error of transformers trained on 3 different task-vector subspace dimensions, shown for [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: Performance on prompts from D( Pw = PA, Px = Id) (In-Distribution) and D( Pw = PB, Px = Id) (OOD) for several methods [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: Singular-value spectrum of representations for the weight-space restricted transformer. [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: Cosine similarity between singular vectors of residual representations of the weight-space restricted [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14: Prediction error as we blend an in-distribution task with an OOD task. The horizontal axis is the convex combination coefficient t, and the vertical axis shows the resulting regression error. H Gaussian Distribution Test for Out-of-Distribution Spectral Sig￾natures He…
Figure 15
Figure 15. Figure 15: Comparison of singular-value spectrum between representations for [PITH_FULL_IMAGE:figures/full_fig_p020_15.png]
Figure 16
Figure 16. Figure 16: Comparison of cosine similarity between singular vectors of residual representations for [PITH_FULL_IMAGE:figures/full_fig_p020_16.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

20 extracted references · 7 canonical work pages

  1. [1]

    What learning algorithm is in-context learning? investigations with linear models

    Ekin Aky \"u rek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=0g0X4H8yN4I

    work page 2023

  2. [2]

    Bayesian scaling laws for in-context learning

    Aryaman Arora, Dan Jurafsky, Christopher Potts, and Noah D Goodman. Bayesian scaling laws for in-context learning. arXiv preprint arXiv:2410.16531, 2024

    arXiv 2024

  3. [3]

    Learning theory from first principles

    Francis Bach. Learning theory from first principles. MIT press, 2024

    2024

  4. [4]

    Training with noise is equivalent to tikhonov regularization

    Chris M Bishop. Training with noise is equivalent to tikhonov regularization. Neural computation, 7 0 (1): 0 108--116, 1995

    work page 1995

  5. [5]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    arXiv 2020

  6. [6]

    A survey on in-context learning

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, et al. A survey on in-context learning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1107--1128, 2024

    work page 2024

  7. [7]

    A mathematical framework for transformer circuits

    Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1 0 (1): 0 12, 2021

    2021

  8. [8]

    What can transformers learn in-context? a case study of simple function classes, 2023

    Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes, 2023. URL https://arxiv.org/abs/2208.01066

    arXiv 2023

Show all 20 references
  1. [9]

    Smith, Vudtiwat Ngampruetikorn, and David J

    Chase Goddard, Lindsay M. Smith, Vudtiwat Ngampruetikorn, and David J. Schwab. Specialization-generalization transition in exemplar-based in-context learning. In NeurIPS 2024 Workshop on Scientific Methods for Understanding Deep Learning, 2024. URL https://openreview.net/forum...

    2024

  2. [10]

    Cauchy's method of minimization

    Allen A Goldstein. Cauchy's method of minimization. Numerische Mathematik, 4 0 (1): 0 146--150, 1962

    1962

  3. [11]

    Understanding catastrophic forgetting in language models via implicit inference, 2024

    Suhas Kotha, Jacob Mitchell Springer, and Aditi Raghunathan. Understanding catastrophic forgetting in language models via implicit inference, 2024. URL https://arxiv.org/abs/2309.10105

    2024 arXiv

  4. [12]

    Predictive multiplicity in classification

    Charles Marx, Flavio Calmon, and Berk Ustun. Predictive multiplicity in classification. In International conference on machine learning, pages 6765--6774. PMLR, 2020

    2020

  5. [13]

    Embers of autoregression: Understanding large language models through the problem they are trained to solve

    R Thomas McCoy, Shunyu Yao, Dan Friedman, Matthew Hardy, and Thomas L Griffiths. Embers of autoregression: Understanding large language models through the problem they are trained to solve. arXiv preprint arXiv:2309.13638, 2023

    2023 arXiv

  6. [14]

    More data can hurt for linear regression: Sample-wise double descent, 2019

    Preetum Nakkiran. More data can hurt for linear regression: Sample-wise double descent, 2019. URL https://arxiv.org/abs/1912.07242

    2019 arXiv

  7. [15]

    In-context learning and induction heads

    Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11895, 2022

    2022 arXiv

  8. [16]

    Pretraining task diversity and the emergence of non-bayesian in-context learning for regression

    Allan Raventos, Mansheej Paul, Feng Chen, and Surya Ganguli. Pretraining task diversity and the emergence of non-bayesian in-context learning for regression. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=BtAz4a5xDg

    2023

  9. [17]

    Do pretrained transformers really learn in-context by gradient descent? arXiv preprint arXiv:2310.08540, 2023

    Lingfeng Shen, Aayush Mishra, and Daniel Khashabi. Do pretrained transformers really learn in-context by gradient descent? arXiv preprint arXiv:2310.08540, 2023

    2023 arXiv

  10. [18]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2017. URL https://arxiv.org/abs/1706.03762

    2017 arXiv

  11. [19]

    Transformers learn in-context by gradient descent

    Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, Jo \ a o Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference on Machine Learning, pages 35151--35174. PMLR, 2023

    2023

  12. [20]

    Qwen3 technical report, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jia...

    2025 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.