REVIEW 14 cited by
What Can Transformers Learn In-Context? A Case Study of Simple Function Classes
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
In-context learning refers to the ability of a model to condition on a prompt sequence consisting of in-context examples (input-output pairs corresponding to some task) along with a new query input, and generate the corresponding output. Crucially, in-context learning happens only at inference time without any parameter updates to the model. While large language models such as GPT-3 exhibit some ability to perform in-context learning, it is unclear what the relationship is between tasks on which this succeeds and what is present in the training data. To make progress towards understanding in-context learning, we consider the well-defined problem of training a model to in-context learn a function class (e.g., linear functions): that is, given data derived from some functions in the class, can we train a model to in-context learn "most" functions from this class? We show empirically that standard Transformers can be trained from scratch to perform in-context learning of linear functions -- that is, the trained model is able to learn unseen linear functions from in-context examples with performance comparable to the optimal least squares estimator. In fact, in-context learning is possible even under two forms of distribution shift: (i) between the training data of the model and inference-time prompts, and (ii) between the in-context examples and the query input during inference. We also show that we can train Transformers to in-context learn more complex function classes -- namely sparse linear functions, two-layer neural networks, and decision trees -- with performance that matches or exceeds task-specific learning algorithms. Our code and models are available at https://github.com/dtsip/in-context-learning .
Forward citations
Cited by 14 Pith papers
-
Learning Pseudorandom Numbers with Transformers: Permuted Congruential Generators, Curricula, and Interpretability
Transformers can in-context predict PCG outputs on unseen parameters; required context length scales as sqrt(m), and curriculum training with smaller moduli is necessary for large moduli.
-
Pretraining Curricula Enable Selective Fine-tuning
Imbalanced pretraining curricula disentangle task circuits in transformers, improving in-context learning and the selectivity of refusal fine-tuning relative to balanced training.
-
Jacobian Scopes: token-level causal attributions in LLMs
Jacobian Scopes attributes LLM next-token predictions to input tokens by taking norms of vector-Jacobian products, covering a target logit, the full output distribution, and forecast confidence.
-
ICR-RL: Deep Reinforcement Learning via In-Context Regression
A pre-trained tabular regression transformer (TabPFN) can act as a Q-function approximator in fitted Q iteration, matching DQN on CartPole, MountainCar, and Acrobot without gradient updates.
-
Customizing the Inductive Biases of Softmax Attention using Structured Matrices
Structured-matrix scoring functions, BTT and MLR, let attention escape the low-rank bottleneck and add a distance-dependent compute bias, improving accuracy for fixed compute on regression, language modeling, and forecasting.
-
Bottom-up Domain-specific Superintelligence: A Reliable Knowledge Graph is What We Need
A language model fine-tuned on knowledge-graph-path reasoning tasks (QwQ-Med-3) beats strong baselines on a same-style benchmark but shows mixed gains on external medical QA tests.
-
LLMs are Bayesian, In Expectation, Not in Realization
Transformers can be Bayes-competitive in prequential log loss even when their predictive distributions are not invariant to example order, provided the cumulative predictive KL to the Bayesian reference stays small.
-
Transformers Don't In-Context Learn Least Squares Regression
In-context regression transformers do not approximate OLS: they underperform it even in-distribution, fail on out-of-subspace prompts, and their failures correlate with a low-rank spectral signature in the residual stream.
-
Context Tuning for In-Context Optimization
Context Tuning initializes a trainable prompt or KV-cache prefix from the few-shot demonstrations and refines it by gradient descent, matching Test-Time Training accuracy on MMLU, BBH, and ARC at up to 2-3x less train...
-
Entangled by Design: Spurious Intra-Variable Signal Routing in Tabular In-Context Learners
In-context learners route predictions through a spurious component inside a composite feature whenever that component correlates with the label, and the routing persists as context grows.
-
In-context learning of closed form solution to simple linear regression task using transformer with linear self-attention
A two-layer linear-attention transformer can approximate the closed-form least-squares solution by using layer normalization for division, and one l1-regularized training run mostly reproduces this construction.
-
In Context Learning with Vision Transformers: Case Study
A decoder-only transformer with a CNN or ViT image encoder learns random linear, convolutional, and ViT functions on 8x8 CIFAR-10 images in-context from a handful of examples.
-
Thinking About Thinking: SAGE-nano's Inverse Reasoning for Self-Aware Language Models
A 4B-parameter model is claimed to explain its own reasoning through inverse attention analysis, but the paper offers no consistent evidence or artifacts.
-
Eliciting Fine-Tuned Transformer Capabilities via Inference-Time Techniques
The paper claims that in-context learning with finite example sets can approximate supervised fine-tuning in transformers, but the proof assumes the very approximation it sets out to establish.
Discussion (0). Sign in to comment.