Pith. sign in

REVIEW 14 cited by

Uncovering mesa-optimization algorithms in Transformers

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2309.05858 v2 pith:K5OFQJOG submitted 2023-09-11 cs.LG cs.AI

Johannes von Oswald , Maximilian Schlegel , Alexander Meulemans , Seijin Kobayashi , Eyvind Niklasson , Nicolas Zucchet , Nino Scherrer , Nolan Miller
show 5 more authors
Mark Sandler Blaise Agüera y Arcas Max Vladymyrov Razvan Pascanu João Sacramento
This is my paper · ORCID
classification cs.LGcs.AI
keywords learningautoregressivein-contextminimizationmodelspredictionsequencetrained
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Some autoregressive models exhibit in-context learning capabilities: being able to learn as an input sequence is processed, without undergoing any parameter changes, and without being explicitly trained to do so. The origins of this phenomenon are still poorly understood. Here we analyze a series of Transformer models trained to perform synthetic sequence prediction tasks, and discover that standard next-token prediction error minimization gives rise to a subsidiary learning algorithm that adjusts the model as new inputs are revealed. We show that this process corresponds to gradient-based optimization of a principled objective function, which leads to strong generalization performance on unseen sequences. Our findings explain in-context learning as a product of autoregressive loss minimization and inform the design of new optimization-based Transformer layers.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 14 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Training with (Swap) Regret Loss in a Single-Layer Self-Attention Model: A Case Study on the Probability Simplex

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Training single-layer attention with squared regret loss has stationary points that implement smoothed fictitious play (external regret) and, via a new swap-regret loss, the Blum–Mansour no-swap-regret algorithm.

  2. Induction Heads Interpolate N-Grams

    cs.LG 2026-07 accept novelty 7.0 of 10

    Induction-head circuits implement soft context-matching (Jelinek–Mercer-style interpolation over partial matches) plus BOS-induced Dirichlet pseudo-counts, and trained transformers recover both mechanisms.

  3. The Orthogonalized Read Is a Removable Training Scaffold for Recurrent Memory

    cs.LG 2026-07 accept novelty 7.0 of 10

    The orthogonalized read on noisy recall is a removable training scaffold: it multiplies escape hazard ~6.4x during plateau training, but a stock mLSTM achieves full accuracy once the read is annealed away.

  4. Beyond Test-Time Memory: State-Space Optimal Control for LLM Reasoning

    cs.LG 2026-03 conditional novelty 7.0 of 10

    A symplectic LQR layer inserted as an adapter into pretrained LLMs yields large gains on MATH-500, AMC and AIME by solving a latent optimal-control problem at inference time.

  5. MesaNet: Sequence Modeling by Locally Optimal Test-Time Training

    cs.LG 2025-06 conditional novelty 7.0 of 10

    MesaNet uses conjugate-gradient-optimal test-time regression in a chunkwise-parallelizable recurrent layer, achieving strong language modeling and benchmark performance at up to 1B scale.

  6. Grounding latent algorithm routing in transformer reasoning

    cs.CL 2026-07 conditional novelty 6.5 of 10

    On ROUTEBENCH, 306M–612M dense transformers close most of the oracle routing gap with decodable, nuisance-stable, intervention-sensitive solver-family preferences.

  7. Learning to Remember, Learn, and Forget in Attention-Based Models

    cs.LG 2026-02 conditional novelty 6.0 of 10

    Palimpsa adds a per-slot importance/precision state to gated linear attention, letting a fixed-size memory forget stale information and protect important information, and recovers Mamba2 as a high-forgetting limit.

  8. Selective Induction Heads: How Transformers Select Causal Structures In Context

    cs.LG 2025-09 conditional novelty 6.0 of 10

    Transformers can learn to select the correct lag of an interleaved Markov chain in context via a circuit the authors call a selective induction head, whose asymptotic optimality proof is incomplete.

  9. Transformers Meet In-Context Learning: A Universal Approximation Theory

    cs.LG 2025-06 accept novelty 6.0 of 10

    A constructive theorem shows that transformers can perform in-context learning for any Barron-type function class by combining universal features with an emulated Lasso solver.

  10. The Role of Diversity in In-Context Learning for Large Language Models

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Diversity-aware selection of in-context examples improves performance on complex and out-of-distribution tasks, though effect sizes are often modest.

  11. Beyond Induction Heads: In-Context Meta Learning Induces Multi-Phase Circuit Emergence

    cs.CL 2025-05 conditional novelty 6.0 of 10

    A two-layer transformer solving an in-context meta-learning task acquires skill in three abrupt phases, each corresponding to a distinct attention circuit: bigram, label attention, then chunking plus label attention.

  12. Position: We Need An Algorithmic Understanding of Generative AI

    cs.AI 2025-07 conditional novelty 5.0 of 10

    The paper argues for a systematic algorithmic understanding of LLMs and presents a case study suggesting that Llama models do not implement BFS or DFS on graph navigation tasks.

  13. ATLAS: Learning to Optimally Memorize the Context at Test Time

    cs.CL 2025-05 conditional novelty 5.0 of 10

    A new recurrent memory module (Atlas) that optimizes its memory over a window of past tokens and uses Muon-style second-order updates, reporting gains over Transformers and existing linear RNNs on long-context tasks.

  14. Brewing Knowledge in Context: Distillation Perspectives on In-Context Learning

    cs.LG 2025-06 reject novelty 3.0 of 10

    In-context learning is reframed as implicit knowledge distillation, but the main claims either restate the known attention-equals-gradient-descent result or build the prompt-shift bound into the definition of MMD.

Pith tools