REVIEW 16 cited by
Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large pretrained language models have shown surprising in-context learning (ICL) ability. With a few demonstration input-label pairs, they can predict the label for an unseen input without parameter updates. Despite the great success in performance, its working mechanism still remains an open question. In this paper, we explain language models as meta-optimizers and understand in-context learning as implicit finetuning. Theoretically, we figure out that Transformer attention has a dual form of gradient descent. On top of it, we understand ICL as follows: GPT first produces meta-gradients according to the demonstration examples, and then these meta-gradients are applied to the original GPT to build an ICL model. We comprehensively compare the behaviors of in-context learning and explicit finetuning on real tasks to provide empirical evidence that supports our understanding. Experimental results show that in-context learning behaves similarly to explicit finetuning from multiple perspectives. Inspired by the dual form between Transformer attention and gradient descent, we design a momentum-based attention by analogy with gradient descent with momentum. The improved performance over vanilla attention further supports our understanding from another perspective, and more importantly, shows the potential to utilize our understanding for future model design. The code is available at \url{https://aka.ms/icl}.
Forward citations
Cited by 16 Pith papers
-
Provable Low-Frequency Bias of In-Context Learning of Representations
In-context learning biases hidden representations toward low-frequency eigenvectors of a reweighted graph Laplacian, a phenomenon the authors prove and test.
-
Can Gradient Descent Simulate Prompting?
A MAML-style meta-training objective makes a single gradient step on new text recover part of the performance that prompting achieves, on reversal-curse and passage-QA tasks.
-
Test-Time Scaling via Error Localization
TTEL uses feedback-induced token probability drops to localize the first error in a failed reasoning trace and branch a new generation from that prefix, improving pass@k per token on coding and math benchmarks.
-
Train Once, Reuse Everywhere: Generalizable Implicit In-Context Learning by Routing Attention
ICR extracts shared attention directions from in-context learning and routes them at inference time, enabling zero-shot reuse across tasks.
-
FedChip: Federated LLM for Artificial Intelligence Accelerator Chip Design
FedChip applies federated fine-tuning to LLM-based AI accelerator design, adding a 30k-sample dataset and a Chip@k metric, with a reported 77% quality improvement over high-end LLMs.
-
Transformers Meet In-Context Learning: A Universal Approximation Theory
A constructive theorem shows that transformers can perform in-context learning for any Barron-type function class by combining universal features with an emulated Lasso solver.
-
ConText: Driving In-context Learning for Text Removal and Segmentation
ConText is the first visual in-context learning model for text removal and segmentation, chaining the two tasks and using self-prompting to reach new state-of-the-art scores.
-
Adaptive Task Vectors for Large Language Models
Adaptive Task Vectors use a small model to generate query-specific steering vectors for frozen LLMs, reporting strong accuracy and generalization, though the theoretical equivalences to LoRA and Prefix-Tuning are not ...
-
Transformers as Multi-task Learners: Decoupling Features in Hidden Markov Models
A hand-constructed O(ln L + T)-layer Transformer is shown to approximate low-rank hidden Markov models in-context, with lower layers extracting local history features and upper layers performing regression-based decoding.
-
Optimization-Inspired Few-Shot Adaptation for Large Language Models
OFA tunes LayerNorm parameters as optimization preconditioners and adds step-ratio and sharpness penalties, reporting consistent few-shot accuracy gains over baselines on Llama and GPT-2 models.
-
Solving Empirical Bayes via Transformers
A transformer pre-trained on synthetic Poisson data can beat the classical NPMLE estimator on several empirical Bayes tasks and run about 100x faster.
-
Enhancing Chain-of-Thought Reasoning with Critical Representation Fine-tuning
CRFT selects critical internal representations via attention and saliency scores and fine-tunes only them, improving GSM8K accuracy over ReFT from 29.0% to 32.8% on LLaMA-2-7B.
-
Prompting Wireless Networks: Reinforced In-Context Learning for Power Control
Prompting LLMs with a few reward-ranked state-action examples controls base station power at a level comparable to a trained DQN on a small simulated problem.
-
The Prompt is Mightier than the Example
Injecting domain knowledge into prompts can substitute for many in-context examples in LLM-based synthetic tabular data generation, cutting required example counts by 40-90%.
-
In-context learning of closed form solution to simple linear regression task using transformer with linear self-attention
A two-layer linear-attention transformer can approximate the closed-form least-squares solution by using layer normalization for division, and one l1-regularized training run mostly reproduces this construction.
-
Thinking About Thinking: SAGE-nano's Inverse Reasoning for Self-Aware Language Models
A 4B-parameter model is claimed to explain its own reasoning through inverse attention analysis, but the paper offers no consistent evidence or artifacts.
Discussion (0). Continue with ORCID to comment.