REVIEW 7 cited by
Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
In recent years, pre-trained large language models (LLMs) have demonstrated remarkable efficiency in achieving an inference-time few-shot learning capability known as in-context learning. However, existing literature has highlighted the sensitivity of this capability to the selection of few-shot demonstrations. Current understandings of the underlying mechanisms by which this capability arises from regular language model pretraining objectives remain disconnected from the real-world LLMs. This study aims to examine the in-context learning phenomenon through a Bayesian lens, viewing real-world LLMs as latent variable models. On this premise, we propose an algorithm to select optimal demonstrations from a set of annotated data with a small LM, and then directly generalize the selected demonstrations to larger LMs. We demonstrate significant improvement over baselines, averaged over eight GPT models on eight real-world text classification datasets. We also demonstrate the real-world usefulness of our algorithm on GSM8K, a math word problem dataset. Our empirical findings support our hypothesis that LLMs implicitly infer a latent variable containing task information.
Forward citations
Cited by 7 Pith papers
-
Adaptive Task Vectors for Large Language Models
Adaptive Task Vectors use a small model to generate query-specific steering vectors for frozen LLMs, reporting strong accuracy and generalization, though the theoretical equivalences to LoRA and Prefix-Tuning are not ...
-
Transformers as Multi-task Learners: Decoupling Features in Hidden Markov Models
A hand-constructed O(ln L + T)-layer Transformer is shown to approximate low-rank hidden Markov models in-context, with lower layers extracting local history features and upper layers performing regression-based decoding.
-
The Role of Diversity in In-Context Learning for Large Language Models
Diversity-aware selection of in-context examples improves performance on complex and out-of-distribution tasks, though effect sizes are often modest.
-
Scaling sparse feature circuit finding for in-context learning
Task vectors for in-context learning decompose into sparse SAE features that detect and execute tasks, linked by an attention/MLP circuit in Gemma-1 2B.
-
SketchFlex: Facilitating Spatial-Semantic Coherence in Text-to-Image Generation with Region-Based Sketches
SketchFlex combines sketch-aware prompt recommendation with decompose-and-recompose shape refinement to help novices generate multi-object images from rough region sketches.
-
DICE: Dynamic In-Context Example Selection in LLM Agents via Efficient Knowledge Transfer
DICE dynamically retrieves the most relevant in-context demonstrations at each agent step, and in this preprint it raises exact-match and success-rate scores on HotpotQA, ALFWorld, and Webshop across ReAct, Reflexion,...
-
Towards Contamination Resistant Benchmarks
The authors define contamination resistance as a benchmark property and show that most tested LLMs score near zero on Caesar-cipher encoding and decoding when the shift is not 3 and the text is random nonsense.
Discussion (0). Continue with ORCID to comment.