Pith. sign in

REVIEW 9 cited by

MetaICL: Learning to Learn In Context

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2110.15943 v2 pith:2WQDSFQN submitted 2021-10-29 cs.CL cs.AI

Sewon Min , Mike Lewis , Luke Zettlemoyer , Hannaneh Hajishirzi This is my paper
classification cs.CLcs.AI
keywords meta-traininglearningmetaicltasksin-contexttargetapproachescontext
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We introduce MetaICL (Meta-training for In-Context Learning), a new meta-training framework for few-shot learning where a pretrained language model is tuned to do in-context learning on a large set of training tasks. This meta-training enables the model to more effectively learn a new task in context at test time, by simply conditioning on a few training examples with no parameter updates or task-specific templates. We experiment on a large, diverse collection of tasks consisting of 142 NLP datasets including classification, question answering, natural language inference, paraphrase detection and more, across seven different meta-training/target splits. MetaICL outperforms a range of baselines including in-context learning without meta-training and multi-task learning followed by zero-shot transfer. We find that the gains are particularly significant for target tasks that have domain shifts from the meta-training tasks, and that using a diverse set of the meta-training tasks is key to improvements. We also show that MetaICL approaches (and sometimes beats) the performance of models fully finetuned on the target task, and outperforms much bigger models with nearly 8x parameters. Finally, we show that MetaICL is complementary to human-written instructions, and the best performance can be achieved by combining both approaches.

Discussion (0). Sign in to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MimicDroid: In-Context Learning for Humanoid Robot Manipulation from Human Play Videos

    cs.RO 2025-09 conditional novelty 7.0 of 10

    Trained only on unlabeled human play videos, MimicDroid lets a GR1 humanoid perform new manipulation tasks from one to three demonstration videos, with roughly twice the real-world success of prior video-conditioned methods.

  2. Can Gradient Descent Simulate Prompting?

    cs.CL 2025-06 conditional novelty 7.0 of 10

    A MAML-style meta-training objective makes a single gradient step on new text recover part of the performance that prompting achieves, on reversal-curse and passage-QA tasks.

  3. Retrieval-Augmented Generation as Noisy In-Context Learning: A Unified Theory and Risk Bounds

    cs.LG 2025-06 conditional novelty 7.0 of 10

    RAG in in-context linear regression has an exact bias-variance tradeoff and a finite-sample bound revealing a generalization ceiling as retrieved examples grow.

  4. MicLog: Towards Accurate and Efficient LLM-based Log Parsing via Progressive Meta In-Context Learning

    cs.SE 2026-01 conditional novelty 6.0 of 10

    A small open-source LLM trained with progressive meta in-context learning parses logs more accurately and faster than prior state-of-the-art LLM parsers on Loghub-2.0.

  5. Train Once, Reuse Everywhere: Generalizable Implicit In-Context Learning by Routing Attention

    cs.CL 2025-09 conditional novelty 6.0 of 10

    ICR extracts shared attention directions from in-context learning and routes them at inference time, enabling zero-shot reuse across tasks.

  6. Agentic Services Computing

    cs.SE 2025-09 conditional novelty 5.0 of 10

    A position and survey paper that defines Agentic Services Computing, a lifecycle-based framework for engineering LLM agents as governed, first-class services.

  7. InSQuAD: In-Context Learning for Efficient Retrieval via Submodular Mutual Information to Enforce Quality and Diversity

    cs.LG 2025-08 conditional novelty 5.0 of 10

    A submodular mutual information framework for selecting and training in-context learning exemplars improves average accuracy on nine benchmarks by about five points over the IDEAL baseline.

  8. ICM-Fusion: In-Context Meta-Optimized LoRA Fusion for Multi-Task Adaptation

    cs.CV 2025-08 reject novelty 5.0 of 10

    ICM-Fusion uses a conditional VAE plus task-vector guidance to fuse multiple LoRA adapters into one model, reporting marginal average gains on vision and language benchmarks and larger gains in a few-shot long-tail setup.

  9. DICE: Dynamic In-Context Example Selection in LLM Agents via Efficient Knowledge Transfer

    cs.AI 2025-07 conditional novelty 5.0 of 10

    DICE dynamically retrieves the most relevant in-context demonstrations at each agent step, and in this preprint it raises exact-match and success-rate scores on HotpotQA, ALFWorld, and Webshop across ReAct, Reflexion,...

Pith tools