Pith. sign in

REVIEW 6 cited by

In-context Learning Distillation: Transferring Few-shot Learning Ability of Pre-trained Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2212.10670 v1 pith:BF353SZE submitted 2022-12-20 cs.CL cs.LG

Yukun Huang , Yanda Chen , Zhou Yu , Kathleen McKeown This is my paper
classification cs.CLcs.LG
keywords in-contextlearningobjectiveslanguagemodelsfew-shotmultitask-ictability
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Given the success with in-context learning of large pre-trained language models, we introduce in-context learning distillation to transfer in-context few-shot learning ability from large models to smaller models. We propose to combine in-context learning objectives with language modeling objectives to distill both the ability to read in-context examples and task knowledge to the smaller models. We perform in-context learning distillation under two different few-shot learning paradigms: Meta In-context Tuning (Meta-ICT) and Multitask In-context Tuning (Multitask-ICT). Multitask-ICT performs better on multitask few-shot learning but also requires more computation than Meta-ICT. Our method shows consistent improvements for both Meta-ICT and Multitask-ICT on two benchmarks: LAMA and CrossFit. Our extensive experiments and analysis reveal that in-context learning objectives and language modeling objectives are complementary under the Multitask-ICT paradigm. In-context learning objectives achieve the best performance when combined with language modeling objectives.

Discussion (0). Sign in to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 9 citations worldwide. Full citation record

  1. Sample-Efficient Learning from Agent Experience

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Agents can consolidate in-context trial-and-error learning into their weights by distilling one-step teacher decisions at recorded histories, needing no additional environment interaction.

  2. Can Smaller LLMs do better? Unlocking Cross-Domain Potential through Parameter-Efficient Fine-Tuning for Text Summarization

    cs.CL 2025-09 reject novelty 5.0 of 10

    PEFT adapters trained on high-resource summarization domains can improve Llama-3-8B's summaries on unseen domains, but the reported gains are weakened by test-set selection and missing significance tests.

  3. AgentDistill: Training-Free Agent Distillation with Generalizable MCP Boxes

    cs.AI 2025-06 reject novelty 5.0 of 10

    AgentDistill distills agent capabilities without any training by having a teacher generate reusable MCP tool boxes that small-model students invoke at inference time.

  4. Exploring the Limits of Model Compression in LLMs: A Knowledge Distillation Study on QA Tasks

    cs.CL 2025-07 conditional novelty 4.0 of 10

    Distilled students at 43% to 50% of teacher size keep over 90% of teacher Exact Match on SQuAD and MLQA, though one-shot gains reverse on SQuAD test for Pythia.

  5. Large Language models for Time Series Analysis: Techniques, Applications, and Challenges

    cs.LG 2025-05 reject novelty 3.0 of 10

    A review of LLM-based time series analysis that proposes several taxonomies, but is undermined by citation errors and a lack of systematic methodology.

  6. RLRC: Reinforcement Learning-based Recovery for Compressed Vision-Language-Action Models

    cs.RO 2025-06

Pith tools