Pith. sign in

REVIEW 8 cited by

Making Pre-trained Language Models Better Few-shot Learners

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2012.15723 v2 pith:ARI72DIS submitted 2020-12-31 cs.CL cs.LG

Tianyu Gao , Adam Fisch , Danqi Chen This is my paper
classification cs.CLcs.LG
keywords few-shotfine-tuninglanguagemodelsapproachcontextdemonstrationslearning
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

The recent GPT-3 model (Brown et al., 2020) achieves remarkable few-shot performance solely by leveraging a natural-language prompt and a few task demonstrations as input context. Inspired by their findings, we study few-shot learning in a more practical scenario, where we use smaller language models for which fine-tuning is computationally efficient. We present LM-BFF--better few-shot fine-tuning of language models--a suite of simple and complementary techniques for fine-tuning language models on a small number of annotated examples. Our approach includes (1) prompt-based fine-tuning together with a novel pipeline for automating prompt generation; and (2) a refined strategy for dynamically and selectively incorporating demonstrations into each context. Finally, we present a systematic evaluation for analyzing few-shot performance on a range of NLP tasks, including classification and regression. Our experiments demonstrate that our methods combine to dramatically outperform standard fine-tuning procedures in this low resource setting, achieving up to 30% absolute improvement, and 11% on average across all tasks. Our approach makes minimal assumptions on task resources and domain expertise, and hence constitutes a strong task-agnostic method for few-shot learning.

Discussion (0). Sign in to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. What Should LLMs Forget? Quantifying Personal Data in LLMs for Right-to-Be-Forgotten Requests

    cs.CL 2025-07 conditional novelty 6.0 of 10

    WikiMem, a Wikidata-derived canary dataset and a calibrated NLL-ranking metric, identifies which human-fact associations an LLM has memorized, with higher rates for famous people and larger models.

  2. Stabilizing Black-Box Prompt Optimization with Textual Regularization and Signal Aggregation

    cs.LG 2025-07 conditional novelty 6.0 of 10

    TRAS adds success-based textual regularization and Monte Carlo signal aggregation to black-box prompt optimization, improving accuracy and reducing instruction loss when moving prompts across models.

  3. Structuralist Approach to AI Literary Criticism: Leveraging Greimas Semiotic Square for Large Language Models

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A structured prompting framework based on Greimas semiotic square helps LLMs produce Greimas-style literary analyses, and the paper claims these outputs score above human expert criticism under LLM-as-judge metrics.

  4. The Few-shot Dilemma: Over-prompting Large Language Models

    cs.CL 2025-09 conditional novelty 5.0 of 10

    Across seven LLMs on two requirements datasets, F1 scores rise then fall as more few-shot examples are added, and TF-IDF-selected examples at small counts match or beat larger prompts, including a 1% gain over prior SOTA.

  5. Membership Inference Attacks on In-Context Examples in LLM-based Recommender Systems

    cs.IR 2025-08 conditional novelty 5.0 of 10

    Simply asking a large language model 'have you seen this user?' or comparing its recommendations after prompt poisoning can reveal whether a user's interactions are in the hidden prompt of an ICL-based recommender.

  6. TrackRec: Iterative Alternating Feedback with Chain-of-Thought via Preference Alignment for Recommendation

    cs.IR 2025-08 conditional novelty 5.0 of 10

    TrackRec trains a small LLM to generate user-preference summaries and a validator to score them, alternating the training so each improves the other, and reports gains on public and industrial recommendation benchmarks.

  7. Unveiling Effective In-Context Configurations for Image Captioning: An External & Internal Analysis

    cs.CL 2025-07 conditional novelty 5.0 of 10

    For Flamingo-style models, increasing the number of in-context examples improves language coherence but degrades visual-text alignment, and similarity-based image retrieval inflates CIDEr scores by encouraging caption...

  8. Hierarchical Self-Prompting SAM: A Prompt-Free Medical Image Segmentation Framework

    cs.CV 2025-06 conditional novelty 5.0 of 10

    HSP-SAM adds learned abstract prompt pairs to SAM, achieving prompt-free medical image segmentation with reported zero-shot improvements of up to 14.04 percent Dice.

Pith tools