Pith. sign in

REVIEW 10 cited by

Calibrate Before Use: Improving Few-Shot Performance of Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2102.09690 v2 pith:MIUIVO5A submitted 2021-02-19 cs.CL cs.LG

Tony Z. Zhao , Eric Wallace , Shi Feng , Dan Klein , Sameer Singh This is my paper · ORCID
classification cs.CLcs.LG
keywords prompttrainingexampleslanguagenearaccuracyacrossanswers
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

GPT-3 can perform numerous tasks when provided a natural language prompt that contains a few training examples. We show that this type of few-shot learning can be unstable: the choice of prompt format, training examples, and even the order of the training examples can cause accuracy to vary from near chance to near state-of-the-art. We demonstrate that this instability arises from the bias of language models towards predicting certain answers, e.g., those that are placed near the end of the prompt or are common in the pre-training data. To mitigate this, we first estimate the model's bias towards each answer by asking for its prediction when given the training prompt and a content-free test input such as "N/A". We then fit calibration parameters that cause the prediction for this input to be uniform across answers. On a diverse set of tasks, this contextual calibration procedure substantially improves GPT-3 and GPT-2's average accuracy (up to 30.0% absolute) and reduces variance across different choices of the prompt.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 72 citations worldwide. Full citation record

  1. EsBBQ and CaBBQ: The Spanish and Catalan Bias Benchmarks for Question Answering

    cs.CL 2025-07 conditional novelty 7.0 of 10

    EsBBQ and CaBBQ are new Spanish and Catalan bias benchmarks for multiple-choice QA, built with survey-validated stereotypes from Spain and evaluated on 17 language models.

  2. Controlling the Risk of Corrupted Contexts for Language Models via Early-Exiting

    cs.AI 2025-10 conditional novelty 6.0 of 10

    An early-exit rule with a zero-shot fallback, calibrated by Learn-then-Test risk control, keeps the average loss from corrupted in-context demonstrations under a preset bound.

  3. Characterizing Fitness Landscape Structures in Prompt Engineering

    cs.AI 2025-09 reject novelty 6.0 of 10

    Prompt fitness autocorrelation appears smooth under systematic enumeration but rugged with an intermediate-distance peak under novelty-driven sampling, yet the two analyses cover non-overlapping distance ranges.

  4. AURA: Affordance-Understanding and Risk-aware Alignment Technique for Large Language Models

    cs.CL 2025-08 unverdicted novelty 5.0 of 10

    AURA uses step-level reward models, self-critique, and safety-aware decoding to reduce affordance-based safety failures in LLM outputs.

  5. (Towards) Scalable Reliable Automated Evaluation with Large Language Models

    cs.CL 2026-07 conditional novelty 4.0 of 10

    Multi-LLM pairwise Elo ranking with adjustable consensus thresholds produces rankings of competency profiles that average Spearman ρ≈0.83 with expert judgments.

  6. Fine-tuning on simulated data outperforms prompting for agent tone of voice

    cs.LG 2025-07 conditional novelty 4.0 of 10

    Fine-tuning a 1B-parameter LLM on as few as 100 synthetically generated, readability-filtered samples achieved conversational tone more reliably than a verbose system prompt.

  7. Dynamic Context-Aware Prompt Recommendation for Domain-Specific AI Applications

    cs.AI 2025-06 conditional novelty 4.0 of 10

    A dynamic prompt recommendation system for skill-based security copilots combines retrieval, hierarchical skill selection, and telemetry-based ranking, reporting high usefulness in internal evaluations.

  8. Extending Epistemic Uncertainty Beyond Parameters Would Assist in Designing Reliable LLMs

    cs.LG 2025-06 conditional novelty 4.0 of 10

    Bayesian Modeling of Experiments is proposed as a unifying framework for quantifying and reducing the many sources of uncertainty in LLM deployments, beyond abstention.

  9. Recalibrating the Compass: Integrating Large Language Models into Classical Research Methods

    cs.AI 2025-05 accept novelty 4.0 of 10

    LLMs extend, rather than replace, classical social science methods, with a proposed three-tier bias framework for LLM-augmented surveys.

  10. Few-Shot Optimization for Sensor Data Using Large Language Models: A Case Study on Fatigue Detection

    cs.CL 2025-05 reject novelty 4.0 of 10

    A hybrid Euclidean-distance and LLM-relevance example selector for few-shot sensor classification reports a small, statistically fragile gain over distance-only selection on a fatigue detection dataset.

Pith tools