Pith. sign in

REVIEW 18 cited by

Larger language models do in-context learning differently

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2303.03846 v2 pith:WBLOU7HT submitted 2023-03-07 cs.CL

Jerry Wei , Jason Wei , Yi Tay , Dustin Tran , Albert Webson , Yifeng Lu , Xinyun Chen , Hanxiao Liu
show 3 more authors
Da Huang Denny Zhou Tengyu Ma
This is my paper · ORCID
classification cs.CL
keywords modelspriorssemanticin-contextlanguagelabelsflippedinput-label
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We study how in-context learning (ICL) in language models is affected by semantic priors versus input-label mappings. We investigate two setups-ICL with flipped labels and ICL with semantically-unrelated labels-across various model families (GPT-3, InstructGPT, Codex, PaLM, and Flan-PaLM). First, experiments on ICL with flipped labels show that overriding semantic priors is an emergent ability of model scale. While small language models ignore flipped labels presented in-context and thus rely primarily on semantic priors from pretraining, large models can override semantic priors when presented with in-context exemplars that contradict priors, despite the stronger semantic priors that larger models may hold. We next study semantically-unrelated label ICL (SUL-ICL), in which labels are semantically unrelated to their inputs (e.g., foo/bar instead of negative/positive), thereby forcing language models to learn the input-label mappings shown in in-context exemplars in order to perform the task. The ability to do SUL-ICL also emerges primarily with scale, and large-enough language models can even perform linear classification in a SUL-ICL setting. Finally, we evaluate instruction-tuned models and find that instruction tuning strengthens both the use of semantic priors and the capacity to learn input-label mappings, but more of the former.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 18 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 100 citations worldwide. Full citation record

  1. Epistemic Familiarity is Associated With Belief Stability in Large Language Models

    cs.CL 2025-11 conditional novelty 7.0 of 10

    Language models retract previously true answers far more often after seeing unfamiliar synthetic statements than after seeing familiar fictional statements, in both internal probes and prompted behavior.

  2. Test-Time Scaling via Error Localization

    cs.LG 2026-07 conditional novelty 6.0 of 10

    TTEL uses feedback-induced token probability drops to localize the first error in a failed reasoning trace and branch a new generation from that prefix, improving pass@k per token on coding and math benchmarks.

  3. Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatch

    stat.ML 2026-07 conditional novelty 6.0 of 10

    Within-context token correlations reduce ICL to an effective shorter i.i.d. context length, while query–context correlations lower error and favor softmax over linear attention.

  4. Dissecting Multimodal In-Context Learning: Modality Asymmetries and Circuit Dynamics in modern Transformers

    cs.CL 2026-01 conditional novelty 6.0 of 10

    In a two-modality transformer, a primary-modality pretraining stage installs an induction circuit, so the secondary modality needs only low class diversity to learn in-context from examples.

  5. HalluJudge: A Reference-Free Hallucination Detection for Context Misalignment in Code Review Automation

    cs.SE 2026-01 conditional novelty 6.0 of 10

    HalluJudge detects hallucinated LLM code-review comments by scoring context alignment between comment and diff, reaching F1 0.85 on 143 annotated enterprise comments.

  6. Filtering Learning Histories Enhances In-Context Reinforcement Learning

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Filtering ICRL pretraining datasets by a simple improvement-and-stability score boosts downstream in-context learning performance across AD, DICP, and DPT baselines.

  7. Investigating the Zone of Proximal Development of Language Models for In-Context Learning

    cs.CL 2025-02 conditional novelty 6.0 of 10

    A framework that predicts, per query, whether an LLM can solve it directly, only with demonstrations, or not at all, and uses those predictions for selective in-context learning and curriculum fine-tuning.

  8. TACTIC-KG: Toward Small Agent Teams for Cyber Threat Intelligence Knowledge Graph Construction

    cs.CR 2026-07 conditional novelty 5.5 of 10

    Specialized 3B–8B LLM agents for extraction, typing, verification, and curation outperform much larger monolithic ICL models on CTI knowledge-graph construction.

  9. Data-Efficient Adaptation of LLMs via Attention Head Reweighting

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Learning a single scalar per attention head lets LLMs adapt to few-shot text classification better than LoRA, with 200–1000x fewer trainable parameters.

  10. TRUE: A Trustworthy Unified Explanation Framework for Large Language Model Reasoning

    cs.LG 2026-02 reject novelty 5.0 of 10

    TRUE checks whether LLM reasoning traces are self-sufficient by executing them blind, maps neighboring reasoning paths into a DAG, and ranks recurring failure modes by Shapley values.

  11. The Few-shot Dilemma: Over-prompting Large Language Models

    cs.CL 2025-09 conditional novelty 5.0 of 10

    Across seven LLMs on two requirements datasets, F1 scores rise then fall as more few-shot examples are added, and TF-IDF-selected examples at small counts match or beat larger prompts, including a 1% gain over prior SOTA.

  12. ALLabel: Three-stage Active Learning for LLM-based Entity Recognition using Demonstration Retrieval

    cs.CL 2025-09 conditional novelty 5.0 of 10

    ALLabel combines diversity, similarity, and uncertainty sampling to pick the most useful texts for human annotation, reaching near-full-data NER accuracy with 5-10% of labels.

  13. Rethinking the Chain-of-Thought: The Roles of In-Context Learning and Pre-trained Priors

    cs.CL 2025-09 conditional novelty 5.0 of 10

    The paper finds that as exemplar count increases, LLMs shift from pretrained priors to in-context signals, and long chain-of-thought prompts induce longer reasoning chains that often improve accuracy.

  14. InSQuAD: In-Context Learning for Efficient Retrieval via Submodular Mutual Information to Enforce Quality and Diversity

    cs.LG 2025-08 conditional novelty 5.0 of 10

    A submodular mutual information framework for selecting and training in-context learning exemplars improves average accuracy on nine benchmarks by about five points over the IDEAL baseline.

  15. Enhancing Reasoning Capabilities of Small Language Models with Blueprints and Prompt Template Search

    cs.LG 2025-06 conditional novelty 5.0 of 10

    Small language models achieve higher accuracy on math, coding, and logic benchmarks when their prompts contain LLM-generated reasoning blueprints and a per-model, per-task searched template.

  16. PromptGuard: An Orchestrated Prompting Framework for Principled Synthetic Text Generation for Vulnerable Populations using LLMs with Enhanced Safety, Fairness, and Controllability

    cs.CV 2025-09 reject novelty 4.0 of 10

    A framework paper that claims its VulnGuard prompt technique cuts harmful LLM outputs by 25-30% via theoretical bounds, without a real proof or empirical test.

  17. Thinking About Thinking: SAGE-nano's Inverse Reasoning for Self-Aware Language Models

    cs.AI 2025-06 reject novelty 3.0 of 10

    A 4B-parameter model is claimed to explain its own reasoning through inverse attention analysis, but the paper offers no consistent evidence or artifacts.

  18. Efficient Knowledge Feeding to Language Models: A Novel Integrated Encoder-Decoder Architecture

    cs.CL 2025-02 reject novelty 3.0 of 10

    A retrieval-augmented encoder-decoder that injects 'in-context vectors' into latent states is presented, with claims of competing with much larger RAG models on three QA benchmarks.

Pith tools