Pith. sign in

REVIEW 11 cited by

Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2305.14160 v4 pith:EMM72R4U submitted 2023-05-23 cs.CL cs.LG

Lean Wang , Lei Li , Damai Dai , Deli Chen , Hao Zhou , Fandong Meng , Jie Zhou , Xu Sun This is my paper
classification cs.CLcs.LG
keywords informationlabeldemonstrationllmsmechanismwordsanchorsexamples
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

In-context learning (ICL) emerges as a promising capability of large language models (LLMs) by providing them with demonstration examples to perform diverse tasks. However, the underlying mechanism of how LLMs learn from the provided context remains under-explored. In this paper, we investigate the working mechanism of ICL through an information flow lens. Our findings reveal that label words in the demonstration examples function as anchors: (1) semantic information aggregates into label word representations during the shallow computation layers' processing; (2) the consolidated information in label words serves as a reference for LLMs' final predictions. Based on these insights, we introduce an anchor re-weighting method to improve ICL performance, a demonstration compression technique to expedite inference, and an analysis framework for diagnosing ICL errors in GPT2-XL. The promising applications of our findings again validate the uncovered ICL working mechanism and pave the way for future studies.

Discussion (0). Sign in to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Multimodal Function Vectors for Visual Relations

    cs.AI 2025-10 conditional novelty 6.0 of 10

    Multimodal function vectors extracted from a handful of attention heads in OpenFlamingo-4B encode spatial relations and can be steered, fine-tuned, and composed to improve zero-shot relational reasoning.

  2. Semantic Causality-Aware Vision-Based 3D Occupancy Prediction

    cs.CV 2025-09 conditional novelty 6.0 of 10

    A class-conditional gradient loss (Causal Loss) plus channel-grouped lifting, learnable camera offsets, and normalized convolution raises Occ3D mIoU by 1.2/0.8 points and cuts the camera-noise mIoU drop from 32% to 7%.

  3. Not All Tokens and Heads Are Equally Important: Dual-Level Attention Intervention for Hallucination Mitigation

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A dual-level attention intervention that boosts salient visual-token attention and suppresses text/system attention during decoding reduces hallucination rates in LLaVA, MiniGPT-4, and mPLUG-Owl2 on POPE and CHAIR.

  4. ConText: Driving In-context Learning for Text Removal and Segmentation

    cs.CV 2025-06 conditional novelty 6.0 of 10

    ConText is the first visual in-context learning model for text removal and segmentation, chaining the two tasks and using self-prompting to reach new state-of-the-art scores.

  5. Pierce the Mists, Greet the Sky: Decipher Knowledge Overshadowing via Knowledge Circuit Analysis

    cs.CL 2025-05 conditional novelty 6.0 of 10

    PhantomCircuit traces knowledge overshadowing to attention circuits during training and prunes circuit edges to recover the overshadowed answer.

  6. Enhancing Chain-of-Thought Reasoning with Critical Representation Fine-tuning

    cs.CL 2025-07 conditional novelty 5.0 of 10

    CRFT selects critical internal representations via attention and saliency scores and fine-tunes only them, improving GSM8K accuracy over ReFT from 29.0% to 32.8% on LLaMA-2-7B.

  7. Unveiling Effective In-Context Configurations for Image Captioning: An External & Internal Analysis

    cs.CL 2025-07 conditional novelty 5.0 of 10

    For Flamingo-style models, increasing the number of in-context examples improves language coherence but degrades visual-text alignment, and similarity-based image retrieval inflates CIDEr scores by encouraging caption...

  8. Understanding the learned look-ahead behavior of chess neural networks

    cs.AI 2025-05 conditional novelty 5.0 of 10

    The Leela Chess Zero policy network encodes information about destination squares of moves up to seven plies ahead, with attention heads that copy future-square information backward in time in a pattern-dependent way.

  9. NEAT: Concept driven Neuron Attribution in LLMs

    cs.CL 2025-08 reject novelty 4.0 of 10

    NEAT identifies concept neurons by feeding a single mean hidden-state vector through the model and ranking neurons by their effect on concept-word probabilities.

  10. MCA-LLaVA: Manhattan Causal Attention for Reducing Hallucination in Large Vision-Language Models

    cs.CV 2025-07 conditional novelty 4.0 of 10

    MCA-LLaVA reindexes image tokens by sums of mirrored 2D coordinates so instruction tokens attend across the whole image, reducing hallucination on POPE, CHAIR, and MME.

  11. Large Language Models as Computable Approximations to Solomonoff Induction

    cs.LG 2025-05 reject novelty 2.0 of 10

    The paper argues LLMs are computable approximations of Solomonoff induction, but its central derivation recovers the model's own probabilities by construction.

Pith tools