Pith. sign in

REVIEW 5 cited by

Language Models Implement Simple Word2Vec-style Vector Arithmetic

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2305.16130 v3 pith:3RKLZ72P submitted 2023-05-25 cs.CL cs.LG

Jack Merullo , Carsten Eickhoff , Ellie Pavlick This is my paper · ORCID
classification cs.CLcs.LG
keywords taskslanguagemechanismmodelssimplearithmeticdespitemodel
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

A primary criticism towards language models (LMs) is their inscrutability. This paper presents evidence that, despite their size and complexity, LMs sometimes exploit a simple vector arithmetic style mechanism to solve some relational tasks using regularities encoded in the hidden space of the model (e.g., Poland:Warsaw::China:Beijing). We investigate a range of language model sizes (from 124M parameters to 176B parameters) in an in-context learning setting, and find that for a variety of tasks (involving capital cities, uppercasing, and past-tensing) a key part of the mechanism reduces to a simple additive update typically applied by the feedforward (FFN) networks. We further show that this mechanism is specific to tasks that require retrieval from pretraining memory, rather than retrieval from local context. Our results contribute to a growing body of work on the interpretability of LMs, and offer reason to be optimistic that, despite the massive and non-linear nature of the models, the strategies they ultimately use to solve tasks can sometimes reduce to familiar and even intuitive algorithms.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Verbalizable Representations Form a Global Workspace in Language Models

    cs.CL 2026-07 conditional novelty 7.0 of 10

    Language models represent their current reasoning in a small, readable set of verbalizable vectors (the J-space) that functions like a global workspace.

  2. Train Once, Reuse Everywhere: Generalizable Implicit In-Context Learning by Routing Attention

    cs.CL 2025-09 conditional novelty 6.0 of 10

    ICR extracts shared attention directions from in-context learning and routes them at inference time, enabling zero-shot reuse across tasks.

  3. Paths Not Taken: Understanding and Mending the Multilingual Factual Recall Pipeline

    cs.CL 2025-05 conditional novelty 6.0 of 10

    LLMs recall facts through an English-centric internal path and then translate the answer; injecting a translation vector and a recall vector raises accuracy by over 35 percentage points in the weakest language.

  4. Linear Relational Decoding of Morphology in Language Models

    cs.CL 2025-07 conditional novelty 5.0 of 10

    A single matrix multiplication on middle-layer word states reproduces morphological transformations in GPT-J and Llama-7b with roughly 90% faithfulness.

  5. SkipGPT: Dynamic Layer Pruning Reinvented with Token Awareness and Module Decoupling

    cs.CL 2025-06 conditional novelty 4.0 of 10

    SkipGPT learns per-token, per-module routers that dynamically skip attention and MLP blocks, then uses LoRA to recover most of the original model's quality.

Pith tools