Pith. sign in

REVIEW 10 cited by

Universal Neurons in GPT2 Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2401.12181 v1 pith:C6HGH2UC submitted 2024-01-22 cs.LG cs.AIcs.CL

Wes Gurnee , Theo Horsley , Zifan Carl Guo , Tara Rezaei Kheirkhah , Qinyi Sun , Will Hathaway , Neel Nanda , Dimitris Bertsimas This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords neuronsuniversalneuronacrossdifferentmodelsgpt2mechanisms
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

A basic question within the emerging field of mechanistic interpretability is the degree to which neural networks learn the same underlying mechanisms. In other words, are neural mechanisms universal across different models? In this work, we study the universality of individual neurons across GPT2 models trained from different initial random seeds, motivated by the hypothesis that universal neurons are likely to be interpretable. In particular, we compute pairwise correlations of neuron activations over 100 million tokens for every neuron pair across five different seeds and find that 1-5\% of neurons are universal, that is, pairs of neurons which consistently activate on the same inputs. We then study these universal neurons in detail, finding that they usually have clear interpretations and taxonomize them into a small number of neuron families. We conclude by studying patterns in neuron weights to establish several universal functional roles of neurons in simple circuits: deactivating attention heads, changing the entropy of the next token distribution, and predicting the next token to (not) be within a particular set.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. What, Where, and How: Disentangling the Roles of Task, Language, and Model in Code Model Representations

    cs.CL 2026-07 conditional novelty 7.0 of 10

    Across Python and Rust, two code models conserve the ranking of which constructs get dedicated circuits (ρ≈0.65) but differ by 12-13 layers in where those circuits sit and whether they show an early spike.

  2. Verbalizable Representations Form a Global Workspace in Language Models

    cs.CL 2026-07 conditional novelty 7.0 of 10

    Language models represent their current reasoning in a small, readable set of verbalizable vectors (the J-space) that functions like a global workspace.

  3. After the Euclidean Highway: Hyperbolic Expert AI as the Next Innovation

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Hyperbolic losses at the loss layer alone train stably at 18M-sample scale, while the same geometry inside a trainable adapter is claimed to collapse training.

  4. Transformers converge to invariant algorithmic cores

    cs.LG 2026-02 conditional novelty 6.0 of 10

    Trained transformers contain low-dimensional causal subspaces — algorithmic cores — that recur across runs and scales and can be extracted, characterized, and steered.

  5. Establishing Trustworthy LLM Evaluation via Shortcut Neuron Analysis

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Shortcut neuron patching suppresses benchmark-contamination shortcuts in LLMs and yields evaluation scores that strongly correlate with the external MixEval benchmark.

  6. Understanding Gated Neurons in Transformers from Their Input-Output Functionality

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Across 12 language models, neurons in early-middle layers tend to add the direction they detect (enrichment), while later layers tend to reduce it (depletion), based on input-output weight cosine similarity.

  7. Expand Neurons, Not Parameters

    cs.LG 2025-10 reject novelty 5.0 of 10

    Fixed Parameter Expansion — duplicating neurons and partitioning their incoming weights into disjoint sparse sub-neurons at constant non-zero parameter count — reduces measured feature interference and improves classi...

  8. From Indirect Object Identification to Syllogisms: Exploring Binary Mechanisms in Transformer Circuits

    cs.CL 2025-08 conditional novelty 5.0 of 10

    GPT-2 small performs syllogisms through truth-copying attention heads and a suppression-plus-MLP pathway that can output a negated truth value.

  9. Detecting and Pruning Prominent but Detrimental Neurons in Large Language Models

    cs.CL 2025-07 conditional novelty 5.0 of 10

    Pruning the most attribution-prominent MLP neurons in a single layer, chosen via a 10-sample validation sweep, consistently improves multiple-choice accuracy across four instruction-tuned LLMs.

  10. Semantic Convergence: Investigating Shared Representations Across Scaled LLMs

    cs.CL 2025-07 conditional novelty 4.0 of 10

    Gemma-2-2B and Gemma-2-9B align most strongly on SAE-derived features in middle layers, with preliminary evidence for shared multi-token concept subspaces.

Pith tools