REVIEW 14 cited by
Dissecting Recall of Factual Associations in Auto-Regressive Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Transformer-based language models (LMs) are known to capture factual knowledge in their parameters. While previous work looked into where factual associations are stored, only little is known about how they are retrieved internally during inference. We investigate this question through the lens of information flow. Given a subject-relation query, we study how the model aggregates information about the subject and relation to predict the correct attribute. With interventions on attention edges, we first identify two critical points where information propagates to the prediction: one from the relation positions followed by another from the subject positions. Next, by analyzing the information at these points, we unveil a three-step internal mechanism for attribute extraction. First, the representation at the last-subject position goes through an enrichment process, driven by the early MLP sublayers, to encode many subject-related attributes. Second, information from the relation propagates to the prediction. Third, the prediction representation "queries" the enriched subject to extract the attribute. Perhaps surprisingly, this extraction is typically done via attention heads, which often encode subject-attribute mappings in their parameters. Overall, our findings introduce a comprehensive view of how factual associations are stored and extracted internally in LMs, facilitating future research on knowledge localization and editing.
Forward citations
Cited by 14 Pith papers
-
Verbalizable Representations Form a Global Workspace in Language Models
Language models represent their current reasoning in a small, readable set of verbalizable vectors (the J-space) that functions like a global workspace.
-
Aligned but Blind: Alignment Increases Implicit Bias by Reducing Awareness of Race
Alignment on Llama 3 reduces explicit bias but amplifies implicit bias, because aligned models no longer represent 'black' and 'white' as racial concepts in ambiguous contexts.
-
Prototype Transformer: Towards Language Model Architectures Interpretable by Design
ProtoT is an autoregressive language model whose attention is replaced by learned prototype channels that are claimed to capture nameable concepts and allow targeted edits, at linear sequence cost but with slightly lo...
-
Geometry of Reason: Spectral Signatures of Valid Mathematical Reasoning
Spectral features of attention are claimed to classify proof validity with near-perfect effect sizes, but the main evaluation relabels proofs using the classifier's own outputs.
-
Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimization
LLM attention maps reveal a preplan-and-anchor pattern, and reweighting RL credit toward the flagged tokens improves math/QA reasoning.
-
Potemkin Understanding in Large Language Models
LLMs frequently pass definition questions yet fail to use the same concepts in classification, generation, and editing tasks, a gap the authors call potemkin understanding.
-
Distinct Computations Emerge From Compositional Curricula in In-Context Learning
When transformer models see easy component examples before a harder combined math problem in one prompt, they solve unseen versions of the combined problem and store intermediate steps internally, unlike models traine...
-
Mamba Knockout for Unraveling Factual Information Flow
Fine-grained token-to-token knockout reveals that Mamba models, like Transformers, rely on subject-token information flow in late-intermediate layers, with architecture-specific variations in relation-token and first-...
-
Paths Not Taken: Understanding and Mending the Multilingual Factual Recall Pipeline
LLMs recall facts through an English-centric internal path and then translate the answer; injecting a translation vector and a recall vector raises accuracy by over 35 percentage points in the weakest language.
-
Linear Relational Decoding of Morphology in Language Models
A single matrix multiplication on middle-layer word states reproduces morphological transformations in GPT-J and Llama-7b with roughly 90% faithfulness.
-
On the Fundamental Impossibility of Hallucination Control in Large Language Models
The paper claims a mathematical impossibility: every capable LLM must violate at least one of four idealized response properties, so hallucination is structurally inevitable.
-
Understanding the learned look-ahead behavior of chess neural networks
The Leela Chess Zero policy network encodes information about destination squares of moves up to seven plies ahead, with attention heads that copy future-square information backward in time in a pattern-dependent way.
-
NEAT: Concept driven Neuron Attribution in LLMs
NEAT identifies concept neurons by feeding a single mean hidden-state vector through the model and ranking neurons by their effect on concept-word probabilities.
-
Neural Incompatibility: The Unbridgeable Gap of Cross-Scale Parametric Knowledge Transfer in Large Language Models
Directly transferring parameters between differently-sized language models is unreliable; the paper proposes a pre-alignment method (LaTen) and explains the failure via 'Neural Incompatibility'.
Discussion (0). Sign in to comment.