REVIEW 5 cited by
Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Causal abstraction is a promising theoretical framework for explainable artificial intelligence that defines when an interpretable high-level causal model is a faithful simplification of a low-level deep learning system. However, existing causal abstraction methods have two major limitations: they require a brute-force search over alignments between the high-level model and the low-level one, and they presuppose that variables in the high-level model will align with disjoint sets of neurons in the low-level one. In this paper, we present distributed alignment search (DAS), which overcomes these limitations. In DAS, we find the alignment between high-level and low-level models using gradient descent rather than conducting a brute-force search, and we allow individual neurons to play multiple distinct roles by analyzing representations in non-standard bases-distributed representations. Our experiments show that DAS can discover internal structure that prior approaches miss. Overall, DAS removes previous obstacles to conducting causal abstraction analyses and allows us to find conceptual structure in trained neural nets.
Forward citations
Cited by 5 Pith papers
-
LAWFUL: Law-Aligned Witness for Faithful Use of Latents
LAWFUL defines coverage-aware physical-consistency scores and circuit tests, reporting that a MoCap-to-Radar transformer's 9-component temporal circuit carries Doppler-law consistency via attention patterns.
-
Emergent Misalignment Recruits a Pre-existing Persona Subspace
Fine-tuning on narrow bad data recruits a low-rank persona subspace already present in a frozen instruction-tuned model; holding that subspace out of activations prevents broad misalignment, and injecting it into the ...
-
Persistent Sparse Autoencoders: Learning Feature Timescales in Language Models
Persistent SAEs learn per-feature persistence coefficients from reconstruction, splitting features into fast local detectors and slow topic-tracking states that retain prompt-injection signals over long contexts.
-
What is a Number, That a Large Language Model May Know It?
LLM similarity ratings over number pairs are best explained by combining Levenshtein string edit distance with a log-linear numerical distance, indicating entangled string and numeric representations.
-
Activation Reward Models for Few-Shot Model Alignment
Mean attention-head activations from a few labeled examples, injected into selected heads, turn a frozen vision-language model into a few-shot reward model that beats prompting and scoring baselines and a new reward-h...
Discussion (0). Continue with ORCID to comment.