NEAT: Concept driven Neuron Attribution in LLMs

Gargi Shroff; Rahul Mishra; Vivek Hruday Kavuri

REVIEW 4 major objections 6 minor 1 cited by

NEAT: Concept driven Neuron Attribution in LLMs

T0 review · 4 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Ten neurons can carry a concept—NEAT finds them in one pass

desk verdict New recipe, plausible claims, but the layer-skipping approximation and circular evaluation leave the main claim unproven. read the letter →

arxiv 2508.15875 v1 pith:Z453SW4V submitted 2025-08-21 cs.CL cs.AIcs.LG

Vivek Hruday Kavuri , Gargi Shroff , Rahul Mishra This is my paper · ORCID

classification cs.CLcs.AIcs.LG

keywords neuronattributionconceptvectormechanisticinterpretabilitylargelanguagemodelsgenderbiashatespeechablationclustering

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

NEAT sets out to show that the neurons responsible for an abstract concept, such as hate speech or a gender stereotype, can be located in a large language model by compressing that concept into a single dense vector and scoring each neuron's effect in a single forward pass. If the method works, it gives a cheap, reusable way to find and switch off concept-level behavior without running per-example gradient or causal tracing. The paper reports that deactivating 10–11 of the top-scored neurons all but eliminates the concept's influence, and that the neuron ranking transfers across sentences rather than overfitting to one example. It also reduces the attribution cost from O(n×m) to O(n) forward passes, where n is the number of neurons and m the number of examples.

What carries the argument

The target-effect score E_{l,j}: feed the concept vector c as the input embedding, zero out the j-th column of the projection matrix at layer l, and measure the absolute log-probability change over a fixed target-word set using only the modified layer output, the final layer norm, and the language-model head. This score ranks all neurons in one O(n) sweep, and the same readout is reused for male/female bias by subtracting effects on opposite-gender word sets.

What would settle it

Take NEAT's top-ranked hate-concept neurons and evaluate them by full-model forward passes: zero each neuron in the real 36-layer model on held-out hate sentences and check whether the true next-token probability falls to near zero as the layer-local method predicts. If the full-model drop is small, or if the same drop occurs for randomly chosen neurons, the layer-local ranking is not measuring concept representation.

Watch

Extended reading notes

Core claim

NEAT claims that a concept can be compressed into a single vector—the mean of the last-layer hidden states over representative examples—and that this vector, fed to the model as if it were a token, lets every neuron be scored by how much its removal changes the log-probability of concept-related target words. On a 36-layer, 774M-parameter GPT-2 model, deactivating the top 10 hate-concept neurons drops the average reciprocal rank of the true next token to 6.50e-5 and its average probability to 5.28e-6, while deactivating 11 male- or female-bias neurons drives that gender's resolution accuracy from 98.37% or 95.08% down to 5.00% or 2.90%, respectively. The paper argues this shows concept-speci

Load-bearing premise

The ranking assumes that reading logits from the layer being ablated, with a synthetic concept vector as input and without running the other 35 layers, faithfully measures how much each neuron contributes to the concept in real sentences.

Editorial extensions

If this is right

Neuron attribution for a concept becomes O(n) in forward passes instead of O(n×m), making concept-level search feasible on larger models and bigger concept sets.
The top-scored concept neurons generalize across test sentences, so one global ranking can be computed once and reused for interventions on that concept.
Removing 10–11 of the top neurons can reduce hate-speech next-token accuracy to near zero and collapse gender resolution, showing concept behavior is concentrated in a handful of parameters.
Clustering the coefficient scores also yields groups whose deactivation changes behavior, suggesting concept information is partially organized in parameter-space clusters.
The same pipeline brings IndiBias stereotype preference from 61.3%/38.7% stereotypical/anti-stereotypical to 50.6%/49.4% after deactivating one male and one female neuron.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the layer-local score faithfully tracks full-model behavior, the same O(n) search could be run for many concepts at once, revealing shared versus disjoint concept neurons; the paper does not test multi-concept localization.
The concept vector is a plain mean of hidden states, which can wash out direction-specific structure; a natural refinement is to use the difference between contrasting concept vectors (e.g., male minus female) to sharpen rankings.
Cluster-level ablation in the appendix degrades resolution accuracy broadly rather than selectively removing one gender, so a useful next test is whether editing cluster centroids can produce the same selective effects as individual neuron removal.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Hi,

You should know this paper is a mixed bag: the recipe is genuinely new, but the scoring approximation is unvalidated and the evaluation is partly circular. I would not trust the central claim that NEAT finds concept-specific neurons, but the idea is worth a look.

What's new: they define a concept vector as the mean of last-layer hidden states over concept examples, feed it as a token embedding, and score each FFN neuron by the change in log-prob of target words when that neuron is zeroed. That specific combination is not in the literature, and it is genuinely cheaper than per-example attribution: one forward pass to collect activations, then per-neuron logit evaluations. The clustering ablation is a reasonable extra. The empirical results are dramatic — deactivating 10 neurons tanks MRR to 1e-5 and dropping male accuracy to 5% — so the selected neurons are clearly influential.

The problems: first, Eqs. 11-14 read logits directly from layer l's output, skipping layers l+1..36. No comparison with full-model ablation is provided anywhere. That is a load-bearing approximation; without validation, the ranked list could be an artifact of the proxy. Second, the selection criterion (Eq. 17, 23-25) maximizes the same quantity the evaluation measures: absolute log-prob change on target words (hate) or asymmetric effects on male/female tokens (bias). The bias resolution metric uses masculine/feminine tokens that likely overlap the target sets, so the 'predictions' are partly circular. Third, the hate test uses only 10 sentences, no error bars, no released code or target word lists. The running header 'Trovato et al.' is a template artifact, which is sloppy.

On the plus side, the paper is honest in the appendix that the bias neurons are largely pronoun detectors, and the IndiBias evaluation is a nice addition. The complexity claim O(n*m) to O(n) is roughly right when n is total neurons and m is examples.

My bottom line: the method is new and the write-up is clear, but the central claim is not supported because the scoring proxy is unvalidated and the evaluation is circular. This deserves a serious referee — not a desk reject — because the flaws are fixable: add a full-model ablation comparison, use held-out target words, and report on more sentences. As it stands, I would not cite it as evidence for concept neurons, but I'd assign it to a reviewer.

Recommendation: send to peer review with the expectation of major revision.

Referee Report

4 major / 6 minor

Summary. The paper proposes NEAT, a concept-vector-based neuron attribution method for LLMs. A concept vector is formed by averaging last-layer hidden states across concept examples; this vector is then fed as an input embedding, and each FFN neuron is scored by the absolute log-probability change on concept-related target tokens when that neuron is deactivated in layer l, with logits read directly from the ablated layer's output (Eqs. 11–14). The top-M neurons are taken as 'concept neurons.' The method is evaluated by deactivating these neurons in the full model on hate speech (10 test sentences) and gender bias (Bias in Bios and IndiBias), reporting large performance drops and near-balanced stereotype preferences. The central claim is that a small set of neurons selectively encodes a concept and that their removal yields concept-specific behavioral changes.

Significance. If the method were validated, efficient concept-level neuron attribution would be a useful contribution to mechanistic interpretability, and the reduction from O(n×m) to O(n) forward passes is an appealing practical feature. The application to bias and hate-speech intervention is also timely. The paper provides a straightforward comparison against several attribution baselines. However, the current evidence does not establish the validity of the scoring mechanism or the concept-specificity of the selected neurons. The scoring method rests on two unvalidated assumptions—using an averaged last-layer hidden state as an input embedding, and computing target effects from the ablated layer alone—and the bias evaluation is substantially circular. These issues undermine the central claims, and the paper therefore does not yet meet the standard for a serious journal publication.

major comments (4)

[§4.2, Eqs. (11)–(14)] The target-effect score E_{l,j} is computed by decoding directly from the ablated layer l's output (after LayerNorm and the LM head), ignoring all layers l+1..L. In GPT-2-large (36 layers), later layers can amplify, cancel, or redirect local changes, so this score is not a valid estimate of the neuron's causal effect on the final distribution. The paper provides no comparison between Eq. (17) and a full-model ablation (e.g., activation patching) for the same neurons. The full-model deactivation results in Tables 1, 4, and 5 show only that the selected neurons are impactful when ablated in the full model, not that the approximate ranking is correct or concept-specific.
[§4.1–4.2, Eqs. (8)–(9)] The concept vector c is an average of last-layer hidden states across tokens and examples, then fed to the model as an input embedding. This vector is out-of-distribution for the token-embedding space and for early layers; no justification is given that running layers 0..l-1 on c produces meaningful intermediate activations. Without this justification or a sanity check (e.g., comparing the resulting neuron rankings with those from real concept examples), the scores and the top-M list may be artifacts of feeding an atypical input.
[§4.3 and §6.1.2] The bias evaluation is circular. The selection criterion for male-biased neurons (Eq. 23) directly maximizes the asymmetric absolute log-prob effect on the masculine token set T_M versus the feminine token set T_F, and the evaluation in Section 6.1.2 measures resolution accuracy using the same masculine/feminine token sets. Thus the observed drop in male accuracy after deactivating male-biased neurons is, by construction, not an independent validation of concept-specificity. A held-out bias metric that does not rely on the token sets used in selection is needed to support the claim that these are 'male-biased' neurons rather than neurons that happen to affect the selected tokens.
[§6.1.1, §6.3.1, Tables 1–3] The hate speech evaluation is based on only 10 test sentences. The per-sentence results in Table 3 show stable MRR values, but with N=10 no confidence intervals or significance tests are reported, and the claim of generalization across examples is not statistically supported. Furthermore, the selection target set T consists of hate-related words, and the evaluation sentences are hateful; the overlap between the scoring objective and the evaluation metric creates a selection bias that the baselines do not share, making the comparison in Table 1 less informative. A larger, independently annotated test set and a random-neuron control would be required.

minor comments (6)

[§4.2, Eq. (7)] The notation in Eq. (7) uses fc1_l^k · (h + A) with a dot product, but the earlier definition of subkeys is not fully precise; the dimensions and position indices should be clarified.
[§5.1, §6.1.2] The phrase 'identified using absolute values of neuron effect scores, hereafter denoted as “with abs”' is never explained in the equations; the bias scores in Eqs. (23)–(25) do not use a simple absolute value. Please define 'with abs' or remove it.
[§1 and §6.3.1] The complexity claim 'from O(n×m) to O(n)' is not fully accurate: computing the concept vector c requires m forward passes over the concept examples, so the total is O(m + n). The paper should state this explicitly, or clarify how m is absorbed.
[§6.1.1] The evaluation says 'tested on the top 10 sentences from the sampled 2000 sentences,' but the criterion for 'top' is not specified. This should be described to ensure reproducibility.
[References [31] and [32]] References [31] and [32] appear to be the same work (the arXiv and EMNLP versions of 'Neuron-Level Knowledge Attribution in Large Language Models'). Please merge or distinguish them appropriately.
[General] The paper does not mention code or data release. Given the small evaluation sizes and the manual labeling of professions, a reproducibility statement would be valuable.

Circularity Check

2 steps flagged · score 7.0 of 10

Bias selection score (Eqs. 23-25) is the same asymmetric gendered-token log-prob change the evaluation measures; hate-speech top-M is selected on the same absolute log-prob-change objective, partly forcing the reported deactivation drop.

self definitional [Section 4.3 (Eqs. 23-26) and Section 6.1.2 (Bias evaluation), Tables 4-5]
"E^{M-bias}_{l,j} = sum_{t in T_M} |Delta log p^{(l,j)}(t)| - sum_{t in T_F} |Delta log p^{(l,j)}(t)| ... The neurons with the largest positive values of E^{M-bias}_{l,j} are identified as male-stereotypical neurons: N^{M-bias} = arg topM_{l,j} E^{M-bias}_{l,j}. ... Given a bio, the model is prompted with the text, and we inspect the probability distribution over the vocabulary for the next token. Let M and F denote the sets of masculine and feminine tokens ... P_masc = sum_{t in M} p(t) ... The predicted gender g-hat is determined as: g-hat = (0 if P_masc > P_fem else 1)."

A male-stereotypical neuron is defined, by Eq. 23, as one whose deactivation maximizes the positive difference between absolute log-prob changes on male-gendered tokens T_M and female-gendered tokens T_F. The evaluation then reports that deactivating these neurons collapses male resolution accuracy, where resolution accuracy is computed by comparing the summed probabilities over masculine versus feminine token sets. This is the same asymmetric gendered-token effect used to rank the neurons. The observed 'validation' is therefore not an independent behavioral prediction but a restatement of the selection rule: the neurons are chosen precisely because their removal distorts male/female token probabilities in the direction the evaluation measures. The female-neuron case (Eq. 25) is symmetric
fitted input called prediction [Section 4.2 (Eqs. 17-18), Section 6.1.1, Section 6.3.1]
"Finally, the target effect score for neuron (l, j) is computed as the total absolute log-probability change over the target token set T: E_{l,j} = sum_{t in T} |log p_mod(t) - log p_orig(t)|. We rank all neurons by E_{l,j} in descending order and select the top M neurons ... N_concept = arg topM_{l,j} E_{l,j}. ... deactivating the top 10 neurons identified by our method causes a dramatic performance degradation, with MRR_avg = 6.50e-05, Prob_avg = 5.28e-06."

The top-M concept-neuron set is selected to maximize the summed absolute log-probability change over concept-representative words T. The headline claim is that deactivating these same neurons causes a dramatic drop in probability/MRR of the true last token in hateful sentences. Because the test sentences are drawn from the same 2000-example pool used to compute the concept vector c (Sec. 5.1, 6.1.1), and T consists of hate-related terms, the evaluation measures essentially the same quantity that was optimized during selection: how much the deactivated model's log-probabilities move on concept-related tokens. The drop is thus heavily forced by the selection objective rather than being an independent confirmation that the neurons encode the concept. The paper's own framing ('demonstrates the

full rationale

The paper does not rely on a self-citation chain to justify its central method; the cited prior work (e.g., [31]/[32]) serves as baselines and external context, not as load-bearing evidence. However, the main empirical validation is substantially circular by construction. For the bias setting, the selection score E^{M-bias} (Eq. 23) and E^{F-bias} (Eq. 25) are literally differences of absolute log-prob changes over male and female gendered token sets, while the evaluation's gender-resolution accuracy is computed from the summed probabilities over masculine and feminine token sets. A neuron selected by those equations is, by definition, one whose deactivation shifts the male/female token-probability balance; the reported accuracy collapse is the same effect, not a prediction. For the hate-speech setting, the ranking score (Eq. 17) is the absolute log-prob change over concept words, and the evaluation measures probability/MRR degradation on concept-related sentences from the same data used to build the concept vector. That makes the result partially selection-bound. The IndiBias evaluation and the clustering study provide some independent or side evidence, and the paper does contain genuine methodological content (concept-vector computation, O(n) forward-pass reduction), so the work is not entirely reducible to its inputs. Still, the central 'concept neurons cause large concept-specific shifts' claim is largely a restatement of the optimization objective in both main evaluations, warranting a high circularity score rather than a clean bill of health.

Assumptions & free parameters 4 free parameters · 4 assumptions · 2 invented entities

The method rests on several hand-chosen parameters (target words, top-M, number of examples) and two unvalidated modeling assumptions: the synthetic concept-vector input and the layer-skipping logits approximation. These are not derived from the model architecture or theory.

free parameters (4)

Number of top neurons M = 10 for hate, 1-11 for bias
Chosen as fixed small numbers; no sensitivity analysis or criterion for selecting M.
Concept vector c = Mean of hidden states from 2000 examples
Data-derived vector; the averaging over tokens and examples (Eqs. 8-9) is a hand-designed representational choice.
Target word sets (T for hate, male/female token sets) = Not specified in paper
Hand-picked by authors; without these, the scoring function is undefined.
Number of concept examples N = 2000 per concept
Fixed subset; no analysis of how N affects results.

assumptions (4)

ad hoc to paper Feeding the concept vector c as an input embedding yields intermediate representations suitable for neuron scoring.
Sec 4.2: 'we feed it to the model as an input embedding to obtain intermediate representations at every transformer layer.' No validation against real-text activations.
ad hoc to paper The target-effect score can be computed from the ablated layer's output alone, ignoring downstream layers.
Eqs. 11-14 pass o_orig and o_mod through LN_f and W_lm directly, skipping layers l+1...L. This is a major approximation with no empirical check.
domain assumption Averaging final-layer hidden states over tokens and examples produces a vector that linearly represents the concept.
Eqs. 8-9. No evidence that averaging preserves concept direction; the model's representation space may not support this.
domain assumption The hand-picked target words are representative of the concept.
Sec 4.2, the set W is 'representative of the concept of interest'. The paper does not list the words or analyze their coverage.

invented entities (2)

Concept neuron
purpose: A neuron whose deactivation most changes the probability of concept-representative words.
The category is defined entirely by the paper's own scoring function; the experimental evidence for its existence is the same function, so there is no independent handle.
Concept vector
purpose: A single dense vector representing a concept, used as a synthetic input to the model for scoring neurons.
A mean of activations treated as an input embedding; its validity is assumed, not demonstrated with independent evidence.

how reviews work

0 comments

Cite this review

Pith. "Pith review of NEAT: Concept driven Neuron Attribution in LLMs." pith.science (2026). https://pith.science/paper/Z453SW4V

@misc{pith2026250815875,
  author       = {Pith},
  title        = {Pith review of: NEAT: Concept driven Neuron Attribution in LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z453SW4V}},
  note         = {Machine review of arXiv:2508.15875}
}

read the original abstract

Locating neurons that are responsible for final predictions is important for opening the black-box large language models and understanding the inside mechanisms. Previous studies have tried to find mechanisms that operate at the neuron level but these methods fail to represent a concept and there is also scope for further optimization of compute required. In this paper, with the help of concept vectors, we propose a method for locating significant neurons that are responsible for representing certain concepts and term those neurons as concept neurons. If the number of neurons is n and the number of examples is m, we reduce the number of forward passes required from O(n*m) to just O(n) compared to the previous works and hence optimizing the time and computation required over previous works. We also compare our method with several baselines and previous methods and our results demonstrate better performance than most of the methods and are more optimal when compared to the state-of-the-art method. We, as part of our ablation studies, also try to optimize the search for the concept neurons by involving clustering methods. Finally, we apply our methods to find, turn off the neurons that we find, and analyze its implications in parts of hate speech and bias in LLMs, and we also evaluate our bias part in terms of Indian context. Our methodology, analysis and explanations facilitate understating of neuron-level responsibility for more broader and human-like concepts and also lay a path for future research in this direction of finding concept neurons and intervening them.

Figures

Figures reproduced from arXiv: 2508.15875 by the authors.

**Figure 2.** This image describes our complete overall architecture, from obtaining concept vector to turning a neuron off and [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Beyond Transcription: Mechanistic Interpretability in ASR
cs.SD 2025-08 reject novelty 5.0 of 10

The abstract promises new findings on how ASR models encode acoustic and semantic information, but the body is an unrelated, truncated paper about neuron attribution in LLMs, leaving the claims unsupported.

Reference graph

Works this paper leans on

34 extracted references · 7 canonical work pages · cited by 1 Pith paper

[1]

Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, et al
[2]

Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. 2023. Analyzing Trans- formers in Embedding Space. arXiv:2209.02535 [cs.CL] https://arxiv.org/abs/ 2209.02535

arXiv 2023
[3]

Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Chris- tian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. 2019. Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting. InProceedings of the Conference on Fairness, Account- ability, and Transparency (Atlanta, GA, USA)...

arXiv 2019
[4]

Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Mar- tin Wattenberg, and Christopher Olah. 2022. Toy Models of Superposition. arXiv:2209.10652 [cs.LG] https://arxiv.org/abs/2209.10652

arXiv 2022
[5]

Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. 2023. Dis- secting Recall of Factual Associations in Auto-Regressive Language Models. arXiv:2304.14767 [cs.CL] https://arxiv.org/abs/2304.14767

arXiv 2023
[6]

Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. 2022. Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space. arXiv:2203.14680 [cs.CL] https://arxiv.org/abs/2203.14680

arXiv 2022
[7]

Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer Feed-Forward Layers Are Key-Value Memories. arXiv:2012.14913 [cs.CL] https: //arxiv.org/abs/2012.14913 NEAT: Concept driven Neuron Attribution in LLMs

arXiv 2021
[8]

Nicholas Goldowsky-Dill, Chris MacLeod, Lucas Sato, and Aryaman Arora. 2023. Localizing Model Behavior with Path Patching. arXiv:2304.05969 [cs.LG] https: //arxiv.org/abs/2304.05969

arXiv 2023

Show all 34 references

[9]

Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. 2023. Finding Neurons in a Haystack: Case Studies with Sparse Probing. arXiv:2305.01610 [cs.LG] https://arxiv.org/abs/2305.01610

2023 arXiv
[10]

Michael Hanna, Ollie Liu, and Alexandre Variengien. 2023. How does GPT- 2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model. arXiv:2305.00586 [cs.CL] https://arxiv.org/abs/2305.00586

2023 arXiv
[11]

Zhengfu He, Xuyang Ge, Qiong Tang, Tianxiang Sun, Qinyuan Cheng, and Xipeng Qiu. 2024. Dictionary Learning Improves Patch-Free Circuit Discovery in Mechanistic Interpretability: A Case Study on Othello-GPT. arXiv:2402.12201 [cs.LG] https://arxiv.org/abs/2402.12201

2024 arXiv
[12]

Théo Jaunet, Corentin Kervadec, Romain Vuillemot, Grigory Antipov, Moez Baccouche, and Christian Wolf. 2022. VisQA: X-raying Vision and Language Reasoning in Transformers. IEEE Transactions on Visualization and Computer Graphics 28, 1 (Jan. 2022), 976–986. doi:10.1109/TVCG.202...

2022
[13]

Schütt, Sven Dähne, Dumitru Erhan, and Been Kim

Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T. Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. 2022. The (Un)reliability of Saliency Methods. Springer-Verlag, Berlin, Heidelberg, 267–280. https://doi.org/ 10.1007/978-3-030-28954-6_14

2022 doi
[14]

Kum- merfeld, and Rada Mihalcea

Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K. Kum- merfeld, and Rada Mihalcea. 2024. A Mechanistic Understanding of Align- ment Algorithms: A Case Study on DPO and Toxicity. arXiv:2401.01967 [cs.CL] https://arxiv.org/abs/2401.01967

2024 arXiv
[15]

Fuxiao Liu, Paiheng Xu, Zongxia Li, Yue Feng, and Hyemi Song. 2024. To- wards Understanding In-Context Learning with Contrastive Demonstrations and Saliency Maps. arXiv:2307.05052 [cs.CL] https://arxiv.org/abs/2307.05052

2024 arXiv
[16]

Daniel D Lundstrom, Tianjian Huang, and Meisam Razaviyayn. 2022. A Rigorous Study of Integrated Gradients Method and Extensions to Internal Neuron Attri- butions. In Proceedings of the 39th International Conference on Machine Learning (Proceedings of Machine Learning Research,...

2022
[17]

Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in GPT. In Proceedings of the 36th International Conference on Neural Information Processing Systems (New Orleans, LA, USA) (NIPS ’22). Curran Associates Inc., Red Hook,...

2022
[18]

Vivek Miglani, Narine Kokhlikyan, Bilal Alsallakh, Miguel Martin, and Orion Reblitz-Richardson. 2020. Investigating Saturation Effects in Integrated Gradients. doi:10.48550/arXiv.2010.12697

2020 doi arXiv
[19]

Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Con- erly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, K...

2022 arXiv
[20]

Koyena Pal, Jiuding Sun, Andrew Yuan, Byron Wallace, and David Bau. 2023. Future Lens: Anticipating Subsequent Tokens from a Single Hidden State. In Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL). Association for Computational Linguistics...

2023 doi
[21]

2022.Direct and Indirect Effects(1 ed.)

Judea Pearl. 2022.Direct and Indirect Effects(1 ed.). Association for Computing Ma- chinery, New York, NY, USA, 373–392. https://doi.org/10.1145/3501714.3501736

2022
[22]

Nihar Sahoo, Pranamya Kulkarni, Arif Ahmad, Tanu Goyal, Narjis Asad, Aparna Garimella, and Pushpak Bhattacharyya. 2024. IndiBias: A Benchmark Dataset to Measure Social Biases in Language Models for Indian Context. In Proceedings of the 2024 Conference of the North American Cha...

2024
[23]

Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan. 2023. A Mecha- nistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysis. arXiv:2305.15054 [cs.CL] https://arxiv.org/abs/2305.15054

2023 arXiv
[24]

Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic Attribution for Deep Networks. InProceedings of the 34th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 70) , Doina Precup and Yee Whye Teh (Eds.). PMLR, 3319–3328. htt...

2017
[25]

Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020. Investigating gender bias in language models using causal mediation analysis. In Proceedings of the 34th International Conference on Neural Information Processing...

2020
[26]

Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2022. Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small. arXiv:2211.00593 [cs.LG] https://arxiv.org/abs/ 2211.00593

2022 arXiv
[27]

Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. 2023. Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning. arXiv:2305.14160 [cs.CL] https://arxiv. org/abs/2305.14160

2023 arXiv
[28]

Zhengxuan Wu, Atticus Geiger, Thomas Icard, Christopher Potts, and Noah D. Goodman. 2023. Interpretability at scale: identifying causal mechanisms in alpaca. In Proceedings of the 37th International Conference on Neural Information Processing Systems (New Orleans, LA, USA) (NI...

2023
[29]

Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017. Ex machina: Personal attacks seen at scale. In Proceedings of the 26th international conference on world wide web. 1391–1399

2017
[30]

Catherine Yeh, Yida Chen, Aoyu Wu, Cynthia Chen, Fernanda Viégas, and Martin Wattenberg. 2024. AttentionViz: A Global View of Transformer Attention. IEEE Transactions on Visualization and Computer Graphics 30, 1 (Jan. 2024), 262–272. doi:10.1109/TVCG.2023.3327163

2024
[31]

Zeping Yu and Sophia Ananiadou. 2024. Neuron-Level Knowledge Attribution in Large Language Models. arXiv:2312.12141 [cs.CL] https://arxiv.org/abs/2312. 12141

2024 arXiv
[32]

Zeping Yu and Sophia Ananiadou. 2024. Neuron-Level Knowledge Attribution in Large Language Models. In Proceedings of the 2024 Conference on Empiri- cal Methods in Natural Language Processing , Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Association for Computatio...

2024 doi
[33]

Fred Zhang and Neel Nanda. 2024. Towards Best Practices of Activation Patching in Language Models: Metrics and Methods. arXiv:2309.16042 [cs.LG] https: //arxiv.org/abs/2309.16042 A Appendix Table 7: Resolution Accuracy and Gender Gap for Different Clustering Configurations (KM...

2024 arXiv
[2023]

Transformer Circuits Thread 2 (2023)

Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread 2 (2023)

2023

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, et al

[2] [2]

Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. 2023. Analyzing Trans- formers in Embedding Space. arXiv:2209.02535 [cs.CL] https://arxiv.org/abs/ 2209.02535

arXiv 2023

[3] [3]

Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Chris- tian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. 2019. Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting. InProceedings of the Conference on Fairness, Account- ability, and Transparency (Atlanta, GA, USA)...

arXiv 2019

[4] [4]

Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Mar- tin Wattenberg, and Christopher Olah. 2022. Toy Models of Superposition. arXiv:2209.10652 [cs.LG] https://arxiv.org/abs/2209.10652

arXiv 2022

[5] [5]

Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. 2023. Dis- secting Recall of Factual Associations in Auto-Regressive Language Models. arXiv:2304.14767 [cs.CL] https://arxiv.org/abs/2304.14767

arXiv 2023

[6] [6]

Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. 2022. Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space. arXiv:2203.14680 [cs.CL] https://arxiv.org/abs/2203.14680

arXiv 2022

[7] [7]

Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer Feed-Forward Layers Are Key-Value Memories. arXiv:2012.14913 [cs.CL] https: //arxiv.org/abs/2012.14913 NEAT: Concept driven Neuron Attribution in LLMs

arXiv 2021

[8] [8]

Nicholas Goldowsky-Dill, Chris MacLeod, Lucas Sato, and Aryaman Arora. 2023. Localizing Model Behavior with Path Patching. arXiv:2304.05969 [cs.LG] https: //arxiv.org/abs/2304.05969

arXiv 2023