Pith. sign in

REVIEW 4 major objections 6 minor 66 references

Discovering Chunks in Neural Embeddings for Interpretability

T0 review · 4 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Recurring chunks in neural activity mirror the training data

desk verdict Honest cognitive framing with a solid RNN proof-of-concept, but the LLM chunk evidence needs null controls before the central claim can land. read the letter →

arxiv 2502.01803 v1 pith:WG5VE2QD submitted 2025-02-03 cs.LG cs.AI

Shuchen Wu , Stephan Alaniz , Eric Schulz , Zeynep Akata This is my paper · ORCID
classification cs.LGcs.AI
keywords interpretabilitychunkingneuralpopulationactivityreflectionhypothesislargelanguagemodelsrecurrentnetworkscausalinterventionembeddingdictionary
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes the Reflection Hypothesis: a well-trained neural network's internal activity, tracked over time, mirrors the regularities in the data it was trained on. If this holds, the high-dimensional mess of hidden states can be segmented the way perception segments sensory streams, into recurring chunks—small subpopulations of neurons whose activity is stable whenever a concept appears. The paper tests this across scales, from a 12-unit RNN trained on artificial ABCD sequences to LLaMA-3 processing natural language, and shows that extracted chunks identify inputs, can be grafted or frozen to change what the model predicts, and can be learned without supervision. A sympathetic reader would care because it offers a data-structure-first answer to interpretability: instead of mapping neurons to words, map trajectories to chunks.

What carries the argument

The central objects are recurring chunks of neural population activity, and the paper contributes three extractors for them. Discrete sequence chunking converts each neuron's activity into a cluster index, concatenates the indices into strings, and iteratively merges frequent adjacent pairs into a chunk vocabulary—this works when the embedding dimension is small. Neural population averaging assumes each recurring signal $s$ has a neuron subset $C(s) = \{i : |h_{i,j} - \bar h_i| \le \text{tol} \ \forall j \in V(s)\}$, estimates the mean chunk $\bar h_{C(s)}$ and allowed deviation $\Delta = \max_j \|h_{C(s),j} - \bar h_{C(s)}\|_2^2 / d$, and classifies a new state as the chunk if it falls in the ball $B(\bar h_{C(s)}, \Delta)$. Unsupervised chunk discovery learns a dictionary $D \in \mathbb{R}^{K \times d}$ by the loss $L = -\frac{1}{M}\sum_m \max_k \mathrm{SIM}(D_k, X_m)$ with normalized cosine similarity, so each embedding is assigned to its most similar chunk. The first method proves the concept in small networks; the second provides causal, signal-tagged chunks in LLMs; the third shows chunks can be found without labeled patterns.

What would settle it

Take a word with strongly context-dependent behavior, such as "in" or "bank", collect hundreds of occurrences in varied sentences, apply the paper's tolerance-optimized population averaging, and measure true-positive and false-positive rates per context cluster. If occurrences from different contexts do not share a stable subpopulation—so that detection at a fixed false-positive budget approaches chance once the embedding layer is excluded—the Reflection Hypothesis as operationalized here would be refuted for those signals. A complementary causal check: graft the extracted chunk for a concept into a prompt that has nothing to do with that concept; if topic-biased generation only appears when the prompt is semantically close, the chunk is a correlate rather than a cause.

Watch

Extended reading notes

Core claim

The central claim is that neural population activity is not arbitrary: trajectories of hidden states recapitulate the statistical structure of the training sequence. The paper demonstrates this by injecting known regularities (the pattern ABCD) into RNN training data and recovering those regularities from the hidden states: the extracted chunk dictionary decodes the input with perfect accuracy, grafting the population state onto the chunk for A or C changes the network's next prediction from B to C or D to E, and grafting can force the network to compose previously learned words into a new word. It then extends the same principle to LLaMA-3, where averaging hidden states over occurrences of a word identifies a subpopulation $C(s)$ and a radius $\Delta$; the resulting chunk detectors find "cheese", "cake", and "cheesecake" with high true-positive and low false-positive rates in the layers that process them, and grafting or zeroing these subpopulations biases or suppresses topic-related generation. An unsupervised dictionary of $K=2000$ chunks, trained by maximizing normalized cosine similarity to hidden states, activates in layer-wise patterns that correlate with part-of-speech tags, suggesting chunks are the atoms of computation rather than a visualization aid.

Load-bearing premise

The load-bearing premise is that each recurring signal is encoded by the same small set of neurons whose activity stays close to its average on every occurrence; if context recruits different neurons each time, averaging across occurrences produces a mixed template that is not a real chunk.

Editorial extensions

If this is right

  • If the Reflection Hypothesis is right, interpretability reduces to finding the right level of description: a model's computation is a sequence of chunks that can be read as a dictionary, rather than a soup of polysemantic neurons.
  • Causal control follows directly: because grafting a chunk state into an RNN or LLaMA-3 biases the next prediction, extracted chunks are not merely correlational labels but usable levers for steering or suppressing concepts.
  • Training-induced representational structure becomes measurable: trained networks acquire more and larger chunk vocabularies than untrained ones, and hierarchical input produces hierarchical chunk structure, giving a principled way to compare models.
  • Unsupervised chunk dictionaries align with linguistic structure such as part-of-speech tags, so the method scales to settings where no labeled concept list exists.
  • The approach transfers across architectures and scales, from 12-unit RNNs to 8-billion-parameter LLMs, suggesting the underlying regularity is in the data rather than in a particular model family.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the assumption of stable subpopulations fails for context-dependent tokens, the averaging method would silently replace a genuine recurring chunk with a mixed template; a natural extension is to learn multiple chunks per signal, one per context, and test whether context-clustered chunks detect better than the global average.
  • The chunk dictionary is conceptually adjacent to sparse autoencoders but assigns whole embedding states to dictionary entries rather than decomposing them into sparse features; a direct comparison on the same LLM layers would clarify whether chunks are complementary to or redundant with sparse features.
  • If the Reflection Hypothesis holds across modalities and models, chunk dictionaries should transfer between models trained on similar data—an untested prediction that would connect the paper's framework to representation-convergence results.
  • A practical extension would use frozen chunks as a safety or content-control mechanism, but that requires measuring side effects on unrelated generation; the paper does not report such measurements.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes the "Reflection Hypothesis": that a well-trained neural network's population activity contains recurring subpopulation states, called chunks, that mirror regularities in its training data. The authors develop three extraction methods: discrete sequence chunking for low-dimensional RNN hidden states, neural population averaging for large-scale LLM hidden states, and unsupervised dictionary learning for LLM embeddings. In RNNs trained on artificial sequences with injected regularities, the discrete chunking method recovers a symbolic vocabulary of population trajectories, and grafting population states changes RNN predictions and speeds transfer learning on a composite word. In LLaMA-3-8B, the population averaging method is used to identify neurons whose activity stays near a word-specific mean, yielding high true positive rates and low false positive rates for detecting words such as "cheese", "cake", and "cheesecake" in held-out prompts; grafting these subpopulations biases generation toward the corresponding topic, and freezing them suppresses the topic. An unsupervised dictionary of K=2000 chunks is also learned and analyzed for correlations with part-of-speech tags across layers. The central claim is that neural population activity can be segmented into interpretable, causally relevant chunks that reflect data structure.

Significance. If established, the chunk-based interpretability framework would offer a new unit of analysis distinct from individual neurons, SAE features, and low-dimensional probes, and it would connect cognitive chunking to mechanistic interpretability. The RNN proof of concept is internally coherent and goes beyond mere correlation: the population grafting experiments show a causal influence on predictions, and the trained-versus-untrained comparisons in Appendix E include ten independent runs with error bars. The LLaMA results, however, are more suggestive than conclusive: the central evidence for LLM chunk identifiability rests on a detector whose thresholds are fit on training data and whose reported success is not benchmarked against null controls, and the causal grafting/freezing demonstrations are qualitative. The unsupervised dictionary method is a reasonable formulation but needs a statistical baseline to support its interpretability claims. The paper's strengths are its clearly stated hypothesis, the range of models considered, and the transparent admission in Section 3.1 and Appendix H that the averaging method may fail when context-dependent multiple chunks exist.

major comments (4)
  1. [§3.1, Eqs. (2)–(4), and Figure 6] The population averaging detector is, by construction, optimized on the training prompt: the tolerance toli is tuned on training data to maximize TPR/minimize FPR, and the radius Delta(s) is the maximum training deviation. Consequently, the training-side performance is expected to be near-perfect, and the held-out evaluation in Figure 6 is not compared against any null model. This leaves open two degenerate alternatives: the selected subpopulation C(s) may consist mostly of globally low-variance neurons that happen to lie near the word mean in the test prompt, and the high TPR may be inflated because the train and test prompts are topically redundant (the "cheese" prompt discusses cheese in nearly every sentence). The authors should add null baselines, for example random subpopulations of the same size, shuffled occurrence labels, or a signal-independent word matched by frequency, and report held-out TPR/FPR for those baselines alongside the reported values.
  2. [Figure 6 and Appendix G] The central quantitative claim for LLaMA chunk identifiability, namely that population averaging produces high TPR and low FPR across layers, is presented without error bars or multiple train/test prompt splits. The paper's own Appendix G shows layer-wise statistics for individual words but no variability measures. Since the method has several free choices (the tolerance schedule, the frequency threshold, the number of occurrences used to estimate C(s), and the single train/test prompt pair), the reader cannot assess whether the reported TPR/FPR curves are robust. I ask for error bars over at least several independent train/test prompt pairs and, ideally, over different random subpopulations, and for a statement of how sensitive the layer-wise trends are to the tolerance schedule.
  3. [Figure 7 and §4.2, Grafting and Freezing Neural Populations] The causal claim that "perturbing the hidden units to embeddings corresponding to words such as 'cheese' biases the network towards generating sequences related to the grafted topic" is supported only by qualitative example generations. There is no control perturbation of equal magnitude applied to randomly selected neurons, no quantification of the topic shift (e.g., n-gram overlap, keyword frequency, or human rating), and no evaluation of whether freezing a random subpopulation of the same size has any effect. Without these controls, the observed generation changes could arise from any large-norm intervention on the residual stream rather than from the specific semantic content of the discovered chunk. Please provide quantitative, controlled steering results, including negative controls and multiple prompts per concept.
  4. [§4.2, Unsupervised Chunk Discovery, Figure 8c and Appendix K] The claim that unsupervised chunks correlate with part-of-speech tags is not supported by any null model. The paper reports the maximum correlation between each POS tag and its most correlated chunk among K=2000 chunks; such a maximum is inflated by selection over many random candidates. A proper baseline would compare these correlations to those obtained from a randomly initialized dictionary, from shuffled chunk labels, or from a dictionary trained on randomly permuted embeddings. Additionally, no confidence intervals are reported for the correlation curves. The visual claim in Figure 8 that chunks are "purely responsible" for certain POS tags requires this baseline to be interpretable.
minor comments (6)
  1. [Section 3.1, Eq. (1)] The notation in Eq. (1) is confusing: the mean vector h_C(s) is written identically on the left-hand side and inside the sum on the right-hand side, and the limit statement refers to a different quantity mu. Please use distinct symbols, e.g., \hat{h}_{C(s)} for the empirical mean and \mu for the true mean.
  2. [Section 3.1, Eq. (4)] The chunk identification function f_chunk takes h_{C(s)} as an explicit argument but the condition on the right-hand side only uses the norm of h_{C(s)} -- the formula should specify whether the numerator is \|h_{C(s)} - \hat{h}_{C(s)}\|_2^2, as the text suggests. As written, Eq. (4) is dimensionally inconsistent with Eq. (3).
  3. [Section 4.1, Figure 3] Figure 3 caption says 'Left: Hidden states can be grafted...' and 'Right: Embedding grafting...', but the figure itself is not reproduced in the submitted text; the reader cannot verify the learning-curve claims from the figure alone. Please ensure the figure is part of the submission and that the axes and error bars (if any) are described.
  4. [Appendix F] The training and test prompts are extremely topically redundant: the 'cheesecake' training prompt mentions cheese, cake, and cheesecake in almost every sentence. This is a confound for the word-detection evaluation and should be acknowledged explicitly in the main text, not only in the appendix. Please also report the number of occurrences of each target word in the train and test prompts.
  5. [Appendix D] Algorithm 1 (LearnChunks) leaves several choices unspecified, including the number of iterations n_iter, the pair count n, and the rule for 'stateparse[:−1], stateparse[1:]'. These choices are free parameters of the method, and their effect on the RNN results is not analyzed. A sensitivity analysis or at least a statement of the chosen values would improve reproducibility.
  6. [References] The reference to Wu, Thalmann, and Schulz (2025) contains a placeholder URL ('some-url-placeholder'), and the reference to Miller and Neo contains a typo ('We found an neuron'). Please fix these before publication.

Circularity Check

1 steps flagged · score 4.0 of 10

LLM chunk identifiability is partially definitional: population averaging defines chunks as means over labeled occurrences, although held-out and unsupervised POS evaluations provide independent content.

  1. self definitional [Section 3.1 (Eqs. 1-4) and Section 4.2 (Figure 6)]
    "we first compute the mean population response of signal s by averaging over the token-specific hidden state representations ... A neuron i is hypothesized to be inside C(s) if its activity at the time of the pattern-specific index fluctuates within a pre-set tolerance level around the mean signal-relevant activity C(s) = {i ∈ W : |h_{i,j} − h_i| ≤ tol ∀j ∈ V(s)}. ... ∆ = max_{j∈V(s)} ||hC(s),j − hC(s)||^2_2 / d ... We assessed the extracted chunks and thresholds by measuring how well chunk detection predicted the occurrence of s."

    The chunk for a word s is constructed as the mean of the labeled training occurrences of s, with support C(s) defined as neurons staying within tol of that mean and radius ∆ defined as the maximum training deviation. Consequently every training occurrence of s lies inside the ball B(hC(s), ∆) by construction, so the existence of a cheese/cake/cheesecake template is guaranteed by the averaging operation rather than discovered. The reported held-out TPR/FPR is not logically forced, but because the tolerance is optimized on the training prompt and the test prompt (Appendix F) is topically redundant, the evaluation does not separate a genuine recurring semantic subpopulation from a mean template that classifies well on overlapping text.

full rationale

The RNN proof-of-concept is self-contained: hidden states of networks trained on engineered sequences are parsed with a standard chunking algorithm, and the causal grafting and trained-versus-untrained comparisons provide falsifiable content beyond the engineered regularities. The unsupervised dictionary learning is validated externally against Penn Treebank POS tags, so it is not circular. The main circularity concern is confined to the Neural Population Averaging method for LLMs: the chunk template, its neuron support, and its radius are all defined from the labeled occurrences of the target word, making training-set identifiability definitional, and the held-out evaluation lacks null controls and uses topically redundant prompts. This is a partial, method-internal circularity rather than a self-citation chain; the paper's central reflection hypothesis still has independent empirical content. Score 4 reflects one partially definitional prediction with independent grounding elsewhere.

Assumptions & free parameters 4 free parameters · 5 assumptions · 1 invented entities

The central claim rests on the stated Reflection Hypothesis plus several representational assumptions: stable subpopulations, fixed dictionary size, discretization fidelity, and SLLN. The main fitted quantities are the tolerance threshold, dictionary size, and unspecified chunking hyperparameters. No new physical entities are posited; 'chunks' are descriptive patterns with causal and correlational evidence.

free parameters (4)
  • tolerance threshold toli = i = 0,...,39 in toli = 2*0.8^i, chosen by TPR/FPR optimization
    Section 3.1 says tolerance is optimized on training data to maximize TPR while minimizing FPR, so the reported chunk detector depends on a fitted threshold.
  • chunk dictionary size K = 2000
    Section 4.2 fixes K=2000 for LLaMA-3 hidden states without sensitivity analysis.
  • frequency threshold and pair count n in LearnChunks = not specified in the paper
    Algorithm 1 in Appendix D requires a frequency threshold and a number of most common pairs per iteration; values are not reported, so the discrete chunk vocabulary is under-specified.
  • number of clusters for per-neuron discretization = not specified
    Section 3.1 discretizes each neuron's activity into cluster indices; the cluster count influences which strings form chunks and is not reported.
assumptions (5)
  • domain assumption Reflection Hypothesis: well-trained networks mirror training-data regularities in their internal states
    Section 3 states the hypothesis and uses it as the foundation for all chunk extraction methods; it is the premise under test, not derived.
  • domain assumption A recurring pattern s induces a subpopulation C(s) whose activity stays near a mean over occurrences
    Section 3.1, Eqs. 2-3 define C(s) by a tolerance around the mean; if no such low-variance subpopulation exists, the extracted 'chunk' is an artifact of averaging.
  • standard math Strong Law of Large Numbers applies to occurrence sets
    Eq. 1 invokes SLLN for convergence of the average hidden state; sequential token states are correlated, so the i.i.d. idealization is an approximation.
  • domain assumption Recurring states are representable by a fixed dictionary with max cosine similarity
    Eq. 5 defines unsupervised chunk discovery as maximizing similarity to one of K dictionary entries; this assumes a finite discrete chunk vocabulary captures the structure.
  • domain assumption Per-neuron clustering preserves the relevant trajectory information
    Discrete sequence chunking converts continuous hidden states to strings of cluster indices in Section 3.1; the method assumes this mapping is lossless enough for chunk discovery.
invented entities (1)
  • neural trajectory chunk independent evidence
    purpose: Interpretable recurring unit of neural population activity that can be extracted, predicted, and manipulated
    The paper introduces chunks as a conceptual unit and provides falsifiable handles: grafting and freezing change model outputs, and discovered chunks correlate with POS tags. These are independent of the definitional averaging step.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Discovering Chunks in Neural Embeddings for Interpretability." pith.science (2026). https://pith.science/paper/WG5VE2QD

@misc{pith2026250201803,
  author       = {Pith},
  title        = {Pith review of: Discovering Chunks in Neural Embeddings for Interpretability},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WG5VE2QD}},
  note         = {Machine review of arXiv:2502.01803}
}
read the original abstract

Understanding neural networks is challenging due to their high-dimensional, interacting components. Inspired by human cognition, which processes complex sensory data by chunking it into recurring entities, we propose leveraging this principle to interpret artificial neural population activities. Biological and artificial intelligence share the challenge of learning from structured, naturalistic data, and we hypothesize that the cognitive mechanism of chunking can provide insights into artificial systems. We first demonstrate this concept in recurrent neural networks (RNNs) trained on artificial sequences with imposed regularities, observing that their hidden states reflect these patterns, which can be extracted as a dictionary of chunks that influence network responses. Extending this to large language models (LLMs) like LLaMA, we identify similar recurring embedding states corresponding to concepts in the input, with perturbations to these states activating or inhibiting the associated concepts. By exploring methods to extract dictionaries of identifiable chunks across neural embeddings of varying complexity, our findings introduce a new framework for interpreting neural networks, framing their population activity as structured reflections of the data they process.

Figures

Figures reproduced from arXiv: 2502.01803 by the authors.

Figure 1
Figure 1. (Top) Naturalistic data is highly redundant and compo￾sitional, e.g. in language sequences. Cognitive systems segment redundancies by chunking recurring patterns. The reflection hy￾pothesis posits that ANNs neural activities can be interpreted as chunks that reflect the structured regularities in reality. (Bottom) In simple networks that contain a small number of neurons, chunking methods can be used to learn a dict… view at source ↗
Figure 2
Figure 2. Testing the reflection hypothesis with simple RNNs and artificial sequences. a. RNN updates predictions and memory states based on inputs and previous hidden state. b. Neural popula￾tion activity (of the first 5 neurons) in response to repeating chunk (ABCD); c. Sparse occurrence of ABCD within a default sequence (E); d. ABCD persists as a cohesive chunk amid background noise (random E, F, G). To test this hypothesi… view at source ↗
Figure 3
Figure 3. Left: Hidden states can be grafted to causally change net￾work memory and prediction. Right: Embedding grafting enables faster transfer learning of a compositional vocabulary [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (18 more)
Figure 4
Figure 4. Figure 4: Left: Training creates extra chunks inside the embedding space. Right: The number of embedding states increases with the complexity of the input sequence. 1995) or tokenization. The idea is to cluster the neuron activations individually, use the cluster indices to repr…
Figure 5
Figure 5. Figure 5: a. Neural embedding activity of the first 50 neurons (unselected) across all layers (33) processing prompt up until the end of each highlighted word. b. Extracted neural activity chunks in response to word at different sequence positions. Grafting neural population act…
Figure 6
Figure 6. Figure 6: The identifiability of the presence of extracted chunks evaluated by signal detection measures. Hierarchically structured input sequence corresponds to hierarchically structured states in neural state space Another implication of the reflection hypothesis is that when …
Figure 7
Figure 7. Figure 7: Grafting and freezing word-related population chunk alters network’s sequence generation. LLaMA-3. To do this, we fed in a prompt and graft the neural subpopulation C(s) to hC (s) at a specific token po￾sition of the prompt. Shown in [PITH_FULL_IMAGE:figures/full_fig_…
Figure 8
Figure 8. Figure 8: c visualizes the maximum correlation between each POS tag and its most correlated discovered chunk across net￾work layers (excluding the embedding layer). Our findings align with prior research showing that certain POS tags are processed in the earlier layers of the ne…
Figure 9
Figure 9. Figure 9: Left: deviation of the hidden population activity from the mean population activity, when the input is ABCD. Middle, of the hidden population activity from the mean population activity, when the input is not ABCD. Right: count of word occurrences and the population tra…
Figure 10
Figure 10. Figure 10: Comparison between trained and untrained RNNs on sequences with overlapping words across 10 independent runs and randomly initialized RNNs. a. Sequence parse length. Dashed line is the ground truth sequence parsing length. b. Number of unique neural population states.…
Figure 11
Figure 11. Figure 11: Example of layer-wise statistics of subpopulation encoding the next step prediction of “cake” [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: Example of layer-wise statistics of subpopulation encoding the next step prediction of “in” [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]
Figure 13
Figure 13. Figure 13: Example of layer-wise statistics of subpopulation encoding the previous word being “of” [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14: Example of layer-wise statistics of subpopulation encoding the latest word being “people”. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_14.png]
Figure 15
Figure 15. Figure 15: Visualization of the extracted neural subpopulation for the words “cheese”, “cake”, and “cheesecake”, respectively, in both directions [PITH_FULL_IMAGE:figures/full_fig_p020_15.png]
Figure 16
Figure 16. Figure 16: 21 [PITH_FULL_IMAGE:figures/full_fig_p021_16.png]
Figure 17
Figure 17. Figure 17: 22 [PITH_FULL_IMAGE:figures/full_fig_p022_17.png]
Figure 18
Figure 18. Figure 18: 23 [PITH_FULL_IMAGE:figures/full_fig_p023_18.png]
Figure 19
Figure 19. Figure 19: b shows the distribution of cosine similarities between embedding chunks and the embeddings in the 10th layer. The average cosine similarity between the maximally similar dictionary chunk and the embeddings is concentrated around 0.5 [PITH_FULL_IMAGE:figures/full_fig…
Figure 20
Figure 20. Figure 20: Interactions of neural population chunks upon parsing the beginning sentence of Emma. K. Full plot maximal POS tag and chunk correlation across all layers [PITH_FULL_IMAGE:figures/full_fig_p025_20.png]
Figure 21
Figure 21. Figure 21: Part-of-speech tags and their maximal correlation across all layers in LLaMA-3. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_21.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

66 extracted references · 31 canonical work pages

  1. [1]

    and Berrada, M

    Adadi, A. and Berrada, M. Peeking inside the black-box: A survey on explainable artificial intelligence (xai). IEEE Access, 6: 0 52138--52160, 2018. doi:10.1109/ACCESS.2018.2870052

    work page arXiv 2018

  2. [2]

    and Srikant, R

    Agrawal, R. and Srikant, R. Mining sequential patterns. In Proceedings of the Eleventh International Conference on Data Engineering, pp.\ 3--14. IEEE, 1995. doi:10.1109/ICDE.1995.380415

    work page arXiv 1995

  3. [3]

    and richard baraniuk

    Balestriero, R. and richard baraniuk. A spline theory of deep learning. In Dy, J. and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp.\ 374--383. PMLR, 10--15 Jul 2018. URL https://proceedings.mlr.press/v80/balestriero18b.html

    work page 2018

  4. [4]

    Revisiting model stitching to compare neural representations

    Bansal, Y., Nakkiran, P., and Barak, B. Revisiting model stitching to compare neural representations. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp.\ 225--236. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/paper/2021/file/...

    work page 2021

  5. [5]

    Extrapolative-Interpolative Cycle-Consistency Learning for Video Frame Extrapolation

    Bau, D., Zhu, J.-Y., et al. Understanding the role of individual neurons in deep learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020. URL https://arxiv.org/abs/2005.13194

    work page Pith review arXiv 2020

  6. [6]

    Probing classifiers: Promises, shortcomings, and advances

    Belinkov, Y. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48 0 (1): 0 207--219, March 2022. doi:10.1162/coli_a_00422. URL https://aclanthology.org/2022.cl-1.7/

    doi:10.1162/coli_a_00422 2022

  7. [7]

    Eliciting latent predictions from transformers with the tuned lens

    Belrose, N., Furman, Z., Smith, L., Halawi, D., Ostrovsky, I., McKinney, L., Biderman, S., and Steinhardt, J. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint, arXiv:2303.08112, 2023

    arXiv 2023

  8. [8]

    Natural Language Processing with Python

    Bird, S., Klein, E., and Loper, E. Natural Language Processing with Python. O’Reilly Media, 2009. URL https://www.nltk.org/

    work page 2009

Show all 66 references
  1. [9]

    Identifying functionally important features with end-to-end sparse dictionary learning, 2024

    Braun, D., Taylor, J., Goldowsky-Dill, N., and Sharkey, L. Identifying functionally important features with end-to-end sparse dictionary learning, 2024. URL https://arxiv.org/abs/2405.12241

    2024 arXiv

  2. [10]

    L., Anil, C., Denison, C., Askell, A., Lasenby, R., Wu, Y., Kravec, S., Schiefer, N., Maxwell, T., Joseph, N., Tamkin, A., Nguyen, K., McLean, B., Burke, J

    Bricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, A., Conerly, T., Turner, N. L., Anil, C., Denison, C., Askell, A., Lasenby, R., Wu, Y., Kravec, S., Schiefer, N., Maxwell, T., Joseph, N., Tamkin, A., Nguyen, K., McLean, B., Burke, J. E., Hume, T., Carter, S., Henighan...

    2023

  3. [11]

    Chase, W. G. and Simon, H. A. Perception in chess . Cognitive Psychology, 4(1) 0 (55-81), 1973. doi:10.1016/0010-0285(73)90004-2

    1973 doi

  4. [12]

    and Geiger, A

    Chaudhary, M. and Geiger, A. Evaluating open-source sparse autoencoders on disentangling factual knowledge in gpt-2 small, 2024. URL https://arxiv.org/abs/2409.04478

    2024 arXiv

  5. [13]

    M., Cunningham, J

    Churchland, M. M., Cunningham, J. P., Kaufman, M. T., Foster, J. D., Nuyujukian, P., Ryu, S. I., and Shenoy, K. V. Neural population dynamics during reaching. Nature, 487 0 (7405): 0 51--56, Jul 2012. doi:10.1038/nature11129

    2012 doi

  6. [14]

    Cohen, M. R. and Kohn, A. Measuring and interpreting neuronal correlations. Nature Neuroscience, 14 0 (7): 0 811--819, Jun 2011. doi:10.1038/nn.2842

    2011 doi

  7. [15]

    Sparse autoencoders find highly interpretable features in language models, 2023

    Cunningham, H., Ewart, A., Riggs, L., Huben, R., and Sharkey, L. Sparse autoencoders find highly interpretable features in language models, 2023. URL https://arxiv.org/abs/2309.08600

    2023 arXiv

  8. [16]

    Knowledge neurons in pretrained transformers

    Dai, D., Dong, L., Hao, Y., Sui, Z., Chang, B., and Wei, F. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 8493--8502, 2022

    2022

  9. [17]

    Y., Karidi, T., Choshen, L., and Geva, M

    Din, A. Y., Karidi, T., Choshen, L., and Geva, M. Jump to conclusions: Short-cutting transformers with linear transformations. arXiv preprint, arXiv:2303.09435, 2023

    2023 arXiv

  10. [18]

    A., and Shocher, A

    Dravid, A., Gandelsman, Y., Efros, A. A., and Shocher, A. Rosetta neurons: Mining the common units in a model zoo, 2023. URL https://arxiv.org/abs/2306.09346

    2023 arXiv

  11. [19]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  12. [20]

    Egan, D. E. and Schwartz, B. J. Chunking in recall of symbolic drawings . Memory & Cognition , 7 0 (2), 1979. ISSN 0090502X. doi:10.3758/BF03197595

    1979 doi

  13. [21]

    Toy models of superposition

    Elhage, N., Hume, T., Olsson, C., Schiefer, N., Henighan, T., Kravec, S., Hatfield-Dodds, Z., Lasenby, R., Drain, D., Chen, C., Grosse, R., McCandlish, S., Kaplan, J., Amodei, D., Wattenberg, M., and Olah, C. Toy models of superposition. Transformer Circuits Thread, 2022. URL ...

    2022

  14. [22]

    Ellis, N. C. Sequencing in SLA: Phonological memory, chunking, and points of order . Studies in Second Language Acquisition, 18 0 (1), 1996. ISSN 14701545. doi:10.1017/S0272263100014698

    1996 doi

  15. [23]

    K., Fries, P., and Singer, W

    Engel, A. K., Fries, P., and Singer, W. Dynamic predictions: Oscillations and synchrony in top-down processing. Nature Reviews Neuroscience, 2 0 (10): 0 704--716, 2001. doi:10.1038/35094565

    2001 doi

  16. [24]

    A new algorithm for data compression

    Gage, P. A new algorithm for data compression. C Users Journal, 1994

    1994

  17. [25]

    R., and Goldberg, Y

    Geva, M., Caciularu, A., Wang, K. R., and Goldberg, Y. Transformer Feed - Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space , October 2022. URL http://arxiv.org/abs/2203.14680. arXiv:2203.14680 [cs]

    2022 arXiv

  18. [26]

    C., Croker, S., Cheng, P

    Gobet, F., Lane, P. C., Croker, S., Cheng, P. C., Jones, G., Oliver, I., and Pine, J. M. Chunking mechanisms in human learning . Trends in Cognitive Sciences, 5 0 (6), 2001. ISSN 13646613. doi:10.1016/S1364-6613(00)01662-4

    2001 doi

  19. [27]

    Goh, G. et al. Multimodal neurons in artificial neural networks. Distill, 2021. URL https://distill.pub/2021/multimodal-neurons/

    2021

  20. [28]

    Graybiel, A. M. The basal ganglia and chunking of action repertoires . In Neurobiology of Learning and Memory, volume 70, pp.\ 1--2, 1998. doi:10.1006/nlme.1998.3843

    1998

  21. [29]

    Finding neurons in a haystack: Case studies with sparse probing, 2023

    Gurnee, W., Nanda, N., Pauly, M., Harvey, K., Troitskii, D., and Bertsimas, D. Finding neurons in a haystack: Case studies with sparse probing, 2023. URL https://arxiv.org/abs/2305.01610

    2023 arXiv

  22. [30]

    Hart, M. S. Project gutenberg. projectgutenberg.org, 1971. URL https://www.gutenberg.org/

    1971

  23. [31]

    The platonic representation hypothesis, 2024

    Huh, M., Cheung, B., Wang, T., and Isola, P. The platonic representation hypothesis, 2024. URL https://arxiv.org/abs/2405.07987

    2024 arXiv

  24. [32]

    What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.\ 3651--3657, 2019

    Jawahar, G., Sagot, B., and Seddah, D. What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.\ 3651--3657, 2019. doi:10.18653/v1/P19-1356. URL https://aclanthology.org/P19-1356

    2019 doi

  25. [33]

    Evaluating sparse autoencoders on targeted concept erasure tasks, 2024

    Karvonen, A., Rager, C., Marks, S., and Nanda, N. Evaluating sparse autoencoders on targeted concept erasure tasks, 2024. URL https://arxiv.org/abs/2411.18895

    2024 arXiv

  26. [34]

    and Hoffmann, J

    Koch, I. and Hoffmann, J. Patterns, chunks, and hierarchies in serial reaction-time tasks. Psychological Research Psychologische Forschung, 63 0 (1): 0 22--35, March 2000. ISSN 0340-0727. doi:10.1007/PL00008165. URL http://link.springer.com/10.1007/PL00008165

    doi

  27. [35]

    Similarity of neural network representations revisited

    Kornblith, S., Norouzi, M., Lee, H., and Hinton, G. Similarity of neural network representations revisited. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Researc...

    2019

  28. [36]

    E., Rosenbloom, P

    Laird, J. E., Rosenbloom, P. S., and Newell, A. Towards chunking as a general learning mechanism. In AAAI, pp.\ 188--192, 1984

    1984

  29. [37]

    and Vedaldi, A

    Lenc, K. and Vedaldi, A. Understanding image representations by measuring their equivariance and equivalence, 2015. URL https://arxiv.org/abs/1411.5908

    2015 arXiv

  30. [38]

    Lipton, Z. C. The mythos of model interpretability, 2017. URL https://arxiv.org/abs/1606.03490

    2017 arXiv

  31. [39]

    P., Santorini, B., and Marcinkiewicz, M

    Marcus, M. P., Santorini, B., and Marcinkiewicz, M. A. Building a large annotated corpus of english: The penn treebank. Computational Linguistics, 19 0 (2): 0 313--330, 1993

    1993

  32. [41]

    J., Belinkov, Y., Bau, D., and Mueller, A

    Marks, S., Rager, C., Michaud, E. J., Belinkov, Y., Bau, D., and Mueller, A. Sparse Feature Circuits : Discovering and Editing Interpretable Causal Graphs in Language Models , March 2024 b . URL http://arxiv.org/abs/2403.19647. arXiv:2403.19647 [cs]

    2024 arXiv

  33. [42]

    Miller, G. A. The magical number seven, plus or minus two: some limits on our capacity for processing information . Psychological Review, 1956. ISSN 0033295X. doi:10.1037/h0043158

    1956 doi

  34. [43]

    and Neo, C

    Miller, J. and Neo, C. We found an neuron in gpt-2. AI Alignment Forum, February 2023. Linkpost from https://clementneo.com

    2023

  35. [44]

    Explanation in artificial intelligence: Insights from the social sciences, 2018

    Miller, T. Explanation in artificial intelligence: Insights from the social sciences, 2018. URL https://arxiv.org/abs/1706.07269

    2018 arXiv

  36. [45]

    Relative representations enable zero-shot latent space communication, 2023

    Moschella, L., Maiorca, V., Fumero, M., Norelli, A., Locatello, F., and Rodolà, E. Relative representations enable zero-shot latent space communication, 2023. URL https://arxiv.org/abs/2209.15430

    2023 arXiv

  37. [46]

    and Andreas, J

    Mu, J. and Andreas, J. Compositional explanations of neurons, 2021. URL https://arxiv.org/abs/2006.14032

    2021 arXiv

  38. [47]

    Synthesizing the preferred inputs for neurons in neural networks via deep generator networks

    Nguyen, A., Dosovitskiy, A., et al. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. Advances in Neural Information Processing Systems (NeurIPS), 2016. URL https://arxiv.org/abs/1605.09304

    2016 arXiv

  39. [48]

    An overview of early vision in inceptionv1

    Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M., and Carter, S. An overview of early vision in inceptionv1. Distill, 2020 a . https://distill.pub/2020/circuits/early-vision

    2020

  40. [49]

    Zoom in: An interpretability method for deep neural networks

    Olah, C., Satyanarayan, A., Carter, S., et al. Zoom in: An interpretability method for deep neural networks. Distill, 2020 b . URL https://distill.pub/2020/circuits/

    2020

  41. [50]

    C., and Bau, D

    Pal, K., Sun, J., Yuan, A., Wallace, B. C., and Bau, D. Future lens: Anticipating subsequent tokens from a single hidden state. arXiv preprint, arXiv:2311.04897, 2023

    2023 arXiv

  42. [51]

    RISE : Randomized Input Sampling for Explanation of Black -box Models , September 2018

    Petsiuk, V., Das, A., and Saenko, K. RISE : Randomized Input Sampling for Explanation of Black -box Models , September 2018. URL http://arxiv.org/abs/1806.07421. arXiv:1806.07421 [cs]

    2018 arXiv

  43. [52]

    Learning to generate reviews and discovering sentiment, 2017

    Radford, A., Jozefowicz, R., and Sutskever, I. Learning to generate reviews and discovering sentiment, 2017. URL https://arxiv.org/abs/1704.01444

    2017 arXiv

  44. [53]

    Why Should I Trust You ?

    Ribeiro, M. T., Singh, S., and Guestrin, C. " Why Should I Trust You ?": Explaining the Predictions of Any Classifier , August 2016. URL http://arxiv.org/abs/1602.04938. arXiv:1602.04938 [cs]

    2016 arXiv

  45. [54]

    On linear identifiability of learned representations

    Roeder, G., Metz, L., and Kingma, D. On linear identifiability of learned representations. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp.\ 9030--9039. PMLR, 18...

    2021

  46. [55]

    E pluribus unum: From complexity, universality

    Tao, T. E pluribus unum: From complexity, universality. Daedalus, 141 0 (3): 0 23--34, 07 2012. ISSN 0011-5266. doi:10.1162/DAED_a_00158. URL https://doi.org/10.1162/DAED\_a\_00158

    2012 doi

  47. [56]

    Word equations: Inherently interpretable sparse word embeddings through sparse coding

    Templeton, A. Word equations: Inherently interpretable sparse word embeddings through sparse coding. In Bastings, J., Belinkov, Y., Dupoux, E., Giulianelli, M., Hupkes, D., Pinter, Y., and Sajjad, H. (eds.), Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Inter...

    2021 doi

  48. [57]

    Bert rediscovers the classical nlp pipeline

    Tenney, I., Das, D., and Pavlick, E. Bert rediscovers the classical nlp pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.\ 4593--4601, 2019. doi:10.18653/v1/P19-1452. URL https://aclanthology.org/P19-1452

    2019 doi

  49. [58]

    Neurons in large language models: Dead, n-gram, positional, 2023

    Voita, E., Ferrando, J., and Nalmpantis, C. Neurons in large language models: Dead, n-gram, positional, 2023. URL https://arxiv.org/abs/2309.04827

    2023 arXiv

  50. [59]

    Interpretability in the wild: a circuit for indirect object identification in gpt-2 small, 2022

    Wang, K., Variengien, A., Conmy, A., Shlegeris, B., and Steinhardt, J. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small, 2022. URL https://arxiv.org/abs/2211.00593

    2022 arXiv

  51. [60]

    Large language models are interpretable learners, 2024

    Wang, R., Si, S., Yu, F., Wiesmann, D., Hsieh, C.-J., and Dhillon, I. Large language models are interpretable learners, 2024. URL https://arxiv.org/abs/2406.17224

    2024 arXiv

  52. [61]

    Learning Structure from the Ground up— Hierarchical Representation Learning by Chunking

    Wu, S., Elteto, N., Dasgupta, I., and Schulz, E. Learning Structure from the Ground up— Hierarchical Representation Learning by Chunking . In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems , vol...

    2022

  53. [62]

    Chunking as a rational solution to the speed–accuracy trade-off in a serial reaction time task

    Wu, S., Éltető, N., Dasgupta, I., and Schulz, E. Chunking as a rational solution to the speed–accuracy trade-off in a serial reaction time task. Scientific Reports, 13 0 (1): 0 7680, May 2023. ISSN 2045-2322. doi:10.1038/s41598-023-31500-3. URL https://doi.org/10.1038/s41598-0...

    2023 doi

  54. [63]

    Two types of motifs enhance human recall and generalization of long sequences

    Wu, S., Thalmann, M., and Schulz, E. Two types of motifs enhance human recall and generalization of long sequences. Communications Psychology, 3 0 (1): 0 3, 2025. URL https://www.nature.com/articles/some-url-placeholder

    2025

  55. [64]

    Zaki, M. J. Sequence mining in categorical domains: Incorporating constraints. In Proceedings of the Ninth International Conference on Information and Knowledge Management, pp.\ 422--429. ACM, 2000. doi:10.1145/354756.354807

  56. [65]

    and Zhu, S.-c

    Zhang, Q.-s. and Zhu, S.-c. Visual interpretability for deep learning: a survey. Frontiers of Information Technology & Electronic Engineering, 19 0 (1): 0 27--39, January 2018. ISSN 2095-9184, 2095-9230. doi:10.1631/FITEE.1700808. URL http://link.springer.com/10.1631/FITEE.1700808

    2018 doi

  57. [66]

    J., Wang, Z., Mallen, A., Basart, S., Koyejo, S., Song, D., Fredrikson, M., Kolter, J

    Zou, A., Phan, L., Chen, S., Campbell, J., Guo, P., Ren, R., Pan, A., Yin, X., Mazeika, M., Dombrowski, A.-K., Goel, S., Li, N., Byun, M. J., Wang, Z., Mallen, A., Basart, S., Koyejo, S., Song, D., Fredrikson, M., Kolter, J. Z., and Hendrycks, D. Representation Engineering : A...

    2023 arXiv

  58. [67]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.