Pith. sign in

REVIEW 5 major objections 8 minor 22 references

Making Sense Of Distributed Representations With Activation Spectroscopy

T0 review · 5 major / 8 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Activation Spectroscopy claims that the joint influence of a neuron group in a distributed representation can be found by searching for high-valued, non-redundant Fourier coefficients, and that this search works on in-distribution data…

desk verdict Genuinely new interpretability method with a load-bearing theoretical claim that is asserted, not shown; worth reading and refereeing, but only after the missing proof and error bars are supplied. read the letter →

arxiv 2501.15435 v1 pith:KKXWFUXP submitted 2025-01-26 cs.LG cs.CV

Kyle Reing , Greg Ver Steeg , Aram Galstyan This is my paper · ORCID
classification cs.LGcs.CV MSC 06E3068Q3268T07
keywords interpretabilitydistributedrepresentationsactivationspectroscopypseudo-BooleanfunctionsFourieranalysisGoldreich-Levinneuronsubsetsinfluence
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to make distributed representations legible: instead of asking which single neurons matter, it asks which groups of neurons jointly drive a network's output. The authors cast the sub-network between a layer and an output logit as a pseudo-Boolean function over binarized activations and search its Fourier spectrum for high-valued coefficients that are not redundant with simpler subsets. This search, Activation Spectroscopy (ActSpec), extends the Goldreich-Levin algorithm with an extra filter for redundant variables and with a projection argument that lets the search run on in-distribution data alone. If the method is right, it returns the specific neuron subsets whose joint states contribute to the output, and the size and redundancy of those subsets become a measurable fingerprint of how distributed the representation is. The paper demonstrates the approach on synthetic functions with known ground truth, an MNIST classifier, and a transformer fine-tuned for sentiment analysis.

What carries the argument

The central object is the pseudo-Boolean Fourier expansion of the sub-network between a chosen layer and an output logit. With binarized activations as input, the logit is written as $f(x)=\sum_{S\subseteq[n]} \hat f(S)\prod_{i\in S} x_i$, where each coefficient $\hat f(S)$ measures how much the joint state of neuron subset $S$ contributes to the output. ActSpec searches this spectrum with a Goldreich-Levin branch-and-bound procedure, adding a redundancy filter that estimates inner products between a candidate variable and combinations of already-selected variables, rejecting variables whose apparent contribution is just a linear artifact of what was already found. The in-distribution guarantee comes from treating the layer function as a pseudo-Boolean projection that is zero outside the observed samples, so uniform in-distribution sampling is proportional to uniform sampling over the whole cube.

What would settle it

Build a small pseudo-Boolean projection whose exact Fourier coefficients can be computed by enumeration, run ActSpec using draws only from its in-distribution points, and compare the estimated high coefficients with the exact values within the algorithm's stated sample error; a systematic mismatch would refute the in-distribution proportionality assumption.

Watch

Extended reading notes

Core claim

The central discovery is that a tractable search for jointly influential neuron subsets remains valid when a network layer is evaluated only on the data it was trained on. The key object is a pseudo-Boolean projection: a function that agrees with the layer's logit on in-distribution samples and is zero elsewhere. For such projections, uniform sampling over the in-distribution subspace is proportional to uniform sampling over the full Boolean cube, so Goldreich-Levin's guarantees for estimating sums of squared Fourier coefficients transfer unchanged. ActSpec then searches for coefficients that are both high-valued and minimal, rejecting a candidate neuron when its in-distribution pattern is nearly a linear combination of already-selected variables. The returned subsets trace which joint states matter; across experiments, ActSpec estimates influence exactly on a hard-coded synthetic network, survives 96 noise variables, tracks the representational changes caused by dropout, and identifies group-level interventions in a sentiment transformer that individual-neuron interventions do not reproduce.

Load-bearing premise

The whole method rests on the claim that uniformly sampling only the in-distribution data is equivalent, up to a constant factor, to uniform sampling over the entire space of possible activations, so the standard sample-size guarantees still hold.

Editorial extensions

If this is right

  • ActSpec can estimate boolean influence exactly in synthetic settings where the ground truth is known, beating standard attribution methods and staying unaffected by an added constant variable.
  • On a real MNIST classifier, ActSpec identifies pixel sets that change meaningfully with the task: distinguishing 0 from 8 highlights the inner bar, while the full ten-way task highlights digit boundaries.
  • Under stronger dropout regularization, the size of the returned neuron subsets shrinks and the amount of redundancy grows, which the authors read as information being concentrated in fewer, more robustly coded neurons.
  • In a transformer fine-tuned for sentiment, ablating the returned neuron subsets changes the classification decision 32.4% of the time, and group-level interventions sometimes change the decision when individual-neuron interventions do not.
  • Because ActSpec filters out constant and redundant variables, it is immune to spurious features that other attribution methods flag as important, such as an input variable that is constant across all observed samples.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, ActSpec could be run layer-by-layer on any trained model to produce a depth profile of distributedness, turning a qualitative intuition into a number that can be tracked across training, architecture, or regularizer.
  • The paper's binarization of activations is acknowledged as potentially lossy; a natural test is whether the same Fourier search still identifies the same subsets when activations are quantized to more than one bit or when continuous states are not uniquely mapped to Boolean patterns.
  • The redundancy filter may itself be a polysemanticity detector: neurons that are redundant with many existing subsets are plausible shared coordinates through which multiple features route.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 8 minor

Summary. The paper proposes Activation Spectroscopy (ActSpec), an interpretability method that treats the subnetwork from a chosen layer to an output logit as a pseudo-Boolean function over binarized activation patterns. ActSpec searches for high-valued, non-redundant Fourier coefficients of this function using an extension of the Goldreich-Levin algorithm; the returned coefficients indicate subsets of neurons whose joint states influence the output, and the sizes and redundancy of these subsets are used to measure how distributed the representation is. The method is evaluated on synthetic pseudo-Boolean functions, an MNIST MLP (input and intermediate layers), and a RoBERTa sentiment model, with comparisons against standard attribution methods.

Significance. If the central in-distribution sampling guarantee holds, ActSpec is a novel and potentially valuable tool for mechanistic interpretability: it addresses a real combinatorial problem, namely discovering joint contributions of neuron subsets without enumerating them, and the synthetic noise experiments (including a 100-variable setting) are a reasonable stress test. The paper also deserves credit for explicitly acknowledging limitations such as the lossiness of binarization and the hand-chosen thresholds. However, the main theoretical claim is only sketched and deferred to a missing appendix, and the experimental sections omit variance estimates and key hyperparameters. The significance is therefore conditional: the contribution is promising but not yet established in this manuscript.

major comments (5)
  1. [Section 4.1] The central claim that, for pseudo-Boolean projections, uniform sampling over an in-distribution subspace is equivalent to uniform sampling over the full space up to a constant factor, and hence that all Goldreich-Levin guarantees carry over, is asserted without proof; the appendix containing the derivation is absent from this version. This is not a routine detail. For a function that is zero outside D, the full-space expectation of f(X)χ_S(X) equals (|D|/2^n) times the expectation over D, so the scale of every estimated coefficient changes and the threshold τ for 'high-valued' coefficients must be rescaled by a factor that depends on the unknown |D|/2^n. The paper's own Section 4.2 example shows that conditioning on in-distribution samples creates linear dependencies among variables, so it cannot be taken for granted that the sum-of-squares estimators used in Goldreich-Levin remain unbiased up to a known constant under the conditional draw. Please supply a complete lemma stating the estimator, the exact normalization constant, and the resulting sample bound; without this, the experimental coefficients could be artifacts of the sampling distribution.
  2. [Section 3.2, Theorem 3.2] The stated sample complexity 'O(log(1/δ / η^2))' is not dimensionally coherent as a sample count, and the standard Goldreich-Levin/Kushilevitz-Mansour bound takes the form O(log(1/δ)/η^2) (up to constants). Since this theorem is the quantitative basis for the algorithm and for the in-distribution sampling argument in Section 4.1, the statement should be corrected and accompanied by a proof that specifies the random variable being sampled and how the sum of squared Fourier coefficients is estimated. As written, the reader cannot verify that the branch-and-bound procedure has the claimed sample efficiency.
  3. [Section 4.2] The redundancy filtering step is not formally specified. The text says the algorithm checks whether 'the inner product between the new variable and all subsets of existing variables in the set' is small, but no definition is given of the inner product between a variable and a subset, no estimator is described for evaluating sums of these inner products, and no guidance is provided for choosing the threshold γ. Consequently, the complexity statement 'O(2nS log 1/δ)' is not justified: if every candidate variable must be tested against exponentially many subsets before rejection, the runtime could be exponential in n even when the number S of output subsets is small. Please provide pseudocode, a formal definition of the redundancy test, and a complexity proof.
  4. [Section 4, final paragraph; Section 5.3] The claim that 'for every experiment we ran, all joint activation patterns corresponded to a unique continuous representation vector' is unquantified and unsupported. No experiment or statistic is described that verifies injectivity of the binarized activation map, and for the token-level activations in the RoBERTa experiment this uniqueness is far from obvious. If binarization is lossy, the Fourier coefficients are computed for a function different from the network's actual computation, so interpreting the returned subsets as joint influences of neurons is unjustified. Please provide a quantitative check of the binarization's fidelity or substantially weaken the claim.
  5. [Section 5, Tables 1 and Figure 3] No error bars, standard deviations, or repeated runs are reported for any experiment, despite the high variability indicated by the noise experiments (ActSpec TV distance 0.0424 at 1000 samples vs 0.1875 at 50 samples). The threshold parameters τ and γ, the exact sample counts, and the RoBERTa experimental details are deferred to a missing appendix, and the pretrained weights are cited as '(anon, [?])'. These omissions make it impossible to assess whether the comparisons in Table 1 are statistically meaningful or whether the dropout trends in Figure 3 are robust. Please report hyperparameters, multiple seeds, and confidence intervals, or explicitly present the results as illustrative rather than definitive.
minor comments (8)
  1. [Abstract] The paper is marked 'Preliminary work, in preparation'; if this is intended as a submission, the header should be removed or the submission status clarified.
  2. [Section 2] The related-work discussion would benefit from citing recent Fourier-based interpretability work beyond logit spectroscopy, especially work on the Boolean Fourier spectrum of neural network components, to better position the contribution.
  3. [Section 4.1] The notation D for the in-distribution set is used informally; please define it explicitly at first use and state whether it is a fixed set of observed activations or the support of a distribution.
  4. [Section 4.2, Table] The table in Section 4.2 is referenced as 'Table 4.2' in the text but is not numbered in the displayed version; please add a caption and number.
  5. [Section 5.1, Table 1] The column labels 'Constant Noise (1000) Noise (50)' are ambiguous; clarify that 'Constant' refers to the spurious constant-variable experiment and state how total variation distance is computed for each method.
  6. [Section 5.2] The anonymized link '(anon, [?])' for pretrained weights must be replaced with a permanent citation or repository identifier.
  7. [Section 5.4] The sentence 'Out of the 43410 sentences in the training set, we tested on a random sample of 6400' is unclear: was the model fine-tuned on 43,410 sentences and tested on a separate 6,400 sample, or is the 43,410 figure the available corpus? Please clarify the data split.
  8. [References] Several references (e.g., [2], [14]) are to lecture notes or preprints without DOI/venue information; please provide the most complete publication details available.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: ActSpec's Fourier-coefficient search is self-contained, and the deferred in-distribution proof is a support gap rather than a circular reduction.

full rationale

The paper's derivation chain runs from the standard Fourier Expansion Theorem and the standard Goldreich-Levin sampling theorem to a modified combinatorial search over Fourier coefficients. None of these steps defines ActSpec's output in terms of the phenomenon it is used to test: the reported coefficients are estimated from network activations and labels, not fitted to ground-truth importance, and the comparison values (Influence) are independently defined functions of the same Fourier spectrum rather than quantities ActSpec is tuned to match. The redundancy filter is a construction, not a renamed target. The paper contains no self-citations and invokes no author-specific uniqueness theorem; the citation-heavy premises (Fourier basis, Hoeffding-based Goldreich-Levin guarantees) are standard external results. The main weakness flagged in Section 4.1—the claim that uniform sampling over an in-distribution subspace is equivalent to full-space sampling up to a constant, with proof deferred to an Appendix that is absent from this version—is an omitted proof and an unsupported assumption, not a circular step: even if that claim fails, ActSpec would be making an unjustified methodological leap rather than deriving its conclusion from its own premises. Similarly, the placeholder reference for pretrained weights and the 'Preliminary work, in preparation' note are completeness and reproducibility issues, not circularity. No passage in the manuscript exhibits an equation or definition that reduces a predicted quantity to a fitted input or to an author-imported uniqueness claim.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper relies on standard Boolean analysis results plus two assumptions introduced by the authors: the in-distribution projection equivalence and the lossless-ness of binarization. The projection equivalence is load-bearing and unproven in the main text. The free parameters are thresholds and sample counts chosen by hand, which affect all experimental conclusions.

free parameters (4)
  • Fourier coefficient threshold tau = Not specified in the text
    Chosen per experiment to accept high-valued coefficients; no sensitivity analysis is provided.
  • Redundancy threshold gamma = Not specified in the text
    Used in the redundancy filter to decide when an inner product is small enough; the value is not reported.
  • Sample count for coefficient estimation = 1000 and 50 in synthetic noise experiments; not specified for MNIST or roBERTa
    The number of in-distribution samples drawn affects the accuracy of Fourier coefficient estimates.
  • Binarization threshold for activations = Sign function (0 for negative, 1 for positive), applied to continuous activations
    Activations are binarized to Boolean values; the paper notes this can be lossy and checks one-to-one mapping only empirically.
assumptions (4)
  • standard math Every pseudo-Boolean function has a unique Fourier expansion (Theorem 3.1).
    This is the classical Fourier expansion theorem for functions on the Boolean hypercube, cited from O'Donnell.
  • standard math The Goldreich-Levin theorem allows efficient estimation of sums of squared Fourier coefficients over subsets (Theorem 3.2).
    This is a known result from the analysis of Boolean functions, cited from O'Donnell.
  • ad hoc to paper Sampling uniformly over the in-distribution set is proportional to sampling uniformly over the full Boolean hypercube, so Goldreich-Levin guarantees are preserved for the projected function.
    This is the key theoretical claim of Section 4.1. The paper calls it a pseudo-Boolean projection and defers the full proof to the Appendix, which is not present in this text.
  • domain assumption Binarized activation patterns correspond uniquely to continuous representation vectors in the tested networks.
    Section 4.2 states that all joint activation patterns corresponded to a unique continuous representation vector in every experiment, but the authors say they are unsure how generally applicable this is across all networks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Making Sense Of Distributed Representations With Activation Spectroscopy." pith.science (2026). https://pith.science/paper/KKXWFUXP

@misc{pith2026250115435,
  author       = {Pith},
  title        = {Pith review of: Making Sense Of Distributed Representations With Activation Spectroscopy},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KKXWFUXP}},
  note         = {Machine review of arXiv:2501.15435}
}
read the original abstract

In the study of neural network interpretability, there is growing evidence to suggest that relevant features are encoded across many neurons in a distributed fashion. Making sense of these distributed representations without knowledge of the network's encoding strategy is a combinatorial task that is not guaranteed to be tractable. This work explores one feasible path to both detecting and tracing the joint influence of neurons in a distributed representation. We term this approach Activation Spectroscopy (ActSpec), owing to its analysis of the pseudo-Boolean Fourier spectrum defined over the activation patterns of a network layer. The sub-network defined between a given layer and an output logit is cast as a special class of pseudo-Boolean function. The contributions of each subset of neurons in the specified layer can be quantified through the function's Fourier coefficients. We propose a combinatorial optimization procedure to search for Fourier coefficients that are simultaneously high-valued, and non-redundant. This procedure can be viewed as an extension of the Goldreich-Levin algorithm which incorporates additional problem-specific constraints. The resulting coefficients specify a collection of subsets, which are used to test the degree to which a representation is distributed. We verify our approach in a number of synthetic settings and compare against existing interpretability benchmarks. We conclude with a number of experimental evaluations on an MNIST classifier, and a transformer-based network for sentiment analysis.

Figures

Figures reproduced from arXiv: 2501.15435 by the authors.

Figure 1
Figure 1. Showcasing redundant variables, where a large inner product between variables leads to identical Fourier coefficients first time a variant of pseudo-Boolean Fourier analy￾sis has been used. The classic algorithm of (Mansour, [11]) is utilized to this end to learn decision trees. How￾ever, our approach does not do any learning, and in￾stead analyses a pre-existing network that has already been trained without such li… view at source ↗
Figure 2
Figure 2. Top Row: Experiments on samples from two classes. Bottom Row: Experiments on all of MNIST. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Left: Size of Subsets returned by ActSpec as a function of Dropout regularization. Right: Amount of [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Hypergraph representations of sets output by ActSpec for the intermediate representations of an [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 19 canonical work pages

  1. [1]

    To- wards monosemanticity: Decomposing language models with dictionary learning

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Con- erly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and ...

    work page 2023

  2. [2]

    Spectral filters, dark signals, and attention sinks, 2024

    Nicola Cancedda. Spectral filters, dark signals, and attention sinks, 2024

    work page 2024

  3. [3]

    Visualizing and measuring the geometry of bert, 2019

    Andy Coenen, Emily Reif, Ann Yuan, Been Kim, Adam Pearce, Fernanda Vi´ egas, and Martin Wat- tenberg. Visualizing and measuring the geometry of bert, 2019

    work page 2019

  4. [4]

    Goemotions: A dataset of fine- grained emotions, 2020

    Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. Goemotions: A dataset of fine- grained emotions, 2020

    work page 2020

  5. [5]

    The shapley taylor interaction in- dex, 2020

    Kedar Dhamdhere, Ashish Agarwal, and Mukund Sundararajan. The shapley taylor interaction in- dex, 2020

    work page 2020

  6. [6]

    Goldreich and L

    O. Goldreich and L. A. Levin. A hard-core pred- icate for all one-way functions. In Proceedings of the Twenty-First Annual ACM Symposium on Theory of Computing , STOC ’89, page 25–32, New York, NY, USA, 1989. Association for Com- puting Machinery

    work page 1989

  7. [7]

    Does localization inform editing? surprising differences in causality-based localiza- tion vs

    Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. Does localization inform editing? surprising differences in causality-based localiza- tion vs. knowledge editing in language models, 2023

    work page 2023

  8. [8]

    Sarthak Jain and Byron C. Wallace. Attention is not explanation, 2019

    work page 2019

Show all 22 references
  1. [9]

    Sch¨ utt, Sven D¨ ahne, Dumitru Erhan, and Been Kim

    Pieter-Jan Kindermans, Sara Hooker, Julius Ade- bayo, Maximilian Alber, Kristof T. Sch¨ utt, Sven D¨ ahne, Dumitru Erhan, and Been Kim. The (un)reliability of saliency methods, 2017

    2017

  2. [10]

    Captum: A unified and generic model interpretability library for pytorch, 2020

    Narine Kokhlikyan, Vivek Miglani, Miguel Mar- tin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushk- ina, Carlos Araya, Siqi Yan, and Orion Reblitz- Richardson. Captum: A unified and generic model interpretability library for pytorch, 2020

    2020

  3. [11]

    Learning decision trees using the fourier spectrum

    Eyal Kushilevitz and Yishay Mansour. Learning decision trees using the fourier spectrum. SIAM Journal on Computing , 22(6):1331–1348, 1993

    1993

  4. [12]

    Roberta: A robustly optimized bert pretraining approach, 2019

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019

    2019

  5. [13]

    Locating and editing factual associations in gpt, 2023

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt, 2023

    2023

  6. [14]

    Goldreich-levin lecture, 2012

    Ryan O’Donnell. Goldreich-levin lecture, 2012

    2012

  7. [15]

    Analysis of boolean functions, 2021

    Ryan O’Donnell. Analysis of boolean functions, 2021

    2021

  8. [16]

    Feature visualization

    Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2017. https://distill.pub/2017/feature-visualization

    2017

  9. [17]

    A primer in bertology: What we know about how bert works, 2020

    Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in bertology: What we know about how bert works, 2020

    2020

  10. [18]

    Jermyn, Joe Benton, and Buck Shlegeris

    Adam Scherlis, Kshitij Sachan, Adam S. Jermyn, Joe Benton, and Buck Shlegeris. Polysemanticity and capacity in neural networks, 2023

    2023

  11. [19]

    Rethinking interpretability in the era of large language mod- els

    Chandan Singh, Jeevana Priya Inala, Michel Gal- ley, Rich Caruana, and Jianfeng Gao. Rethinking interpretability in the era of large language mod- els. arXiv preprint arXiv:2402.01761 , 2024

    2024 arXiv

  12. [20]

    Daniel Freeman, Theodore R

    Adly Templeton, Tom Conerly, Jonathan Mar- cus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Ed- ward Rees, Jos...

    2024

  13. [21]

    In- terpretability in the wild: a circuit for indirect object identification in gpt-2 small, 2022

    Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. In- terpretability in the wild: a circuit for indirect object identification in gpt-2 small, 2022

    2022

  14. [22]

    Attention is not not explanation, 2019

    Sarah Wiegreffe and Yuval Pinter. Attention is not not explanation, 2019

    2019

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.