{"id":"07d61de6-fcab-4a3c-bdcd-49adb838cb3b","arxiv_id":"2501.15435","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Activation Spectroscopy uses Fourier analysis of binarized neural activations to find high-value, non-redundant sets of neurons that jointly drive a network's output.","lead":"This paper introduces Activation Spectroscopy, a method that finds which groups of neurons in a neural network jointly influence a prediction by treating a layer's activations as a Boolean function and examining its Fourier coefficients. It could help interpretability researchers move beyond individual neuron importance and toward understanding how information is distributed across many neurons.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"In-distribution sampling guarantee is asserted, not shown; without the deferred Appendix proof or a numeric test, the core claim that ActSpec returns genuine joint contributions is unverified.","rationale":"The reader's weakest_assumption pinpoints the same Section 4.1 in-distribution sampling equivalence as the load-bearing premise. The paper labels itself as preliminary work and leaves the proof to a missing appendix, so the concern is real but not a demonstrated error. A concrete numeric or analytic test on small projections would settle whether the proportionality claim holds. Until then, the CONDITIONAL verdict remains appropriate; the central claim is plausible but not yet supported. I do not see grounds to strengthen the verdict to ACCEPT, and nothing here demonstrates a fatal flaw that would require REJECT. The reader's recommendation to provide the appendix proof, add error bars and baselines, and release code is consistent with the identified gap. The synthetic and MNIST results are suggestive, and the paper does flag its own limitations (e.g., the binarization caveat in Section 4.2), but the missing mathematical support for the main sampling guarantee keeps the verdict at CONDITIONAL.","tokens_in":9975,"tokens_out":1535,"duration_ms":12648,"concrete_test":"Verify the Section 4.1 proportionality claim computationally on a small synthetic projection where the full-space Fourier coefficients are known exactly. Implement f as a projection onto a set of D in-distribution samples, compute its exact Fourier coefficients, and compare them with the estimates obtained by ActSpec using D, 2D, and 4D samples drawn from the in-distribution set, across 50 random sample draws. If the estimated coefficients do not concentrate on the exact high-valued coefficients within a tolerance consistent with the Hoeffding bounds claimed, the in-distribution guarantee fails. Additionally, independently re-derive the Goldreich-Levin sample bound for the projection case without assuming full-space uniform sampling.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central theoretical load-bearing step is in Section 4.1: for a pseudo-Boolean projection, uniform sampling over an In-distribution subspace is claimed equivalent to uniform sampling over the full space up to a constant factor, so that Goldreich-Levin sample guarantees carry over. The derivation is deferred to an Appendix that is absent from this manuscript. This matters because ActSpec's experiments run on real data draws, and the returned Fourier coefficients are estimated from those draws. If the stated proportionality fails, or if the complexity estimate in Theorem 3.2 assumes independence/uniformity that is not preserved under conditional draws, then the coefficients found could reflect sampling artifacts rather than actual high-valued, non-redundant contributions of the network. The paper's own example in Section 4.2 shows that conditioning on in-distribution data creates linear dependencies among variables, which is precisely a place where sums over squared coefficients can behave differently than under full uniform sampling. The claims that 'all existing guarantees still hold' and that sample bounds from Hoeffding apply are not substantiated in the text. This is a placeholder rather than an internal contradiction, but it is the single assumption on which every experimental conclusion rests.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Activation Spectroscopy (ActSpec), an interpretability method that treats the subnetwork from a chosen layer to an output logit as a pseudo-Boolean function over binarized activation patterns. ActSpec searches for high-valued, non-redundant Fourier coefficients of this function using an extension of the Goldreich-Levin algorithm; the returned coefficients indicate subsets of neurons whose joint states influence the output, and the sizes and redundancy of these subsets are used to measure how distributed the representation is. The method is evaluated on synthetic pseudo-Boolean functions, an MNIST MLP (input and intermediate layers), and a RoBERTa sentiment model, with comparisons against standard attribution methods.","tokens_in":10182,"tokens_out":6164,"duration_ms":62622,"significance":"If the central in-distribution sampling guarantee holds, ActSpec is a novel and potentially valuable tool for mechanistic interpretability: it addresses a real combinatorial problem, namely discovering joint contributions of neuron subsets without enumerating them, and the synthetic noise experiments (including a 100-variable setting) are a reasonable stress test. The paper also deserves credit for explicitly acknowledging limitations such as the lossiness of binarization and the hand-chosen thresholds. However, the main theoretical claim is only sketched and deferred to a missing appendix, and the experimental sections omit variance estimates and key hyperparameters. The significance is therefore conditional: the contribution is promising but not yet established in this manuscript.","major_comments":[{"comment":"The central claim that, for pseudo-Boolean projections, uniform sampling over an in-distribution subspace is equivalent to uniform sampling over the full space up to a constant factor, and hence that all Goldreich-Levin guarantees carry over, is asserted without proof; the appendix containing the derivation is absent from this version. This is not a routine detail. For a function that is zero outside D, the full-space expectation of f(X)χ_S(X) equals (|D|/2^n) times the expectation over D, so the scale of every estimated coefficient changes and the threshold τ for 'high-valued' coefficients must be rescaled by a factor that depends on the unknown |D|/2^n. The paper's own Section 4.2 example shows that conditioning on in-distribution samples creates linear dependencies among variables, so it cannot be taken for granted that the sum-of-squares estimators used in Goldreich-Levin remain unbiased up to a known constant under the conditional draw. Please supply a complete lemma stating the estimator, the exact normalization constant, and the resulting sample bound; without this, the experimental coefficients could be artifacts of the sampling distribution.","section":"Section 4.1"},{"comment":"The stated sample complexity 'O(log(1/δ / η^2))' is not dimensionally coherent as a sample count, and the standard Goldreich-Levin/Kushilevitz-Mansour bound takes the form O(log(1/δ)/η^2) (up to constants). Since this theorem is the quantitative basis for the algorithm and for the in-distribution sampling argument in Section 4.1, the statement should be corrected and accompanied by a proof that specifies the random variable being sampled and how the sum of squared Fourier coefficients is estimated. As written, the reader cannot verify that the branch-and-bound procedure has the claimed sample efficiency.","section":"Section 3.2, Theorem 3.2"},{"comment":"The redundancy filtering step is not formally specified. The text says the algorithm checks whether 'the inner product between the new variable and all subsets of existing variables in the set' is small, but no definition is given of the inner product between a variable and a subset, no estimator is described for evaluating sums of these inner products, and no guidance is provided for choosing the threshold γ. Consequently, the complexity statement 'O(2nS log 1/δ)' is not justified: if every candidate variable must be tested against exponentially many subsets before rejection, the runtime could be exponential in n even when the number S of output subsets is small. Please provide pseudocode, a formal definition of the redundancy test, and a complexity proof.","section":"Section 4.2"},{"comment":"The claim that 'for every experiment we ran, all joint activation patterns corresponded to a unique continuous representation vector' is unquantified and unsupported. No experiment or statistic is described that verifies injectivity of the binarized activation map, and for the token-level activations in the RoBERTa experiment this uniqueness is far from obvious. If binarization is lossy, the Fourier coefficients are computed for a function different from the network's actual computation, so interpreting the returned subsets as joint influences of neurons is unjustified. Please provide a quantitative check of the binarization's fidelity or substantially weaken the claim.","section":"Section 4, final paragraph; Section 5.3"},{"comment":"No error bars, standard deviations, or repeated runs are reported for any experiment, despite the high variability indicated by the noise experiments (ActSpec TV distance 0.0424 at 1000 samples vs 0.1875 at 50 samples). The threshold parameters τ and γ, the exact sample counts, and the RoBERTa experimental details are deferred to a missing appendix, and the pretrained weights are cited as '(anon, [?])'. These omissions make it impossible to assess whether the comparisons in Table 1 are statistically meaningful or whether the dropout trends in Figure 3 are robust. Please report hyperparameters, multiple seeds, and confidence intervals, or explicitly present the results as illustrative rather than definitive.","section":"Section 5, Tables 1 and Figure 3"}],"minor_comments":[{"comment":"The paper is marked 'Preliminary work, in preparation'; if this is intended as a submission, the header should be removed or the submission status clarified.","section":"Abstract"},{"comment":"The related-work discussion would benefit from citing recent Fourier-based interpretability work beyond logit spectroscopy, especially work on the Boolean Fourier spectrum of neural network components, to better position the contribution.","section":"Section 2"},{"comment":"The notation D for the in-distribution set is used informally; please define it explicitly at first use and state whether it is a fixed set of observed activations or the support of a distribution.","section":"Section 4.1"},{"comment":"The table in Section 4.2 is referenced as 'Table 4.2' in the text but is not numbered in the displayed version; please add a caption and number.","section":"Section 4.2, Table"},{"comment":"The column labels 'Constant Noise (1000) Noise (50)' are ambiguous; clarify that 'Constant' refers to the spurious constant-variable experiment and state how total variation distance is computed for each method.","section":"Section 5.1, Table 1"},{"comment":"The anonymized link '(anon, [?])' for pretrained weights must be replaced with a permanent citation or repository identifier.","section":"Section 5.2"},{"comment":"The sentence 'Out of the 43410 sentences in the training set, we tested on a random sample of 6400' is unclear: was the model fine-tuned on 43,410 sentences and tested on a separate 6,400 sample, or is the 43,410 figure the available corpus? Please clarify the data split.","section":"Section 5.4"},{"comment":"Several references (e.g., [2], [14]) are to lecture notes or preprints without DOI/venue information; please provide the most complete publication details available.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is clearly preliminary: the central proof is deferred to an absent appendix, the algorithm is not fully specified, and the experiments lack variance reporting. None of these issues appears irreparable, but the revision must supply the missing derivation and a more rigorous empirical evaluation before the claims can be accepted. I would also ask the editor to ensure that the comparison against attribution methods is fair, since ActSpec is a global method operating on binarized activations while several baselines are local gradient-based methods."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is worth your time for one reason: it introduces a genuinely new tool for interpretability. The authors treat binarized layer activations as a pseudo-Boolean function and use a modified Goldreich-Levin algorithm to find high-valued, non-redundant Fourier coefficients — which map to neuron subsets whose joint states matter. That is a real departure from logit spectroscopy, which uses eigenspectra, and from standard attribution methods, which are local. The synthetic experiments back this up: ActSpec handles constant and random noise variables far better than Integrated Gradients, Saliency, and friends, and the redundancy filter is a sensible fix for a real problem the paper identifies clearly.\n\nThe soft spot is the central theoretical claim, not the experiments. Section 4.1 asserts that for a pseudo-Boolean projection, uniform sampling over the in-distribution subspace is equivalent to uniform sampling over the full space up to a constant, so Goldreich-Levin guarantees carry over. That is the load-bearing step for every real-data experiment, and it is only sketched; the proof is deferred to an appendix that is not present in this manuscript. As written, the proportionality argument is hand-wavy, and the paper's own example in 4.2 shows that conditioning on in-distribution data creates linear dependencies between variables, which is precisely where summed Fourier coefficients can misbehave. The claim that all existing guarantees still hold needs either a proof or a numeric check. This is a placeholder rather than an internal contradiction, but it is the single assumption on which the method's validity rests.\n\nThe other issues are minor-to-moderate: no error bars or repeated runs, no code or data released, a placeholder link for pretrained weights, and threshold parameters tau and gamma chosen by hand. The paper is honestly labeled 'Preliminary work, in preparation,' which matches the state of the analysis.\n\nWho gets value from this: interpretability researchers working on distributed representations and polysemanticity. The idea is likely to be useful even if some details need correction. I would send this to a serious referee — it deserves careful reading — but I would expect heavy revision before publication. The authors need to supply the appendix proof, add variance and baseline comparisons, and release code. If they do, this could be a solid contribution. If not, the current draft is a promising preprint, not a finished paper.\n\nRecommendation: engage with it, but treat the theoretical guarantee as unverified until the appendix appears.","headline":"Genuinely new interpretability method with a load-bearing theoretical claim that is asserted, not shown; worth reading and refereeing, but only after the missing proof and error bars are supplied.","tokens_in":10711,"tokens_out":2443,"would_cite":false,"duration_ms":22990,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["06E30","68Q32","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"Activation Spectroscopy claims that the joint influence of a neuron group in a distributed representation can be found by searching for high-valued, non-redundant Fourier coefficients, and that this search works on in-distribution data…","keywords":["interpretability","distributed representations","activation spectroscopy","pseudo-Boolean functions","Fourier analysis","Goldreich-Levin","neuron subsets","influence"],"falsifier":"Build a small pseudo-Boolean projection whose exact Fourier coefficients can be computed by enumeration, run ActSpec using draws only from its in-distribution points, and compare the estimated high coefficients with the exact values within the algorithm's stated sample error; a systematic mismatch would refute the in-distribution proportionality assumption.","tokens_in":9724,"feed_emoji":"🔬","tokens_out":12048,"duration_ms":97204,"temperature":0.7,"pith_summary":"This paper tries to make distributed representations legible: instead of asking which single neurons matter, it asks which groups of neurons jointly drive a network's output. The authors cast the sub-network between a layer and an output logit as a pseudo-Boolean function over binarized activations and search its Fourier spectrum for high-valued coefficients that are not redundant with simpler subsets. This search, Activation Spectroscopy (ActSpec), extends the Goldreich-Levin algorithm with an extra filter for redundant variables and with a projection argument that lets the search run on in-distribution data alone. If the method is right, it returns the specific neuron subsets whose joint states contribute to the output, and the size and redundancy of those subsets become a measurable fingerprint of how distributed the representation is. The paper demonstrates the approach on synthetic functions with known ground truth, an MNIST classifier, and a transformer fine-tuned for sentiment analysis.","feed_headline":"Fourier scan traces which neuron groups drive a network's output","feed_subtitle":"ActSpec isolates non-redundant neuron subsets, making distributed representations measurable and intervenable","key_machinery":"The central object is the pseudo-Boolean Fourier expansion of the sub-network between a chosen layer and an output logit. With binarized activations as input, the logit is written as $f(x)=\\sum_{S\\subseteq[n]} \\hat f(S)\\prod_{i\\in S} x_i$, where each coefficient $\\hat f(S)$ measures how much the joint state of neuron subset $S$ contributes to the output. ActSpec searches this spectrum with a Goldreich-Levin branch-and-bound procedure, adding a redundancy filter that estimates inner products between a candidate variable and combinations of already-selected variables, rejecting variables whose apparent contribution is just a linear artifact of what was already found. The in-distribution guarantee comes from treating the layer function as a pseudo-Boolean projection that is zero outside the observed samples, so uniform in-distribution sampling is proportional to uniform sampling over the whole cube.","core_discovery":"The central discovery is that a tractable search for jointly influential neuron subsets remains valid when a network layer is evaluated only on the data it was trained on. The key object is a pseudo-Boolean projection: a function that agrees with the layer's logit on in-distribution samples and is zero elsewhere. For such projections, uniform sampling over the in-distribution subspace is proportional to uniform sampling over the full Boolean cube, so Goldreich-Levin's guarantees for estimating sums of squared Fourier coefficients transfer unchanged. ActSpec then searches for coefficients that are both high-valued and minimal, rejecting a candidate neuron when its in-distribution pattern is nearly a linear combination of already-selected variables. The returned subsets trace which joint states matter; across experiments, ActSpec estimates influence exactly on a hard-coded synthetic network, survives 96 noise variables, tracks the representational changes caused by dropout, and identifies group-level interventions in a sentiment transformer that individual-neuron interventions do not reproduce.","pith_inferences":["Beyond the paper, ActSpec could be run layer-by-layer on any trained model to produce a depth profile of distributedness, turning a qualitative intuition into a number that can be tracked across training, architecture, or regularizer.","The paper's binarization of activations is acknowledged as potentially lossy; a natural test is whether the same Fourier search still identifies the same subsets when activations are quantized to more than one bit or when continuous states are not uniquely mapped to Boolean patterns.","The redundancy filter may itself be a polysemanticity detector: neurons that are redundant with many existing subsets are plausible shared coordinates through which multiple features route."],"forward_implications":["ActSpec can estimate boolean influence exactly in synthetic settings where the ground truth is known, beating standard attribution methods and staying unaffected by an added constant variable.","On a real MNIST classifier, ActSpec identifies pixel sets that change meaningfully with the task: distinguishing 0 from 8 highlights the inner bar, while the full ten-way task highlights digit boundaries.","Under stronger dropout regularization, the size of the returned neuron subsets shrinks and the amount of redundancy grows, which the authors read as information being concentrated in fewer, more robustly coded neurons.","In a transformer fine-tuned for sentiment, ablating the returned neuron subsets changes the classification decision 32.4% of the time, and group-level interventions sometimes change the decision when individual-neuron interventions do not.","Because ActSpec filters out constant and redundant variables, it is immune to spurious features that other attribution methods flag as important, such as an input variable that is constant across all observed samples."],"supporting_citations":[{"why":"The original hard-core predicate search that ActSpec extends for finding high Fourier coefficients.","marker":"[6]"},{"why":"The standard reference supplying the Fourier expansion theorem, the influence identity, and the Goldreich-Levin sample bound.","marker":"[15]"},{"why":"Earlier Fourier-spectrum-based decision tree learning that ActSpec's no-learning analysis contrasts with.","marker":"[11]"},{"why":"The attribution library providing the gradient-based baselines for comparison.","marker":"[10]"},{"why":"Feature-visualization observations of single neurons responding to multiple inputs, which motivates the distributed-encoding question.","marker":"[16]"},{"why":"The formal account of polysemanticity and capacity that frames the need to study joint neuron subsets.","marker":"[18]"},{"why":"The pretrained transformer whose layer-10 representations are analyzed in the sentiment study.","marker":"[12]"},{"why":"The fine-grained emotion dataset that defines the sentiment classification task.","marker":"[4]"},{"why":"The ablation protocol used to test whether group-level interventions change predictions.","marker":"[21]"},{"why":"The source of the multi-tiered boolean function used as synthetic ground truth.","marker":"[14]"}],"fun_headline_variants":["Activation Spectroscopy maps joint neuron impact","Fourier scan finds non-redundant neuron subsets","Distributed representations made measurable via ActSpec","New probe isolates influential neuron coalitions","Interpreting AI neurons through activation spectra"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method rests on the claim that uniformly sampling only the in-distribution data is equivalent, up to a constant factor, to uniform sampling over the entire space of possible activations, so the standard sample-size guarantees still hold.","fun_headline_variants_meta":{"raw":{"variants":["Activation Spectroscopy maps joint neuron impact","Fourier scan finds non-redundant neuron subsets","Distributed representations made measurable via ActSpec","New probe isolates influential neuron coalitions","Interpreting AI neurons through activation spectra"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000513,"raw_usage":{"total_tokens":2502,"prompt_tokens":961,"completion_tokens":1541,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":1474}},"tokens_in":577,"tokens_out":1541,"duration_ms":10795,"temperature":1.0,"reasoning_tokens":1474,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:17:22.565311+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a small pseudo-Boolean projection whose exact Fourier coefficients can be computed by enumeration, run ActSpec using draws only from its in-distribution points, and compare the estimated high coefficients with the exact values within the algorithm's stated sample error; a systematic mismatch would refute the in-distribution proportionality assumption.","supporting_citations":[{"cited_title":"Captum: A unified and generic model interpretability library for pytorch, 2020","cited_arxiv_id":null,"evidence_quote":"The attribution library providing the gradient-based baselines for comparison."},{"cited_title":"Feature visualization","cited_arxiv_id":null,"evidence_quote":"Feature-visualization observations of single neurons responding to multiple inputs, which motivates the distributed-encoding question."},{"cited_title":"Jermyn, Joe Benton, and Buck Shlegeris","cited_arxiv_id":null,"evidence_quote":"The formal account of polysemanticity and capacity that frames the need to study joint neuron subsets."},{"cited_title":"Goemotions: A dataset of fine- grained emotions, 2020","cited_arxiv_id":null,"evidence_quote":"The fine-grained emotion dataset that defines the sentiment classification task."},{"cited_title":"In- terpretability in the wild: a circuit for indirect object identification in gpt-2 small, 2022","cited_arxiv_id":null,"evidence_quote":"The ablation protocol used to test whether group-level interventions change predictions."},{"cited_title":"Goldreich-levin lecture, 2012","cited_arxiv_id":null,"evidence_quote":"The source of the multi-tiered boolean function used as synthetic ground truth."},{"cited_title":"Goldreich and L","cited_arxiv_id":null,"evidence_quote":"The original hard-core predicate search that ActSpec extends for finding high Fourier coefficients."},{"cited_title":"Analysis of boolean functions, 2021","cited_arxiv_id":null,"evidence_quote":"The standard reference supplying the Fourier expansion theorem, the influence identity, and the Goldreich-Levin sample bound."},{"cited_title":"Learning decision trees using the fourier spectrum","cited_arxiv_id":null,"evidence_quote":"Earlier Fourier-spectrum-based decision tree learning that ActSpec's no-learning analysis contrasts with."}],"review_version":1}