{"id":"f8d5855c-3447-478d-96d2-1854a26087be","arxiv_id":"2608.13538","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"SAEVerbalizer fine-tunes an LLM so that injecting an SAE decoder direction into its hidden states makes it generate a natural-language explanation, and this learned skill transfers to unseen features, new SAE dictionaries, and other LLMs through a small adapter.","lead":"This paper trains large language models to write natural-language explanations of sparse autoencoder features by injecting the feature's internal direction into the model and fine-tuning the later layers. The approach could make AI interpretability much cheaper and more scalable, because explanations are generated directly from internal representations rather than by scanning massive text corpora.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2's 'transfer across SAE dictionaries' may be confounded by Matryoshka nesting: the width-65k Gemma Scope 2 test dictionary is likely a sub-dictionary of the width-262k training dictionary, so test directions may overlap training features; an independent-SAe transfer test is required before…","rationale":"I read the paper as a solid but conditionally acceptable contribution. The reader's weakest assumption (reference validity) is real and is explicitly disclosed in the Limitations; it weakens absolute claims about explanation correctness but does not undermine the large RA gaps over the untrained backbone or the relative comparisons between configurations. The more load-bearing issue is the cross-dictionary transfer claim in Table 2, because the paper's own invocation of Gemma Scope 2's Matryoshka objective makes it plausible that the width-65k dictionary is not independently trained but is a nested sub-dictionary of the width-262k dictionary used for training. If test directions overlap training directions, the high RA is explained by memorization of seen directions, not by a transferable verbalization capability. This confound is unacknowledged and directly threatens a headline abstract claim. It is also easily testable by computing direction overlap and, if needed, repeating the experiment with a genuinely independent SAE. The within-dictionary generalization and cross-LLM adapter results are not affected by this concern, so the overall method remains credible. Since the reader already recommended CONDITIONAL and this concern reinforces conditionality rather than overturning the paper, I leave the verdict unchanged.","tokens_in":18362,"tokens_out":6958,"duration_ms":75556,"concrete_test":"Extract all width-65k test decoder directions and compute the maximum cosine similarity (and index identity) against all width-262k decoder columns, especially in the low-index range. If a substantial fraction of test features matches a width-262k direction at cosine > 0.999, repeat the §4.3.1 transfer experiment using a width-65k SAE trained from a different random seed (or a non-Matryoshka SAE) with no shared feature directions; report RA on that genuinely held-out dictionary and compare it with the RA on the nested dictionary.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim includes transfer 'across separately trained SAE dictionaries' (abstract; §4.3.1, Table 2). The evidence is applying the default verbalizer to a width-65k Gemma Scope 2 SAE on the same LLM and layer. However, Gemma Scope 2 SAEs are Matryoshka SAEs: the paper cites Bussmann et al. 2025 and invokes the Matryoshka reconstruction objective in §4.1. In a Matryoshka SAE, narrower dictionaries are nested subsets of wider ones, with shared low-index feature directions. The width-65k test features may therefore coincide with width-262k directions already present in the 48k training pairs, and the high RA in Table 2 would reflect memorized directions rather than a learned capability that generalizes to a genuinely unseen dictionary. The paper never checks decoder-direction overlap between the two dictionaries, and the Limitations section only notes limited LLM/SAE coverage; it does not flag this nested-dictionary confound. This concern is concrete and separable from the acknowledged RA-is-not-correctness limitation: even if RA were a perfect correctness metric, a transfer result obtained on overlapping directions would not establish cross-dictionary generalization. The within-dictionary unseen-feature results and the cross-LLM adapter results are not affected by this particular concern, so the method remains plausible, but the headline transfer claim requires an independent test.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SAEVerbalizer, a method that fine-tunes the downstream layers of an LLM so that, given only a sparse autoencoder decoder direction injected into a designated prompt span, the model generates a natural-language explanation of the feature. The authors train on feature–explanation pairs from filtered Neuronpedia references, evaluate with Reference Agreement (RA) against the same reference pool, and report generalization to held-out features across Gemma 3 1B/4B/27B backbones and multiple Gemma Scope 2 layers (Table 1), reuse on a width-65k SAE on the same LLM and layer (Table 2), adapter-based cross-LLM transfer (Table 3), and ablations over supervision size, prompt, injection span, and injection strength. They also report qualitative cases for joint injection and sign reversal. The central claimed capability is that a single fine-tuned LLM can explain unseen SAE features directly from decoder directions without per-feature activation analysis.","tokens_in":18598,"tokens_out":6492,"duration_ms":66152,"significance":"If the capability claims hold, the paper makes a useful practical contribution: it replaces corpus-scale activation mining with a single forward pass on a fixed prompt for each feature, and it demonstrates that explanation of unseen features can be learned from a few thousand pairs. Strengths include the disjoint test splits (GTS/LIG/GG), the clear zero-supervision backbone baseline in Table 4, the careful documentation of prompts and filtering, and the explicit limitations section. The main quantitative effects—for example, 1.6% backbone versus 52.3% RA for the default 27B-L16 configuration—are large and support the existence of a learned verbalization capability. However, the cross-dictionary transfer claim is weakened by the Matryoshka nesting of the two Gemma Scope 2 dictionaries, and the evaluation metric partly measures agreement with the same filtered Neuronpedia pool used for supervision, so the headline 'explains features' wording overstates what RA establishes.","major_comments":[{"comment":"The cross-dictionary transfer experiment is confounded by Matryoshka nesting. The width-65k and width-262k Gemma Scope 2 SAEs are from the same Matryoshka family, and the paper itself invokes the Matryoshka reconstruction objective in §4.1. In such dictionaries, narrower dictionaries are nested subsets of wider ones, so the width-65k test directions may coincide with low-index width-262k directions already present in the 48k training pairs. The paper never measures decoder-direction overlap between the two dictionaries. As a result, the high RA in Table 2 could reflect memorized directions rather than a learned capability that transfers to a genuinely separate dictionary. This is load-bearing because the abstract and §4.3.1 claim transfer 'across separately trained SAE dictionaries.' Please report the distribution of cosine similarities between the width-65k directions and the width-262k training directions, and repeat the transfer test either with an independently trained SAE or with the width-65k directions restricted to those with no near-neighbor in the training dictionary. The within-dictionary unseen-feature results and the adapter results are not affected by this particular confound.","section":"§4.3.1 (Table 2)"},{"comment":"RA is a permissive judge-based agreement metric against references drawn from the same filtered Neuronpedia pool used to construct the training supervision. The judge counts synonyms, subset/superset, and 'strong direct logical/contextual association' as agreement, and the model is explicitly fine-tuned to reproduce filtered Neuronpedia explanations. Consequently, the reported RA values partly measure reproduction of the reference distribution rather than explanation correctness. The Limitations section acknowledges that RA does not establish absolute correctness, but the concern is stronger: for the central claim that the verbalizer 'explains SAE features directly from decoder directions,' the evaluation should include at least a human-annotated subset, a hard-negative or distractor-reference condition, or an inter-judge agreement measure to show that RA is not merely style matching. This point affects the interpretation of every quantitative table.","section":"§4.1 (Evaluation Protocol) and Limitations"},{"comment":"All configurations are single runs, so the quantitative claims lack variability estimates. Differences such as 48.1% versus 52.3% on GTS across supervision sizes (Table 4) and 50.6% versus 52.3% across injection strengths (Table 5) may be within noise. The Limitations section notes this, but since the paper's central claim rests on these RA differences, please provide multiple seeds for at least the default configuration and the main comparison points, or bootstrap confidence intervals over test features. Without this, the claims about scaling with supervision and robustness to prompt and injection variation are not statistically supported.","section":"Appendix F and Tables 1, 4, 5"},{"comment":"The adapter transfer assumes that a linear map trained to reconstruct token representations can be applied to decoder directions via A(h+v)-A(h)=Wv. Decoder directions are not validated to behave as difference vectors in the source representation space, and the MSE objective only aligns pointwise representations. The empirical RA in Table 3 is encouraging, but the paper should provide a direct check that mapped directions land near target-space features (for example, nearest-neighbor directions in a target SAE) or that Wv produces the intended steering effect. Otherwise the cross-LLM transfer claim rests on an unverified linearity assumption.","section":"§3.2 (Eq. 2)"}],"minor_comments":[{"comment":"Please clarify that the cosine similarity column reports the cosine between the two decoder directions in each row (for example, the directions of features #40105 and #111800), and state this explicitly in the table caption.","section":"Table 8"},{"comment":"The width-65k SAE is described as 'unseen'; 'not used during training' would be more precise, since the dictionary belongs to the same Gemma Scope 2 Matryoshka family as the training dictionary.","section":"§4.3.1"},{"comment":"Configurations with very low RA, such as 1B-L22 with 3.9% on GTS at 12k pairs, are still summarized as 'generalize to unseen features across all configurations'; please report the corresponding backbone baseline for each configuration or add a statistical threshold to support this blanket statement.","section":"Table 1"},{"comment":"The evaluation judge prompt permits YES for 'strong direct logical/contextual association,' which may inflate agreement; please report a stricter variant that requires synonymy or near-synonymy, or at least quantify how often the permissive criterion is the deciding factor.","section":"Appendix D"}],"recommendation":"major_revision","confidential_remarks":"The main blocking issue is the Matryoshka confound in Table 2. The authors cite Bussmann et al. (2025) and use the Matryoshka reconstruction objective in their test-set design, so they are aware of the nesting; a straightforward overlap analysis should resolve it. If the transfer result survives an independent-dictionary test, the paper is a solid contribution. I would not reject on the RA circularity alone, because the disjoint unseen-feature splits and the backbone baseline provide partial support, but the abstract and conclusion wording should be tempered to match what RA actually measures."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take. The clean result here is that you can fine-tune an LLM's layers downstream of an injection point to turn SAE decoder directions into short natural-language explanations, and that this generalizes to held-out features from the same dictionary. That result is solid. The second headline claim—transfer across \"separately trained\" SAE dictionaries—does not hold up as stated. The width-65k Gemma Scope 2 SAE at the same layer is almost certainly a Matryoshka sub-dictionary of the width-262k training dictionary, meaning the test directions are likely already present in the training data. The paper never checks decoder-direction overlap, and the Limitations section does not flag this.\n\nWhat is new and good. The method is a direct extension of the activation-verbalization line (Activation Oracles, LatentQA), but the specific combination—norm-matched additive injection of decoder directions, freezing everything before the injection layer, fine-tuning only downstream components, then a linear adapter for cross-LLM transfer—is clean and clearly described. The data filtering pipeline is unusually careful: the authors manually find that only about 11.5% of raw Neuronpedia explanations are both coherent and accurate, and they retain only qualified pairs for training and evaluation. Within-dictionary RA numbers are strong relative to the untrained backbone (1.6% to 52.3% on GTS with the 27B model), robust across injection strengths, and the qualitative cases illustrate real differences between verbalizer output and observational references.\n\nSoft spots. The Table 2 transfer result is the load-bearing weakness. Gemma Scope 2's Matryoshka objective means low-index directions in narrower dictionaries are shared with wider ones; without an overlap analysis, the high RA on the width-65k SAE could simply be memorization of training directions. This is a concrete, separable flaw, not just a quibble about RA. The rest are moderate: all configurations are single runs (acknowledged); RA is agreement with filtered Neuronpedia references that are also the supervision source, so it partly measures reproduction of the reference distribution (also acknowledged); and adapter checkpoints were selected on the LIG test set that later appears in the adapter results—mild leakage, easy to fix.\n\nWho it is for. People working on scalable SAE interpretability tools and on activation verbalization more generally. The paper deserves a serious referee: the method is plausible, well-engineered, and the within-dictionary effect looks real. A referee should require an independent cross-dictionary transfer test (e.g., a non-nested SAE on the same LLM and layer, or at minimum a direction-overlap analysis), seed averaging, and unbiased checkpoint selection. I would engage with it, and I'd send it to review, but the transfer claim needs to be re-supported before publication.","headline":"Within-dictionary verbalization of SAE decoder directions is real and well-engineered; the cross-dictionary transfer headline is likely confounded by Matryoshka nesting and needs an independent test before it can stand.","tokens_in":19169,"tokens_out":2328,"would_cite":true,"duration_ms":23758,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning an LLM on feature–explanation pairs lets it explain sparse autoencoder features directly from decoder directions, generalizing to unseen features, transferring across SAE dictionaries, and extending to other LLMs through a…","keywords":["sparse autoencoders","feature interpretation","representation verbalization","decoder directions","LLM interpretability","cross-LLM transfer","fine-tuning language models","reference agreement"],"falsifier":"Construct a gold set of features with independently verified semantics by having several human annotators label features from both their top-activating examples and targeted behavioral probes, retaining only features where annotators converge. If the verbalizer's Reference Agreement with the filtered references stays high while its agreement with those gold labels is near chance, the central claim would be falsified; if the verbalizer tracks the gold labels even where they disagree with the filtered references, the claim would be strengthened. A complementary strong test is synthetic: add a known concept direction to representations at the injection layer and check whether the verbalizer names that concept at above-chance rates.","tokens_in":18107,"feed_emoji":"🧠","tokens_out":10642,"duration_ms":99781,"temperature":0.7,"pith_summary":"The paper claims that a sparse autoencoder feature can be explained in natural language by a fine-tuned LLM that never sees the feature's activating text. The method injects the feature's decoder direction into the LLM's hidden states during prompt prefilling and fine-tunes only the layers downstream of the injection to complete a fixed sentence naming the concept. If correct, feature explanation becomes a one-pass inference from the internal representation itself, removing the corpus-scale activation search that current bottom-up interpretability pipelines require. The authors demonstrate that the trained verbalizer generalizes to unseen features, transfers to a different SAE dictionary over the same representation space without retraining, and—through a lightweight affine adapter trained on unlabeled aligned representations—explains features from other LLMs. They also show that jointly injecting two directions yields combined meanings and that reversing a direction shifts the generated meaning, indicating sensitivity to compositional and signed structure in decoder directions.","feed_headline":"A fine-tuned LLM explains SAE features from their internal directions","feed_subtitle":"It needs no activation examples or corpus scans, just one forward pass on a fixed prompt per feature.","key_machinery":"The load-bearing object is the decoder direction: the column of the SAE decoder that defines a feature as a direction in the LLM's representation space. The paper uses it as the only feature-specific signal. At inference the direction is normalized to unit norm, scaled by the mean representation norm of the injection span, and added to every token representation in that span at a chosen layer; the prompt is a fixed sentence ending in 'The target concept is:', and the downstream layers generate the explanation. Partial fine-tuning freezes layers at or before the injection layer, preserving the representation space in which decoder directions live, and trains only layers after it plus the final norm and language-model head. For cross-LLM transfer, a single affine layer maps source-layer representations into the injection-layer space; because injection is additive, the adapter's bias cancels and only its weight matrix $W$ maps the direction itself. This whole interface is what makes explanation a one-pass readout from internal geometry rather than a search over activation examples.","core_discovery":"The paper's central claim is that the semantic content of an SAE feature is accessible in its decoder direction alone, and that an LLM can be trained to read that direction aloud. SAEVerbalizer injects a normalized decoder direction into the token representations of a fixed prompt at a chosen layer, then fine-tunes only the layers after that layer to continue the prompt with an explanation. The trained verbalizer produces explanations for features it never saw during training, transfers directly to a different SAE dictionary over the same representation space, and, via an affine adapter trained on aligned hidden states from unlabeled text, explains features from a different LLM. In the main evaluation, the best configuration attains Reference Agreement—the fraction of generated explanations an LLM judge considers to agree with the reference explanation—of 52.3 percent on a global train-standard set, 80.5 percent on low-index gold features, and 56.1 percent on a global gold set. The authors interpret these results as evidence that representation verbalization is a trainable, partially reusable capability rather than a byproduct of studying activation examples.","pith_inferences":["A natural extension the paper does not test: features whose activations are rare or absent in text corpora might still be explained from their decoder directions, because the method never depends on collecting activation examples; this could be checked by comparing verbalizer outputs on low-frequency features against human labels.","The same trained interface could be pushed beyond naming a concept, for example to produce test sentences that activate a feature or to answer questions about its behavior; the paper's fixed 'name the concept' prompt is only one instantiation of representation verbalization.","If adapters transfer direction semantics across LLMs, then decoder-direction geometry itself may be largely shared across models at matched depths; a stronger test would be whether adapter-mapped directions compose consistently, for instance whether $\\text{adapter}(A+B) \\approx \\text{adapter}(A)+\\text{adapter}(B)$ across many features.","An important caveat that follows from the paper's own evaluation choices: Reference Agreement measures consistency with filtered references, not correctness, so a human-gold validation study would be needed before treating verbalizer outputs as verified descriptions of model behavior."],"forward_implications":["A single trained verbalizer can explain any feature of any SAE defined on the same LLM layer, since decoder directions from different dictionaries share that representation space.","Explanation cost drops to one forward pass on a short fixed prompt per feature, eliminating corpus-scale inference and example retrieval.","Cross-LLM explanation can be obtained from as little as one affine layer trained on aligned unlabeled text, so the method can be extended to new LLMs without feature-level supervision.","Adding supervision beyond a small amount (1.5k pairs already produce large gains) is what creates the capability; raw backbones show almost no spontaneous explanation ability, with Reference Agreement around 1–2 percent.","The verbalizer's sensitivity to joint injection and sign reversal offers a direct way to probe compositional and signed structure in decoder directions."],"supporting_citations":[{"why":"It defines sparse autoencoders and the activation-example interpretation setup that frames the paper's problem.","marker":"Bricken et al. (2023)"},{"why":"It supplies the Gemma Scope 2 SAEs whose decoder directions and representation spaces are used in all verbalizer and adapter experiments.","marker":"McDougall et al. (2025)"},{"why":"It provides the online feature-explanation database and cached activation examples from which training pairs and evaluation references are drawn.","marker":"Lin (2023)"},{"why":"It represents the scaled bottom-up interpretation pipeline whose corpus-scale cost SAEVerbalizer claims to avoid.","marker":"Paulo et al. (2025)"},{"why":"It shows that LLM computation can decode hidden representations into natural language, motivating direct representation verbalization.","marker":"Ghandeharioun et al. (2024)"},{"why":"It demonstrates that explicit training can teach LLMs to decode activations into natural language, the basis for the fine-tuning objective.","marker":"Pan et al. (2026)"},{"why":"It supports the matched-depth layer correspondence used to choose source and target layers for the cross-LLM adapters.","marker":"Wolfram and Schein (2025)"},{"why":"It supplies the judge model used to filter feature–explanation pairs and to compute Reference Agreement, the paper's main evaluation metric.","marker":"Qwen Team (2025)"}],"fun_headline_variants":["LLM trained to read SAE decoder directions explains unseen features","One forward pass explains SAE features from decoder directions","SAEVerbalizer: LLM explains features from internal directions alone","Decoder directions suffice to explain SAE features after fine-tuning","Trained verbalizer transfers SAE explanations across dictionaries"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the filtered reference explanations used as training targets and evaluation references correctly capture what each SAE feature means; if those references are wrong or too broad, high Reference Agreement does not show the verbalizer produces correct explanations.","fun_headline_variants_meta":{"raw":{"variants":["LLM trained to read SAE decoder directions explains unseen features","One forward pass explains SAE features from decoder directions","SAEVerbalizer: LLM explains features from internal directions alone","Decoder directions suffice to explain SAE features after fine-tuning","Trained verbalizer transfers SAE explanations across dictionaries"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000301,"raw_usage":{"total_tokens":1727,"prompt_tokens":928,"completion_tokens":799,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":716}},"tokens_in":544,"tokens_out":799,"duration_ms":8217,"temperature":1.0,"reasoning_tokens":716,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:35:57.396934+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a gold set of features with independently verified semantics by having several human annotators label features from both their top-activating examples and targeted behavioral probes, retaining only features where annotators converge. If the verbalizer's Reference Agreement with the filtered references stays high while its agreement with those gold labels is near chance, the central claim would be falsified; if the verbalizer tracks the gold labels even where they disagree with the filtered references, the claim would be strengthened. A complementary strong test is synthetic: add a known concept direction to representations at the injection layer and check whether the verbalizer names that concept at above-chance rates.","supporting_citations":[{"cited_title":"2025 , type =","cited_arxiv_id":null,"evidence_quote":"It supplies the Gemma Scope 2 SAEs whose decoder directions and representation spaces are used in all verbalizer and adapter experiments."},{"cited_title":"2023 , url =","cited_arxiv_id":null,"evidence_quote":"It provides the online feature-explanation database and cached activation examples from which training pairs and evaluation references are drawn."},{"cited_title":"Proceedings of the 42nd International Conference on Machine Learning , pages =","cited_arxiv_id":null,"evidence_quote":"It represents the scaled bottom-up interpretation pipeline whose corpus-scale cost SAEVerbalizer claims to avoid."},{"cited_title":"2024 , editor =","cited_arxiv_id":null,"evidence_quote":"It shows that LLM computation can decode hidden representations into natural language, motivating direct representation verbalization."},{"cited_title":"2025 , url=","cited_arxiv_id":null,"evidence_quote":"It supports the matched-depth layer correspondence used to choose source and target layers for the cross-LLM adapters."}],"review_version":1}