{"id":"64971747-799f-446a-bfe1-88f346aef088","arxiv_id":"2512.10092","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Sparse autoencoder activations can be used as interpretable, controllable text embeddings for data diffing, correlation mining, clustering, and property-based retrieval.","lead":"This paper shows that sparse autoencoders trained on LLM internals can turn documents into interpretable feature vectors, and uses them for four data-analysis tasks. The result is a cheaper, controllable alternative to LLM labeling and dense embeddings for auditing model behavior and training data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Feature absorption and label inaccuracy on the reused Goodfire latents are unquantified for the latents actually used in the headline tasks; the paper's own F1 ablation samples frequent latents and uses LLM judges, so it does not resolve the risk that the reported diffing/correlation/retrieval resul","rationale":"The reader's weakest assumption identifies the correct load-bearing point: the entire toolkit rests on the reused latent descriptions being accurate and stable on the corpora analyzed. The paper itself concedes feature absorption as a limitation but never measures its impact on the specific latents that drive the headline results. The only quantitative attempt (Appendix I) uses a non-representative sample (frequent latents, >10% activation) and an LLM judge as ground truth, so it cannot settle the question. I agree with the reader that this is the most important gap. The proposed concrete test—human annotation of the actual top-diff and top-correlation latents, followed by recomputation of the headline metrics using human labels—would directly resolve whether the reported frequency differences and NPMI values are trustworthy. If the test passes, the central claim is substantially supported; if it fails, the abstract's reliability claim would need to be softened or the method restricted to settings where label quality is verified. The reader's CONDITIONAL verdict remains appropriate pending this evidence, so I recommend no change to the verdict.","tokens_in":50349,"tokens_out":8333,"duration_ms":88581,"concrete_test":"Select the top 20 latents by frequency difference from the Grok-4 diff (Table 7) and the top 20 latent pairs from the correlation analysis (Figure 4). For each, sample 200 documents from the relevant datasets (arena-human-preference-55k responses, CivilComments, Pile). Have three human annotators independently label whether each document exhibits the concept described by the latent label (yes/no). Compare majority human labels to SAE binary activation (max activation > 0). Compute precision and recall per latent. Then recompute the headline frequency differences and NPMI values using human labels instead of SAE activations. If average recall or precision < 0.8 on the selected latents, or if the ranking of top differences/correlations changes materially, the paper's reliability claim needs substantial qualification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that SAE embeddings are more reliable than LLMs and more controllable than dense embeddings—depends on the 61,521 reused Goodfire latent descriptions (from LMSYS-Chat-1M activations of Llama 3.3 70B layer 50) being accurate, monosemantic, and stable on out-of-distribution corpora. The paper acknowledges in Limitations that SAEs are 'imperfect labelers due to feature absorption' but does not quantify how absorption or other label errors affect any of the four reported tasks. The only direct evidence for label quality is Appendix I, which measures F1 of SAE latents as classifiers on Pile/arXiv/GPT-5 responses, but (i) it samples only 100 latents active in >10% of documents—not the top-frequency-difference latents, high-NPMI pairs, or retrieval latents used in the headline experiments; (ii) the ground truth is an LLM judge, so the evaluation is partly circular with the label-generation process; and (iii) median F1 appears modest (roughly 0.5–0.7), leaving room for systematic false positives/negatives. If absorption causes a latent to miss a subset of its concept (low recall) or to fire on an unrelated concept (low precision), then frequency differences (Section 4.1), NPMI correlations (Section 4.2), cluster filtering (Section 4.3), and property retrieval (Section 4.4) all inherit these errors. The 'more reliable than LLMs' comparison is therefore only as strong as the unverified label quality of the specific latents used.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes using pretrained sparse autoencoders (SAEs) on an LLM's hidden states to construct 'SAE embeddings': per-document vectors whose dimensions are labeled, human-interpretable concepts. It applies these embeddings to four data-analysis tasks — dataset diffing, correlation mining, clustering, and property-based retrieval — and reports that SAE embeddings are more cost-effective and reliable than LLM-based annotation and more controllable than dense embeddings. Two case studies apply the method to OpenAI model generations and to finding a spurious 'I hope it is correct' trigger in Tulu-3's post-training data. The core method is clearly specified, code is released, and each task includes at least one validation component; the Tulu-3 case study generates new prompts to verify the discovered trigger.","tokens_in":50763,"tokens_out":3963,"duration_ms":46220,"significance":"If the central claims hold, this paper would make a useful contribution: it shows that a single reader model's SAE can serve as a reusable, interpretable labeler across many text-analysis tasks, and it provides a concrete toolkit with code. Strengths include the explicit task-specific validations, the synthetic injection experiments for correlations, the cost comparison with token counts, and the Tulu-3 trigger study, which is genuinely testable and tested with newly generated prompts. The paper is also honest in its Limitations section about feature absorption and sensitivity to the SAE's training distribution. However, the strength of the headline claim — 'more reliable than LLMs' — depends on the quality of the 61,521 reused latent descriptions, and that dependency is not quantified for the specific latents used in the headline experiments. The evidence for label quality in Appendix I is partial, samples an unrepresentative subset of latents, and uses an LLM judge as ground truth. These issues are fixable but currently leave a gap between the evidence and the abstract's reliability claim.","major_comments":[{"comment":"The central reliability claim depends on the 61,521 reused Goodfire latent descriptions. The paper acknowledges in Limitations that SAEs are 'imperfect labelers due to feature absorption' but does not quantify absorption or label error for the latents that actually drive the reported results. Appendix I evaluates latent-as-classifier F1 on Pile/arXiv/GPT-5, but it samples only 100 latents active in >10% of documents, which is not the same distribution as the top-frequency-difference latents (§4.1), high-NPMI pairs (§4.2), or retrieval latents (§4.4). Additionally, the F1 ground truth is an LLM judge, which is partly circular with the LLM-based label-generation process. I would like to see a targeted evaluation: sample latents from the top of each task-specific ranking, measure precision/recall against human annotations, and report how many hypotheses would be invalidated by absorption or","section":"§3, Limitations, and Appendix I"},{"comment":"The abstract's 'more reliable than LLMs' is not uniformly supported by the paper's own ground-truth diffing comparison. In Appendix D.3, the SAE achieves average surface similarity 0.75 (movies) and 0.80 (tones), while the GPT-5 baseline achieves 0.90 (movies) and 0.78 (tones). Thus on one of the two grounded settings the LLM baseline is stronger. The later verification-rate results (Figure 11) support a more specific claim: SAE-generated hypotheses have higher judge-verified frequency differences in multi-model settings. The paper should qualify the 'more reliable' claim to this setting, or provide additional evidence that the SAE is more reliable across the four tasks as a whole.","section":"§4.1 and Appendix D.3"},{"comment":"The 'verified NPMI' evaluation in Figure 4 is partly self-referential. Latent labels are produced by an LLM, and the judge that verifies their presence is also an LLM. The CDF therefore measures consistency between the latent label and the LLM judge, not necessarily ground-truth semantic correlation. The synthetic injection experiments provide a genuinely external anchor and are a strength, but the real-world correlation signal (CivilComments, Pile) would be more convincing with a human-annotated subset or an independent, non-LLM validation. Without this, the claim that SAEs 'identify biases more reliably' is only as strong as the unverified label quality.","section":"§4.2, Figure 4"},{"comment":"The retrieval results show substantial query-level variance that is not discussed in the main text. For example, on ChatbotArena prompts the query about fictional story generation drops from 0.719 (OpenAI+LLM) to 0.110 (SAE), while 'user includes emojis' improves from 0.066 to 0.480. These swings suggest that the SAE's advantage is concentrated in certain property types (formatting, repetition, multilingual patterns) and not in others (narrative, philosophical content). Reporting aggregate MAP alone may overstate the 'generally outperform or match' conclusion. I would like an error analysis or at least a sentence acknowledging that the SAE is better for some property families and worse for others.","section":"§4.4, Tables 23–28"}],"minor_comments":[{"comment":"'we find 61521 existing latent descriptions that we reuse' is ungrammatical; suggest 'we reuse 61,521 existing latent descriptions'.","section":"§3"},{"comment":"The cost table reports token usage including latent relabeling, but the SAE's fixed cost of obtaining the pretrained SAE and labels is excluded. This is acceptable if the SAE is reused across many tasks, but the paper should state this amortization assumption explicitly in the main text, not only in the caption.","section":"§4.1, Table 3"},{"comment":"The cluster labels in Figure 5 are useful, but the scores in the 'Acc.' column are not defined until later in the text. Please define 'Acc.' and the z-score metric when the figure is first referenced, so that a reader does not have to jump to §4.3 or Appendix F.","section":"§4.3, Figure 5"},{"comment":"The appendix reports that SAEs 'were unable to find a good combination of queries and k' for sentiment and emotion clustering. This is an honest negative result, but it is not mentioned in the main-text limitations. A one-sentence acknowledgment in §4.3 would help readers calibrate the controllability claim.","section":"Appendix F.4"}],"recommendation":"major_revision","confidential_remarks":"The Tulu-3 case study is the strongest part of the paper and demonstrates the method's potential. The main risk to the published version is the gap between the abstract's reliability claim and the unquantified quality of the reused latent descriptions. I would encourage the editor to ask for a focused human-annotation study on the latents used in the headline tasks rather than a broad expansion of benchmarks."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper is worth engaging seriously. The core contribution is a reusable toolkit—max-pooled SAE activations from a reader LLM (Goodfire SAE on Llama 3.3 70B) treated as interpretable embeddings—applied to dataset diffing, correlation mining, clustering, and property-based retrieval. The combination is new, and the empirical findings are real: the Grok-4 clarification behavior, the Tulu-3 “I hope it is correct” trigger, and the OpenAI trends all look like actual discoveries, not artifacts.\n\nWhat's done well: the method is clearly specified, code is linked, and each task has a validation component. The Tulu-3 case study stands out—they found the correlation in the training data, generated new prompts along hypothesis axes, and showed the model genuinely learned the behavior. That's a proper testable prediction. The retrieval benchmark includes strong baselines (OpenAI, Gemini, Qwen, BM25+LLM) and the qualitative examples showing the SAE captures implicit properties (repetitive loops) are convincing.\n\nSoft spots: the label quality of the 61,521 reused Goodfire latents is the load-bearing assumption, and it's not directly validated for the latents used in the headline experiments. The only quality check (Appendix I) samples 100 frequent latents, uses an LLM judge as ground truth, and reports modest median F1s. Feature absorption is acknowledged but not quantified. So the frequency differences, NPMI correlations, cluster filters, and retrieval scores all inherit whatever label error exists. That's a real limitation, but not fatal—the ground-truth evaluations (IMDB, GSM8k, synthetic correlations) provide external anchors, and the Tulu result was verified with fresh prompts. The abstract's “more reliable than LLMs” is a bit strong given that in two-model diffing the SAE isn't clearly better on verification rate. The reliance on LLM judges throughout, without human agreement, is a weakness but a standard one in this area.\n\nWho it's for: researchers doing data-centric interpretability, dataset auditing, or looking for practical uses of SAEs. The paper is honest about its limitations and doesn't oversell the mechanism. I'd cite it and would want a serious referee to engage with the label-quality question.\n\nRecommendation: send to peer review. The main fixes are addressable: calibrate the reliability claim, add a dedicated latent-label-quality experiment that includes the latents used in the main analyses, and report at least a small human-validated sample to break the LLM-judge circularity.","headline":"A well-executed empirical toolkit paper demonstrating SAE embeddings' value across four data-analysis tasks, with a genuinely testable Tulu-3 case study; the main weakness is unquantified latent-label quality, an addressable limitation rather than a fatal flaw.","tokens_in":51275,"tokens_out":2478,"would_cite":true,"duration_ms":26978,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Sparse autoencoder embeddings, whose dimensions are named concepts, make text-corpus analysis cheaper, more reliable, and more controllable than LLM labeling or dense embeddings.","keywords":["sparse autoencoders","interpretable embeddings","dataset diffing","concept correlation mining","property-based retrieval","controllable clustering","data-centric interpretability","LLM behavior auditing"],"falsifier":"Take a held-out corpus with known concept labels (e.g., the synthetic tone and genre datasets plus an out-of-distribution set) and measure precision/recall of each binarized SAE dimension against the label; if a substantial fraction of high-frequency dimensions have near-chance precision, the label layer fails. Alternatively, rerun all four tasks with a second independently labeled SAE on the same reader model and check whether the top dataset differences, correlations, clusters, and retrieval rankings replicate; non-replication would show the results are tied to one label set rather than to t","tokens_in":50227,"feed_emoji":"📊","tokens_out":8280,"duration_ms":79007,"temperature":0.7,"pith_summary":"The paper tries to establish that sparse autoencoder (SAE) embeddings—document vectors built by max-pooling a reader LLM's SAE activations over tokens—are a practical tool for analyzing text corpora, because each dimension is a human-interpretable concept. With these embeddings, a single pass labels every document with thousands of named properties, and the paper argues this makes dataset diffing and correlation mining cheaper and more reliable than LLM-based annotation, while making clustering and retrieval more controllable than dense embeddings. Across four tasks it reports larger judge-verified differences at 2–8x lower token cost, recovery of injected correlations that an LLM baseline missed, clusters along user-selected axes such as reasoning style, and retrieval that matches or beats dense embedding baselines on implicit properties. The payoff is a data-centric way to interpret models: the same tool uncovers distinctive behaviors of a specific chat model, tracks how another model family's outputs shift across generations, and finds a spurious prompt–response pattern learned by a fine-tuned model.","feed_headline":"Concept-labeled embeddings find data differences at 2-8x lower cost","feed_subtitle":"Each dimension is a named concept, so diffing, clustering, and retrieval can be steered by meaning.","key_machinery":"SAE embeddings: a document is embedded as the max over token-level sparse autoencoder activations from a reader LLM, so each of roughly 65,000 dimensions is a fixed, human-readable concept (a 'latent' with an auto-generated label, produced by prompting an LLM with ten activating and ten non-activating phrases). These dimensions supply a large, fixed hypothesis space of concept detectors. Frequencies of binarized dimensions enable dataset diffing; pairwise normalized pointwise mutual information between dimensions, filtered by low label similarity, surfaces surprising correlations; restricting the embedding to query-relevant dimensions gives controllable clustering and retrieval.","core_discovery":"At the center is a simple construction: feed each document to a reader LLM, extract per-token internal activations, apply a pretrained sparse autoencoder whose 65,536 dimensions have been labeled with concept descriptions, and max-pool activations across tokens. The resulting vector is an embedding whose axes are named properties (e.g., 'asks a clarifying question,' 'personifies objects'), and binarizing it turns the SAE into a mass labeler for thousands of concepts. The paper claims this construction outperforms LLM-based baselines on dataset diffing and correlation discovery—finding larger, judge-verified differences at 2–8x lower token cost and recovering injected correlations that an LLM","pith_inferences":["If the label layer is trustworthy, the reader-model SAE becomes a shared measurement instrument: one could audit any model's outputs without access to that model's internals, which would make cross-company behavioral comparisons much cheaper.","The controllability result suggests a natural extension: composing multiple property queries into a single retrieval or clustering objective could support open-ended 'find me texts that are X but not Y' analyses, which dense embeddings cannot express.","Since performance drops on corpora far from the SAE's training distribution, a testable extension is to train or select domain-specific reader SAEs and pool their latents; the paper hints at this but does not evaluate it.","The reliability of every downstream number inherits from label accuracy, so an automated calibration pass—relabeling or pruning low-precision latents per corpus—would likely sharpen the paper's results; the authors acknowledge feature absorption but do not quantify its effect."],"forward_implications":["A single pretrained SAE plus reader LLM can replace repeated LLM prompting for dataset comparison: once the embedding is built, datasets can be diffed and re-diffed at near-zero marginal cost.","Differences and correlations found this way come with human-readable names, so a practitioner can go from a statistical signal to a testable hypothesis without a separate interpretation step.","Filtering the embedding to concept-relevant dimensions changes what clustering and retrieval optimize for: instead of topic similarity, a user can cluster by reasoning style, tone, or formatting.","The same tool can audit training data: identifying prompt features correlated with unwanted response phrases, then generating new prompts to confirm the model has learned the correlation.","Because the embeddings are reused across comparisons, adding more models or datasets to an analysis is cheaper than rerunning LLM baselines.","Cost-effectiveness is most pronounced in multi-model settings, where the SAE embedding is computed once and then compared across many targets."],"fun_headline_variants":["Concept-tagged embeddings spot data gaps at 2-8x lower cost","SAE embeddings give every axis a name, cheaper than LLMs","Interpretable axes expose dataset biases at a fraction of cost","Turn model internals into labeled concepts for sharper data mining","Sparse autoencoders make embeddings explainable and cost-efficient"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The reused auto-generated labels for the 61,521 SAE dimensions are accurate, monosemantic, and stable on out-of-distribution corpora; if many labels fire on the wrong concepts or absorb multiple meanings, the reliability and controllability claims weaken—the paper's own Limitations section flags feature absorption without quantifying its effect.","fun_headline_variants_meta":{"raw":{"variants":["Concept-tagged embeddings spot data gaps at 2-8x lower cost","SAE embeddings give every axis a name, cheaper than LLMs","Interpretable axes expose dataset biases at a fraction of cost","Turn model internals into labeled concepts for sharper data mining","Sparse autoencoders make embeddings explainable and cost-efficient"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000164,"raw_usage":{"total_tokens":1120,"prompt_tokens":820,"completion_tokens":300,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":564,"completion_tokens_details":{"reasoning_tokens":212}},"tokens_in":564,"tokens_out":300,"duration_ms":4005,"temperature":1.0,"reasoning_tokens":212,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T17:14:23.827581+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out corpus with known concept labels (e.g., the synthetic tone and genre datasets plus an out-of-distribution set) and measure precision/recall of each binarized SAE dimension against the label; if a substantial fraction of high-frequency dimensions have near-chance precision, the label layer fails. Alternatively, rerun all four tasks with a second independently labeled SAE on the same reader model and check whether the top dataset differences, correlations, clusters, and retrieval rankings replicate; non-replication would show the results are tied to one label set rather than to t","supporting_citations":[],"review_version":1}