{"id":"c1b6ce19-e956-4cfd-839a-b8b40d534a3b","arxiv_id":"2509.02565","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"When loss improvement from tiling a common feature manifold decays slower than feature frequency decays, sparse autoencoders allocate most latents to that manifold and discover sublinearly many features.","lead":"Sparse autoencoders that decode neural activity into simple pieces can waste a large share of their capacity on common high-dimensional features, a regime this paper shows through a scaling model adapted from prior neural-scaling work. The paper identifies when this waste happens and gives preliminary evidence that real models may avoid it.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The allocation model's central decomposition (Eq. 1) depends on mutually orthogonal, feature-specific latents; the paper concedes these fail in practice, so the pathological-regime prediction is not yet connected to real SAEs.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing concern: Section 2.2's reduction to a latent allocation problem depends on feature-specific latents, subspace-respecting decoders, and mutual orthogonality, all of which the paper concedes are violated in practice. The mathematical derivation from those assumptions is sound; the two-regime result follows from the stated power-law premises, and the toy experiments are informative about isolated manifold geometry. The gap is in transferring the model to real SAEs, not in the internal logic. A concrete synthetic test with non-orthogonal features would settle whether the decomposition holds approximately, and if it fails, the central claim about real SAE scaling would need to be weakened to a purely theoretical statement. Since the paper already presents itself as preliminary framing and the reader's conditional verdict reflects exactly this uncertainty, no change to the verdict is needed.","tokens_in":10830,"tokens_out":8104,"duration_ms":78496,"concrete_test":"Train ReLU and JumpReLU SAEs on synthetic data with two ground-truth features whose subspaces have angle θ, frequencies p1 > p2, and known per-feature loss curves L_i(n_i) measured from SAEs trained on each feature in isolation. Sweep θ from 0° to 90° and latent budgets N; compare the actual total loss and D(N)/N to the prediction obtained by minimizing Σ p_i L_i(n_i) with Σ n_i = N. If Eq. (1) is violated for θ > 0 — for example, if the joint solution mixes latents across subspaces, absorbs features, or produces cross-feature encoder interference — then the allocation model does not transfer to non-orthogonal features. Also count how many latents fire for both ground-truth features to quantify absorption.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central result (Section 2.5 and Appendix B) minimizes Σ_i p_i L_i(n_i) subject to Σ_i n_i = N. This objective follows from the additivity assumption L(x) = Σ_{i∈A(x)} L(S_i f_i), which requires three conditions: each latent is specific to one feature, decoder columns for a feature lie in that feature's subspace, and feature subspaces are mutually orthogonal. The authors explicitly state in Section 2.2 that 'feature absorption and related phenomena and the non-orthogonality of features in practice violate these assumptions.' If real SAE solutions mix features across subspaces or absorb features, then the per-feature loss curves L_i(n_i) measured in Figure 3 on isolated manifolds are not the actual marginal cost curves inside a joint SAE, and the two-regime prediction D(N) ∝ N^{(1+β)/(1+α)} is not established for real networks. This is not an internal inconsistency, but it is the load-bearing transfer step between an idealized allocation model and actual SAE scaling. The paper honestly flags this and frames the work as preliminary, so a conditional verdict is appropriate, but the key premise deserves a direct empirical check.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper adapts Brill's (2024) capacity-allocation model to sparse autoencoders. It assumes activations are sums of sparse features living in orthogonal subspaces, and that an SAE assigns latents to features so that total loss is Σ_i p_i L_i(n_i) with Σ_i n_i = N. Assuming feature frequencies p_i ∝ i^{-(1+α)} and identical per-feature loss curves L_i(n) = n^{-β}, the authors derive two scaling regimes: when β < α, L(N) ∝ N^{-β} and the number of discovered features grows only as D(N) ∝ N^{(1+β)/(1+α)}, so D(N)/N → 0; when α < β, D(N) ∝ N and L(N) ∝ N^{-α}. They support the per-feature loss curves with SAE experiments on hyperspheres and spherical shells, run a numerical simulation of the allocation model, and present preliminary measurements of latent firing frequencies and decoder-neighbor cosine similarities on Gemma Scope and Inception-v1 SAEs. The paper concludes that whether real SAEs are in the pathological regime is unresolved.","tokens_in":11051,"tokens_out":9234,"duration_ms":79073,"significance":"If the model's assumptions hold, the derivation in Appendix B is clean and yields a crisp, falsifiable scaling prediction that connects SAE scaling to the geometry of feature manifolds. The paper is explicit about the idealized nature of its assumptions and does not overclaim empirical confirmation. The toy experiments directly show that hollow hyperspheres permit gradual loss reduction while radial-variation manifolds saturate, which is a useful empirical fact. The framework provides a concrete vocabulary (α, β, D(N)) for future work on SAE scaling. Its main value is as a formal model and a source of hypotheses; its main weakness is the unvalidated transfer from the allocation model to real SAE training.","major_comments":[{"comment":"The allocation objective in Eq. (1) is derived from three premises: feature-specific latents, decoder columns lying in each feature's subspace, and mutually orthogonal feature subspaces. The paper explicitly concedes that feature absorption [27] and non-orthogonality violate these assumptions in practice. This matters because the per-feature loss curves L_i(n_i) used in the allocation model are measured on isolated manifolds (Figure 3); if in a joint SAE latents mix features or absorb them, these curves are not the actual marginal cost curves inside the SAE. I recommend adding a direct empirical check on synthetic data: train a single SAE on data containing one manifold feature plus several discrete features, then measure (i) the number of latents whose decoder direction lies in each feature subspace, (ii) whether latents fire only when their feature is active, and (iii) whether the per-feature loss as a function of allocated latents matches the isolated curves. Without such a check, the paper's prediction of a pathological regime, while correct for the idealized model, is not yet connected to actual SAE training.","section":"§2.2, Eq. (1)"},{"comment":"The asymptotic results D(N) ∝ N^{(1+β)/(1+α)} (β < α) and D(N) ∝ N (α < β) assume that every feature has the same per-feature loss L_i(n_i) = n_i^{-β} for all n_i, with no saturation. The paper's own experiments in Figure 3 and Appendix A.1 show that L(n) curves can plateau sharply (spherical shells at n ≈ 2d_i), and even the hypersphere curves are only measured over 10^2 to 10^4 latents, with β ≈ 0.05 for dimensions 6–8. If a real manifold's L(n) eventually saturates, the infinite-N limit in Section 2.5 no longer follows: once the manifold is fully tiled, additional latents would go to discrete features and D(N)/N would recover. I ask for an analysis of the allocation problem with saturating per-feature loss curves (e.g., L(n) = n^{-β} + c, or a sharp cutoff at some n_max), to characterize when the pathological regime is realized and how large N must be before it sets in. The current numerical simulation in Figure 2 (right) uses a non-saturating power-law manifold plus step-function features, so it does not address this.","section":"§2.5 and Appendix B.3/B.4"}],"minor_comments":[{"comment":"The caption says a 'simulated SAE' discovers only 3 million features, but the simulation optimizes the allocation objective Σ_i p_i L_i(n_i) rather than training an SAE; please rename it a 'simulation of the allocation model' to avoid implying end-to-end SAE training.","section":"Figure 2 (right) caption"},{"comment":"The paragraph estimating α from Gemma Scope latent firing frequencies notes the distortion caused by feature absorption and compositional features, but the figure may still be read as evidence about α; add an explicit statement that this estimate cannot be used to test the model because the mapping from features to latents is exactly what the model is trying to characterize.","section":"Appendix A.1, Figure 4"},{"comment":"The definition 'L(n) = 1 if n_i < 1 else λ' uses n_i inconsistently with the function argument n; use a single symbol (e.g., L_i(n_i)) throughout the section.","section":"§2.3"}],"recommendation":"major_revision","confidential_remarks":"This is a promising framework paper, but the gap between the idealized allocation model and real SAEs is the central issue. The authors are honest about the limitations, and my recommendation of major_revision is driven by the need for a direct empirical test of the decomposition, not by any flaw in the mathematics."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a framing paper, and a good one. The new thing here is importing Brill's capacity-allocation model into SAE scaling and showing that when per-feature loss decays slower than feature frequency (beta < alpha), the number of discovered features grows sublinearly with latent count. The math in Appendix B is correct and readable; the two regimes follow cleanly from the stated power-law assumptions. The toy experiments are also genuinely useful: hollow hyperspheres let SAEs shave loss incrementally by tiling, while radial-variation shells saturate quickly around 2d latents. That geometric contrast is the most concrete result in the paper.\n\nThe authors also deserve credit for honesty. They explicitly say they do not resolve whether real SAEs are in the pathological regime, they flag that the central assumptions (feature-specific latents, orthogonal subspaces) are violated in practice, and they offer a plausible argument that radial-variation geometry might save real models. That is the right scientific posture.\n\nThe soft spots are in proportion. The load-bearing transfer step is not yet connected to real SAEs. The loss decomposition L(x) = sum_i L(S_i f_i) requires orthogonal, feature-specific latents, and the paper concedes that feature absorption and non-orthogonality break this. The isolated L_i(n_i) curves measured on toy manifolds are not demonstrated to be the actual marginal cost curves inside a jointly trained SAE. So the D(N)/N -> 0 prediction is a warning, not a demonstrated phenomenon. The empirical sections are also thin: no error bars, no code release, and the cosine-similarity exploration on real SAEs is merely suggestive. The novelty is primarily in the application, not the underlying derivation, which the paper acknowledges by following Brill closely.\n\nThis is a paper for interpretability researchers, especially those working on SAE scaling or the 'dark matter' question. It gives them a clean model and a vocabulary for thinking about manifold tiling. It deserves a serious referee. The main thing I would ask a referee to require is a direct empirical check of the allocation assumptions—for example, testing whether isolated per-feature loss curves predict behavior when multiple features are present in a single SAE—plus code and error bars for the toy experiments.\n\nI would take it to a reading group and would cite it as a useful conceptual reference, but I would not treat the pathological regime as established for real networks.","headline":"A clean, honest adaptation of Brill's allocation model to SAE scaling that identifies a plausible pathological regime but stops short of showing it occurs in real SAEs; worth refereeing, not worth over-trusting.","tokens_in":11570,"tokens_out":2215,"would_cite":true,"duration_ms":21734,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Sparse autoencoder scaling under feature manifolds reduces to a latent-allocation problem, and one exponent comparison decides whether SAEs discover new features or spend latents tiling common manifolds.","keywords":["sparse autoencoders","scaling laws","feature manifolds","latent allocation","feature discovery","interpretability","neural scaling","dictionary learning"],"falsifier":"Train an SAE on a synthetic mixture containing one common hypersphere manifold plus many rare discrete features, with parameters chosen so $\\beta < \\alpha$, and measure the discovered-feature count $D(N)$ as width grows by orders of magnitude: linear growth in $N$ would falsify the sublinear-discovery prediction. On real data, directly measuring $L_i(n_i)$ for identified features at several widths would show whether any manifold sustains the slow decay needed for the pathological regime.","tokens_in":10621,"feed_emoji":"🧩","tokens_out":13026,"duration_ms":110703,"temperature":0.7,"pith_summary":"This paper aims to explain SAE scaling laws as the outcome of allocating a fixed number of latents among features, and to show that feature manifolds (multi-dimensional features) can make that allocation pathological. Under the model, feature frequencies follow a power law $p_i \\propto i^{-(1+\\alpha)}$, each feature's loss with $n_i$ latents follows $L_i(n_i) \\propto n_i^{-\\beta}$, and the SAE's total loss is $\\sum_i p_i L_i(n_i)$. Solving the allocation problem yields a sharp regime switch: when $\\beta < \\alpha$, the optimal SAE keeps shaving loss by tiling common feature manifolds, and the number of features it discovers grows sublinearly with the number of latents. The authors show that such shallow per-feature loss curves are plausible for hollow hyperspheres but plateau when features have radial intensity variation, and they leave open whether real SAEs are in the pathological regime. This matters because SAEs are a main tool for interpreting neural networks: if many latents are silently spent re-tiling common manifolds, rare features remain undiscovered.","feed_headline":"Feature manifolds can make sparse autoencoders miss rare features","feed_subtitle":"A capacity-allocation model shows when SAE loss improves while discovered features grow sublinearly.","key_machinery":"The carrying object is the latent-allocation reduction: under assumptions of feature-specific latents and orthogonal feature subspaces, SAE training is equivalent to choosing $n_i$ latents per feature to minimize $\\sum_i p_i L_i(n_i)$ subject to $\\sum_i n_i = N$. Solving the continuous relaxation gives $n_i \\propto i^{-(1+\\alpha)/(1+\\beta)}$ for power-law frequencies and loss curves. The decisive identity is the comparison of $\\alpha$ and $\\beta$: when $\\beta < \\alpha$, the cutoff feature index grows like $N^{(1+\\beta)/(1+\\alpha)}$, so the discovery count $D(N)$ grows sublinearly; when $\\alpha < \\beta$, $D(N)$ grows linearly with $N$. This machinery converts a question about neural interpretability into a measurable relationship between two power-law exponents.","core_discovery":"The paper's central claim is that, under explicit idealizing assumptions, SAE training is exactly a latent-allocation problem, and feature manifolds introduce a pathology the discrete-feature picture misses. When feature frequencies obey $p_i \\propto i^{-(1+\\alpha)}$ and per-feature loss obeys $L_i(n_i) \\propto n_i^{-\\beta}$, the optimal allocation solves a Lagrange-multiplier problem with $n_i \\propto i^{-(1+\\alpha)/(1+\\beta)}$. Total loss then scales as $L(N) \\propto N^{-\\min(\\alpha,\\beta)}$, but discovered features behave differently in the two regimes: $D(N) \\propto N$ when $\\alpha < \\beta$, while $D(N) \\propto N^{(1+\\beta)/(1+\\alpha)}$ when $\\beta < \\alpha$. In the latter case $D(N)/N \\to 0$, so an SAE keeps improving reconstruction loss while its latent budget is overwhelmingly spent on common feature manifolds rather than on rarer features. Toy-manifold experiments show that the shape of $L_i(n_i)$ depends on geometry, and the real-network question is left open.","pith_inferences":["A practical diagnostic follows directly: fit per-feature loss curves at small widths for frequent features and measure the feature-frequency exponent; if the per-feature curve is shallower, scaling up will mostly re-tile manifolds.","The same allocation logic should apply to crosscoders and transcoders, whose features span shared layer spaces; manifold tiling could be more or less severe there.","If real-world features carry radial intensity variation, the benign regime holds; the open risk is ripples, where intrinsically low-dimensional manifolds wander through many dimensions, a case the toy experiments do not cover."],"forward_implications":["A clean power-law SAE loss curve does not reveal which regime the SAE is in; because total loss falls as $N^{-\\min(\\alpha,\\beta)}$ in both cases, scaling behavior alone cannot detect pathological tiling.","Even very large SAEs can have most latents allocated to a few common feature manifolds while rare features go undiscovered, decoupling reconstruction quality from interpretability coverage.","Per-feature loss curves are geometry-dependent: hollow hyperspheres can absorb thousands of latents, while radial intensity variation makes loss plateau near $2d_i$ latents and would suppress the pathology.","Latent-firing-frequency slopes measured on real SAEs give tentative $\\alpha$ estimates around 0.5–0.7, but feature absorption and tiling distort the mapping from latents to true features."],"supporting_citations":[{"why":"Adapts the capacity-allocation model with continuous relaxation that the paper re-derives for SAE latent budgets.","marker":"[25]"},{"why":"Provides prior empirical SAE scaling laws that the model is built to reproduce.","marker":"[4]"},{"why":"Supplies the observation that SAEs learn features roughly in decreasing order of frequency, supporting the Zipfian frequency assumption.","marker":"[15]"},{"why":"Documents language-model features that are not one-dimensionally linear, motivating the feature-manifold setting.","marker":"[22]"},{"why":"Supplies the large trained SAE suite used for latent-firing-frequency and decoder-geometry measurements.","marker":"[16]"},{"why":"Documents feature absorption, which the paper cites as a violation of its core assumptions.","marker":"[27]"},{"why":"Provides the 'quanta' neural-scaling model that the discrete-feature picture mirrors.","marker":"[28]"}],"fun_headline_variants":["SAE scaling law hides a feature-discovery trap","Feature manifolds make SAEs over-invest in common features","When SAE loss improves but feature count stagnates","Capacity allocation reveals why SAEs undercount rare features","Sparse autoencoders can scale loss without scaling features"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The model treats each feature as a separate, non-overlapping piece of activation space served by its own dedicated latents, so total loss decomposes into per-feature terms; if real SAEs absorb features or let latents mix across overlapping feature subspaces, that decomposition no longer holds, and the authors explicitly admit this happens in practice.","fun_headline_variants_meta":{"raw":{"variants":["SAE scaling law hides a feature-discovery trap","Feature manifolds make SAEs over-invest in common features","When SAE loss improves but feature count stagnates","Capacity allocation reveals why SAEs undercount rare features","Sparse autoencoders can scale loss without scaling features"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000215,"raw_usage":{"total_tokens":1419,"prompt_tokens":929,"completion_tokens":490,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":412}},"tokens_in":545,"tokens_out":490,"duration_ms":5072,"temperature":1.0,"reasoning_tokens":412,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:35:55.693537+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train an SAE on a synthetic mixture containing one common hypersphere manifold plus many rare discrete features, with parameters chosen so $\\beta < \\alpha$, and measure the discovered-feature count $D(N)$ as width grows by orders of magnitude: linear growth in $N$ would falsify the sublinear-discovery prediction. On real data, directly measuring $L_i(n_i)$ for identified features at several widths would show whether any manifold sustains the slow decay needed for the pathological regime.","supporting_citations":[{"cited_title":"The quantization model of neural scaling","cited_arxiv_id":null,"evidence_quote":"Provides the 'quanta' neural-scaling model that the discrete-feature picture mirrors."}],"review_version":2}