{"id":"d9c215ec-07c4-4c6d-a734-80dec436c920","arxiv_id":"2506.04859","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"VAEase gates the VAE decoder input by the encoder's variance, combining sparse-autoencoder adaptive sparsity with a hyperparameter-free loss; a global-minimizer theorem says active latent dimensions recover per-manifold dimensions.","lead":"VAEase is a modified variational autoencoder that uses the encoder's own uncertainty estimates to gate the decoder input, so that different latent dimensions can turn on for different inputs. The paper reports sparser codes and better reconstruction than sparse autoencoders and VAEs on images and language-model activations, with a theory tying latent usage to manifold dimension.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 4.5 is unproven as stated: the proof's feasible construction uses non-Lipschitz indicator maps and assumes a bi-Lipschitz lower bound, so the central adaptive-sparsity guarantee lacks support.","rationale":"The reader's weakest assumption focuses on the gap between asymptotic global-minimizer theory and finite-gamma SGD solutions, which is a valid practical concern. However, on careful reading, the theoretical statement itself is not secure: the proof of Theorem 4.5 uses a feasible construction with indicator functions that are not Lipschitz on X when manifolds overlap, even though Definition 4.1 explicitly permits overlap, and the decoder uses inverses of diffeomorphisms that need not be Lipschitz. The lower-bound half of the proof then asserts a local bi-Lipschitz inequality for arbitrary Lipschitz decoders, which is false in general. These are internal inconsistencies in the argument, not mere omissions of convergence diagnostics. Because Theorem 4.5 and Corollary 4.6 are the theoretical core of the paper, the claim that VAEase 'provably' recovers per-manifold active dimensions is currently unverified. The empirical results are suggestive but do not compensate for a broken proof, and the lack of multi-seed reporting and learned-γ values would matter even for the empirical claim. I therefore recommend keeping the verdict CONDITIONAL, with explicit conditions: repair or restate Theorem 4.5 for admissible Lipschitz maps, and add multi-seed results with learned γ reported. If the proof cannot be repaired, the theoretical contribution should be downgraded to a conjecture or heuristic.","tokens_in":28531,"tokens_out":10681,"duration_ms":139843,"concrete_test":"Check the proof's feasible construction on X equal to the union of the x-axis and y-axis in R^2, with probability mass on both lines. Compute the Lipschitz constant of the proposed μz and μx: points (ε,0) and (0,ε) are distance √2 ε apart while indicator-based encoder outputs differ by O(1), so the constant diverges as ε→0; this violates Definition 4.2. Separately, test the lower-bound step with a Lipschitz injective decoder μ~(u)=u^3 on [-1,1]: if no l>0 satisfies ∥μ~(u)−μ~(v)∥ ≥ l|u−v|, the claimed bi-Lipschitz lower bound is not a consequence of the stated assumptions.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central theoretical claim is Theorem 4.5, but its proof in Appendix E.1 is internally inconsistent with Definition 4.2. The feasible solution defines the encoder via terms like I(x∈M_i)ψ_i(x) and the decoder as a sum over ψ_i^{-1} of selected coordinates. Indicator functions of manifolds are not Lipschitz on X when manifolds overlap or merely cross, which Definition 4.1 explicitly permits: for two crossing lines, points (ε,0) and (0,ε) are arbitrarily close while the indicator-based encoder entries differ by 1, so the Lipschitz constant diverges. The proof also uses ψ_i^{-1} as a decoder map, but a diffeomorphism to R^{r_i} does not imply its inverse is Lipschitz (e.g., ψ(u)=u^3 has non-Lipschitz inverse). Thus the upper-bound solution lies outside the model class being analyzed. Separately, the lower-bound argument asserts without proof that there is a compact set Z and constant l>0 with ∥μ~(z1)−μ~(z2)∥ ≥ l∥z1−z2∥ for all z1,z2∈Z. Lipschitz injectivity does not imply such a uniform lower bound, as shown by μ~(u)=u^3 on [-1,1]. This bi-Lipschitz assumption is exactly what forces σ_z^2=O(γ) on r_i dimensions; without it, the necessary-condition part of the proof fails. Since Corollary 4.6 and the paper's claim of provable manifold recovery depend on Theorem 4.5, the theoretical foundation is not currently established. This is a proof gap, not a disagreement with consensus, and it is more fundamental than the finite-sample/SGD gap: even the asymptotic global-minimizer statement is unsupported as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes VAEase, a modification of the variational autoencoder in which the decoder receives ez = (1 - sigma_z) ⊙ z instead of z, so that inactive latent dimensions are gated to zero while active dimensions transmit near-deterministic information. The authors claim that (i) global minimizers of the VAEase population loss recover per-manifold latent dimensions for data on a union of manifolds (Theorem 4.5), (ii) vanilla VAEs provably fail at this task for some datasets (Corollary 4.6), and (iii) the VAEase loss has fewer local minima than an analogous SAE loss (Theorems 4.7 and 4.8). Empirically, VAEase reports lower active-dimension counts than SAE and VAE baselines at comparable or better reconstruction error on synthetic linear and nonlinear manifolds, MNIST, Fashion-MNIST, LLM intermediate activations, text embeddings, and a GAN-generated pseudo-MNIST dataset.","tokens_in":28787,"tokens_out":8485,"duration_ms":92413,"significance":"If the theoretical claims were fully established, VAEase would be a valuable contribution: it is a hyperparameter-free (up to the learned decoder variance gamma) adaptive-sparse autoencoder with a plausible mechanism for avoiding the fixed-sparsity trap of vanilla VAEs, and the empirical results are genuinely strong. The synthetic experiments in Table 2 are clean and directly falsifiable, with ground-truth manifold dimensions recovered almost exactly (VAEase 5,5,11,11 versus ground truth 5,5,10,10), and the real-data results in Table 3, especially the LLM activation AD of 22.5 versus SAE-Tk 30.0, are substantial. The paper also ships a code link and provides detailed experimental settings. However, the central theoretical result, Theorem 4.5, has load-bearing proof gaps in Appendix E.1, and the empirical evaluation does not bridge the gap between asymptotic global-minimizer statements and finite-sample SGD training with learned gamma. These issues are significant but appear repairable, so the appropriate decision is major revision rather than rejection.","major_comments":[{"comment":"The claimed feasible solution defines the encoder with terms I(x∈M_i)ψ_i(x) and the effective decoder through ψ_i^{-1}. Under Definition 4.2, both µz and µx must be Lipschitz on X. Definition 4.1 and the discussion around it explicitly allow overlapping manifolds, and for two crossing manifolds the indicator values differ by 1 at arbitrarily close points, so the Lipschitz constant diverges. Moreover, the proof requires ψ_i^{-1} to be L-Lipschitz, but Definition 4.1 only assumes a diffeomorphism; for example, ψ(u)=u^3 is a diffeomorphism of R with non-Lipschitz inverse. Thus the upper-bound construction lies outside the model class analyzed, and the claimed upper rate bound is not established.","section":"Appendix E.1, feasible construction for Theorem 4.5"},{"comment":"The proof asserts without derivation that there exist a compact set Z and a constant l>0 such that ∥μ~(z1)−μ~(z2)∥ ≥ l∥z1−z2∥ for all z1,z2∈Z. Lipschitz continuity and injectivity do not imply such a uniform lower bound, as shown by μ~(u)=u^3 on [-1,1]. This bi-Lipschitz assumption is exactly what forces σ_z^2=O(γ) on the r_i active dimensions; without a proof that such Z and l exist under the stated assumptions, the necessary-condition part of the theorem fails. In addition, the reduction 'without loss of generality' to the first r_i channels of μ~ is not justified for nonlinear decoders, and the limiting arguments involving the events {B_1(...) ⊆ Z} and Ai'(C,l') are stated informally and require proof.","section":"Appendix E.1, lower-bound argument around Eq. (17)"},{"comment":"Theorem 4.5 concerns global minimizers of the population loss in the limit γ→0, whereas the experiments in Section 5 minimize finite-sample losses using SGD and learn γ as a parameter. The paper does not report the learned γ values, any multi-seed variation, or convergence diagnostics, and the active-dimension estimates in Tables 2-4 use a heuristic variance-threshold criterion described in Section 5.1 rather than the O(γ) criterion of Definition 4.3. Without this information it is not possible to determine whether the empirical solutions are in the regime covered by the theorem, so the claim that the experiments confirm the theoretical recovery guarantee is not yet substantiated.","section":"Section 5 versus Theorem 4.5"}],"minor_comments":[{"comment":"The displayed reconstruction term should be (1/γ)∥x − W((1−σ_z)⊙µ_z(x;ϕ))∥², since the mean of ez is (1−σ_z)⊙µ_z; the current expression writes ∥x−Wµ_z(x;ϕ)∥², which omits the gating factor on the mean. The qualitative conclusion about σ_z-dependent regularization vanishing at both σ_z^2→0 and σ_z^2→1 remains correct, but the equation as written is misleading.","section":"Appendix B.3, Eq. (16)"},{"comment":"The heading 'Pesudo-MNIST' and the caption 'samples from the proposed pesudo-MNIST' contain typographical errors; they should read 'pseudo-MNIST'.","section":"Appendix D.4"},{"comment":"Definition 4.3 defines an active dimension via σ_z^2=O(γ) as γ→0, but in the experiments γ is a learned finite scalar. The paper should clarify how the threshold criterion used in Section 5.1 relates to this asymptotic definition, and ideally report the distribution of σ_z values alongside the learned γ.","section":"Definition 4.3 and Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"The empirical contribution is strong and the gating idea is elegant, but the proof of Theorem 4.5 currently has gaps that are substantial enough that the advertised theoretical guarantee cannot be considered established. The issues seem fixable: one could either strengthen Definition 4.1 to assume separated manifolds and Lipschitz inverse diffeomorphisms, or replace the indicator-based construction with a Lipschitz approximation and provide the missing bi-Lipschitz lower-bound argument. Given the clean synthetic results and the code availability, I would encourage the editor to invite a major revision rather than reject."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know up front. The VAEase gating trick is genuinely new and well motivated. Feeding ez = (1 - sigma_z) * z to the decoder, with the sigma-dependent regularizer in Eq. (16) vanishing at both sigma^2 -> 0 and sigma^2 -> 1, is a neat fix for the fixed-sparsity failure of vanilla VAEs. The synthetic results in Table 2 are also clean: VAEase recovers per-manifold dimensions 5,5,11,11 versus ground truth 5,5,10,10 while all baselines misfire. That part deserves real credit.\n\nThe soft spot is the theory. The stress-test lands. Theorem 4.5 is stated for Lipschitz encoder/decoder networks, but the proof's feasible construction uses indicator functions I(x in M_i), which are not Lipschitz on the union when manifolds cross or overlap, and Definition 4.1 explicitly permits that. The lower-bound argument also asserts a uniform bi-Lipschitz lower bound on the decoder map without proof; Lipschitz injectivity does not imply it. Without that bound, the necessary condition sigma_z^2 = O(gamma) on active dimensions is not established. So the asymptotic global-minimizer statement is unproven as written, and Corollary 4.6 inherits the problem. This is a proof gap, not a disagreement with the conclusion, but it is load-bearing for the paper's central theoretical claim.\n\nThe empirical section has secondary but addressable issues: no multi-seed variation, no learned gamma values reported, and SAE baselines are tuned to roughly match reconstruction error rather than compared at strictly matched RE, which matters for the active-dimension comparisons. The diffusion comparison is informative but not controlled, since NB and FLIPD are known to overestimate intrinsic dimension. None of this kills the mechanism; it just means the practical claims are weaker than the text suggests.\n\nWhat is solid: the mechanism, the local-minima contrast (Theorem 4.7 versus Corollary 4.8), the Appendix B.3 derivation, and the synthetic validation. What is not: the proof of Theorem 4.5 as stated and the strength of the real-data claims. If the proof gaps are repaired or explicitly flagged, and the empirical reporting is tightened, the central claim likely stands. This paper is for the SAE/interpretability community and anyone working on adaptive sparsity without sensitive hyperparameters. I would send it to serious peer review, but with referees who check the proof carefully and request the missing diagnostics.","headline":"VAEase is a genuinely new gating idea with clean synthetic results, but the main theorem has a real proof gap and the real-data claims outrun the evidence.","tokens_in":29502,"tokens_out":1723,"would_cite":true,"duration_ms":20983,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A minimal gating change lets variational autoencoders provably recover per-manifold latent dimensions that plain SAEs and VAEs miss.","keywords":["sparse autoencoders","variational autoencoders","adaptive sparsity","manifold dimension estimation","union of manifolds","latent gating","LLM activation sparsity","unsupervised representation learning"],"falsifier":"Train VAEase on the paper's nonlinear synthetic multi-manifold dataset while logging the learned decoder variance $\\gamma$ and the per-manifold active-dimension counts throughout training: if the counts approach ground truth $r_i$ only when $\\gamma$ is forced to values far below what training selects, or if freezing $\\gamma$ at a large value does not worsen recovery, then finite-$\\gamma$ dynamics rather than the $\\gamma\\to 0$ global-minimizer mechanism are producing the reported result.","tokens_in":28163,"feed_emoji":"🧠","tokens_out":11633,"duration_ms":107452,"temperature":0.7,"pith_summary":"Classic sparse autoencoders (SAEs) and variational autoencoders (VAEs) have complementary flaws, this paper argues: SAEs can shift which latent units are active from input to input but need tuned penalty hyperparameters and suffer many local minima, while VAEs need no hyperparameters but lock into fixed sparsity patterns that cannot vary per sample. The paper proposes VAEase, a minimal VAE modification in which the encoder's variance gates the latent code before the decoder sees it, restoring sample-adaptive sparsity while keeping the VAE's smooth, hyperparameter-free objective. The central theoretical claim is that VAEase global minimizers align with ground-truth structure: for data supported on a union of manifolds, as the decoder noise vanishes, the model reconstructs near-perfectly while activating exactly as many latent dimensions as each manifold has. The empirical claim is that trained VAEase models report fewer active dimensions than SAE, VAE, and diffusion-based baselines at equal or better reconstruction across synthetic manifolds, MNIST, Fashion-MNIST, LLM activation layers, and text embeddings.","feed_headline":"One-line VAE tweak provably recovers each manifold's dimension","feed_subtitle":"The gated-code model beats SAEs and VAEs on images and LLM activity with fewer active latents.","key_machinery":"The load-bearing object is the gated latent code $\\tilde{z} := (1-\\sigma_z(x;\\phi))\\odot z$, which converts the VAE encoder's variance output into a deterministic, input-dependent mask. For an inactive dimension $\\sigma_z^2 \\approx 1$, so the decoder receives a near-zero deterministic input instead of white noise and need not permanently zero its incoming weights; for an active dimension $\\sigma_z^2 = O(\\gamma)$ as $\\gamma \\to 0$, so $\\tilde{z} \\approx \\mu_z(x;\\phi)$ and the signal passes through. The decoder can therefore keep all of its columns alive while the set of active dimensions varies sample to sample, precisely the adaptive sparsity that SAEs provide and plain VAEs cannot. The proof of Theorem 4.5 bounds the loss rate of a constructed feasible solution and shows that deviations, too few active dimensions on a positive-measure subset or inactive dimensions whose variance is not $1-O(\\gamma)$, force the reconstruction term to a higher order in $\\gamma$, making the per-manifold active count $r_i$ the unique rate-optimal behavior.","core_discovery":"On the paper's own terms: a VAE whose decoder receives $\\tilde{z} := (1-\\sigma_z(x;\\phi))\\odot z$, with the encoder variance network acting as a per-input sparsity gate, achieves the adaptive sparsity that a vanilla VAE provably cannot. Theorem 4.5 states that for data adhering to a union-of-manifolds structure with $\\sum_i r_i \\le \\kappa$, as the decoder variance $\\gamma \\to 0$, every global minimizer of the VAEase loss attains reconstruction error $R = o(1)$ and uses exactly $r_i$ active latent dimensions on almost every point of each manifold $M_i$, so the active set matches each manifold's intrinsic dimension per sample. Corollary 4.6 shows there exist such datasets where vanilla VAE global minimizers fail, favoring one fixed active set that captures only the aggregated dimension. Empirically, VAEase recovers per-manifold dimensions on synthetic data (e.g., 5,5,11,11 against ground truth 5,5,10,10), and on real data reaches the lowest active-dimension counts at equal or better reconstruction: 22.5 active dimensions on LLM activations versus 30.0 for the closest SAE baseline, and 16.2 on MNIST.","pith_inferences":["The gating recipe likely generalizes: any latent-variable model whose encoder emits a per-dimension variance could feed $(1-\\sigma)\\odot z$ to its decoder and inherit sample-adaptive sparsity; trying this on hierarchical or diffusion-style autoencoders is a direct test the paper does not run.","The paper leaves the finite-sample regime unexamined: since theory covers $\\gamma\\to 0$ global optima but training learns $\\gamma$, measuring per-manifold active-dimension recovery while sweeping $\\gamma$ on the synthetic datasets would show how much of the result survives outside the asymptotic regime.","If the label-aligned active-dimension differences the paper reports generalize (intra-class 0.31 versus inter-class 0.49 on MNIST; 0.68 versus 2.11 on Fashion-MNIST), VAEase becomes a label-free probe of class-conditional intrinsic dimension with testable predictions on datasets of known subpopulation structure."],"forward_implications":["On LLM activation data VAEase reaches 22.5 active dimensions with reconstruction error 39.5, versus 30.0 active dimensions and error 45.1 for the best SAE baseline, so interpretable sparse codes can be produced without hand choosing a sparsity level $k$.","Per-manifold dimension estimation improves: on synthetic multi-manifold data VAEase reports {5,5,11,11} against ground truth {5,5,10,10}, while SAE-$\\ell_1$, SAE-log, SAE-Tk, and plain VAE overestimate by factors of two or more.","Diffusion-based dimension estimators are outperformed on pseudo-MNIST: VAEase reports 14.93 active dimensions (true dimension at most 16) versus 105.94 and 169.81 for two recent estimators.","The local-minima analysis predicts smoother optimization: in a simplified linear setting the VAEase loss has a unique minimum while the analogous SAE objective has $2^d$ local minima, and VAEase needs no sparsity hyperparameters at all."],"supporting_citations":[{"why":"supplies the active-dimension formalism and the conditional-VAE setting that VAEase's variance gating extends.","marker":"Zheng et al., 2022"},{"why":"establishes that VAEs exhibit emergent sparsity as $\\gamma \\to 0$, the mechanism VAEase inherits.","marker":"Dai et al., 2021"},{"why":"provides the result that VAEs selectively smooth away bad local minima, which VAEase claims to retain.","marker":"Wipf, 2023"},{"why":"motivates the union-of-manifolds data model formalized in Definition 4.1 and used in Theorem 4.5.","marker":"Brown et al., 2022"},{"why":"defines the SAE-$\\ell_1$ baseline and supplies the LLM activation extraction code used in Section 5.2.","marker":"Cunningham et al., 2024"},{"why":"defines the top-$k$ SAE-Tk baseline and scaling practices compared against across experiments.","marker":"Gao et al., 2024"},{"why":"provides the NB diffusion-based manifold-dimension estimator compared in Table 4.","marker":"Stanczuk et al., 2024"},{"why":"provides the FLIPD diffusion-based estimator compared in Table 4.","marker":"Kamkari et al., 2024"}],"fun_headline_variants":["Gated VAE provably recovers each manifold's intrinsic dimension","One-line VAE tweak provably finds per-manifold latent dimension","Gated-code VAE beats SAEs and VAEs with fewer active latents","Provable manifold-dimension recovery via a gated VAE decoder","Sparse gating VAE provably outperforms SAEs on real data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proofs characterize ideal global optima as the decoder-noise parameter $\\gamma$ tends to zero, while the experiments run stochastic gradient training on finite samples with $\\gamma$ learned from data; the empirical claims hold only if those trained solutions inherit the per-manifold active-dimension behavior of the $\\gamma\\to 0$ global minimizers, and only if real datasets approximate the assumed union of low-dimensional manifolds.","fun_headline_variants_meta":{"raw":{"variants":["Gated VAE provably recovers each manifold's intrinsic dimension","One-line VAE tweak provably finds per-manifold latent dimension","Gated-code VAE beats SAEs and VAEs with fewer active latents","Provable manifold-dimension recovery via a gated VAE decoder","Sparse gating VAE provably outperforms SAEs on real data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00024,"raw_usage":{"total_tokens":1582,"prompt_tokens":1075,"completion_tokens":507,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":691,"completion_tokens_details":{"reasoning_tokens":409}},"tokens_in":691,"tokens_out":507,"duration_ms":5526,"temperature":1.0,"reasoning_tokens":409,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:34:38.167962+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train VAEase on the paper's nonlinear synthetic multi-manifold dataset while logging the learned decoder variance $\\gamma$ and the per-manifold active-dimension counts throughout training: if the counts approach ground truth $r_i$ only when $\\gamma$ is forced to values far below what training selects, or if freezing $\\gamma$ at a large value does not worsen recovery, then finite-$\\gamma$ dynamics rather than the $\\gamma\\to 0$ global-minimizer mechanism are producing the reported result.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the active-dimension formalism and the conditional-VAE setting that VAEase's variance gating extends."},{"cited_title":"On the value of infinite gradients in variational autoencoder models","cited_arxiv_id":null,"evidence_quote":"establishes that VAEs exhibit emergent sparsity as $\\gamma \\to 0$, the mechanism VAEase inherits."},{"cited_title":"Marginalization is not marginal: N o bad VAE local minima when learning optimal sparse representations","cited_arxiv_id":null,"evidence_quote":"provides the result that VAEs selectively smooth away bad local minima, which VAEase claims to retain."},{"cited_title":"Sparse autoencoders find highly interpretable features in language models","cited_arxiv_id":null,"evidence_quote":"defines the SAE-$\\ell_1$ baseline and supplies the LLM activation extraction code used in Section 5.2."},{"cited_title":"P., Batzolis, G., Deveney, T., and Sch \\\"o nlieb, C.-B","cited_arxiv_id":null,"evidence_quote":"provides the NB diffusion-based manifold-dimension estimator compared in Table 4."}],"review_version":1}