Pith. sign in

REVIEW 3 major objections 3 minor 57 references

Sparse Autoencoders, Again?

T0 review · 3 major / 3 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A minimal gating change lets variational autoencoders provably recover per-manifold latent dimensions that plain SAEs and VAEs miss.

desk verdict VAEase is a genuinely new gating idea with clean synthetic results, but the main theorem has a real proof gap and the real-data claims outrun the evidence. read the letter →

arxiv 2506.04859 v2 pith:DHQZX6QU submitted 2025-06-05 cs.LG cs.AI

Yin Lu , Xuening Zhu , Tong He , David Wipf This is my paper
classification cs.LGcs.AI
keywords sparseautoencodersvariationaladaptivesparsitymanifolddimensionestimationunionofmanifoldslatentgatingLLMactivationunsupervisedrepresentationlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Classic sparse autoencoders (SAEs) and variational autoencoders (VAEs) have complementary flaws, this paper argues: SAEs can shift which latent units are active from input to input but need tuned penalty hyperparameters and suffer many local minima, while VAEs need no hyperparameters but lock into fixed sparsity patterns that cannot vary per sample. The paper proposes VAEase, a minimal VAE modification in which the encoder's variance gates the latent code before the decoder sees it, restoring sample-adaptive sparsity while keeping the VAE's smooth, hyperparameter-free objective. The central theoretical claim is that VAEase global minimizers align with ground-truth structure: for data supported on a union of manifolds, as the decoder noise vanishes, the model reconstructs near-perfectly while activating exactly as many latent dimensions as each manifold has. The empirical claim is that trained VAEase models report fewer active dimensions than SAE, VAE, and diffusion-based baselines at equal or better reconstruction across synthetic manifolds, MNIST, Fashion-MNIST, LLM activation layers, and text embeddings.

What carries the argument

The load-bearing object is the gated latent code $\tilde{z} := (1-\sigma_z(x;\phi))\odot z$, which converts the VAE encoder's variance output into a deterministic, input-dependent mask. For an inactive dimension $\sigma_z^2 \approx 1$, so the decoder receives a near-zero deterministic input instead of white noise and need not permanently zero its incoming weights; for an active dimension $\sigma_z^2 = O(\gamma)$ as $\gamma \to 0$, so $\tilde{z} \approx \mu_z(x;\phi)$ and the signal passes through. The decoder can therefore keep all of its columns alive while the set of active dimensions varies sample to sample, precisely the adaptive sparsity that SAEs provide and plain VAEs cannot. The proof of Theorem 4.5 bounds the loss rate of a constructed feasible solution and shows that deviations, too few active dimensions on a positive-measure subset or inactive dimensions whose variance is not $1-O(\gamma)$, force the reconstruction term to a higher order in $\gamma$, making the per-manifold active count $r_i$ the unique rate-optimal behavior.

What would settle it

Train VAEase on the paper's nonlinear synthetic multi-manifold dataset while logging the learned decoder variance $\gamma$ and the per-manifold active-dimension counts throughout training: if the counts approach ground truth $r_i$ only when $\gamma$ is forced to values far below what training selects, or if freezing $\gamma$ at a large value does not worsen recovery, then finite-$\gamma$ dynamics rather than the $\gamma\to 0$ global-minimizer mechanism are producing the reported result.

Watch

Extended reading notes

Core claim

On the paper's own terms: a VAE whose decoder receives $\tilde{z} := (1-\sigma_z(x;\phi))\odot z$, with the encoder variance network acting as a per-input sparsity gate, achieves the adaptive sparsity that a vanilla VAE provably cannot. Theorem 4.5 states that for data adhering to a union-of-manifolds structure with $\sum_i r_i \le \kappa$, as the decoder variance $\gamma \to 0$, every global minimizer of the VAEase loss attains reconstruction error $R = o(1)$ and uses exactly $r_i$ active latent dimensions on almost every point of each manifold $M_i$, so the active set matches each manifold's intrinsic dimension per sample. Corollary 4.6 shows there exist such datasets where vanilla VAE global minimizers fail, favoring one fixed active set that captures only the aggregated dimension. Empirically, VAEase recovers per-manifold dimensions on synthetic data (e.g., 5,5,11,11 against ground truth 5,5,10,10), and on real data reaches the lowest active-dimension counts at equal or better reconstruction: 22.5 active dimensions on LLM activations versus 30.0 for the closest SAE baseline, and 16.2 on MNIST.

Load-bearing premise

The proofs characterize ideal global optima as the decoder-noise parameter $\gamma$ tends to zero, while the experiments run stochastic gradient training on finite samples with $\gamma$ learned from data; the empirical claims hold only if those trained solutions inherit the per-manifold active-dimension behavior of the $\gamma\to 0$ global minimizers, and only if real datasets approximate the assumed union of low-dimensional manifolds.

Editorial extensions

If this is right

  • On LLM activation data VAEase reaches 22.5 active dimensions with reconstruction error 39.5, versus 30.0 active dimensions and error 45.1 for the best SAE baseline, so interpretable sparse codes can be produced without hand choosing a sparsity level $k$.
  • Per-manifold dimension estimation improves: on synthetic multi-manifold data VAEase reports {5,5,11,11} against ground truth {5,5,10,10}, while SAE-$\ell_1$, SAE-log, SAE-Tk, and plain VAE overestimate by factors of two or more.
  • Diffusion-based dimension estimators are outperformed on pseudo-MNIST: VAEase reports 14.93 active dimensions (true dimension at most 16) versus 105.94 and 169.81 for two recent estimators.
  • The local-minima analysis predicts smoother optimization: in a simplified linear setting the VAEase loss has a unique minimum while the analogous SAE objective has $2^d$ local minima, and VAEase needs no sparsity hyperparameters at all.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The gating recipe likely generalizes: any latent-variable model whose encoder emits a per-dimension variance could feed $(1-\sigma)\odot z$ to its decoder and inherit sample-adaptive sparsity; trying this on hierarchical or diffusion-style autoencoders is a direct test the paper does not run.
  • The paper leaves the finite-sample regime unexamined: since theory covers $\gamma\to 0$ global optima but training learns $\gamma$, measuring per-manifold active-dimension recovery while sweeping $\gamma$ on the synthetic datasets would show how much of the result survives outside the asymptotic regime.
  • If the label-aligned active-dimension differences the paper reports generalize (intra-class 0.31 versus inter-class 0.49 on MNIST; 0.68 versus 2.11 on Fashion-MNIST), VAEase becomes a label-free probe of class-conditional intrinsic dimension with testable predictions on datasets of known subpopulation structure.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper proposes VAEase, a modification of the variational autoencoder in which the decoder receives ez = (1 - sigma_z) ⊙ z instead of z, so that inactive latent dimensions are gated to zero while active dimensions transmit near-deterministic information. The authors claim that (i) global minimizers of the VAEase population loss recover per-manifold latent dimensions for data on a union of manifolds (Theorem 4.5), (ii) vanilla VAEs provably fail at this task for some datasets (Corollary 4.6), and (iii) the VAEase loss has fewer local minima than an analogous SAE loss (Theorems 4.7 and 4.8). Empirically, VAEase reports lower active-dimension counts than SAE and VAE baselines at comparable or better reconstruction error on synthetic linear and nonlinear manifolds, MNIST, Fashion-MNIST, LLM intermediate activations, text embeddings, and a GAN-generated pseudo-MNIST dataset.

Significance. If the theoretical claims were fully established, VAEase would be a valuable contribution: it is a hyperparameter-free (up to the learned decoder variance gamma) adaptive-sparse autoencoder with a plausible mechanism for avoiding the fixed-sparsity trap of vanilla VAEs, and the empirical results are genuinely strong. The synthetic experiments in Table 2 are clean and directly falsifiable, with ground-truth manifold dimensions recovered almost exactly (VAEase 5,5,11,11 versus ground truth 5,5,10,10), and the real-data results in Table 3, especially the LLM activation AD of 22.5 versus SAE-Tk 30.0, are substantial. The paper also ships a code link and provides detailed experimental settings. However, the central theoretical result, Theorem 4.5, has load-bearing proof gaps in Appendix E.1, and the empirical evaluation does not bridge the gap between asymptotic global-minimizer statements and finite-sample SGD training with learned gamma. These issues are significant but appear repairable, so the appropriate decision is major revision rather than rejection.

major comments (3)
  1. [Appendix E.1, feasible construction for Theorem 4.5] The claimed feasible solution defines the encoder with terms I(x∈M_i)ψ_i(x) and the effective decoder through ψ_i^{-1}. Under Definition 4.2, both µz and µx must be Lipschitz on X. Definition 4.1 and the discussion around it explicitly allow overlapping manifolds, and for two crossing manifolds the indicator values differ by 1 at arbitrarily close points, so the Lipschitz constant diverges. Moreover, the proof requires ψ_i^{-1} to be L-Lipschitz, but Definition 4.1 only assumes a diffeomorphism; for example, ψ(u)=u^3 is a diffeomorphism of R with non-Lipschitz inverse. Thus the upper-bound construction lies outside the model class analyzed, and the claimed upper rate bound is not established.
  2. [Appendix E.1, lower-bound argument around Eq. (17)] The proof asserts without derivation that there exist a compact set Z and a constant l>0 such that ∥μ~(z1)−μ~(z2)∥ ≥ l∥z1−z2∥ for all z1,z2∈Z. Lipschitz continuity and injectivity do not imply such a uniform lower bound, as shown by μ~(u)=u^3 on [-1,1]. This bi-Lipschitz assumption is exactly what forces σ_z^2=O(γ) on the r_i active dimensions; without a proof that such Z and l exist under the stated assumptions, the necessary-condition part of the theorem fails. In addition, the reduction 'without loss of generality' to the first r_i channels of μ~ is not justified for nonlinear decoders, and the limiting arguments involving the events {B_1(...) ⊆ Z} and Ai'(C,l') are stated informally and require proof.
  3. [Section 5 versus Theorem 4.5] Theorem 4.5 concerns global minimizers of the population loss in the limit γ→0, whereas the experiments in Section 5 minimize finite-sample losses using SGD and learn γ as a parameter. The paper does not report the learned γ values, any multi-seed variation, or convergence diagnostics, and the active-dimension estimates in Tables 2-4 use a heuristic variance-threshold criterion described in Section 5.1 rather than the O(γ) criterion of Definition 4.3. Without this information it is not possible to determine whether the empirical solutions are in the regime covered by the theorem, so the claim that the experiments confirm the theoretical recovery guarantee is not yet substantiated.
minor comments (3)
  1. [Appendix B.3, Eq. (16)] The displayed reconstruction term should be (1/γ)∥x − W((1−σ_z)⊙µ_z(x;ϕ))∥², since the mean of ez is (1−σ_z)⊙µ_z; the current expression writes ∥x−Wµ_z(x;ϕ)∥², which omits the gating factor on the mean. The qualitative conclusion about σ_z-dependent regularization vanishing at both σ_z^2→0 and σ_z^2→1 remains correct, but the equation as written is misleading.
  2. [Appendix D.4] The heading 'Pesudo-MNIST' and the caption 'samples from the proposed pesudo-MNIST' contain typographical errors; they should read 'pseudo-MNIST'.
  3. [Definition 4.3 and Section 5.1] Definition 4.3 defines an active dimension via σ_z^2=O(γ) as γ→0, but in the experiments γ is a learned finite scalar. The paper should clarify how the threshold criterion used in Section 5.1 relates to this asymptotic definition, and ideally report the distribution of σ_z values alongside the learned γ.

Circularity Check

0 steps flagged · score 2.0 of 10

No meaningful circularity: the central VAEase theorem and empirical comparisons are self-contained; only non-load-bearing self-citations and a separate proof gap appear.

full rationale

The main derivation chain is not circular. Theorem 4.5 is proven in Appendix E.1 directly from the VAEase loss, Definitions 4.1-4.4, and Lipschitz assumptions, without invoking any prior result at the load-bearing step. The active-dimension notion (Definition 4.3) is defined via sigma_z^2 = O(gamma), and the gating e_z = (1 - sigma_z) * z in (4) is a design choice; whether global minimizers use exactly r_i active coordinates on each manifold is a substantive conclusion established from the loss and manifold geometry, not a restatement of a definition. The empirical active-dimension estimates are also not fitted-input predictions: synthetic ground truths (linear subspaces, MLP manifolds, pseudo-MNIST upper bound 16) are external to the trained models, and the same variance-partitioning criterion is applied uniformly to all baselines. Self-citations to Zheng et al. (2022), Wipf (2023), and Dai et al. (2021) appear in motivation and background sections, but the paper supplies independent proofs for the claims that matter (e.g., Corollary 4.6, Appendix B.1, Appendix E.5), so these citations are lineage rather than load-bearing. There is, however, a genuine proof-correctness concern in Appendix E.1: the proof asserts without proof a bi-Lipschitz lower bound (existence of Z and l > 0 with ||mu_tilde(z1) - mu_tilde(z2)|| >= l ||z1 - z2||), and the upper-bound feasible construction uses non-Lipschitz indicator functions. This is an omitted-support issue, not a circular reduction, so it does not raise the circularity score.

Assumptions & free parameters 2 free parameters · 4 assumptions · 1 invented entities

The central claims rest on the union-of-manifolds data model (Definition 4.1), the Lipschitz function class (Definition 4.2), an unproved local well-conditioning step in the lower-bound proof, and the transfer from global minimizers at gamma to 0 to SGD solutions at learned gamma. The only tunable constant in the VAEase model itself is the learned decoder variance gamma, which is not reported after training. No exotic entities are introduced; the gating mechanism is the single new component, and it carries direct falsifiable predictions.

free parameters (2)
  • gamma (learned decoder variance) = learned, not reported
    Gamma is the scale of the Gaussian decoder and the small parameter of the asymptotic theory (gamma to 0) in Theorem 4.5. Whether the trained model operates in the asymptotic regime depends on the learned gamma, which the paper never reports.
  • active-dimension threshold criterion = data-adaptive, no explicit number reported
    Active dimensions are counted by partitioning average variance above and below a threshold chosen to minimize intra-cluster variance (adapted from Xia et al. 2015). This is a modeling choice that affects every reported AD number and could systematically favor models with a clear bimodal variance profile, such as VAEase.
assumptions (4)
  • domain assumption Definition 4.1 union-of-manifolds data hypothesis
    All theoretical and interpretable empirical claims presuppose that data lie on a finite union of smooth manifolds with well-defined per-manifold dimensions r_i and zero measure on lower-dimensional submanifolds. If real data violate this, the recovery statements lose their meaning.
  • domain assumption Definition 4.2 Lipschitz encoder and decoder
    Theorem 4.5 requires Lipschitz mu_z and mu_x. The upper-bound construction in Appendix E.1 uses indicator functions I(x in M_i), which are not Lipschitz, so the constructed solution is not strictly inside the admitted model class.
  • ad hoc to paper Local bi-Lipschitz decoder restriction in the lower bound
    The lower-bound argument in Appendix E.1 assumes that on a compact set of near-total measure, the decoder restricted to active coordinates satisfies ||mu_x(z1) - mu_x(z2)|| >= l ||z1 - z2||. This is asserted rather than derived from the Lipschitz assumption, and it is essential for concluding that at least r_i dimensions are required.
  • domain assumption Global-minimizer-to-SGD transfer at finite gamma
    Theorem 4.5 governs global minimizers as gamma approaches 0. The empirical pipeline runs SGD on a finite sample at learned gamma and assumes it lands in the favorable basin; no convergence certificate or gamma-smallness diagnostic is provided.
invented entities (1)
  • VAEase gating mechanism (decoder input ez = (1 - sigma_z) * z) independent evidence
    purpose: Suppress noise from inactive latent dimensions so the decoder can keep nonzero weights on all dimensions, enabling input-adaptive sparsity in a VAE without additional parameters.
    This is a new architectural component rather than a new physical entity. It is directly falsifiable: its predicted consequences, per-manifold active-dimension recovery on synthetic data and sparser codes at equal reconstruction error on real data, are testable externally, and the synthetic ground-truth experiments already constitute such a test.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Sparse Autoencoders, Again?." pith.science (2026). https://pith.science/paper/DHQZX6QU

@misc{pith2026250604859,
  author       = {Pith},
  title        = {Pith review of: Sparse Autoencoders, Again?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DHQZX6QU}},
  note         = {Machine review of arXiv:2506.04859}
}
read the original abstract

Is there really much more to say about sparse autoencoders (SAEs)? Autoencoders in general, and SAEs in particular, represent deep architectures that are capable of modeling low-dimensional latent structure in data. Such structure could reflect, among other things, correlation patterns in large language model activations, or complex natural image manifolds. And yet despite the wide-ranging applicability, there have been relatively few changes to SAEs beyond the original recipe from decades ago, namely, standard deep encoder/decoder layers trained with a classical/deterministic sparse regularizer applied within the latent space. One possible exception is the variational autoencoder (VAE), which adopts a stochastic encoder module capable of producing sparse representations when applied to manifold data. In this work we formalize underappreciated weaknesses with both canonical SAEs, as well as analogous VAEs applied to similar tasks, and propose a hybrid alternative model that circumvents these prior limitations. In terms of theoretical support, we prove that global minima of our proposed model recover certain forms of structured data spread across a union of manifolds. Meanwhile, empirical evaluations on synthetic and real-world datasets substantiate the efficacy of our approach in accurately estimating underlying manifold dimensions and producing sparser latent representations without compromising reconstruction error. In general, we are able to exceed the performance of equivalent-capacity SAEs and VAEs, as well as recent diffusion models where applicable, within domains such as images and language model activation patterns.

Figures

Figures reproduced from arXiv: 2506.04859 by the authors.

Figure 1
Figure 1. Fixed vs. adaptive sparsity. Depicted are two manifolds {M1,M2} with dim[M1] = dim[M2] = 2, and data points x1 ∈ M1 and x2 ∈ M2. Adaptive sparse representations require only two informative/active latent dimensions for either x1 or x2 (the dark colored elements of {z1, z2} which vary across samples). Meanwhile, fixed sparse representations involve shared active dimensions across all samples (the union of dark and li… view at source ↗
Figure 2
Figure 2. Comparison among methods. On the left SAE active (green) and inactive (red) dimensions are easily identifiable by the decoder, so accurate reconstructions are not disrupted when the locations of the nonzero elements adaptively shift from input sample-to-sample. Meanwhile in the middle figure, VAE active and inactive dimensions look similar for any given sample, which favors the VAE simply learning a fixed sparsity p… view at source ↗
Figure 3
Figure 3. Local minima smoothing example. On the left a 1D slice of the SAE loss (details in Appendix D.1) has two distinct minima appear (red dots). Meanwhile, on the right VAEase under identical conditions has a single (global) minimum because of the latent smoothing afforded by σz. symmetric about zero, concave and non-decreasing in |z|. Commonly-used sparsity-promoting penalties typically fall into this category (Chen et … view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Reconstruction error (RE) curve as latent dimen [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Because the ideal ℓ0 norm cannot be optimized via SGD, some form of smooth approximation is required by SAE models in practice. Here we show histograms of |z| values (on a log scale) for SAE-ℓ1 (left) and SAE-log (right) on MNIST data. Because the weaker convex ℓ1 very…
Figure 6
Figure 6. Figure 6: Samples from Pesudo-MNIST. On the left we show samples from the original MNIST, while on the right are samples from the proposed pesudo-MNIST; the latter are generated by a GAN trained on MNIST with a known upper bound on the manifold dimension. Because visually the tw…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

57 extracted references · 35 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    H., and Zoccolan, D

    Ansuini, A., Laio, A., Macke, J. H., and Zoccolan, D. Intrinsic dimension of data representations in deep neural networks. Advances in Neural Information Processing Systems, 32, 2019

    work page 2019

  3. [3]

    L., Foster, D

    Bartlett, P. L., Foster, D. J., and Telgarsky, M. J. Spectrally-normalized margin bounds for neural networks. Advances in neural information processing systems, 30, 2017

    2017

  4. [4]

    Representation learning: A review and new perspectives

    Bengio, Y., Courville, A., and Vincent, P. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35 0 (8): 0 1798--1828, 2013

    work page 2013

  5. [5]

    S., Li, Y., and Xu, Y

    Berahmand, K., Daneshfar, F., Salehi, E. S., Li, Y., and Xu, Y. Autoencoders and their applications in machine learning: a survey. Artificial Intelligence Review, 57 0 (2): 0 28, 2024

    2024

  6. [6]

    G., Bradley, H., O’Brien, K., Hallahan, E., Khan, M

    Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O’Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp.\ 2397--2430. PMLR, 2023

    2023

  7. [7]

    and Kamp, Y

    Bourlard, H. and Kamp, Y. Auto-association by multilayer perceptrons and singular value decomposition. Biological cybernetics, 59 0 (4): 0 291--294, 1988

    work page 1988

  8. [8]

    and Cranmer, K

    Brehmer, J. and Cranmer, K. Flows for simultaneous manifold learning and density estimation. Advances in neural information processing systems, 33: 0 442--453, 2020

    work page 2020

Show all 57 references
  1. [9]

    Towards monosemanticity: Decomposing language models with dictionary learning

    Bricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, A., Conerly, T., Turner, N., Anil, C., Denison, C., Askell, A., et al. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2, 2023

    2023

  2. [10]

    C., Caterini, A

    Brown, B. C., Caterini, A. L., Ross, B. L., Cresswell, J. C., and Loaiza-Ganem, G. Verifying the union of manifolds hypothesis for image data. arXiv preprint arXiv:2207.02862, 2022

    2022 arXiv

  3. [11]

    J., Wakin, M

    Candes, E. J., Wakin, M. B., and Boyd, S. P. Enhancing sparsity by reweighted _1 minimization. Journal of Fourier analysis and applications, 14: 0 877--905, 2008

    2008

  4. [12]

    L., Loaiza-Ganem, G., Pleiss, G., and Cunningham, J

    Caterini, A. L., Loaiza-Ganem, G., Pleiss, G., and Cunningham, J. P. Rectangular flows for manifold learning. Advances in Neural Information Processing Systems, 34: 0 30228--30241, 2021

    2021

  5. [13]

    and Geiger, A

    Chaudhary, M. and Geiger, A. Evaluating open-source sparse autoencoders on disentangling factual knowledge in gpt-2 small. arXiv preprint arXiv:2409.04478, 2024

    2024 arXiv

  6. [14]

    Strong NP -hardness for sparse optimization with concave penalty functions

    Chen, Y., Ge, D., Wang, M., Wang, Z., Ye, Y., and Yin, H. Strong NP -hardness for sparse optimization with concave penalty functions. In International Confernece on Machine Learning, 2017

    2017

  7. [15]

    Sparse autoencoders find highly interpretable features in language models

    Cunningham, H., Ewart, A., Riggs, L., Huben, R., and Sharkey, L. Sparse autoencoders find highly interpretable features in language models. International Conference on Learning Representations, 2024

    2024

  8. [16]

    Connections with robust PCA and the role of emergent sparsity in variational autoencoder models

    Dai, B., Wang, Y., Aston, J., Hua, G., and Wipf, D. Connections with robust PCA and the role of emergent sparsity in variational autoencoder models. Journal of Machine Learning Research, 19 0 (41): 0 1--42, 2018

    2018

  9. [17]

    On the value of infinite gradients in variational autoencoder models

    Dai, B., Wenliang, L., and Wipf, D. On the value of infinite gradients in variational autoencoder models. Advances in Neural Information Processing Systems, 34: 0 7180--7192, 2021

    2021

  10. [18]

    The mnist database of handwritten digit images for machine learning research [best of the web]

    Deng, L. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE signal processing magazine, 29 0 (6): 0 141--142, 2012

    2012

  11. [19]

    and Li, R

    Fan, J. and Li, R. Variable selection via nonconcave penalized likelihood and its oracle properties. JASTA, 96 0 (456): 0 1348--1360, 2001

    2001

  12. [20]

    Fazel, M., Hindi, H., and Boyd, S. P. Log-det heuristic for matrix rank minimization with applications to hankel and euclidean distance matrices. In Proceedings of the 2003 American Control Conference, 2003., volume 3, pp.\ 2156--2162. IEEE, 2003

    2003

  13. [21]

    Testing the manifold hypothesis

    Fefferman, C., Mitter, S., and Narayanan, H. Testing the manifold hypothesis. Journal of the American Mathematical Society, 29 0 (4): 0 983--1049, 2016

    2016

  14. [22]

    The pile: An 800gb dataset of diverse text for language modeling

    Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020

    2020 arXiv

  15. [23]

    D., Tillman, H., Goh, G., Troll, R., Radford, A., Sutskever, I., Leike, J., and Wu, J

    Gao, L., la Tour, T. D., Tillman, H., Goh, G., Troll, R., Radford, A., Sutskever, I., Leike, J., and Wu, J. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093, 2024

    2024 arXiv

  16. [24]

    Generative adversarial networks

    Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial networks. In ar X iv preprint ar X iv:1406.2661 , 2014

    2014 arXiv

  17. [25]

    Deep Learning, Chapter 14

    Goodfellow, I., Bengio, Y., and Courville, A. Deep Learning, Chapter 14. MIT Press, 2016

    2016

  18. [26]

    and Zemel, R

    Hinton, G. and Zemel, R. Autoencoders, minimum description length and helmholtz free energy. Advances in Neural Information Processing Systems, 6, 1993

    1993

  19. [27]

    Denoising diffusion probabilistic models

    Ho, J., Jain, A., and Abbeel, P. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33: 0 6840--6851, 2020

    2020

  20. [28]

    and Pfister, J.-P

    Horvat, C. and Pfister, J.-P. On gauge freedom, conservativity and intrinsic dimensionality estimation in diffusion models. arXiv preprint arXiv:2402.03845, 2024

    2024 arXiv

  21. [29]

    L., Hosseinzadeh, R., Cresswell, J

    Kamkari, H., Ross, B. L., Hosseinzadeh, R., Cresswell, J. C., and Loaiza-Ganem, G. A geometric view of data complexity: Efficient local intrinsic dimension estimation with diffusion models. arXiv preprint arXiv:2406.03537, 2024

    2024 arXiv

  22. [30]

    and Welling, M

    Kingma, D. and Welling, M. Auto-encoding variational B ayes. In International Conference on Learning Representations, 2014

    2014

  23. [31]

    Variational diffusion models

    Kingma, D., Salimans, T., Poole, B., and Ho, J. Variational diffusion models. Advances in Neural Information Processing Systems, 34: 0 21696--21707, 2021

    2021

  24. [32]

    Sparse autoencoders reveal universal feature spaces across large language models

    Lan, M., Torr, P., Meek, A., Khakzar, A., Krueger, D., and Barez, F. Sparse autoencoders reveal universal feature spaces across large language models. arXiv preprint arXiv:2410.06981, 2024

    2024 arXiv

  25. [33]

    Mod\`eles connexionnistes de l'apprentissage

    LeCun, Y. Mod\`eles connexionnistes de l'apprentissage. PhD thesis, Université de Paris VI, 1987

    1987

  26. [34]

    Understanding diffusion models: A unified perspective

    Luo, C. Understanding diffusion models: A unified perspective. arxiv 2022. arXiv preprint arXiv:2208.11970

    2022 arXiv

  27. [35]

    and Frey, B

    Makhzani, A. and Frey, B. K-sparse autoencoders. arXiv preprint arXiv:1312.5663, 2013

    2013 arXiv

  28. [36]

    and Nickel, M

    Mathieu, E. and Nickel, M. Riemannian continuous normalizing flows. Advances in Neural Information Processing Systems, 33: 0 2503--2515, 2020

    2020

  29. [37]

    Sparse autoencoders for hypothesis generation

    Movva, R., Peng, K., Garg, N., Kleinberg, J., and Pierson, E. Sparse autoencoders for hypothesis generation. arXiv preprint arXiv:2502.04382, 2025

    2025 arXiv

  30. [38]

    Exploring generalization in deep learning

    Neyshabur, B., Bhojanapalli, S., McAllester, D., and Srebro, N. Exploring generalization in deep learning. Advances in neural information processing systems, 30, 2017

    2017

  31. [39]

    Sparse autoencoder

    Ng, A. Sparse autoencoder. CS294A Lecture notes, 72 0 (2011): 0 1--19, 2011

    2011

  32. [40]

    X., Duderstadt, B., and Mulyar, A

    Nussbaum, Z., Morris, J. X., Duderstadt, B., and Mulyar, A. Nomic embed: Training a reproducible long context text embedder, 2024

    2024

  33. [41]

    O'Neill, C., Ye, C., Iyer, K., and Wu, J. F. Disentangling dense embeddings with sparse autoencoders. arXiv preprint arXiv:2408.00657, 2024

    2024 arXiv

  34. [42]

    Sparse autoencoders learn monosemantic features in vision-language models

    Pach, M., Karthik, S., Bouniot, Q., Belongie, S., and Akata, Z. Sparse autoencoders learn monosemantic features in vision-language models. arXiv preprint arXiv:2504.02821, 2025

    2025

  35. [43]

    Variational EM algorithms for non- G aussian latent variable models

    Palmer, J., Wipf, D., Kreutz-Delgado, K., and Rao, B. Variational EM algorithms for non- G aussian latent variable models. Advances in Neural Information Processing Systems, pp.\ 1059--1066, 2006

    2006

  36. [44]

    Ponti, E. M. and Martins, A. Dynamic sparsity in machine learning. NeurIPS Tutorial, 2024

    2024

  37. [45]

    The intrinsic dimension of images and its impact on learning

    Pope, P., Zhu, C., Abdelkader, A., Goldblum, M., and Goldstein, T. The intrinsic dimension of images and its impact on learning. arXiv preprint arXiv:2104.08894, 2021

    2021 arXiv

  38. [46]

    Efficient learning of sparse representations with an energy-based model

    Ranzato, M., Poultney, C., Chopra, S., and LeCun, Y. Efficient learning of sparse representations with an energy-based model. Advances in Neural Information Processing Systems, 19, 2006

    2006

  39. [47]

    Sparse feature learning for deep belief networks

    Ranzato, M., Boureau, Y.-L., LeCun, Y., et al. Sparse feature learning for deep belief networks. Advances in Neural Information Processing Systems, 20, 2007

    2007

  40. [48]

    J., Mohamed, S., and Wierstra, D

    Rezende, D. J., Mohamed, S., and Wierstra, D. Stochastic backpropagation and approximate inference in deep generative models. In International Conference on Machine Learning, 2014

    2014

  41. [49]

    Learning internal representations by error propagation

    Rumelhart, D., Hinton, G., and Williams, R. Learning internal representations by error propagation. Parallel Distributed Processing: Explorations in the Microstructures of Cognition, MIT Press, Vol. I: 0 318--362, 1986

    1986

  42. [50]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pp.\ 2256--2265, 2015

    2015

  43. [51]

    P., Batzolis, G., Deveney, T., and Sch \"o nlieb, C.-B

    Stanczuk, J. P., Batzolis, G., Deveney, T., and Sch \"o nlieb, C.-B. Diffusion models encode the intrinsic dimension of data manifolds. In Forty-first International Conference on Machine Learning, 2024

    2024

  44. [52]

    Sparse autoencoders for scientifically rigorous interpretation of vision models

    Stevens, S., Chao, W.-L., Berger-Wolf, T., and Su, Y. Sparse autoencoders for scientifically rigorous interpretation of vision models. arXiv preprint arXiv:2502.06755, 2025

    2025

  45. [53]

    Lidl: Local intrinsic dimension estimation using approximate likelihood

    Tempczyk, P., Michaluk, R., Garncarek, L., Spurek, P., Tabor, J., and Golinski, A. Lidl: Local intrinsic dimension estimation using approximate likelihood. In International Conference on Machine Learning, pp.\ 21205--21231. PMLR, 2022

    2022

  46. [54]

    Marginalization is not marginal: N o bad VAE local minima when learning optimal sparse representations

    Wipf, D. Marginalization is not marginal: N o bad VAE local minima when learning optimal sparse representations. In International Conference on Machine Learning, pp.\ 37108--37132, 2023

    2023

  47. [55]

    Learning discriminative reconstructions for unsupervised outlier removal

    Xia, Y., Cao, X., Wen, F., Hua, G., and Sun, J. Learning discriminative reconstructions for unsupervised outlier removal. In Proceedings of the IEEE international conference on computer vision, pp.\ 1511--1519, 2015

    2015

  48. [56]

    Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms

    Xiao, H., Rasul, K., and Vollgraf, R. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017

    2017 arXiv

  49. [57]

    Zheng, Y., He, T., Qiu, Y., and Wipf, D. P. Learning manifold dimensions with conditional variational autoencoders. Advances in Neural Information Processing Systems, 35: 0 34709--34721, 2022

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.