Pith. sign in

REVIEW 3 major objections 4 minor 2 cited by

ARD-VAE: A Statistical Formulation to Find the Relevant Latent Dimensions of Variational Autoencoders

T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A VAE with a data-learned hierarchical prior automatically finds the relevant latent dimensions without changing the ELBO.

desk verdict Useful empirical package, overstated mechanism: the Jacobian, not the ARD prior, does the dimension selection, and the missing VAE+Jacobian baseline leaves the prior's contribution untested. read the letter →

arxiv 2501.10901 v2 pith:SRDRYHH7 submitted 2025-01-18 cs.LG

Surojit Saha , Sarang Joshi , Ross Whitaker This is my paper · ORCID
classification cs.LG
keywords automaticrelevancedeterminationhierarchicalpriorvariationalautoencoderlatentdimensionalityStudent-tdisentanglementgenerativemodelintrinsicdimension
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that a VAE can discover how many latent factors a dataset really needs by replacing the fixed Gaussian prior with a hierarchical prior whose per-axis precision is estimated from the data the encoder produces, leaving the rest of the ELBO untouched. On synthetic datasets with six known generative factors and a latent space of size ten, the method reports $5.80 \pm 0.40$ and $6.40 \pm 0.49$ active dimensions, close to the ground truth of six. On real datasets it uses fewer dimensions than a plain VAE and improves FID on MNIST (22.24 vs 28.78), CIFAR10 (87.56 vs 147.74), and ImageNet (121.21 vs 180.44), while staying comparable on CelebA. If true, this gives practitioners a principled way to size the bottleneck for a new dataset without trial-and-error retraining.

What carries the argument

The carrying object is a hierarchical prior on the latent space: $p(z\mid\alpha)=\prod_l \mathcal{N}(z_l;0,\alpha_l^{-1})$ with $p(\alpha)=\prod_l \mathrm{Gamma}(\alpha_l;a_l^0,b_l^0)$. Integrating out $\alpha$ gives a per-axis Student-$t$ marginal whose heavy tails encourage sparsity, and the conjugate Gaussian-Gamma update produces posterior parameters $a_l=a_l^0+n/2$ and $b_l=b_l^0+\frac{1}{2}\sum_i (z_i^l)^2$ from the encoded data $D_z$. The estimated variance $\hat{\sigma}^2=b_L/a_L$ is the raw relevance signal, and the final relevance score multiplies it by $\mathbf{w}_{\hat{\sigma}}$, the average magnitude of the decoder Jacobian with respect to each latent mean; this weighting suppresses noise on dimensions whose changes do not alter the decoder output.

What would settle it

Train the ARD-VAE on a synthetic dataset with a known number of generative factors and track the estimated variances over training. If the axes the decoder ignores do not drift to clearly lower variance than active axes, for example if their variance sits at arbitrary fixed points, or if the number of active dimensions depends strongly on the Jacobian weighting, then the hierarchical prior alone is not carrying the discovery.

Watch

Extended reading notes

Core claim

The central claim is that the relevant latent dimensions of a VAE can be identified by a data-dependent prior rather than by a trainable mask or an added regularizer. Setting $p(z\mid \alpha)=\prod_l \mathcal{N}(z_l;0,\alpha_l^{-1})$ with a Gamma hyperprior on each precision $\alpha_l$, then updating the Gamma parameters from the encoded latent codes using conjugate formulas, yields per-axis estimated variances $\hat{\sigma}^2=b_L/a_L$ that separate the axes the decoder uses from the axes it ignores. The paper shows empirically that the minimum estimated variance sits far below the maximum across training, and that a Jacobian-weighted relevance score $\mathbf{w}_{\hat{\sigma}}\odot \hat{\sigma}^2$ cleanly separates active from inactive axes and stays stable as the initial latent size $L$ grows. Consequently, the ARD-VAE estimates active axes close to the known ground truth on DSprites and 3D Shapes, and produces better or comparable generative samples on real benchmarks.

Load-bearing premise

The scheme depends on unused latent axes ending up with low estimated variance after training, so that active and inactive axes can be told apart by a variance cut-off; the paper shows this empirically but gives no convergence argument that the variance of ignored axes tends to zero.

Editorial extensions

If this is right

  • A fixed reconstruction-balance hyperparameter $\beta$ carries over when the initial latent size is varied (L, 2L, 4L) on MNIST and CIFAR10, so the user does not need to re-tune $\beta$ for each candidate bottleneck.
  • On DSprites and 3D Shapes, the estimated active dimensions stay near the ground-truth value of six for initial sizes 10, 15, 20, and 30.
  • Pruning to the active dimensions loses little information: FID with the active axes is only slightly worse than FID using all latent axes on MNIST, CIFAR10, and ImageNet.
  • The method trains on ImageNet at 32x32 resolution in about 17 hours on a single 12GB GPU, indicating it scales to large datasets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the variance-collapse behaviour is generic rather than dataset-specific, the ARD-VAE is effectively an intrinsic-dimensionality estimator for high-dimensional data, and could be compared directly with classical intrinsic-dimension estimators on the learned latent codes.
  • The Jacobian-weighting principle, that an axis matters only if the decoder output changes along it, is separable from the hierarchical prior and could be applied to any generative latent-variable model, including ones trained by other objectives.
  • A testable prediction of the paper is that the number of active dimensions stays roughly constant as the initial latent size grows; if it instead keeps growing with L, the procedure is measuring model capacity rather than the data's generative complexity.
  • Because $\beta$ is tuned to reconstruction quality, the reported separation between active and inactive axes may shift with the reconstruction-regularization tradeoff, an interaction the paper does not quantify.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes ARD-VAE, a variational autoencoder with a hierarchical prior p(z|α) in which a Gamma hyperprior over per-axis precisions is estimated from encoded data Dz. The authors derive an approximate closed-form KL divergence for the resulting Student-t marginalized prior under a Gaussian approximation, train the model with the standard ELBO, and then determine active latent dimensions from the estimated variance σ̂² (Eq. 18) weighted by a decoder Jacobian norm w_σ̂ (Eq. 19), followed by a 99% variance threshold. Experiments on DSprites, 3D Shapes, MNIST, CelebA, CIFAR10, and ImageNet report active-dimension counts, FID, precision-recall, and disentanglement metrics, with comparisons to VAE, β-TCVAE, DIP-VAE, RAE, WAE, GECO-L0-ARM-VAE, and MaskAAE.

Significance. If the central claim held, the paper would be a practical contribution to automatic bottleneck selection in VAEs, with the attractive property of not modifying the ELBO apart from the prior. The work has real strengths: public code, a clean closed-form KL treatment, extensive ablations across latent sizes and architectures, and a consistent experimental protocol with multiple seeds. However, as presented, the evidence does not isolate the hierarchical prior as the mechanism of relevancy detection. The reported active-dimension counts are produced by the post-hoc Jacobian relevance score, not by the estimated variances, and the supplementary applies the same Jacobian pruning to plain VAE and other baselines. The FID improvements over plain VAE (e.g., CIFAR10 87.56 vs 147.74, ImageNet 121.21 vs 180.44) are interesting, but they are confounded by the fact that the active subset is selected post hoc; the missing control is the same Jacobian pruning rule applied to a plain VAE with a fixed prior.

major comments (3)
  1. [§3.4, Eqs. (18)–(20)] The paper's central claim that the hierarchical prior identifies relevant latent dimensions is not the mechanism that produces the reported active counts. The text explicitly concedes that the raw estimated variance σ̂² has non-zero, variable values on spurious axes and that a threshold on σ̂² alone is non-trivial; the actual selection uses the relevance score σ̂²_w = w_σ̂ ⊙ σ̂² with a 99% threshold. The weight w_σ̂ in Eq. (19) is a Jacobian sensitivity of the decoder output with respect to the latent mean, which can be computed for any trained VAE, and the supplementary (Tab. 13) applies exactly this rule to plain VAE, β-TCVAE, RAE, and WAE. Therefore Tabs. 1–2 provide evidence for the Jacobian pruning rule, not for the ARD prior, unless a controlled comparison is added. Please report an experiment on DSprites and 3D Shapes in which a plain VAE trained with the same L is pruned with the same Eq. (19)–(20) rule, and show whether the ARD prior changes the active count or the quality of the selected axes relative to that baseline.
  2. [§3.2–3.3, Eqs. (8)–(17)] There is a circularity concern in the variance-based signal. The prior variance σ̂² = b_L/a_L is estimated from the encoder's own stochastic latent samples Dz, and the KL term in Eq. (17) is minimized when the posterior variance σ² equals σ̂². Thus an axis ignored by the decoder can sit at an arbitrary variance fixed point with near-zero KL penalty, and the paper provides no convergence argument showing that the variance of unused axes tends to zero. The observed separation of variances (Fig. 1b) is empirical, not a consequence of the statistical formulation alone. This gap matters because the claimed discovery mechanism is that irrelevant axes 'obtain very low variance'; without that property, Eq. (18) is meaningful only after the Jacobian weighting. Please supply either a fixed-point analysis of the update in Algorithm 1 showing that unused axes are driven to low variance, or a synthetic experiment where the ground-truth generative factors are known and the variance-only rule (without Jacobian weighting) is evaluated against a fixed-prior baseline.
  3. [§3.3, Eqs. (8)–(9)] The hyperprior parameters are estimated with µ_α set to zero, so b_L is proportional to the sum of squared latents E[z_l²] rather than the centered variance Var(z_l) unless the encoder outputs are zero-mean. Since Eq. (17) uses σ̂² = b_L/a_L as the target variance, a nonzero mean on an irrelevant or weakly used axis can inflate σ̂² and further weaken the variance-only relevance signal. Please either justify the zero-mean assumption or estimate µ_α from Dz when forming the sufficient statistics in Eq. (9).
minor comments (4)
  1. [Fig. 1 caption] The caption says 'using 18' and 'using 20' instead of referencing Eq. (18) and Eq. (20).
  2. [§3.2, Eq. (15)] There is an empty numbered equation (Eq. 15) after Eq. (14); it should be removed or filled.
  3. [Supplementary, Tab. 10 discussion] There are several typos in the supplementary, including 'MNSIT', 'CIAFR10', and the sentence fragment 'tasest. The choice of β' at the end of the Tab. 10 discussion.
  4. [Table 3] The header '2L 4LACTIVE' is malformed; the active-dimension columns need clear separation, and the caption should state that the bottleneck size is a multiple of L.

Circularity Check

2 steps flagged · score 6.0 of 10

The central 'automatic relevancy detection' claim reduces to a decoder-Jacobian sensitivity score: the reported active-dimension counts come from Eq. (20), not from the hierarchical prior's estimated variance, and the prior variance is itself a self-referential fitted target.

  1. self definitional [Section 3.4, Eqs. (19)-(20); supplementary 'Relevant axes for the VAE using the Jacobian']
    "For axes that are not relevant to the reconstruction, we have observed that the decoder produces virtually no variability in output in response to deviations along these axes. ... This motivates us to consider the deviation of the output, x̂ ∈ R^D (produced by the decoder Dθ), with respect to the mean representation, µx ∈ R^L (µx ← Eϕ(x)), as the measure of relevance. ... The weighted estimated variance defined as σ̂²_w = w_σ̂ ⊙ σ̂² (20) gives us the relevance score that is used to determine the relevant axes of the ARD-VAE."

    Relevance is defined, by construction, as decoder-output Jacobian sensitivity: Eq. (19) computes exactly how much the decoder output varies when each latent mean is perturbed, and Eq. (20) multiplies that by the prior variance. The paper then uses this score to count active dimensions. The supplementary applies the same Eq. (19) Jacobian to select active axes for a plain VAE, β-TCVAE, RAE, and WAE (Tab. 13), showing that the selection mechanism is generic and does not require the hierarchical prior. Thus the reported discovery of relevant latent dimensions reduces to a post-hoc sensitivity measure rather than to the ARD prior's learned variances.

  2. fitted input called prediction [Section 3.2-3.3, Eqs. (8)-(9) and (17); Algorithm 2 in the supplementary]
    "The parameters of the Gamma distribution are estimated analytically using data in the latent space, Dz, produced using the posterior distribution of the VAE estimated by the encoder, Eϕ (refer to algorithm 2 in the supp). ... where µ,σ²← Eϕ(x), x∈Xsgd and σ̂² =bL/aL,aL and bL are estimated using the latestDz."

    The quantity σ̂² that is supposed to reveal relevant latent axes is computed from the encoder's own latent samples: b_L is updated from Dz generated by the same encoder being trained. The KL term in Eq. (17) is minimized when the posterior variance equals σ̂², so the estimated variances are a self-consistent fixed point of the training objective rather than an externally grounded measure of relevance. The paper itself concedes in Section 3.4 that raw σ̂² cannot separate active from inactive axes and that thresholding it is non-trivial; the actual separation used for the reported counts comes from the Jacobian weight in Eq. (20). The fitted prior parameter is therefore not the source of the 'predicted' relevant dimensions.

full rationale

The main circularity is specific and quotable. Section 3.4 states that raw σ̂² is non-zero and variable on collapsed axes and that no threshold works; it then defines a relevance score as w_σ̂⊙σ̂², where w_σ̂ is the Jacobian of decoder outputs with respect to latent means. Since the supplementary uses Eq. (19) alone to determine active axes for plain VAE and other baselines, the active-dimension counts in the tables are produced by a generic decoder-sensitivity rule, not by the hierarchical prior. In addition, σ̂² is estimated from the encoder's own latent samples and then used as the KL target, so any low-variance behavior of inactive axes is a self-consistent training fixed point. Self-citations in the paper ([44,45]) are not load-bearing and do not contribute to this score. There is independent content: the ARD-VAE trained with all latent axes still improves FID over the plain VAE (supp. Tab. 11), and the disentanglement results may reflect real training changes. Therefore the paper is partially circular on its central discovery claim but not a fully tautological derivation; score 6.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim depends on two fitted hyperparameters (β and the 99% threshold) and on two heuristic assumptions (Jacobian relevance, variance collapse). The ARD prior and conjugate updates are standard. No new physical or probabilistic entities are introduced.

free parameters (4)
  • beta = MNIST 0.5; CelebA 1.0; CIFAR10 0.05; ImageNet 0.05; DSprites 5.0; 3D Shapes 5.0 (Tab. 6)
    Scales the KL divergence in the ELBO; the paper tunes it per dataset to match a target reconstruction loss (§3.3, Tab. 6).
  • active_dim_variance_threshold = 0.99 (99% of weighted variance)
    Dimensions whose cumulative weighted variance accounts for 99% are kept (§3.4); this threshold is chosen by hand and is not derived from the statistical formulation.
  • X_alpha_size = 10,000 samples
    Size of the subset used to estimate the Gamma posterior parameters; ablation shows low sensitivity, but the value is a user choice.
  • lag_period_uDz = 1 epoch
    The prior parameters are updated once per epoch; chosen for training stability (§3.3).
assumptions (5)
  • standard math The Gamma-Gaussian conjugate update (Eq. 8-9) yields the exact posterior for the precision α given latent samples, treating z_i as independent draws from N(0, α^{-1}).
    Invoked in §3.2 to derive a_L and b_L; valid if the likelihood is Gaussian.
  • standard math For large ν (= 2a_L ≈ 10,000), the Student-t prior is well approximated by a Gaussian with variance b_L/a_L, allowing the closed-form KL in Eq. (17).
    Invoked in §3.3; valid since the Student-t converges to Gaussian as ν grows.
  • domain assumption Latent dimensions are independent (diagonal covariance q(z|x)).
    Standard VAE assumption, stated in §3.1 Eq. (2).
  • ad hoc to paper The Jacobian norm of the decoder output with respect to a latent mean measures that axis's relevance (Eq. 19).
    This is a heuristic sensitivity measure introduced in §3.4; it is not derived from the ELBO or the hierarchical prior, and the paper admits raw variances alone do not separate axes.
  • ad hoc to paper Unused latent axes obtain low estimated variance after training.
    Section 3.4 relies on this to identify spurious dimensions, but the KL objective has fixed points with arbitrary σ̂² (KL=0 at σ²=σ̂²), so collapse is not guaranteed by the objective.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ARD-VAE: A Statistical Formulation to Find the Relevant Latent Dimensions of Variational Autoencoders." pith.science (2026). https://pith.science/paper/SRDRYHH7

@misc{pith2026250110901,
  author       = {Pith},
  title        = {Pith review of: ARD-VAE: A Statistical Formulation to Find the Relevant Latent Dimensions of Variational Autoencoders},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SRDRYHH7}},
  note         = {Machine review of arXiv:2501.10901}
}
read the original abstract

The variational autoencoder (VAE) is a popular, deep, latent-variable model (DLVM) due to its simple yet effective formulation for modeling the data distribution. Moreover, optimizing the VAE objective function is more manageable than other DLVMs. The bottleneck dimension of the VAE is a crucial design choice, and it has strong ramifications for the model's performance, such as finding the hidden explanatory factors of a dataset using the representations learned by the VAE. However, the size of the latent dimension of the VAE is often treated as a hyperparameter estimated empirically through trial and error. To this end, we propose a statistical formulation to discover the relevant latent factors required for modeling a dataset. In this work, we use a hierarchical prior in the latent space that estimates the variance of the latent axes using the encoded data, which identifies the relevant latent dimensions. For this, we replace the fixed prior in the VAE objective function with a hierarchical prior, keeping the remainder of the formulation unchanged. We call the proposed method the automatic relevancy detection in the variational autoencoder (ARD-VAE). We demonstrate the efficacy of the ARD-VAE on multiple benchmark datasets in finding the relevant latent dimensions and their effect on different evaluation metrics, such as FID score and disentanglement analysis.

Figures

Figures reproduced from arXiv: 2501.10901 by the authors.

Figure 1
Figure 1. (a) Relevance scores and (b) spread (min-max) of the estimated variances on the MNIST dataset while training the ARD-VAE. [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 1
Figure 1. The minimum and maximum variances estimated by the ARD-VAE while training on the MNIST, CelebA and CIFAR10 datasets [PITH_FULL_IMAGE:figures/full_fig_p012_1.png] view at source ↗
Figure 2
Figure 2. Latent traversal of the DSprites data set [ [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗
Figures from the paper (2 more)
Figure 3
Figure 3. Figure 3: Latent traversal of the 3D Shapes data set [ [PITH_FULL_IMAGE:figures/full_fig_p013_3.png]
Figure 4
Figure 4. Figure 4: Latent traversal of the 3D Shapes data set [ [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Disentanglement Analysis in Deep Latent Variable Models Matching Aggregate Posterior Distributions

    cs.LG 2025-01 conditional novelty 5.0 of 10

    The paper proposes a PCA-based method to find latent directions for each generative factor, creating disentanglement metrics that work for latent variable models with non-axis-aligned encodings, and reports improved s...

  2. AdaSemSeg: An Adaptive Few-shot Semantic Segmentation of Seismic Facies

    cs.CV 2025-01 conditional novelty 4.0 of 10

    A shared Gaussian-process few-shot network, split into per-class binary tasks and initialized with SimCLR, labels seismic facies in apparently unseen datasets from just 1 or 5 annotated slices.

Reference graph

Works this paper leans on

14 extracted references · 14 canonical work pages · cited by 2 Pith papers

  1. [1]

    3d shapes dataset

    Chris Burgess and Hyunjik Kim. 3d shapes dataset. https://github.com/deepmind/3d-shapes/, 2018. 3, 4, 5, 6

    work page 2018

  2. [2]

    Dynamic narrowing of vae bottle- necks using geco and l0 regularization

    Cedric De Boom, Samuel Wauthier, Tim Verbelen, and Bart Dhoedt. Dynamic narrowing of vae bottle- necks using geco and l0 regularization. In Interna- tional Joint Conference on Neural Networks (IJCNN),

    work page

  3. [3]

    Imagenet: A large-scale hierar- chical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierar- chical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248– 255, 2009. 6

    work page 2009

  4. [4]

    Partha Ghosh, Mehdi S. M. Sajjadi, Antonio Vergari, Michael Black, and Bernhard Scholk¨opf. From varia- tional to deterministic autoencoders. In International Conference on Learning Representations, 2020. 4, 8

    work page 2020

  5. [5]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium

    Martin Heusel, Hubert Ramsauer, Thomas Un- terthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Conference on Neural Information Processing Systems, 2017. 6

    work page 2017

  6. [6]

    Disentangling by factorising

    Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference on Machine Learning, 2018. 5

    work page 2018

  7. [7]

    Auto-encoding variational bayes

    Diederik P Kingma and Max Welling. Auto-encoding variational bayes. International Conference on Learn- ing Representations, 2014. 1, 3

    work page 2014

  8. [8]

    Variational inference of disentangled latent concepts from unlabeled observations

    Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentangled latent concepts from unlabeled observations. In In- ternational Conference on Learning Representations ,

    work page

Show all 14 references
  1. [9]

    Challenging common assumptions in the unsupervised learning of disentangled representa- tions

    Francesco Locatello, Stefan Bauer, Mario Lucic, Gun- nar R ¨atsch, Sylvain Gelly, Bernhard Sch ¨olkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representa- tions. In International Conference on Machine Learn- ing, 2019. 5

    2019

  2. [10]

    dsprites: Disentanglement testing sprites dataset

    Loic Matthey, Irina Higgins, Demis Has- sabis, and Alexander Lerchner. dsprites: Disentanglement testing sprites dataset. https://github.com/deepmind/dsprites-dataset/, 2017. 3, 5, 6

    2017

  3. [11]

    Maskaae: Latent space optimization for adversarial auto-encoders

    Arnab Kumar Mondal, Sankalan Pal Chowdhury, Ar- avind Jayendran, Parag Singla, Himanshu Asnani, and Prathosh AP. Maskaae: Latent space optimization for adversarial auto-encoders. In Uncertainty in Artificial Intelligence (UAI), 2020. 4

    2020

  4. [12]

    Stochastic backpropagation and approxi- mate inference in deep generative models

    Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approxi- mate inference in deep generative models. In Interna- tional Conference on Machine Learning, pages 1278– 1286, 2014. 1, 3

    2014

  5. [13]

    Mehdi S. M. Sajjadi, Olivier Bachem, Mario Lu ˇci´c, Olivier Bousquet, and Sylvain Gelly. Assessing Gen- erative Models via Precision and Recall. In Advances in Neural Information Processing Systems (NeurIPS),

  6. [14]

    Wasserstein auto-encoders

    Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelk ¨opf. Wasserstein auto-encoders. In International Conference on Learning Representa- tions, 2018. 4 9

    2018

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.