{"id":"271a09d6-1af8-43bd-bf84-bda654aba57e","arxiv_id":"2501.10901","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"ARD-VAE uses a hierarchical prior in the latent space whose per-axis variance is learned from the encoder, and a Jacobian-based relevance score, to identify how many latent dimensions a VAE actually needs.","lead":"This paper modifies the standard variational autoencoder (VAE) by replacing its fixed latent prior with a data-driven prior that shrinks unused latent dimensions, then uses the decoder's sensitivity to each dimension to count which dimensions matter. If the method works, practitioners could avoid trial-and-error choice of the VAE's latent space size.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim that the hierarchical prior discovers relevant dimensions is undercut by §3.4: raw σ̂² does not separate active and inactive axes, and the reported active counts come from the post-hoc Jacobian relevance score (Eq. 19–20).","rationale":"The paper is honest about the limitation: §3.4 explicitly states that the raw estimated variances for collapsed dimensions are low but non-zero, and that threshold selection on σ̂² is non-trivial, which is why the Jacobian weighting is introduced. This is not an internal inconsistency, but it does mean the reported active-dimension counts conflate two mechanisms: the hierarchical prior's variance estimation and the decoder-sensitivity-based Jacobian pruning. The supplementary's Tab. 13 strengthens this concern by showing that the same Jacobian score can prune plain VAE and other variants, yet no synthetic-data comparison is provided to show whether the ARD prior adds anything beyond the pruning rule. The dimensional inconsistency in Eq. (19) is a secondary correctness issue but does not change the central concern; the missing baseline is more load-bearing because it bears directly on whether the paper's stated mechanism causes the observed results. A conditional accept remains appropriate because the empirical FID/disentanglement results are strong and the concern is addressable with one additional experiment: if plain VAE+Jacobian fails to recover the ground-truth six factors, the paper's claim is substantially supported. If it succeeds, the title claim should be softened to describe the Jacobian-based pruning as the discovery mechanism, with the ARD prior as an auxiliary modeling choice.","tokens_in":26339,"tokens_out":3863,"duration_ms":41191,"concrete_test":"Using the same encoder/decoder, L=10, and 5 seeds used for Tab. 1, train a plain VAE with the standard N(0,I) prior on DSprites and 3D Shapes. After convergence, compute the Jacobian weight w_σ̂ from Eq. (19) using the correct column-wise per-axis norm, form the relevance score w_σ̂ ⊙ σ̂² with σ̂² set to the empirical variance of the plain VAE's encoded means, and apply the same 99% cumulative-variance threshold used for ARD-VAE. Count the active dimensions. If plain VAE+Jacobian also yields ≈5.8/6.4 active dims, the prior is not responsible for the discovery; if it yields roughly 10 active dims, the prior is essential.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The most load-bearing concern is that the paper's central claim — that the hierarchical prior discovers the relevant latent dimensions — is not actually supported by the experiments, because the reported active-dimension counts are produced by the post-hoc Jacobian relevance score in Eq. (19)–(20), not by the prior's estimated variances. Section 3.4 concedes that the raw variance σ̂² = b_L/a_L is non-zero and variable across spurious axes, so a threshold on σ̂² alone is non-trivial; the paper then replaces it with w_σ̂ ⊙ σ̂², where w_σ̂ is a decoder-sensitivity measure that could be computed for any trained VAE. The supplementary confirms this by using the same Jacobian score to prune dimensions for plain VAE, β-TCVAE, RAE, and WAE (Tab. 13). If the Jacobian term is what separates active from inactive axes, then the hierarchical prior is not the mechanism of relevancy detection, and the DSprites/3D Shapes results (5.80±0.40 and 6.40±0.49 active dims, Tab. 1) are evidence for the pruning rule, not for the ARD prior. The paper reports no synthetic-data baseline of a plain VAE with the same Jacobian pruning, so the unique contribution of the prior is untested.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ARD-VAE, a variational autoencoder with a hierarchical prior p(z|α) in which a Gamma hyperprior over per-axis precisions is estimated from encoded data Dz. The authors derive an approximate closed-form KL divergence for the resulting Student-t marginalized prior under a Gaussian approximation, train the model with the standard ELBO, and then determine active latent dimensions from the estimated variance σ̂² (Eq. 18) weighted by a decoder Jacobian norm w_σ̂ (Eq. 19), followed by a 99% variance threshold. Experiments on DSprites, 3D Shapes, MNIST, CelebA, CIFAR10, and ImageNet report active-dimension counts, FID, precision-recall, and disentanglement metrics, with comparisons to VAE, β-TCVAE, DIP-VAE, RAE, WAE, GECO-L0-ARM-VAE, and MaskAAE.","tokens_in":26584,"tokens_out":4695,"duration_ms":50999,"significance":"If the central claim held, the paper would be a practical contribution to automatic bottleneck selection in VAEs, with the attractive property of not modifying the ELBO apart from the prior. The work has real strengths: public code, a clean closed-form KL treatment, extensive ablations across latent sizes and architectures, and a consistent experimental protocol with multiple seeds. However, as presented, the evidence does not isolate the hierarchical prior as the mechanism of relevancy detection. The reported active-dimension counts are produced by the post-hoc Jacobian relevance score, not by the estimated variances, and the supplementary applies the same Jacobian pruning to plain VAE and other baselines. The FID improvements over plain VAE (e.g., CIFAR10 87.56 vs 147.74, ImageNet 121.21 vs 180.44) are interesting, but they are confounded by the fact that the active subset is selected post hoc; the missing control is the same Jacobian pruning rule applied to a plain VAE with a fixed prior.","major_comments":[{"comment":"The paper's central claim that the hierarchical prior identifies relevant latent dimensions is not the mechanism that produces the reported active counts. The text explicitly concedes that the raw estimated variance σ̂² has non-zero, variable values on spurious axes and that a threshold on σ̂² alone is non-trivial; the actual selection uses the relevance score σ̂²_w = w_σ̂ ⊙ σ̂² with a 99% threshold. The weight w_σ̂ in Eq. (19) is a Jacobian sensitivity of the decoder output with respect to the latent mean, which can be computed for any trained VAE, and the supplementary (Tab. 13) applies exactly this rule to plain VAE, β-TCVAE, RAE, and WAE. Therefore Tabs. 1–2 provide evidence for the Jacobian pruning rule, not for the ARD prior, unless a controlled comparison is added. Please report an experiment on DSprites and 3D Shapes in which a plain VAE trained with the same L is pruned with the same Eq. (19)–(20) rule, and show whether the ARD prior changes the active count or the quality of the selected axes relative to that baseline.","section":"§3.4, Eqs. (18)–(20)"},{"comment":"There is a circularity concern in the variance-based signal. The prior variance σ̂² = b_L/a_L is estimated from the encoder's own stochastic latent samples Dz, and the KL term in Eq. (17) is minimized when the posterior variance σ² equals σ̂². Thus an axis ignored by the decoder can sit at an arbitrary variance fixed point with near-zero KL penalty, and the paper provides no convergence argument showing that the variance of unused axes tends to zero. The observed separation of variances (Fig. 1b) is empirical, not a consequence of the statistical formulation alone. This gap matters because the claimed discovery mechanism is that irrelevant axes 'obtain very low variance'; without that property, Eq. (18) is meaningful only after the Jacobian weighting. Please supply either a fixed-point analysis of the update in Algorithm 1 showing that unused axes are driven to low variance, or a synthetic experiment where the ground-truth generative factors are known and the variance-only rule (without Jacobian weighting) is evaluated against a fixed-prior baseline.","section":"§3.2–3.3, Eqs. (8)–(17)"},{"comment":"The hyperprior parameters are estimated with µ_α set to zero, so b_L is proportional to the sum of squared latents E[z_l²] rather than the centered variance Var(z_l) unless the encoder outputs are zero-mean. Since Eq. (17) uses σ̂² = b_L/a_L as the target variance, a nonzero mean on an irrelevant or weakly used axis can inflate σ̂² and further weaken the variance-only relevance signal. Please either justify the zero-mean assumption or estimate µ_α from Dz when forming the sufficient statistics in Eq. (9).","section":"§3.3, Eqs. (8)–(9)"}],"minor_comments":[{"comment":"The caption says 'using 18' and 'using 20' instead of referencing Eq. (18) and Eq. (20).","section":"Fig. 1 caption"},{"comment":"There is an empty numbered equation (Eq. 15) after Eq. (14); it should be removed or filled.","section":"§3.2, Eq. (15)"},{"comment":"There are several typos in the supplementary, including 'MNSIT', 'CIAFR10', and the sentence fragment 'tasest. The choice of β' at the end of the Tab. 10 discussion.","section":"Supplementary, Tab. 10 discussion"},{"comment":"The header '2L 4LACTIVE' is malformed; the active-dimension columns need clear separation, and the caption should state that the bottleneck size is a multiple of L.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to be of interest to the VAE and dimensionality-selection community, but the title and abstract currently promise more than the experiments establish. The essential missing control is a plain VAE with the same Jacobian-based pruning rule; without it, the unique contribution of the ARD prior is untested. I would ask for that experiment and for a convergence or fixed-point argument for the variance behavior of unused axes before reconsidering."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a useful, well-run empirical study, but the headline claim doesn't survive contact with Section 3.4. The paper says the hierarchical prior identifies relevant dimensions; then it admits raw variance doesn't separate active from inactive axes and switches to a Jacobian-based relevance score. The reported counts in Tables 1 and 2 come from that post-hoc score, and the supplementary shows the same Jacobian score prunes plain VAEs, β-TCVAE, RAE, and WAE. So we don't actually know what the ARD prior contributes to the detection. The missing experiment is plain VAE plus the same Jacobian pruning on DSprites/3D Shapes; that would settle it.\n\nWhat's genuinely good: the empirical work is broad—synthetic with known ground truth, four real datasets, three distinct architectures (their own, MaskAAE's, GECO's), ablations on |Xα|, L, β, and a nice ARD-VAE-ALL comparison showing pruning costs little FID. The FID gains over the plain VAE on MNIST/CIFAR10/ImageNet are not just from pruning; ARD-VAE-ALL already beats the baseline, so the hierarchical prior changes the learned distribution in a useful way. The paper is also honest about the variance-threshold problem, which is more than many papers do. Code is provided.\n\nSoft spots, in order of importance. First, the conceptual one above: the prior isn't clearly the mechanism. Second, Eq. (19) is dimensionally muddled as written—sum over D of a term that doesn't depend on k—and needs correcting. Third, the \"single hyperparameter\" claim is generous; there is β, the 99% threshold, |Xα|, and the lag, though they ablate the latter two. That's a minor framing point. Fourth, there is no convergence argument for why spurious-axis variance should shrink; the empirical plots are reassuring but not proof.\n\nNone of these are fatal. The paper deserves a proper review. With the VAE+Jacobian baseline added and the framing adjusted from \"the prior discovers\" to \"the prior plus Jacobian pruning discovers,\" it could be a solid conditional accept.","headline":"Useful empirical package, overstated mechanism: the Jacobian, not the ARD prior, does the dimension selection, and the missing VAE+Jacobian baseline leaves the prior's contribution untested.","tokens_in":27136,"tokens_out":3699,"would_cite":true,"duration_ms":38792,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A VAE with a data-learned hierarchical prior automatically finds the relevant latent dimensions without changing the ELBO.","keywords":["automatic relevance determination","hierarchical prior","variational autoencoder","latent dimensionality","Student-t prior","disentanglement","generative model","intrinsic dimension"],"falsifier":"Train the ARD-VAE on a synthetic dataset with a known number of generative factors and track the estimated variances over training. If the axes the decoder ignores do not drift to clearly lower variance than active axes, for example if their variance sits at arbitrary fixed points, or if the number of active dimensions depends strongly on the Jacobian weighting, then the hierarchical prior alone is not carrying the discovery.","tokens_in":26075,"feed_emoji":"🎯","tokens_out":11059,"duration_ms":99813,"temperature":0.7,"pith_summary":"The paper argues that a VAE can discover how many latent factors a dataset really needs by replacing the fixed Gaussian prior with a hierarchical prior whose per-axis precision is estimated from the data the encoder produces, leaving the rest of the ELBO untouched. On synthetic datasets with six known generative factors and a latent space of size ten, the method reports $5.80 \\pm 0.40$ and $6.40 \\pm 0.49$ active dimensions, close to the ground truth of six. On real datasets it uses fewer dimensions than a plain VAE and improves FID on MNIST (22.24 vs 28.78), CIFAR10 (87.56 vs 147.74), and ImageNet (121.21 vs 180.44), while staying comparable on CelebA. If true, this gives practitioners a principled way to size the bottleneck for a new dataset without trial-and-error retraining.","feed_headline":"VAEs can now find their own relevant latent dimensions","feed_subtitle":"Swap the fixed prior for a learned one, and the VAE prunes unused axes and improves FID on MNIST, CIFAR10, ImageNet.","key_machinery":"The carrying object is a hierarchical prior on the latent space: $p(z\\mid\\alpha)=\\prod_l \\mathcal{N}(z_l;0,\\alpha_l^{-1})$ with $p(\\alpha)=\\prod_l \\mathrm{Gamma}(\\alpha_l;a_l^0,b_l^0)$. Integrating out $\\alpha$ gives a per-axis Student-$t$ marginal whose heavy tails encourage sparsity, and the conjugate Gaussian-Gamma update produces posterior parameters $a_l=a_l^0+n/2$ and $b_l=b_l^0+\\frac{1}{2}\\sum_i (z_i^l)^2$ from the encoded data $D_z$. The estimated variance $\\hat{\\sigma}^2=b_L/a_L$ is the raw relevance signal, and the final relevance score multiplies it by $\\mathbf{w}_{\\hat{\\sigma}}$, the average magnitude of the decoder Jacobian with respect to each latent mean; this weighting suppresses noise on dimensions whose changes do not alter the decoder output.","core_discovery":"The central claim is that the relevant latent dimensions of a VAE can be identified by a data-dependent prior rather than by a trainable mask or an added regularizer. Setting $p(z\\mid \\alpha)=\\prod_l \\mathcal{N}(z_l;0,\\alpha_l^{-1})$ with a Gamma hyperprior on each precision $\\alpha_l$, then updating the Gamma parameters from the encoded latent codes using conjugate formulas, yields per-axis estimated variances $\\hat{\\sigma}^2=b_L/a_L$ that separate the axes the decoder uses from the axes it ignores. The paper shows empirically that the minimum estimated variance sits far below the maximum across training, and that a Jacobian-weighted relevance score $\\mathbf{w}_{\\hat{\\sigma}}\\odot \\hat{\\sigma}^2$ cleanly separates active from inactive axes and stays stable as the initial latent size $L$ grows. Consequently, the ARD-VAE estimates active axes close to the known ground truth on DSprites and 3D Shapes, and produces better or comparable generative samples on real benchmarks.","pith_inferences":["If the variance-collapse behaviour is generic rather than dataset-specific, the ARD-VAE is effectively an intrinsic-dimensionality estimator for high-dimensional data, and could be compared directly with classical intrinsic-dimension estimators on the learned latent codes.","The Jacobian-weighting principle, that an axis matters only if the decoder output changes along it, is separable from the hierarchical prior and could be applied to any generative latent-variable model, including ones trained by other objectives.","A testable prediction of the paper is that the number of active dimensions stays roughly constant as the initial latent size grows; if it instead keeps growing with L, the procedure is measuring model capacity rather than the data's generative complexity.","Because $\\beta$ is tuned to reconstruction quality, the reported separation between active and inactive axes may shift with the reconstruction-regularization tradeoff, an interaction the paper does not quantify."],"forward_implications":["A fixed reconstruction-balance hyperparameter $\\beta$ carries over when the initial latent size is varied (L, 2L, 4L) on MNIST and CIFAR10, so the user does not need to re-tune $\\beta$ for each candidate bottleneck.","On DSprites and 3D Shapes, the estimated active dimensions stay near the ground-truth value of six for initial sizes 10, 15, 20, and 30.","Pruning to the active dimensions loses little information: FID with the active axes is only slightly worse than FID using all latent axes on MNIST, CIFAR10, and ImageNet.","The method trains on ImageNet at 32x32 resolution in about 17 hours on a single 12GB GPU, indicating it scales to large datasets."],"supporting_citations":[{"why":"Defines the VAE ELBO and posterior parameterisation that the method extends.","marker":"[19]"},{"why":"Establishes the reparameterized variational inference machinery the objective relies on.","marker":"[41]"},{"why":"Introduces the hierarchical prior over latent variables that the ARD-VAE adopts.","marker":"[36]"},{"why":"Provides the Student-t marginalisation and closed-form KL derivations used in the formulation.","marker":"[3]"},{"why":"Supplies the sparse-Bayesian relevance determination perspective behind the variance-based selection.","marker":"[51]"},{"why":"Gives the conjugate Gaussian-Gamma posterior update used in Eqs. (8)-(9).","marker":"[35]"},{"why":"The L0-masked VAE baseline that the ARD-VAE must beat on active-dimension discovery.","marker":"[6]"},{"why":"The masking-vector AAE baseline that the ARD-VAE contrasts on architecture sensitivity.","marker":"[33]"},{"why":"Provides the MIG disentanglement metric and the beta-TCVAE comparison.","marker":"[50]"}],"fun_headline_variants":["VAE auto-prunes latent dimensions via hierarchical prior","Data-driven prior lets VAEs drop unused latent axes","VAE finds its own latent size, no manual tuning","Relevant latent dimensions emerge from VAE's own data","VAEs learn which latent axes matter from data alone"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The scheme depends on unused latent axes ending up with low estimated variance after training, so that active and inactive axes can be told apart by a variance cut-off; the paper shows this empirically but gives no convergence argument that the variance of ignored axes tends to zero.","fun_headline_variants_meta":{"raw":{"variants":["VAE auto-prunes latent dimensions via hierarchical prior","Data-driven prior lets VAEs drop unused latent axes","VAE finds its own latent size, no manual tuning","Relevant latent dimensions emerge from VAE's own data","VAEs learn which latent axes matter from data alone"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000189,"raw_usage":{"total_tokens":1362,"prompt_tokens":999,"completion_tokens":363,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":615,"completion_tokens_details":{"reasoning_tokens":285}},"tokens_in":615,"tokens_out":363,"duration_ms":4200,"temperature":1.0,"reasoning_tokens":285,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:52:50.196200+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the ARD-VAE on a synthetic dataset with a known number of generative factors and track the estimated variances over training. If the axes the decoder ignores do not drift to clearly lower variance than active axes, for example if their variance sits at arbitrary fixed points, or if the number of active dimensions depends strongly on the Jacobian weighting, then the hierarchical prior alone is not carrying the discovery.","supporting_citations":[{"cited_title":"Imagenet: A large-scale hierar- chical image database","cited_arxiv_id":null,"evidence_quote":"Provides the Student-t marginalisation and closed-form KL derivations used in the formulation."},{"cited_title":"Disentangling by factorising","cited_arxiv_id":null,"evidence_quote":"The L0-masked VAE baseline that the ARD-VAE must beat on active-dimension discovery."}],"review_version":1}