Fixing a Broken ELBO
read the original abstract
Recent work in unsupervised representation learning has focused on learning deep directed latent-variable models. Fitting these models by maximizing the marginal likelihood or evidence is typically intractable, thus a common approximation is to maximize the evidence lower bound (ELBO) instead. However, maximum likelihood training (whether exact or approximate) does not necessarily result in a good latent representation, as we demonstrate both theoretically and empirically. In particular, we derive variational lower and upper bounds on the mutual information between the input and the latent variable, and use these bounds to derive a rate-distortion curve that characterizes the tradeoff between compression and reconstruction accuracy. Using this framework, we demonstrate that there is a family of models with identical ELBO, but different quantitative and qualitative characteristics. Our framework also suggests a simple new method to ensure that latent variable models with powerful stochastic decoders do not ignore their latent code.
This paper has not been read by Pith yet.
Forward citations
Cited by 4 Pith papers
-
Posterior Collapse as Automatic Spectral Pruning
Posterior collapse in β-VAEs is derived as automatic spectral pruning via Landau stability analysis, with collapse thresholds matching normalized PCA spectra in the linear Gaussian case and tested on WorldClim data.
-
Taming Audio VAEs via Target-KL Regularization
The paper introduces target-KL regularization to train audio VAEs at specific bitrates, enabling rate-distortion curves and comparison to discrete audio codecs for improved text-to-sound generation.
-
Shaping Belief States with Generative Environment Models for RL
Multi-step predictive generative models form stable belief states capturing environment layout and agent pose, yielding higher data efficiency on RL tasks than model-free agents.
-
Query-based Deep Improvisation
A VAE trained on one music style is queried with different-style input and uses rate-distortion noise to produce blended output with longer-term structure.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.