Pith. sign in

REVIEW 4 major objections 6 minor 15 references

Improve variational autoEncoder with auxiliary softmax multiclassifier

T0 review · 4 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that appending an auxiliary softmax multiclassifier to a variational autoencoder makes the mutual information between data and latent codes estimable and controllable, giving a direct handle on posterior collapse and…

desk verdict A VAE mutual-information control scheme with an appealing empirical story, but the central estimator is unvalidated and the claimed equivalence to MINE is not correct. read the letter →

arxiv 1908.06966 v3 pith:UZYELXCB submitted 2019-08-17 stat.ML cs.LG

Yao Li This is my paper · ORCID
classification stat.MLcs.LG
keywords variationalautoencoderposteriorcollapsemutualinformationauxiliaryclassifiersoftmaxhierarchicalmarginalKLdivergenceVAE-AS
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Variational autoencoders often stop using their latent codes, a failure called posterior collapse, and their images come out blurred. This paper argues that both symptoms follow from a lack of control over the mutual information between the input and the latent code. It proposes VAE-AS, which adds a softmax multiclassifier that learns to predict which training sample a latent code came from. That classifier supplies estimates of both the mutual information and the marginal KL divergence between the aggregated posterior and the prior, and those estimates become tunable terms in the training objective. Experiments on MNIST and Omniglot are offered as evidence that the added control changes the number of active latent units and the sharpness of reconstructions, which the paper identifies with relieving posterior collapse.

What carries the argument

The load-bearing object is the auxiliary softmax multiclassifier $s_\omega(\hat e|z)$, a neural network that maps a sampled latent code $z$ to a probability distribution over one-hot sample labels. Trained by cross-entropy, it is assumed to converge to the true conditional $q_\phi(x^{(i)}|z)$; equation (9) turns that conditional into the mutual information, and equation (14) turns it into the aggregated posterior via Bayes' rule. This converts an intractable integral over a high-dimensional mixture distribution into a supervised classification problem. Because the cross-entropy loss is itself a MINE-type lower bound on mutual information, the classifier is simultaneously the estimator and part of the optimization target. A hierarchical softmax binary tree is used to reduce the classifier's cost from $O(hN)$ to $O(h\log_2 N)$.

What would settle it

Take a trained VAE-AS encoder, compute the classifier's mutual-information estimate, and compare it with a high-sample Monte Carlo estimate of $I_{q_\phi}(z,x)$ on the same encoder; the paper predicts agreement up to the Fano error bound, so a systematic gap that grows on held-out or duplicated labels would falsify the central claim.

Watch

Extended reading notes

Core claim

The paper's central claim is that an auxiliary softmax classifier can reveal the two quantities a VAE normally cannot see: the mutual information $I_{q_\phi}(z,x)$ and the marginal divergence $D_{KL}(q_\phi(z)\|p_\theta(z))$. For an empirical sample distribution over $N$ training points, the mutual information is written as $\log N - \mathbb{E}_{q_\phi(z)}[H(q_\phi(x|z))]$, so the missing ingredient is just the conditional distribution $q_\phi(x|z)$. A softmax network $s_\omega(\hat e|z)$ trained by cross-entropy against one-hot sample labels is claimed to converge to that conditional, and Bayes' rule then gives the aggregated posterior $q_\phi(z)$. Using these estimates, the ELBO is rewritten with separate Lagrange multipliers $\alpha$ for mutual information and $\beta$ for the marginal divergence, so the user can choose how much information the latent code keeps. The paper further claims that the classifier's cross-entropy objective is equivalent to the MINE lower bound, and bounds the estimation error by the classifier's error rate times $\log N$.

Load-bearing premise

The whole method rests on the auxiliary classifier actually learning the true conditional probability that a latent code came from each training sample; if it overfits or cannot represent that conditional, the mutual-information and marginal-KL estimates are biased and the alpha and beta controls optimize a miscalibrated objective.

Editorial extensions

If this is right

  • The ELBO gains two independent knobs: $\alpha$ controls how much information a latent code retains about its input, and $\beta$ controls how far the aggregate code distribution sits from the prior.
  • Because the auxiliary classifier's objective is the same lower bound the MINE estimator maximizes, no adversarial loop or additional sampling scheme is needed; estimation and optimization are the same computation.
  • The hierarchical softmax version makes the method scale to large label sets, since the classification cost grows logarithmically with the number of training samples rather than linearly.
  • On MNIST and Omniglot the paper reports that lowering $\beta$ increases the number of active latent units and sharpens reconstructions at the cost of generalization, while lowering $\alpha$ has a similar sharpening effect, giving a practical lever for the blur-quality tradeoff.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that the method turns unsupervised VAE training into a self-supervised classification task: any encoder that makes arbitrary sample identities easy to read off from $z$ is implicitly maximizing mutual information, so the same idea could serve as a generic representation-learning objective.
  • The bound that using $V$ labels caps the available mutual-information budget at $\log V$ suggests that for very large datasets the label count, not the encoder, may become the limiting factor; this is not tested in the MNIST and Omniglot experiments.
  • A testable extension is to swap the softmax classifier for other probabilistic classifiers, such as noise-contrastive or energy-based ones, and check whether the same ELBO surgery still holds; the claimed equivalence with MINE suggests the choice of classifier family should not change the core result.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes VAE-AS, a variational autoencoder augmented with an auxiliary softmax multiclassifier that is trained to predict the identity of the training sample from the latent code z. The classifier's output is used to estimate the mutual information I_φ(z,x) and the marginal KL divergence D_KL(q_φ(z)||p_θ(z)), which are then incorporated into the ELBO with adjustable weights α and β to control posterior collapse and image blurring. The authors claim that the auxiliary classifier makes the model equivalent to MINE, that it provides accurate estimates of mutual information, and that experiments on MNIST and Omniglot confirm its effectiveness. The paper also proposes a hierarchical softmax approximation to reduce the classifier's computational cost.

Significance. The proposed control of mutual information via a sample-identity classifier is a potentially practical heuristic, but the manuscript does not establish its theoretical or empirical validity. The claimed equivalence with MINE is not demonstrated and is in fact incorrect as stated because the averaged softmax loss converges to a different functional than the Donsker-Varadhan bound. The mutual-information estimate is subject to an unaddressed memorization bias of the training-set labels, and the only supporting experiments lack ground-truth validation and statistical rigor. The paper's decomposition of the ELBO into mutual information and marginal KL is standard, so the novelty rests entirely on the auxiliary classifier idea, which is not sufficiently validated. I cannot recommend acceptance.

major comments (4)
  1. [3.4, Theorem 3.2] The proof of Theorem 3.2 does not establish equivalence between VAE-AS and MINE. After applying log to the softmax probability, the empirical objective becomes (1/NL) Σ_i Σ_l [ T_ψ(x^(i), z^(l)) − log( (1/N) Σ_j exp(T_ψ(x^(j), z^(l))) ) ]. As L,N → ∞ this converges to E_q(x,z)[T_ψ(x,z)] − E_q(z)[ log E_q(x) exp(T_ψ(x,z)) ], whereas the MINE estimator in Eq. (8) contains the term − log E_q(z) E_q(x) exp(T_ψ(x,z)). Jensen's inequality gives E_z log E_x exp(T) ≤ log E_z E_x exp(T), with equality only in degenerate cases, so the two objectives are different in general. The paragraph following Eq. (12) tries to ignore this by claiming z^(l) is independent of x^(j), but that independence only affects the inner expectation, not the location of the logarithm. Therefore the central theoretical claim of the paper is not supported and must be corrected or removed.
  2. [3.3, Eq. (9)–(13)] The mutual-information estimator in Eq. (9) is only valid if the auxiliary classifier s_ω(e|z) converges to the true conditional q_φ(x|z) of Eq. (10). The paper provides no convergence theorem and no empirical check of this convergence. Because the labels e^(i) are arbitrary indices assigned to the training samples, and because s_ω is trained on the same samples used to fit q_φ, a high-capacity classifier can memorize the training set: the training cross-entropy (SC) can be driven to a small value even when the q_φ(z|x^(i)) distributions have large overlap. The values in Table 2 (MI≈10.78, SC≈0.35, log 55,000≈10.91) are exactly what a near-deterministic classifier would produce, not evidence that the true H(q_φ(x|z)) has been measured. The α/β control loop in Eq. (3) then optimizes a miscalibrated proxy. The manuscript needs a held-out validation of s_ω against Eq. (10), or a theoretical bias bound that accounts for training-set memorization, before the reported MI values can be trusted.
  3. [3.5, Theorem 3.3] Theorem 3.3's statement contains a clear algebraic error: it defines \hat I = log N + E_{q(z)}[P_e log P_e + (1−P_e) log P_e], but the second term should be (1−P_e) log(1−P_e) for a binary entropy term. As printed, the estimator is not a function of the confusion distribution and cannot be an estimate of conditional entropy. The proof then asserts \hat I − I ≤ P_e log N without showing how the Fano bound on H(q(x|z)) translates into an upper bound on the difference between the proposed \hat I and the true MI; the direction of the inequality is also not established. Since this theorem is the basis for the computational shortcut used in the experiments, it must be corrected and the bound re-derived in a fully rigorous way.
  4. [5.2–5.3, Tables 2 and 3] The empirical section does not validate the claim that VAE-AS accurately estimates mutual information and marginal KL. No ground-truth MI is available for a trained VAE, and Figure 2 only compares the VAE-AS estimate with Monte Carlo estimates at different sample sizes; the Monte Carlo estimates are themselves biased and do not provide a reference value. The tables report only single runs, with no variance or significance information, and the central metric AU (active units) is a heuristic proxy for posterior collapse. To establish the paper's main thesis, the authors should validate the estimator on synthetic data with known MI, quantify the classifier's generalization to held-out samples, and show that changes in α and β move the true (or at least an independently estimated) MI in the intended direction, rather than merely changing the classifier's training accuracy.
minor comments (6)
  1. [2.2, Table 1] The header 'ecoder Layers' should be 'Decoder Layers'.
  2. [3.5] The phrase 'bianry tree' should read 'binary tree', and 'V layers binary tree' should be 'V-level binary tree'.
  3. [Figure 4] The caption lists panels (a)–(f) in an order that does not match the panel layout; please relabel the panels or the caption to make the correspondence explicit.
  4. [5.3] The term 'NLLtest is a non-negative likelihood' is non-standard and appears to mean negative log-likelihood; please clarify the definition.
  5. [5.3, Table 2] The column 'NNLtrain' is presumably a typo for 'NLLtrain'; please standardize.
  6. [3.3, Eq. (9)] Eq. (9) is derived under the empirical distribution q_φ(x)=1/N; this assumption should be stated when the equation is introduced, not only in the appendix.

Circularity Check

2 steps flagged · score 6.0 of 10

Classifier-based MI estimate reduces to the classifier's own training loss; reported MI near log N is a memorization artifact, so the central mutual-information control claim is partly circular.

  1. fitted input called prediction [Section 3.3, Eq. 9 and Eq. 12; Section 5.3, Table 2]
    "Iqφ(z, x) = logN− Eqφ(z) [H (qφ(x| z))] (9) ... Minimizing the cross entropy loss function, We hopesω(ˆe| z) will gradually converge to qφ(x| z). After completing the fitting of the distributionqφ(x| z), we can calculate the entropyH (qφ(x| z)) ... then we can calculate the mutual informationIqφ(z, x) with equation 9."

    Equation 9 defines the mutual-information estimate as log N minus the average entropy of qφ(x|z). The paper then replaces qφ(x|z) with the auxiliary softmax classifier s_ω, which is trained by cross-entropy (Eq. 12) on one-hot labels for the same training samples. Therefore the reported I is, up to the entropy of s_ω, a monotone transform of the classifier's training loss: as the classifier memorizes the sample labels, the estimated entropy goes to 0 and the estimated mutual information is forced toward log N by construction, regardless of the true overlap of the encoder conditionals. Table 2 confirms this: whenever SC is small (0.11-0.35), MI is 10.78-10.87, essentially the log 55,000 ceiling of about 10.91.

  2. fitted input called prediction [Section 3.5, Theorem 3.3]
    "Theorem 3.3. For each x(i), Note the probability of multi-classifier prediction error P(ê_i ≠ e_i | z) = s_ω(ê(i) ≠ e(i) | z) as P(i)_e, let P_e = 1/N Σ_i P(i)_e. For VAE-AS, let Î(z, x) = logN + E_qφ(z)[ P(i)_e log P(i)_e + (1− P(i)_e) log (P(i)_e) ] be an estimate of I_qφ(z, x). ... With optimization, the predicted error rate Pe will gradually decrease, The gap between Î(z, x) and Iqφ(z, x) will be gradually reduced to 0."

    The 'estimate' Î is defined directly in terms of P_e, the classifier's training-set error probability. Because the classifier is trained to minimize that error, Î is driven toward log N by construction as P_e decreases; the claimed convergence 'gap between Î and I will be gradually reduced to 0' simply restates that a memorizing classifier has near-zero entropy, not that the true mutual information has been measured. Fano's inequality, used in the proof, would require P_e to be the true Bayes error of predicting X from Z, but the theorem uses the fitted classifier's in-sample error. Hence the theorem's estimator is a fitted error rate renamed as a mutual-information estimate, and the bound does not certify convergence to the true I_qφ(z,x).

full rationale

The paper's strongest empirical claim is that VAE-AS can estimate and control mutual information I(z,x) and marginal KL divergence, and thereby mitigate posterior collapse. The MI estimate in Eq. 9 is computed from a softmax classifier that is trained on the same sample labels used to define the categorical qφ(x); consequently the reported MI is essentially log N minus the classifier's fitted entropy. When the classifier memorizes, the MI estimate approaches its theoretical maximum log N, which is exactly what Table 2 shows (MI ≈ 10.78 with SC ≈ 0.35 for MNIST, log 55000 ≈ 10.91). Theorem 3.3 makes the reduction explicit by defining the estimate through the classifier's training error P_e. This is a fitted-input-called-prediction circularity in the central control claim. However, the paper also reports some externally meaningful metrics (NLL, active units AU) against vanilla VAE, and those comparisons are not themselves circular; there are no load-bearing self-citations. The score is therefore 6 rather than higher: the MI-based justification and the alpha/beta control loop reduce to the classifier fit, while some empirical reconstruction and collapse results retain independent content.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The method depends on user-chosen coefficients alpha and beta and on the unproven convergence of the auxiliary classifier to the true posterior over sample identities. The MI estimate is not anchored to any external benchmark; it is computed from the fitted classifier itself, so the chain of validation is internal.

free parameters (4)
  • alpha (coefficient for mutual information term in ELBO) = varied: 0.1, 0.2, 0.5, 1.0, 2.0, 5.0, 10.0
    Chosen by hand in Tables 2 and 3; controls the strength of the MI regularizer and strongly affects results.
  • beta (coefficient for marginal KL divergence term in ELBO) = varied: 0.1, 0.2, 0.5, 1.0, 2.0, 5.0
    Chosen by hand in Tables 2 and 3; controls the shape of the latent distribution and affects reconstruction and active units.
  • Number of softmax labels V = 55000 or 10000 on MNIST
    Selected by the authors; the maximum achievable mutual information is log V, so this caps the estimator and influences measured MI.
  • Threshold epsilon for active units (AU) = 0.01
    Used to define active dimensions in the AU metric; arbitrary but standard in prior work.
assumptions (6)
  • domain assumption The empirical data distribution q_phi(x) = 1/N over the training set is a valid substitute for the true data distribution.
    Used throughout the derivation (Eq. 4, 9, 14) to define empirical mutual information and aggregated posterior.
  • domain assumption The softmax classifier s_omega(e|z), trained by cross-entropy, converges to the true conditional q_phi(x|z).
    Section 3.3, Eq. 10-13; the entire MI estimate depends on this convergence, which is asserted without proof.
  • domain assumption The aggregated posterior q_phi(z) can be recovered from the classifier via Bayes' rule (Eq. 14).
    Section 3.3, Eq. 14; the identity relies on q_phi(x)=1/N and the classifier being exact.
  • standard math Fano's inequality applies to the classifier's error probability P_e.
    Used in Theorem 3.3 to bound the MI estimate.
  • standard math Donsker-Varadhan representation holds for the KL divergence.
    Theorem 3.1, used to connect MINE and the proposed estimator.
  • domain assumption The chain x -> z -> e_hat is Markov.
    Section 3.5, used to set up Fano's inequality for the classifier.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improve variational autoEncoder with auxiliary softmax multiclassifier." pith.science (2026). https://pith.science/paper/UZYELXCB

@misc{pith2026190806966,
  author       = {Pith},
  title        = {Pith review of: Improve variational autoEncoder with auxiliary softmax multiclassifier},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UZYELXCB}},
  note         = {Machine review of arXiv:1908.06966}
}
read the original abstract

As a general-purpose generative model architecture, VAE has been widely used in the field of image and natural language processing. VAE maps high dimensional sample data into continuous latent variables with unsupervised learning. Sampling in the latent variable space of the feature, VAE can construct new image or text data. As a general-purpose generation model, the vanilla VAE can not fit well with various data sets and neural networks with different structures. Because of the need to balance the accuracy of reconstruction and the convenience of latent variable sampling in the training process, VAE often has problems known as "posterior collapse". images reconstructed by VAE are also often blurred. In this paper, we analyze the main cause of these problem, which is the lack of mutual information between the sample variable and the latent feature variable during the training process. To maintain mutual information in model training, we propose to use the auxiliary softmax multi-classification network structure to improve the training effect of VAE, named VAE-AS. We use MNIST and Omniglot data sets to test the VAE-AS model. Based on the test results, It can be show that VAE-AS has obvious effects on the mutual information adjusting and solving the posterior collapse problem.

Figures

Figures reproduced from arXiv: 1908.06966 by the authors.

Figure 1
Figure 1. VAE-AS Network Structure Based on the structure of VAE, we add a multi layer neural network sω(eˆ | z) to fit the distribution qφ(x | z) . The network sω(eˆ | z) is trained in a supervised manner, we use the one-hot variable e as label, the activation function of the network sω(eˆ | z) is the softmax function. For each sampled z (l) , after a multi-layered neuron network transformation, let h (l) 0 = z (l) , h (l) t… view at source ↗
Figure 2
Figure 2. Estimate mutual information and marginal KL-Divergence. [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Estimate KL-Divergence. 10 [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Compare the effects of image reconstruction when [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]
Figure 5
Figure 5. Figure 5: Compare the effects of image reconstruction for Omniglot dataset when [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 2 canonical work pages

  1. [1]

    Deep variational information bottleneck

    Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410,

    arXiv

  2. [4]

    Understanding disentangling in β-vae

    Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Des- jardins, and Alexander Lerchner. Understanding disentangling in β-vae. arXiv preprint arXiv:1804.03599,

    arXiv

  3. [5]

    Variational lossy autoencoder

    Xi Chen, Diederik P Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, and Pieter Abbeel. Variational lossy autoencoder. arXiv preprint arXiv:1611.02731,

    arXiv

  4. [7]

    Lagging inference networks and posterior collapse in variational autoencoders

    Junxian He, Daniel Spokoyny, Graham Neubig, and Taylor Berg-Kirkpatrick. Lagging inference networks and posterior collapse in variational autoencoders. arXiv preprint arXiv:1901.05534,

    arXiv 1901

  5. [8]

    Auto-encoding variational bayes

    Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114,

    arXiv

  6. [11]

    Opening the black box of deep neural networks via informa- tion

    14 Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via informa- tion. arXiv preprint arXiv:1703.00810,

    arXiv

  7. [13]

    Wasserstein auto- encoders

    Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein auto- encoders. arXiv preprint arXiv:1711.01558,

    arXiv

  8. [14]

    Spherical latent spaces for stable variational autoencoders

    Jiacheng Xu and Greg Durrett. Spherical latent spaces for stable variational autoencoders. arXiv preprint arXiv:1808.10805,

    arXiv

Show all 15 references
  1. [15]

    Infovae: Information maximizing variational autoencoders

    Shengjia Zhao, Jiaming Song, and Stefano Ermon. Infovae: Information maximizing variational autoencoders. arXiv preprint arXiv:1706.02262, 2017a. Shengjia Zhao, Jiaming Song, and Stefano Ermon. Towards deeper understanding of variational autoencoding models. arXiv preprint arX...

    arXiv

  2. [1998]

    Mae: Mutual posterior-divergence regularization for variational autoencoders

    Xuezhe Ma, Chunting Zhou, and Eduard Hovy. Mae: Mutual posterior-divergence regularization for variational autoencoders. arXiv preprint arXiv:1901.01498,

    1901 arXiv

  3. [2015]

    Importance weighted autoencoders

    Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. arXiv preprint arXiv:1509.00519,

    arXiv

  4. [2016]

    Avoiding latent variable collapse with generative skip models

    Adji B Dieng, Yoon Kim, Alexander M Rush, and David M Blei. Avoiding latent variable collapse with generative skip models. arXiv preprint arXiv:1807.04863,

    arXiv

  5. [2017]

    How to train deep variational autoencoders and probabilistic ladder networks

    Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. How to train deep variational autoencoders and probabilistic ladder networks. In 33rd International Conference on Machine Learning (ICML 2016),

    2016

  6. [2018]

    Generating sentences from a continuous space

    13 Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Ben- gio. Generating sentences from a continuous space. arXiv preprint arXiv:1511.06349,

    arXiv

  7. [2019]

    Adversarial autoencoders

    Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders. arXiv preprint arXiv:1511.05644,

    arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.