{"id":"f47bf343-48d6-4de8-a137-84ae6bc901fc","arxiv_id":"1908.06966","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"VAE-AS estimates mutual information in a VAE by fitting a softmax classifier that predicts sample identity from the latent code, and uses it to adjust the ELBO.","lead":"This paper proposes VAE-AS, a variational autoencoder with an auxiliary softmax classifier that estimates the mutual information between data and latent codes. The authors claim this controls posterior collapse and blurring, and they test it on MNIST and Omniglot.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Classifier-based MI estimate is unvalidated and likely biased; Eq. 9/3 control loop may optimize classifier overfitting rather than true mutual information.","rationale":"The central promise of VAE-AS is that the auxiliary softmax classifier makes I(z,x) and D_KL(q(z)||p(z)) estimable in training, and that the alpha/beta-weighted objective then controls posterior collapse. The load-bearing link is therefore the identity between the fitted classifier and q_phi(x|z): if s_omega is not a faithful estimate, every number in Table 2 labeled MI/MD, and every conclusion about mutual-information adjustment, is uninterpretable. This is not merely a missing proof: the classifier is trained discriminatively on the same arbitrary identity labels that define the empirical q_phi(x), and deep classifiers can memorize finite labels; the error bound in Theorem 3.3 is circular in this respect because it is stated in terms of the fitted error rate P_e. The reader's weakest assumption identifies the same point, and the proposed empirical check (comparing s_omega to the closed-form Eq. 10 posterior on a trained model) would settle it. The Theorem 3.2 issue is a separate mathematical error that further weakens the paper, but the classifier-calibration concern is the one that directly bears on the empirical central claim. Since the check has not been done and there is no code or comparison to existing MI-regularized VAEs, the reader's REJECT verdict remains appropriate.","tokens_in":15244,"tokens_out":14529,"duration_ms":156729,"concrete_test":"Take the trained VAE-AS encoder from Table 2 (alpha=beta=1). Draw a fixed set of z samples from q_phi(z|x_i) for a random subset of, say, 500 training images. Compute the exact conditional probabilities q_phi(x_j|z) = q_phi(z|x_j) / sum_k q_phi(z|x_k) using the Gaussian encoder densities (Eq. 10) for all j in the subset, and compare with the auxiliary classifier's s_omega(e_j|z). Report the average exact entropy H(q(x|z)) versus the classifier's entropy and cross-entropy, and repeat for a held-out split of labels. If the classifier entropy is significantly lower than the exact entropy, or if the training SC is much lower than the held-out SC, the MI estimate in Eq. 9 is biased upward and the alpha/beta control loop in Eq. 3 is miscalibrated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3 defines the MI estimate through Eq. 9 as log N - E[H(q_phi(x|z))], then replaces q_phi(x|z) with the softmax classifier s_omega(e|z) trained by cross-entropy. The entire control loop in Eq. 3 and the posterior-collapse claims in Section 5 depend on s_omega converging to the true conditional in Eq. 10. The paper gives no convergence proof, no held-out validation, and no unbiased comparison of s_omega against Eq. 10. Because the labels are arbitrary sample identities and the classifier is trained on the same finite training set, a sufficiently expressive network can drive training cross-entropy toward zero even when the true q_phi(x|z) is broad; then H(q(x|z)) is underestimated and I(z,x) is overestimated. The Table 2 values (MI near 10.78, SC near 0.35, log 55000 near 10.91) are consistent with a near-deterministic classifier, not necessarily with an accurate estimate of q_phi(x|z). Theorem 3.3 does not fix this: its bound uses P_e, the classifier's training-set error rate, which is exactly the quantity inflated by overfitting. Additionally, Theorem 3.2's claimed equivalence with MINE is incorrect (the softmax objective contains E_z log E_x e^T, not log E_z E_x e^T, and these differ by the Jensen gap), so the theoretical support for the estimator is also unsound. Without a check of the classifier against the true posterior, the reported MI and the alpha/beta adjustments cannot be attributed to real mutual-information control.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes VAE-AS, a variational autoencoder augmented with an auxiliary softmax multiclassifier that is trained to predict the identity of the training sample from the latent code z. The classifier's output is used to estimate the mutual information I_φ(z,x) and the marginal KL divergence D_KL(q_φ(z)||p_θ(z)), which are then incorporated into the ELBO with adjustable weights α and β to control posterior collapse and image blurring. The authors claim that the auxiliary classifier makes the model equivalent to MINE, that it provides accurate estimates of mutual information, and that experiments on MNIST and Omniglot confirm its effectiveness. The paper also proposes a hierarchical softmax approximation to reduce the classifier's computational cost.","tokens_in":15612,"tokens_out":7589,"duration_ms":67561,"significance":"The proposed control of mutual information via a sample-identity classifier is a potentially practical heuristic, but the manuscript does not establish its theoretical or empirical validity. The claimed equivalence with MINE is not demonstrated and is in fact incorrect as stated because the averaged softmax loss converges to a different functional than the Donsker-Varadhan bound. The mutual-information estimate is subject to an unaddressed memorization bias of the training-set labels, and the only supporting experiments lack ground-truth validation and statistical rigor. The paper's decomposition of the ELBO into mutual information and marginal KL is standard, so the novelty rests entirely on the auxiliary classifier idea, which is not sufficiently validated. I cannot recommend acceptance.","major_comments":[{"comment":"The proof of Theorem 3.2 does not establish equivalence between VAE-AS and MINE. After applying log to the softmax probability, the empirical objective becomes (1/NL) Σ_i Σ_l [ T_ψ(x^(i), z^(l)) − log( (1/N) Σ_j exp(T_ψ(x^(j), z^(l))) ) ]. As L,N → ∞ this converges to E_q(x,z)[T_ψ(x,z)] − E_q(z)[ log E_q(x) exp(T_ψ(x,z)) ], whereas the MINE estimator in Eq. (8) contains the term − log E_q(z) E_q(x) exp(T_ψ(x,z)). Jensen's inequality gives E_z log E_x exp(T) ≤ log E_z E_x exp(T), with equality only in degenerate cases, so the two objectives are different in general. The paragraph following Eq. (12) tries to ignore this by claiming z^(l) is independent of x^(j), but that independence only affects the inner expectation, not the location of the logarithm. Therefore the central theoretical claim of the paper is not supported and must be corrected or removed.","section":"3.4, Theorem 3.2"},{"comment":"The mutual-information estimator in Eq. (9) is only valid if the auxiliary classifier s_ω(e|z) converges to the true conditional q_φ(x|z) of Eq. (10). The paper provides no convergence theorem and no empirical check of this convergence. Because the labels e^(i) are arbitrary indices assigned to the training samples, and because s_ω is trained on the same samples used to fit q_φ, a high-capacity classifier can memorize the training set: the training cross-entropy (SC) can be driven to a small value even when the q_φ(z|x^(i)) distributions have large overlap. The values in Table 2 (MI≈10.78, SC≈0.35, log 55,000≈10.91) are exactly what a near-deterministic classifier would produce, not evidence that the true H(q_φ(x|z)) has been measured. The α/β control loop in Eq. (3) then optimizes a miscalibrated proxy. The manuscript needs a held-out validation of s_ω against Eq. (10), or a theoretical bias bound that accounts for training-set memorization, before the reported MI values can be trusted.","section":"3.3, Eq. (9)–(13)"},{"comment":"Theorem 3.3's statement contains a clear algebraic error: it defines \\hat I = log N + E_{q(z)}[P_e log P_e + (1−P_e) log P_e], but the second term should be (1−P_e) log(1−P_e) for a binary entropy term. As printed, the estimator is not a function of the confusion distribution and cannot be an estimate of conditional entropy. The proof then asserts \\hat I − I ≤ P_e log N without showing how the Fano bound on H(q(x|z)) translates into an upper bound on the difference between the proposed \\hat I and the true MI; the direction of the inequality is also not established. Since this theorem is the basis for the computational shortcut used in the experiments, it must be corrected and the bound re-derived in a fully rigorous way.","section":"3.5, Theorem 3.3"},{"comment":"The empirical section does not validate the claim that VAE-AS accurately estimates mutual information and marginal KL. No ground-truth MI is available for a trained VAE, and Figure 2 only compares the VAE-AS estimate with Monte Carlo estimates at different sample sizes; the Monte Carlo estimates are themselves biased and do not provide a reference value. The tables report only single runs, with no variance or significance information, and the central metric AU (active units) is a heuristic proxy for posterior collapse. To establish the paper's main thesis, the authors should validate the estimator on synthetic data with known MI, quantify the classifier's generalization to held-out samples, and show that changes in α and β move the true (or at least an independently estimated) MI in the intended direction, rather than merely changing the classifier's training accuracy.","section":"5.2–5.3, Tables 2 and 3"}],"minor_comments":[{"comment":"The header 'ecoder Layers' should be 'Decoder Layers'.","section":"2.2, Table 1"},{"comment":"The phrase 'bianry tree' should read 'binary tree', and 'V layers binary tree' should be 'V-level binary tree'.","section":"3.5"},{"comment":"The caption lists panels (a)–(f) in an order that does not match the panel layout; please relabel the panels or the caption to make the correspondence explicit.","section":"Figure 4"},{"comment":"The term 'NLLtest is a non-negative likelihood' is non-standard and appears to mean negative log-likelihood; please clarify the definition.","section":"5.3"},{"comment":"The column 'NNLtrain' is presumably a typo for 'NLLtrain'; please standardize.","section":"5.3, Table 2"},{"comment":"Eq. (9) is derived under the empirical distribution q_φ(x)=1/N; this assumption should be stated when the equation is introduced, not only in the appendix.","section":"3.3, Eq. (9)"}],"recommendation":"reject","confidential_remarks":"The paper has multiple load-bearing errors. The claimed MINE equivalence is false, the estimator is vulnerable to memorization bias, and the printed theorem contains an algebraic mistake. Even with revisions, the central estimator would need substantial rework and validation; the current submission is not suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know: this paper proposes adding a softmax classifier over sample identities to estimate I(z,x) inside the VAE ELBO, then uses alpha and beta to control mutual information and marginal KL. If the estimator were sound, the method would be useful. As it stands, the theoretical grounding does not hold up and the empirical validation is too thin to rescue it.\n\nWhat is genuinely nice: the identity I(z,x) = log N - E_z H(q(x|z)) is correct, and replacing the intractable posterior over inputs with a learned classifier is a reasonable trick. The hierarchical softmax variant is a practical touch for large N. The experiments also show the expected qualitative trends: larger beta reduces marginal KL and active units, larger alpha reduces MI and sharpens reconstructions at the cost of test error. The author is honest about the blur/overfitting trade-off, which is a point in the paper's favor.\n\nThe soft spots are load-bearing. The classifier is trained on the same data that defines q_phi(x|z), and it is never validated against held-out output or against the true conditional on a tractable problem. A sufficiently flexible network can drive cross-entropy down by memorizing sample identities, which overestimates MI and biases the alpha/beta control loop. Theorem 3.2's proof is wrong: the softmax objective is E_z log E_x e^T, not log E_z E_x e^T, and those differ by the Jensen gap, so the claimed equivalence with MINE is not established. Theorem 3.3 has an apparent algebraic error: the printed estimator has (1-P_e) log P_e where it should be (1-P_e) log(1-P_e). The Fano-based bound does not fix the bias because P_e is the classifier's training-set error rate, which is exactly the quantity inflated by overfitting. The paper also does not cite InfoNCE/CPC, which is essentially the same classifier-based MI bound, and this weakens the novelty claim.\n\nThe empirical section is thin: no error bars, no code, no comparison to InfoVAE, beta-VAE, or other MI-regularized VAEs. The reported MI values near log(55000) are suspiciously close to the upper bound, which is consistent with a near-deterministic classifier rather than an accurate q(x|z) estimate.\n\nThis is not ready for a serious referee. I would desk reject it, but I would also tell the author the empirical direction has value: validate the classifier against the true posterior on a small problem, correct the theorems, and compare against existing methods. As submitted, the central claims do not survive contact with the equations.","headline":"A VAE mutual-information control scheme with an appealing empirical story, but the central estimator is unvalidated and the claimed equivalence to MINE is not correct.","tokens_in":16101,"tokens_out":2469,"would_cite":false,"duration_ms":27624,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that appending an auxiliary softmax multiclassifier to a variational autoencoder makes the mutual information between data and latent codes estimable and controllable, giving a direct handle on posterior collapse and…","keywords":["variational autoencoder","posterior collapse","mutual information","auxiliary classifier","softmax","hierarchical softmax","marginal KL divergence","VAE-AS"],"falsifier":"Take a trained VAE-AS encoder, compute the classifier's mutual-information estimate, and compare it with a high-sample Monte Carlo estimate of $I_{q_\\phi}(z,x)$ on the same encoder; the paper predicts agreement up to the Fano error bound, so a systematic gap that grows on held-out or duplicated labels would falsify the central claim.","tokens_in":15042,"feed_emoji":"🧠","tokens_out":10288,"duration_ms":91443,"temperature":0.7,"pith_summary":"Variational autoencoders often stop using their latent codes, a failure called posterior collapse, and their images come out blurred. This paper argues that both symptoms follow from a lack of control over the mutual information between the input and the latent code. It proposes VAE-AS, which adds a softmax multiclassifier that learns to predict which training sample a latent code came from. That classifier supplies estimates of both the mutual information and the marginal KL divergence between the aggregated posterior and the prior, and those estimates become tunable terms in the training objective. Experiments on MNIST and Omniglot are offered as evidence that the added control changes the number of active latent units and the sharpness of reconstructions, which the paper identifies with relieving posterior collapse.","feed_headline":"Auxiliary softmax classifier lets VAEs tune mutual information","feed_subtitle":"Each training sample becomes a label, so the model can estimate and adjust the code-data information behind posterior collapse.","key_machinery":"The load-bearing object is the auxiliary softmax multiclassifier $s_\\omega(\\hat e|z)$, a neural network that maps a sampled latent code $z$ to a probability distribution over one-hot sample labels. Trained by cross-entropy, it is assumed to converge to the true conditional $q_\\phi(x^{(i)}|z)$; equation (9) turns that conditional into the mutual information, and equation (14) turns it into the aggregated posterior via Bayes' rule. This converts an intractable integral over a high-dimensional mixture distribution into a supervised classification problem. Because the cross-entropy loss is itself a MINE-type lower bound on mutual information, the classifier is simultaneously the estimator and part of the optimization target. A hierarchical softmax binary tree is used to reduce the classifier's cost from $O(hN)$ to $O(h\\log_2 N)$.","core_discovery":"The paper's central claim is that an auxiliary softmax classifier can reveal the two quantities a VAE normally cannot see: the mutual information $I_{q_\\phi}(z,x)$ and the marginal divergence $D_{KL}(q_\\phi(z)\\|p_\\theta(z))$. For an empirical sample distribution over $N$ training points, the mutual information is written as $\\log N - \\mathbb{E}_{q_\\phi(z)}[H(q_\\phi(x|z))]$, so the missing ingredient is just the conditional distribution $q_\\phi(x|z)$. A softmax network $s_\\omega(\\hat e|z)$ trained by cross-entropy against one-hot sample labels is claimed to converge to that conditional, and Bayes' rule then gives the aggregated posterior $q_\\phi(z)$. Using these estimates, the ELBO is rewritten with separate Lagrange multipliers $\\alpha$ for mutual information and $\\beta$ for the marginal divergence, so the user can choose how much information the latent code keeps. The paper further claims that the classifier's cross-entropy objective is equivalent to the MINE lower bound, and bounds the estimation error by the classifier's error rate times $\\log N$.","pith_inferences":["The paper leaves implicit that the method turns unsupervised VAE training into a self-supervised classification task: any encoder that makes arbitrary sample identities easy to read off from $z$ is implicitly maximizing mutual information, so the same idea could serve as a generic representation-learning objective.","The bound that using $V$ labels caps the available mutual-information budget at $\\log V$ suggests that for very large datasets the label count, not the encoder, may become the limiting factor; this is not tested in the MNIST and Omniglot experiments.","A testable extension is to swap the softmax classifier for other probabilistic classifiers, such as noise-contrastive or energy-based ones, and check whether the same ELBO surgery still holds; the claimed equivalence with MINE suggests the choice of classifier family should not change the core result."],"forward_implications":["The ELBO gains two independent knobs: $\\alpha$ controls how much information a latent code retains about its input, and $\\beta$ controls how far the aggregate code distribution sits from the prior.","Because the auxiliary classifier's objective is the same lower bound the MINE estimator maximizes, no adversarial loop or additional sampling scheme is needed; estimation and optimization are the same computation.","The hierarchical softmax version makes the method scale to large label sets, since the classification cost grows logarithmically with the number of training samples rather than linearly.","On MNIST and Omniglot the paper reports that lowering $\\beta$ increases the number of active latent units and sharpens reconstructions at the cost of generalization, while lowering $\\alpha$ has a similar sharpening effect, giving a practical lever for the blur-quality tradeoff."],"supporting_citations":[{"why":"Defines the variational autoencoder and the ELBO that VAE-AS modifies.","marker":"Kingma & Welling, 2013"},{"why":"Provides the ELBO surgery that decomposes the KL term into mutual information plus marginal KL divergence, the identity behind equations (3), (9), and (14).","marker":"Hoffman & Johnson, 2016"},{"why":"Supplies the KL dual representation that underlies the MINE comparison in Theorem 3.2.","marker":"Donsker & Varadhan, 1983"},{"why":"Introduces the MINE estimator whose optimization goal the paper claims is equivalent to VAE-AS.","marker":"Belghazi et al., 2018"},{"why":"Names and motivates the aggregated posterior q_phi(z), the quantity equation (14) estimates.","marker":"Makhzani et al., 2015"},{"why":"Provides the InfoVAE baseline that also adds mutual-information and distribution-matching penalties to the ELBO, the comparison class for VAE-AS.","marker":"Zhao et al., 2017a"},{"why":"Gives the hierarchical softmax construction used to reduce the classifier's cost to O(h log_2 N).","marker":"Morin & Bengio, 2005"},{"why":"Supplies the importance-weighted evaluation and the active-unit metric used in the empirical tables.","marker":"Burda et al., 2015"},{"why":"Provides the MNIST dataset used for the main experiments.","marker":"LeCun et al., 1998"},{"why":"Provides the Omniglot dataset used for the second set of experiments.","marker":"Lake et al., 2013"}],"fun_headline_variants":["Auxiliary softmax gives VAEs a dial for mutual information","Softmax classifier exposes VAE's hidden information gap","VAE-AS: auxiliary softmax rebalances latent information","Tune VAE code information with a softmax multiclassifier","Softmax side network lets VAE control posterior collapse"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method rests on the auxiliary classifier actually learning the true conditional probability that a latent code came from each training sample; if it overfits or cannot represent that conditional, the mutual-information and marginal-KL estimates are biased and the alpha and beta controls optimize a miscalibrated objective.","fun_headline_variants_meta":{"raw":{"variants":["Auxiliary softmax gives VAEs a dial for mutual information","Softmax classifier exposes VAE's hidden information gap","VAE-AS: auxiliary softmax rebalances latent information","Tune VAE code information with a softmax multiclassifier","Softmax side network lets VAE control posterior collapse"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000571,"raw_usage":{"total_tokens":2726,"prompt_tokens":1000,"completion_tokens":1726,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":616,"completion_tokens_details":{"reasoning_tokens":1642}},"tokens_in":616,"tokens_out":1726,"duration_ms":13187,"temperature":1.0,"reasoning_tokens":1642,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:50:10.049484+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a trained VAE-AS encoder, compute the classifier's mutual-information estimate, and compare it with a high-sample Monte Carlo estimate of $I_{q_\\phi}(z,x)$ on the same encoder; the paper predicts agreement up to the Fano error bound, so a systematic gap that grows on held-out or duplicated labels would falsify the central claim.","supporting_citations":[],"review_version":1}