Pith. sign in

REVIEW 4 major objections 4 minor 63 references

PixelVAE++: Improved PixelVAE with Discrete Prior

T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read PixelVAE++ claims state-of-the-art density estimates on MNIST, Omniglot, and CIFAR-10 among latent-variable models, while keeping its discrete latents informative.

desk verdict A credible architecture paper whose headline SOTA margins are within AIS and relaxation uncertainty; worth refereeing, not worth citing as SOTA without code. read the letter →

arxiv 1908.09948 v1 pith:OATXF4ML submitted 2019-08-26 cs.CV cs.LGstat.ML

Hossein Sadeghi , Evgeny Andriyash , Walter Vinci , Lorenzo Buffoni , Mohammad H. Amin This is my paper · ORCID
classification cs.CVcs.LGstat.ML
keywords generativemodelingvariationalautoencoderPixelCNN++decoderdiscretelatentvariablesrestrictedBoltzmannmachinepriorGumbel-SoftmaxrelaxationdensityestimationCIFAR-10
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a variational autoencoder with a strong autoregressive decoder can beat both pure autoregressive models and previous latent-variable models at image density estimation. It combines a PixelCNN++ decoder with three groups of latent variables and a restricted Boltzmann machine prior, and reports log-likelihood values of -78.00 on dynamically binarized MNIST, -88.29 on Omniglot, and 2.90 bits per dimension on CIFAR-10, averaged over three runs. The reason to care is that this is presented as the best likelihood among latent-variable models on CIFAR-10 and better than the published PixelCNN++ result, while the latent code remains informative rather than being ignored by the decoder.

What carries the argument

The central object is a three-tier latent representation: concatenated latents $z_1$, conditioning latents $z_2$, and shared latents $z_3$ that reuse the decoder's down-sampling stack as an encoder. The binary Bernoulli latents are trained through the Gumbel-Softmax continuous relaxation $\zeta = \sigma[(l+\sigma^{-1}(\rho))/\tau]$ at temperature $\tau = 0.25$, and the prior is a restricted Boltzmann machine whose partition function is estimated by annealed importance sampling. This machinery is what lets the model train with a tractable relaxed objective while evaluating the true discrete model at test time.

What would settle it

Recompute the model's likelihood evaluation with a much tighter partition-function estimate (say 100,000 annealed importance sampling samples and 100,000 temperature steps) on CIFAR-10 and check whether 2.90 bits per dimension holds; a shift of more than 0.03 bits per dimension would erase the margin over VLAE. A second check: train the identical architecture with a Gaussian prior under the same schedule and see whether the RBM prior still wins by 0.02 bits per dimension; the paper's own temperature experiments predict that lowering the relaxation temperature to 0.1 should degrade performance to around 2.98 bits per dimension.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that adding discrete latent variables to an autoregressive PixelCNN++ decoder improves likelihood while keeping the latents useful. The model achieves -78.00 on dynamically binarized MNIST and -88.29 on Omniglot, both better than the VLAE numbers the paper compares against, and 2.90 bits per dimension on CIFAR-10, which is below the 2.92 of PixelCNN++ and the 2.95 of VLAE. The authors attribute the gain to the RBM prior: with the discrete prior the KL divergence is larger and conditional reconstructions are sharper than with a Gaussian prior, which they read as evidence that the latent variables encode global structure such as digit class.

Load-bearing premise

The whole comparison rests on the annealed importance sampling estimate of the RBM partition function being accurate to well under 0.03 bits per dimension, and on the continuous Gumbel-Softmax training at temperature 0.25 being a faithful proxy for the discrete model evaluated at temperature 0.

Editorial extensions

If this is right

  • Latent-variable models with a strong autoregressive decoder can surpass pure autoregressive models on CIFAR-10, at 25 percent fewer parameters, if the reported 2.90 bits per dimension holds.
  • A discrete RBM prior yields a sharper conditional distribution than a Gaussian prior, so the latent code carries more information about identity and global structure on MNIST and Omniglot.
  • For binary images, one group of concatenated latent variables is enough to reach optimal likelihood; the conditioning and shared groups add no measurable improvement.
  • The shared encoder-decoder stack means latent variables can be added to an existing PixelCNN architecture for roughly 6 million extra parameters, making the hybrid design practical at 40 million parameters total.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the annealed importance sampling estimate of the RBM partition function carries a bias of even 0.03 bits per dimension on CIFAR-10, the reported margin over VLAE (0.05 bits per dimension) would not survive, so the ranking should be checked with a much tighter partition estimate.
  • The parameter-reuse idea is a general recipe: any autoregressive model with down-sampling layers could be turned into a VAE by reading latents from intermediate activations and feeding them back, which might transfer to audio or video generators.
  • The sharp-versus-broad conditional difference between RBM and Gaussian priors suggests that what matters is the prior's multimodality, not the exact functional form, so a simpler mixture prior might reproduce the effect at lower cost.
  • A testable extension is to anneal the Gumbel-Softmax temperature from high to low with a variance-reduced estimator; the paper reports this hurts, but a different schedule or estimator could change that conclusion.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces PixelVAE++, a hierarchical VAE with three groups of discrete latent variables and a PixelCNN++ autoregressive decoder. A parameter-sharing scheme reuses a part of the decoder as an encoder, and an RBM prior is used for the latent variables, trained with a Gumbel-Softmax relaxation. The authors report negative log-likelihoods of -78.00 nats on dynamically binarized MNIST and -88.29 nats on Omniglot with 1000 importance-weighted samples, and 2.90 bits/dim on CIFAR-10, claiming state-of-the-art among latent-variable models on CIFAR-10 and on binary datasets. They also discuss conditional generation and the informativeness of the latent variables.

Significance. If the reported likelihoods are accurate, the architecture is a useful step for combining autoregressive decoders with discrete latent variables, and the shared encoder-decoder design is a practical way to reduce parameters while retaining global information. The paper uses standard benchmarks and evaluation protocols, and it reports KL values that allow the reader to see the degree of posterior/prior overlap. It also honestly documents limitations in Appendix D. However, the central SOTA claims hinge on two effects that are not quantified in the manuscript: the accuracy of the AIS estimate of the RBM partition function, and the consistency between the continuous relaxation used at training and the discrete model used at evaluation. The small margins over VLAE (0.53 nats on MNIST, 0.05 bpd on CIFAR-10) make these effects load-bearing for the headline claims.

major comments (4)
  1. [Section 3.1, Table 1] The abstract's claim of state-of-the-art performance on binary datasets is contradicted by the paper's own Caltech-101 silhouettes row: the RBM-prior PixelVAE++ reports -77.46 nats versus -77.36 for VLAE, so the claim should be restricted to MNIST and Omniglot, or the Caltech-101 result should be explained.
  2. [Section 3.1] For the RBM-prior models, the reported IWAE likelihood is shifted by the AIS estimate of log Z_RBM; the paper reports only run-to-run standard deviations of the full likelihood (±0.04 nats on MNIST, ±0.001 bpd on CIFAR-10) and no uncertainty for the AIS estimate itself. Because the MNIST margin over VLAE is 0.53 nats and the CIFAR-10 margin is 0.05 bpd, an AIS bias of the size of the margin would erase the headline improvement. Please report multiple independent AIS estimates or an alternative bound on the logZ error.
  3. [Appendix C, Eq. (4)] Training is performed with the continuous Gumbel-Softmax relaxation at τ=0.25, whereas evaluation uses the discrete model at τ=0; the paper reports that lowering τ to 0.1 already degrades CIFAR-10 to 2.98 bpd. This indicates that the discrete model's likelihood is not the objective actually optimized, and the 2.90 bpd figure may overstate the performance of the discrete model. Please provide a quantitative characterization of the relaxation bias, for example by reporting importance-weighted likelihoods of the continuous model at training temperature or by evaluating the discrete model during training.
  4. [Section 3.2 and Appendix D] The claim that latent variables remain informative on CIFAR-10 is hard to reconcile with Appendix D's statement that 'neither the discrete nor continuous variables capture the structure in the CIFAR-10 data set' and with Table 1's KL of only 0.016 bpd, which is nearly zero in information terms. If the latent variables are not capturing class or global structure, the corresponding part of the abstract and Section 3.2 should be revised or supported with quantitative evidence such as mutual information estimates or downstream classification from the latents.
minor comments (4)
  1. [Throughout] The model name appears inconsistently as 'PixelV AE++' with a spurious space; please standardize to 'PixelVAE++'.
  2. [Section 2.1, Eq. (3)] The relaxed prior p_theta(zeta) is not explicitly written; please define how the RBM energy and partition function are evaluated for continuous zeta, since this affects the KL computation and the reader's understanding of the objective.
  3. [Section 3.1] The text says the standard deviation for the binary-set mean is ±0.04, while for CIFAR-10 it is reported as ±0.001; please clarify the number of independent runs used for each dataset and whether the reported standard deviation is over runs or over importance-weighted estimates.
  4. [Appendix A, Table 2] The entry '1↓(Kernel = Stride = 4)' is not self-contained; please provide the full layer parameters or a reference to released code so that the architecture can be reproduced.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the headline results are empirical test-set likelihoods compared against external baselines; the authors' prior RBM/Gumbolt work is used as a method but is not load-bearing for the central claim.

full rationale

The paper makes no derived predictions; all headline numbers are empirical test-set likelihoods from IWAE/AIS evaluated on external benchmark datasets (MNIST, Omniglot, Caltech-101 silhouettes, CIFAR-10), with baseline numbers taken from published PixelCNN++, VLAE, and PixelSNAIL results. The central claim is an empirical comparison, not a derivation from first principles, so there is no equation that reduces to its own input. The RBM prior training follows references [40,43,44,45], including Gumbolt, coauthored by the present authors; the paper says 'Reference [45] showed that a relaxed (biased) objective can be used to train a DVAE with an RBM prior' (Sec. 2.1, Eq. 3). That is a genuine reliance on prior work by the same group, but it is not circular: the relaxed objective (Eq. 3-4) is a standard Gumbel-Softmax/Boltzmann-prior construction whose stated assumptions do not include the target state-of-the-art likelihoods, and the reported likelihoods are measured against held-out data and independent published numbers. Moreover, the central benchmark claim does not depend on the self-cited RBM relaxation alone: Table 1 shows the Gaussian-prior PixelVAE++ already matches or beats VLAE on all reported datasets (e.g., -78.01 vs -78.53 nats on dynamically binarized MNIST and 2.92 vs 2.95 bpd on CIFAR-10), so the SOTA-against-VLAE claim survives even without the discrete RBM prior. The paper's own Appendix D concedes that on CIFAR-10 'neither the discrete nor continuous variables capture the structure in the CIFAR-10 data set,' which is a substantive limitation of the abstract's 'informative latent variables' claim, and the evaluation's reliance on AIS estimates of log Z_RBM is reported only with run-to-run standard deviations rather than AIS uncertainty (Sec. 3.1); these are correctness and validation risks, not circular-derivation steps. No fitted parameter is renamed as a prediction, no equation is defined in terms of the result it is supposed to predict, and no load-bearing argument reduces to a self-citation. Therefore the paper is not circular.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper adds no derivations from first principles; it relies on standard VAE lower bounds, the PixelCNN++ likelihood, RBM priors, and tuned hyperparameters. The main free parameters are architecture choices and the relaxation temperature.

free parameters (5)
  • Gumbel-Softmax temperature tau = 0.25
    Chosen by tuning; values 0.5 and 0.1 degrade performance (Appendix C).
  • Binary dataset latent dimensions = 400 binary latent variables, RBM with 200+200 units
    Reported as sufficient for MNIST and Omniglot; not derived from any principle (Section 3.1).
  • CIFAR-10 latent dimensions = 512 concatenated, 128 conditional, 8x8x3x(n+1) shared
    Hand-selected architecture for the three latent groups (Section 3.1 and Appendix A).
  • Number of ResNet layers per block = n=3 for 28x28 inputs, n=5 for 32x32 inputs
    Chosen by input size; described as optimal after experimentation (Sections 2.2 and 3.1).
  • CIFAR-10 filter count = 128 filters instead of 160
    The paper reports 2.90 bpd at 128 filters and no gain when returning to 160 filters (Section 3.1).
assumptions (4)
  • domain assumption The annealed importance sampling estimate of the RBM log-partition function is accurate enough to support NLL differences of about 0.02 bpd.
    Log-likelihood evaluation in Section 3.1 relies on AIS with 500 or 5000 samples and 1000 or 10,000 temperature steps; no error bars on the partition estimate are given.
  • ad hoc to paper Training with the continuous Gumbel-Softmax relaxation at temperature 0.25 produces a model that is well approximated by the discrete latent model evaluated at temperature 0.
    Appendix C admits high temperature causes mismatch between the smoothed posterior and the RBM and low temperature fails to converge; the chosen temperature is a tuning assumption.
  • standard math PixelCNN++'s raster-order factorization and discretized logistic mixture likelihood provide a valid tractable likelihood model.
    Adopted from Salimans et al. [26] as the decoder; the paper does not derive it.
  • domain assumption The standard binarization schemes and train/validation/test splits for MNIST, Omniglot, Caltech-101, and CIFAR-10 are appropriate for comparing published NLL values.
    Comparison against literature baselines assumes identical data preprocessing.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PixelVAE++: Improved PixelVAE with Discrete Prior." pith.science (2026). https://pith.science/paper/OATXF4ML

@misc{pith2026190809948,
  author       = {Pith},
  title        = {Pith review of: PixelVAE++: Improved PixelVAE with Discrete Prior},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OATXF4ML}},
  note         = {Machine review of arXiv:1908.09948}
}
read the original abstract

Constructing powerful generative models for natural images is a challenging task. PixelCNN models capture details and local information in images very well but have limited receptive field. Variational autoencoders with a factorial decoder can capture global information easily, but they often fail to reconstruct details faithfully. PixelVAE combines the best features of the two models and constructs a generative model that is able to learn local and global structures. Here we introduce PixelVAE++, a VAE with three types of latent variables and a PixelCNN++ for the decoder. We introduce a novel architecture that reuses a part of the decoder as an encoder. We achieve the state of the art performance on binary data sets such as MNIST and Omniglot and achieve the state of the art performance on CIFAR-10 among latent variable models while keeping the latent variables informative.

Figures

Figures reproduced from arXiv: 1908.09948 by the authors.

Figure 1
Figure 1. The hierarchical structure of PixelVAE++ with RBM prior. Arrows indicate convolutional [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Reconstructed samples for a PixelVAE++ trained on MNIST. Each row is generated from [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Reconstructed samples for a PixelVAE++ trained on Omniglot. Each row is generated from [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Reconstructed samples for a PixelVAE++ trained on CIFAR-10. Each row is generated [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Generated samples for a PixelVAE++ trained on MNIST. Each row is generated from latent [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Generated samples for a PixelVAE++ trained on Omniglot. Each row is generated from [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Generated samples. Each row corresponds to the same latent variable. Out of 128 row, we [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

63 extracted references · 26 canonical work pages

  1. [1]

    Conditional image generation with pixelcnn decoders

    Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelcnn decoders. In Advances in Neural Information Processing Systems, pages 4790–4798, 2016

    work page 2016

  2. [2]

    Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio

    Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Networks.arXiv:1406.2661 [cs, stat], June 2014. arXiv: 1406.2661

    arXiv 2014

  3. [3]

    Towards conceptual compression

    Karol Gregor, Frederic Besse, Danilo Jimenez Rezende, Ivo Danihelka, and Daan Wierstra. Towards conceptual compression. In Advances In Neural Information Processing Systems , pages 3549–3557, 2016

    2016

  4. [4]

    Improved variational inference with inverse autoregressive flow

    Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. In Advances in neural informa- tion processing systems, pages 4743–4751, 2016

    work page 2016

  5. [5]

    Density estimation using real nvp

    Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605.08803, 2016

    arXiv 2016

  6. [6]

    Wavenet: A generative model for raw audio

    Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03499, 2016

    arXiv 2016

  7. [7]

    Samplernn: An unconditional end-to-end neural audio generation model

    Soroush Mehri, Kundan Kumar, Ishaan Gulrajani, Rithesh Kumar, Shubham Jain, Jose Sotelo, Aaron Courville, and Yoshua Bengio. Samplernn: An unconditional end-to-end neural audio generation model. arXiv preprint arXiv:1612.07837, 2016

    arXiv 2016

  8. [8]

    Video pixel networks

    Nal Kalchbrenner, Aäron van den Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu. Video pixel networks. InProceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1771–1779. JMLR. org, 2017

    2017

Show all 63 references
  1. [9]

    Unsupervised learning for physical interac- tion through video prediction

    Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interac- tion through video prediction. In Advances in neural information processing systems, pages 64–72, 2016

    2016

  2. [10]

    Photo-realistic single image super-resolution using a generative adversarial network

    Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE c...

    2017

  3. [11]

    Image-to-image translation with conditional adversarial networks

    Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125–1134, 2017

    2017

  4. [12]

    Efficient learning of domain-invariant image representations

    Judy Hoffman, Erik Rodner, Jeff Donahue, Trevor Darrell, and Kate Saenko. Efficient learning of domain-invariant image representations. arXiv preprint arXiv:1301.3224, 2013

    2013 arXiv

  5. [13]

    Domain adaptive neural networks for object recognition

    Muhammad Ghifary, W Bastiaan Kleijn, and Mengjie Zhang. Domain adaptive neural networks for object recognition. In Pacific Rim international conference on artificial intelligence, pages 898–904. Springer, 2014

    2014

  6. [14]

    Unsupervised pixel-level domain adaptation with generative adversarial networks

    Konstantinos Bousmalis, Nathan Silberman, David Dohan, Dumitru Erhan, and Dilip Krishnan. Unsupervised pixel-level domain adaptation with generative adversarial networks. In Proceed- ings of the IEEE conference on computer vision and pattern recognition , pages 3722–3731, 2017

    2017

  7. [15]

    Deep visual domain adaptation: A survey

    Mei Wang and Weihong Deng. Deep visual domain adaptation: A survey. Neurocomputing, 312:135–153, 2018

    2018

  8. [16]

    Optimization as a model for few-shot learning

    Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. 2016

    2016

  9. [17]

    One-shot learning with memory-augmented neural networks

    Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. One-shot learning with memory-augmented neural networks. arXiv preprint arXiv:1605.06065, 2016

    2016 arXiv

  10. [18]

    Prototypical networks for few-shot learning

    Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, pages 4077–4087, 2017. 7

    2017

  11. [19]

    Auto-encoding variational bayes

    Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013

    2013 arXiv

  12. [20]

    Stochastic backpropagation and approximate inference in deep generative models

    Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv:1401.4082, 2014

    2014 arXiv

  13. [21]

    Markov chain monte carlo and variational inference: Bridging the gap

    Tim Salimans, Diederik P Kingma, Max Welling, et al. Markov chain monte carlo and variational inference: Bridging the gap. In ICML, volume 37, pages 1218–1226, 2015

    2015

  14. [22]

    Glow: Generative flow with invertible 1x1 convolutions

    Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. In Advances in Neural Information Processing Systems, pages 10236–10245, 2018

    2018

  15. [23]

    Variational inference with normalizing flows

    Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. arXiv preprint arXiv:1505.05770, 2015

    2015 arXiv

  16. [24]

    Nice: Non-linear independent components estimation

    Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint arXiv:1410.8516, 2014

    2014 arXiv

  17. [25]

    Implicit generation and generalization in energy-based models

    Yilun Du and Igor Mordatch. Implicit generation and generalization in energy-based models. arXiv preprint arXiv:1903.08689, 2019

    1903 arXiv

  18. [26]

    PixelCNN++: Improving the pixelCNN with discretized logistic mixture likelihood and other modifications

    Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. PixelCNN++: Improving the pixelCNN with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv:1701.05517, 2017

    2017 arXiv

  19. [27]

    Pixel recurrent neural networks

    Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In Proceedings of the 33rd International Conference on International Conference on Machine Learning, pages 1747–1756. JMLR. org, 2016

    2016

  20. [28]

    Pixelsnail: An improved autoregressive generative model

    Xi Chen, Nikhil Mishra, Mostafa Rohaninejad, and Pieter Abbeel. Pixelsnail: An improved autoregressive generative model. arXiv preprint arXiv:1712.09763, 2017

    2017 arXiv

  21. [29]

    Made: Masked autoencoder for distribution estimation

    Mathieu Germain, Karol Gregor, Iain Murray, and Hugo Larochelle. Made: Masked autoencoder for distribution estimation. In International Conference on Machine Learning, pages 881–889, 2015

    2015

  22. [30]

    Pixelvae: A latent variable model for natural images

    Ishaan Gulrajani, Kundan Kumar, Faruk Ahmed, Adrien Ali Taiga, Francesco Visin, David Vazquez, and Aaron Courville. Pixelvae: A latent variable model for natural images. arXiv preprint arXiv:1611.05013, 2016

    2016 arXiv

  23. [31]

    Variational lossy autoencoder

    Xi Chen, Diederik P Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, and Pieter Abbeel. Variational lossy autoencoder. arXiv preprint arXiv:1611.02731, 2016

    2016 arXiv

  24. [32]

    Deep learning of representations: Looking forward

    Yoshua Bengio. Deep learning of representations: Looking forward. InInternational Conference on Statistical Language and Speech Processing, pages 1–37. Springer, 2013

    2013

  25. [33]

    Variational recurrent auto-encoders

    Otto Fabius and Joost R van Amersfoort. Variational recurrent auto-encoders. arXiv preprint arXiv:1412.6581, 2014

    2014 arXiv

  26. [34]

    Generating sentences from a continuous space

    Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from a continuous space. arXiv preprint arXiv:1511.06349, 2015

    2015 arXiv

  27. [35]

    Empirical evaluation of gated recurrent neural networks on sequence modeling

    Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014

    2014 arXiv

  28. [36]

    Building end-to-end dialogue systems using generative hierarchical neural network models

    Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. Building end-to-end dialogue systems using generative hierarchical neural network models. In Thirtieth AAAI Conference on Artificial Intelligence, 2016

    2016

  29. [37]

    A hierarchical latent variable encoder-decoder model for generating dialogues

    Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron Courville, and Yoshua Bengio. A hierarchical latent variable encoder-decoder model for generating dialogues. In Thirty-First AAAI Conference on Artificial Intelligence, 2017

    2017

  30. [38]

    Sequential neural models with stochastic layers

    Marco Fraccaro, Søren Kaae Sønderby, Ulrich Paquet, and Ole Winther. Sequential neural models with stochastic layers. In Advances in neural information processing systems, pages 2199–2207, 2016

    2016

  31. [39]

    Vae with a vampprior.arXiv preprint arXiv:1705.07120, 2017

    Jakub M Tomczak and Max Welling. Vae with a vampprior.arXiv preprint arXiv:1705.07120, 2017. 8

    2017 arXiv

  32. [40]

    Dvae++: Discrete variational autoencoders with overlapping transformations

    Arash Vahdat, William G Macready, Zhengbing Bian, Amir Khoshaman, and Evgeny Andriyash. Dvae++: Discrete variational autoencoders with overlapping transformations. arXiv preprint arXiv:1802.04920, 2018

    2018 arXiv

  33. [41]

    Representational power of restricted boltzmann machines and deep belief networks

    Nicolas Le Roux and Yoshua Bengio. Representational power of restricted boltzmann machines and deep belief networks. Neural computation, 20(6):1631–1649, 2008

    2008

  34. [42]

    Discrete variational autoencoders

    Jason Tyler Rolfe. Discrete variational autoencoders. arXiv preprint arXiv:1609.02200, 2016

    2016 arXiv

  35. [43]

    DV AE#: Discrete variational autoencoders with relaxed Boltzmann priors

    Arash Vahdat, Evgeny Andriyash, and William G Macready. DV AE#: Discrete variational autoencoders with relaxed Boltzmann priors. In Neural Information Processing Systems (NIPS), 2018

    2018

  36. [44]

    Learning undirected posteriors by backpropagation through mcmc updates

    Arash Vahdat, Evgeny Andriyash, and William G Macready. Learning undirected posteriors by backpropagation through mcmc updates. arXiv preprint arXiv:1901.03440, 2019

    1901 arXiv

  37. [45]

    Gumbolt: Extending gumbel trick to boltzmann priors

    Amir H Khoshaman and Mohammad Amin. Gumbolt: Extending gumbel trick to boltzmann priors. In Advances in Neural Information Processing Systems, pages 4065–4074, 2018

    2018

  38. [46]

    Estimating or propagating gradients through stochastic neurons for conditional computation

    Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013

    2013 arXiv

  39. [47]

    Techniques for learning binary stochastic feedforward neural networks

    Tapani Raiko, Mathias Berglund, Guillaume Alain, and Laurent Dinh. Techniques for learning binary stochastic feedforward neural networks. arXiv preprint arXiv:1406.2989, 2014

    2014 arXiv

  40. [48]

    Categorical reparameterization with gumbel-softmax

    Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144, 2016

    2016 arXiv

  41. [49]

    The concrete distribution: A continuous relaxation of discrete random variables

    Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv:1611.00712, 2016

    2016 arXiv

  42. [50]

    Gradient-based learning applied to document recognition

    Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998

    1998

  43. [51]

    Inductive principles for re- stricted Boltzmann machine learning

    Benjamin Marlin, Kevin Swersky, Bo Chen, and Nando Freitas. Inductive principles for re- stricted Boltzmann machine learning. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 509–516, 2010

    2010

  44. [52]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009

    2009

  45. [53]

    Importance weighted autoencoders

    Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. arXiv preprint arXiv:1509.00519, 2015

    2015 arXiv

  46. [54]

    Annealed importance sampling

    Radford M Neal. Annealed importance sampling. Statistics and computing, 11(2):125–139, 2001

    2001

  47. [55]

    Fixing a broken elbo

    Alexander A Alemi, Ben Poole, Ian Fischer, Joshua V Dillon, Rif A Saurous, and Kevin Murphy. Fixing a broken elbo. arXiv preprint arXiv:1711.00464, 2017

    2017 arXiv

  48. [56]

    Batch normalization: Accelerating deep network training by reducing internal covariate shift

    Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pages 448–456, 2015

    2015

  49. [57]

    Dropout: a simple way to prevent neural networks from overfitting

    Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929–1958, 2014

    1929

  50. [58]

    Weight normalization: A simple reparameterization to accelerate training of deep neural networks

    Tim Salimans and Durk P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In Advances in Neural Information Processing Systems, pages 901–909, 2016

    2016

  51. [59]

    Ladder variational autoencoders

    Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. In Advances in neural information processing systems, pages 3738–3746, 2016

    2016

  52. [60]

    Sticking the landing: Simple, lower- variance gradient estimators for variational inference

    Geoffrey Roeder, Yuhuai Wu, and David K Duvenaud. Sticking the landing: Simple, lower- variance gradient estimators for variational inference. In Advances in Neural Information Processing Systems, pages 6925–6934, 2017. 9 A Network architecture details The table 2 outlines the...

    2017

  53. [61]

    dense + deconvolutional 64 FC, 128↑(Kernel = Stride = 4) f (zzz2

  54. [62]

    dense + deconvolutional 64 FC, 128↑ f (zzz3

  55. [63]

    The functions with q are encoding distributions pa- rameterized by neural networks while functions with f are deterministic transformation of latent variables

    dense + deconvolutional 64 FC, 128 Table 2: The details of the implementation. The functions with q are encoding distributions pa- rameterized by neural networks while functions with f are deterministic transformation of latent variables. The arrow signs pointing up and down i...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.