Pith. sign in

REVIEW 3 major objections 4 minor 53 references

Bridging the inference gap in Mutimodal Variational Autoencoders

T0 review · 3 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that multimodal VAEs can escape the mixture-aggregation quality ceiling by training a joint VAE first and then fitting normalizing-flow unimodal posteriors, and that shared-information projectors further improve…

desk verdict Solid two-stage multimodal VAE with an internal HMC target bug that needs fixing before the numbers can be trusted. read the letter →

arxiv 2502.03952 v1 pith:B3GMKR5I submitted 2025-02-06 cs.LG stat.ML

Agathe Senellart , Stéphanie Allassonnière This is my paper · ORCID
classification cs.LGstat.ML
keywords multimodalvariationalautoencodersnormalizingflowsmixture-of-expertsaggregationconditionalgenerationsharedinformationcontrastivelearningdeepcanonicalcorrelationanalysisproduct-of-expertsinference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multimodal variational autoencoders that aggregate unimodal encoders with a product or mixture of experts carry a proven generative gap: their approximate joint posterior can approach the true posterior only up to a strictly positive discrepancy. This paper proposes two methods, JNF and JNF-Shared, that avoid aggregation by splitting training into two stages, first learning the joint generative model and then approximating each unimodal posterior with normalizing flows. In the shared variant, the flow posteriors are conditioned on a learned summary of each modality that is supposed to retain exactly the information the other modalities can predict. The paper reports state-of-the-art conditional coherence with competitive FID on MNIST-SVHN, PolyMNIST, Translated PolyMNIST, and MHD, and argues that the approach is a practical, interpretable alternative to mixture-based multimodal VAEs.

What carries the argument

The load-bearing object is a two-stage training scheme with normalizing-flow posterior approximators, plus an optional shared-information projector. Stage 1 fits a standard joint VAE with a $\beta$-weighted ELBO. Stage 2 freezes the generative model and the joint encoder, then minimizes the sum of KL divergences from the joint posterior to each unimodal flow posterior, where each flow is a Gaussian base transformed by Masked Autoregressive Flows. For JNF-Shared, the input $x_j$ is replaced by $g_j(x_j)$, with projectors trained beforehand by Deep Canonical Correlation Analysis or contrastive learning. At inference, any subset posterior is approximated as proportional to $\prod_{j\in S} q_{\phi_j}(z|x_j)\,/\,p_\theta(z)^{|S|-1}$ and sampled with Hamiltonian Monte Carlo, which avoids training dedicated encoders for every subset.

What would settle it

Build or find a paired dataset on which a learned projector $g_j$ provably discards information that another modality depends on; if JNF-Shared's conditional coherence there does not fall to or below plain JNF's, the projector assumption is not doing the claimed work.

Watch

Extended reading notes

Core claim

The paper's central claim is that conditional generation from one or more modalities does not require mixture aggregation. After a joint VAE provides $p_\theta(X|z)$ and $q_\phi(z|X)$, the paper approximates each unimodal posterior $p_\theta(z|x_j)$ by a normalizing-flow distribution $q_{\phi_j}(z|x_j)$, trained by minimizing $\mathrm{KL}(q_\phi(z|X)\|q_{\phi_j}(z|x_j))$. Subset posteriors are obtained at inference from a product-of-experts of these flow posteriors, sampled with Hamiltonian Monte Carlo, so no subset-specific networks are trained. In JNF-Shared, the flows condition on a projector $g_j(x_j)$ instead of the raw input, under the conjecture that $p_\theta(x_i|x_j)=p_\theta(x_i|g_j(x_j))$ for every other modality $i$; the paper states this as an assumption and validates it empirically. It argues that this two-stage scheme sidesteps the mixture-aggregation discrepancy identified in prior work and yields higher conditional coherence while preserving diversity.

Load-bearing premise

For JNF-Shared, the load-bearing premise is that each modality can be compressed by a projector $g_j$ so that every other modality depends on the raw input only through $g_j(x_j)$; the paper calls this a conjecture and tests it only empirically.

Editorial extensions

If this is right

  • Because the joint posterior is trained before the unimodal posteriors, the generative discrepancy bound that limits mixture-based models no longer applies to the conditional path of JNF.
  • Users can condition on any subset of modalities without additional training: the product-of-experts of trained flow posteriors, sampled with HMC, supplies the subset posterior.
  • When the shared-information projectors work, JNF-Shared filters modality-specific background noise, so cross-modal generation preserves the shared semantic content even when full reconstruction of the conditioning modality would not.
  • The parameter cost scales with the number of modalities, one flow posterior per modality, rather than with the number of nonempty subsets, avoiding the exponential growth of surrogate-posterior methods.
  • On the four benchmarks studied, JNF and JNF-Shared reach the highest or second-highest conditional coherence while keeping FID or MFD competitive, including on Translated PolyMNIST where mixture-based baselines degrade sharply.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication left implicit by the paper is that Stage 1 can be swapped for any stronger unimodal VAE, so future improvements in decoders, priors, or diffusion-based generators would carry directly into conditional generation.
  • Equation (13) yields a direct test that does not require retraining: a classifier fitted on $g_j(x_j)$ should predict any other modality $x_i$ as accurately as one fitted on $x_j$ itself, which checks whether the projector really screens off the other modalities.
  • The inference-time HMC sampler is the main practical cost of the method; distilling the product-of-experts posterior into a single amortized network would make the approach cheaper and seems a natural next step for large-scale deployment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes a two-stage multimodal VAE framework. In the first stage, a joint VAE is trained with a standard β-weighted ELBO to model pθ(X, z). In the second stage, the generative model and joint encoder are frozen, and per-modality approximate posteriors q_{φ_j}(z|x_j) are trained, using normalizing flows, to distill the joint posterior. For conditional generation from any subset S of modalities, the paper derives a product-of-experts approximation pθ(z|x_S) ∝ (∏_{j∈S} q_{φ_j}(z|x_j)) / pθ(z)^{|S|-1} and samples it with Hamiltonian Monte Carlo. A variant called JNF-Shared first learns projectors g_j by DCCA or contrastive learning to extract shared information and trains the unimodal flows on g_j(x_j). The method is evaluated on MNIST-SVHN, PolyMNIST, Translated PolyMNIST, and MHD, reporting state-of-the-art conditional coherence and competitive diversity.

Significance. If the results hold, the paper offers a clean alternative to mixture-of-experts aggregation for multimodal VAEs, with the joint model and the conditional inference models trained separately. The distillation objective is non-circular, the use of normalizing flows addresses the limited flexibility of Gaussian unimodal posteriors, and the paper provides code and multiple-seed error bars. However, the empirical claims rest on three load-bearing points that need attention: the implemented HMC target is inconsistent with the stated subset posterior for |S|≠2, the β hyperparameter is selected on the test set, and the improvement of JNF-Shared relies on a conjectural sufficiency assumption on the projectors.

major comments (3)
  1. [§3.3, Eq. (12) vs. Appendix F, Eq. (F.3)] Eq. (12) defines the subset posterior as proportional to (∏_{j∈S} q_{φ_j}(z|x_j)) / pθ(z)^{|S|-1}, so its log-density is C + Σ_{j∈S} log q_{φ_j}(z|x_j) - (|S|-1) log pθ(z). Appendix F, Eq. (F.3), instead states log q(z|(x_i)_{i∈S}) = -log pθ(z) + Σ_{i∈S} log q_{φ_i}(z|x_i), which is correct only for |S|=2. For |S|=1 the sampler targets q_{φ_1}(z|x_1)/pθ(z) instead of q_{φ_1}(z|x_1), and for |S|=3,4 it underweights the prior by omitting the |S|-1 factor. The conditional coherence and conditional FID curves in Figure 4 vary the number of conditioning modalities from 1 to 4, and Section 3.3 claims the framework scales to 'any number of modalities', so a substantial part of the reported results may be sampling a target different from the one in Eq. (12). This is an internal mathematical inconsistency, not a disagreement with prior work. Please correct the target in Eq. (F.3) or in the implementation, rerun or re-report the affected experiments, and check the released code. The same issue also affects the JMVAE baseline, since Appendices E.2–E.4 state that JMVAE uses the same PoE sampling solution as the proposed models.
  2. [§4.3, β selection] Section 4.3 states: 'We train all models with a β-weighted ELBO and keep the β ∈ {0.5, 1, 2.5} that maximizes average coherence for each model.' The coherence values used for this selection are computed on the test set, and the same test-set values are then reported in Tables 1, 2, and Figure 4 (the appendices show the selected β). This is test-set model selection: the reported best-of-three performance is optimistically biased, and the four-seed standard deviations do not account for the selection. The comparison would be more convincing if β were chosen on a validation split, or if the main tables aggregated results over all β values with the selection procedure stated explicitly. For JNF-Shared (DCCA), Appendix D.4 says the β is reused from JNF-Shared (CL) 'because of similarity between models', which is not a principled selection and should be justified or replaced.
  3. [§3.4, Eq. (13)] JNF-Shared's improvement over JNF rests on Eq. (13): for every modality j there exists a projector g_j such that pθ(x_i|x_j) = pθ(x_i|g_j(x_j)) for all i, i.e., g_j is a sufficient statistic for cross-modal generation. The paper explicitly calls this a conjecture and tests it only indirectly through final coherence numbers. Nothing in DCCA or contrastive learning guarantees that the learned projection discards exactly the modality-specific information and retains all shared information, nor that the resulting summary is sufficient for the decoder. If Eq. (13) fails, JNF-Shared may lose its coherence advantage over plain JNF. I would like to see a direct check of the premise, for example measuring how well pθ(x_i|g_j(x_j)) approximates pθ(x_i|x_j) on a held-out set, or an ablation varying projection dimension or capacity and reporting when the JNF-Shared advantage disappears.
minor comments (4)
  1. [Title and Highlights] The title contains a typo: 'Mutimodal' should be 'Multimodal'. The Highlights also say 'two novels methods'; this should be 'two novel methods'.
  2. [§3.2, Eq. (7)] The sum in Eq. (7) is written as ∑_{j=0}^M, but the terms are q_{φ_j}(z|x_j) for j=1,...,M; the index should start at 1.
  3. [Appendix D, Figure D.10 caption] The caption says 'the newt rows' but should be 'the next rows'. There is also a typo 'Constrastive Learning' in Appendix D.1.
  4. [§4.2] For the PolyMNIST experiments, the description of FID says it is computed on 10,000 samples of the first modality, but it is not immediately clear whether this is for conditional or unconditional generation; please state the protocol explicitly in the main text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the JNF derivation is a self-contained distillation objective, and the projector assumption is explicitly conjectural and empirically tested.

full rationale

The central derivation chain is not circular. Step 1 (Section 3.1) trains a joint VAE by maximizing the ELBO (Eq. 3), with no coherence or FID term in the loss. Step 2 (Section 3.2) freezes q_phi(z|X) and fits each unimodal flow q_phi_j(z|x_j) by minimizing KL(q_phi(z|X) || q_phi_j(z|x_j)) (Eq. 7); this is a distillation objective, not a fit to the evaluation metric. The subset posterior (Eq. 12) is an exact Bayes identity under the conditional-independence model of Eq. (1), with p_theta(z|x_j) replaced by the trained flows; sampling from it via HMC does not re-introduce any fitted target metric. JNF-Shared rests on the projector condition Eq. (13), which the paper explicitly labels a conjecture: 'We conjecture that using these methods, we can extract summary statistics (g_j(x_j))_{1<=j<=M} verifying Equation (13) and check this assumption in our experiments' (Section 3.4). This is a testable empirical assumption, not a definitional equivalence. The only self-citation of note, Pythae [48], is an open-source implementation library for Normalizing Flows and is not load-bearing for any stated result. The selection of beta on test coherence is an evaluation-protocol concern, not circularity. One non-circular internal inconsistency exists: Eq. (F.3) omits the |S|-1 prior exponent from Eq. (12), so the HMC target for |S| != 2 differs from the stated subset posterior; this is a correctness/implementation issue, not a circular derivation.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The central claim does not rest on new physical entities; it rests on standard VAE factorization plus two explicit approximations (projector sufficiency and PoE subset posterior). The projectors g_j are learned neural networks, and the latent z is the standard VAE latent.

free parameters (6)
  • β (KL divergence weight) = 0.5, 1, or 2.5 per model
    Tuned per model on the test set by maximizing average coherence (Section 4.3, Tables D.3-D.5); directly affects the reported coherence and FID of every model.
  • Number of normalizing flow blocks (n_flows) = 2 or 3
    Selected per dataset after β was chosen (Appendix D.3); affects the flexibility of the unimodal posteriors.
  • Latent dimension d = 20 (MNIST-SVHN), 190 (PolyMNIST), 200 (Translated PolyMNIST), 64 (MHD)
    Set per dataset, partly to match baseline capacities; specified in Appendix E.
  • Likelihood rescaling factors λ = λ_MNIST = 3*32*32/(28*28), λ_SVHN = 1, λ_audio = 1, λ_trajectory = 20.48
    Hand-set following prior work [7] to balance reconstruction across modalities; affects the ELBO trade-off.
  • Contrastive learning temperature τ = 0.1
    Hyperparameter for the CL projectors (Appendix B.2).
  • HMC sampling parameters = 100 leapfrog steps
    Number of integration steps in HMC used to sample the PoE posterior (Appendix F); no convergence diagnostics reported.
assumptions (6)
  • domain assumption Modalities are conditionally independent given the latent variable z (pθ(X,z) = pθ(z) ∏_j pθ(x_j|z)).
    Invoked in Eq. (1) and used to derive the subset posterior in Eq. (12); standard but often violated in real multimodal data.
  • domain assumption A single latent variable z captures all information needed to generate all modalities.
    Core VAE assumption; used throughout Sections 2 and 3.
  • ad hoc to paper For each modality j there exists a projector g_j such that pθ(x_i|x_j) = pθ(x_i|g_j(x_j)) for all i (Eq. 13).
    Conjectured in Section 3.4; enables the JNF-Shared model to discard modality-specific information. The paper checks this only empirically.
  • domain assumption The frozen joint encoder q_φ(z|X) is a good enough approximation to pθ(z|X) that matching q_{φ_j} to it yields good conditional posteriors.
    Underpins the L_uni objective in Section 3.2; the quality of the joint posterior is inherited from stage one, with no additional guarantee.
  • domain assumption The product-of-experts of trained unimodal posteriors approximates the subset posterior pθ(z|x_S) (Eq. 12).
    Standard PoE approximation used at inference; its error is not quantified.
  • domain assumption The HMC sampler (100 leapfrog steps) converges to the target distribution for all conditioning sets.
    No convergence diagnostics are reported; the target in Eq. (F.3) also appears inconsistent with Eq. (12) for |S|>2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Bridging the inference gap in Mutimodal Variational Autoencoders." pith.science (2026). https://pith.science/paper/B3GMKR5I

@misc{pith2026250203952,
  author       = {Pith},
  title        = {Pith review of: Bridging the inference gap in Mutimodal Variational Autoencoders},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B3GMKR5I}},
  note         = {Machine review of arXiv:2502.03952}
}
read the original abstract

From medical diagnosis to autonomous vehicles, critical applications rely on the integration of multiple heterogeneous data modalities. Multimodal Variational Autoencoders offer versatile and scalable methods for generating unobserved modalities from observed ones. Recent models using mixturesof-experts aggregation suffer from theoretically grounded limitations that restrict their generation quality on complex datasets. In this article, we propose a novel interpretable model able to learn both joint and conditional distributions without introducing mixture aggregation. Our model follows a multistage training process: first modeling the joint distribution with variational inference and then modeling the conditional distributions with Normalizing Flows to better approximate true posteriors. Importantly, we also propose to extract and leverage the information shared between modalities to improve the conditional coherence of generated samples. Our method achieves state-of-the-art results on several benchmark datasets.

Figures

Figures reproduced from arXiv: 2502.03952 by the authors.

Figure 1
Figure 1. Graphical models in the case M = 2. Dashed lines represent decoders, solid lines represent encoders, and red arrows represent the projectors extracting shared information. "NF" refers to Normalizing Flows. step of our method on this dataset (see 3.1), which is training a simple joint VAE with a two-dimensional latent space that we can visualize. In [PITH_FULL_IMAGE:figures/full_fig_p012_1.png] view at source ↗
Figure 2
Figure 2. a) Samples from the toy dataset. b) The joint generative model [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗
Figure 3
Figure 3. On the first row: generation from MNIST to SVHN. On the second row: genera [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: In the two left columns, we present results for conditional generation when [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]
Figure 5
Figure 5. Figure 5: Joint generation in all five modalities when sampling a latent code from the prior. [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

53 extracted references · 34 canonical work pages

  1. [1]

    D. P. Kingma, M. Welling, Auto-Encoding Variational Bayes, arXiv:1312.6114 [cs, stat] (May 2014). URL http://arxiv.org/abs/1312.6114

    arXiv 2014

  2. [2]

    Higgins, N

    I. Higgins, N. Sonnerat, L. Matthey, A. Pal, C. P. Burgess, M. Bosn- jak, M. Shanahan, M. Botvinick, D. Hassabis, A. Lerchner, SCAN: Learning Hierarchical Compositional Visual Concepts (Jun. 2018).doi: 10.48550/arXiv.1707.03389. URL http://arxiv.org/abs/1707.03389

    arXiv doi:10.48550/arxiv.1707.03389 2018

  3. [3]

    Y. Tian, D. Krishnan, P. Isola, Contrastive Multiview Coding, in: A. Vedaldi, H. Bischof, T. Brox, J.-M. Frahm (Eds.), Computer Vi- sion – ECCV 2020, Lecture Notes in Computer Science, Springer International Publishing, Cham, 2020, pp. 776–794. doi:10.1007/ 978-3-030-58621-8_45

    work page 2020

  4. [4]

    H. Yin, F. Melo, A. Billard, A. Paiva, Associate Latent Encodings in Learning from Demonstrations, Proceedings of the AAAI Conference on Artificial Intelligence 31 (1), number: 1 (Feb. 2017).doi:10.1609/ aaai.v31i1.11040. URL https://ojs.aaai.org/index.php/AAAI/article/view/11040

    work page 2017

  5. [5]

    Suzuki, K

    M. Suzuki, K. Nakayama, Y. Matsuo, Joint Multimodal Learning with Deep Generative Models, arXiv:1611.01891 [cs, stat]ArXiv: 1611.01891 41 (Nov. 2016). URL http://arxiv.org/abs/1611.01891

    arXiv 2016

  6. [6]

    M. Wu, N. Goodman, Multimodal Generative Models for Scalable Weakly-Supervised Learning, in: Advances in Neural Information Processing Systems, Vol. 31, Curran Associates, Inc., 2018. URL https://proceedings.neurips.cc/paper/2018/hash/ 1102a326d5f7c9e04fc3c89d0ede88c9-Abstract.html

    work page 2018

  7. [7]

    Y. Shi, N. Siddharth, B. Paige, P. H. S. Torr, Variational Mixture- of-Experts Autoencoders for Multi-Modal Deep Generative Models, arXiv:1911.03393 [cs, stat]ArXiv: 1911.03393 (Nov. 2019). URL http://arxiv.org/abs/1911.03393

    work page Pith review arXiv 1911

  8. [8]

    T. M. Sutter, I. Daunhawer, J. E. Vogt, Generalized Multimodal ELBO, ICLR (2021)

    work page 2021

Show all 53 references
  1. [9]

    Daunhawer, T

    I. Daunhawer, T. M. Sutter, K. Chin-Cheong, E. Palumbo, J. E. Vogt, On the limitations of multimodal VAEs, in: International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=w-CPUXXrAj

    2022

  2. [10]

    Y. Shi, B. Paige, P. Torr, S. N, Relating by contrasting: A data-efficient framework for multimodal generative models, in: International Confer- ence on Learning Representations, 2021. URL https://openreview.net/forum?id=vhKe9UFbrJo

    2021

  3. [11]

    M. I. Jordan, Z. Ghahramani, T. S. Jaakkola, L. K. Saul, An Intro- duction to Variational Methods for Graphical Models, in: M. I. Jordan (Ed.), Learning in Graphical Models, Springer Netherlands, Dordrecht, 1998, pp. 105–161. doi:10.1007/978-94-011-5014-9_5. URL http://link.sp...

    1998 doi

  4. [13]

    Palumbo, I

    E. Palumbo, I. Daunhawer, J. E. Vogt, MMVAE+: Enhancing the generative quality of multimodal VAEs without compromises, in: The 42 Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=sdQGxouELX

    2023

  5. [14]

    Higgins, L

    I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, A. Lerchner, beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework, 2017

    2017

  6. [15]

    Burda, R

    Y. Burda, R. Grosse, R. Salakhutdinov, Importance Weighted Autoen- coders, arXiv:1509.00519 [cs, stat] (Nov. 2016).doi:10.48550/arXiv. 1509.00519. URL http://arxiv.org/abs/1509.00519

    2016 doi arXiv

  7. [16]

    Sutter, I

    T. Sutter, I. Daunhawer, J. Vogt, Multimodal Generative Learning Utilizing Jensen-Shannon-Divergence, in: Advances in Neural Informa- tion Processing Systems, Vol. 33, Curran Associates, Inc., 2020, pp. 6100–6110. URL https://proceedings.neurips.cc/paper/2020/hash/ 43bb733c1b6...

    2020

  8. [17]

    Vedantam, I

    R. Vedantam, I. Fischer, J. Huang, K. Murphy, Generative Models of Visually Grounded Imagination, arXiv:1705.10762 [cs, stat]ArXiv: 1705.10762 (Nov. 2018). URL http://arxiv.org/abs/1705.10762

    2018 arXiv

  9. [18]

    Lawry Aguila, J

    A. Lawry Aguila, J. Chapman, A. Altmann, Multi-modal variational au- toencoders for normative modelling across multiple imaging modalities, in: H. Greenspan, A. Madabhushi, P. Mousavi, S. Salcudean, J. Dun- can, T. Syeda-Mahmood, R. Taylor (Eds.), Medical Image Computing and C...

    2023

  10. [19]

    Hwang, G.-H

    H. Hwang, G.-H. Kim, S. Hong, K.-E. Kim, Multi-View Representation Learning via Total Correlation Objective, in: Advances in Neural Information Processing Systems, Vol. 34, Curran Associates, Inc., 2021, pp. 12194–12207. URL https://proceedings.neurips.cc/paper/2021/hash/ 65a9...

    2021

  11. [20]

    Suzuki, Y

    M. Suzuki, Y. Matsuo, Mitigating the limitations of multimodal VAEs with coordination-based approach (2023). URL https://openreview.net/forum?id=Rn8u4MYgeNJ 43

    2023

  12. [21]

    M. Lee, V. Pavlovic, Private-shared disentangled multimodal vae for learning of latent representations, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2021, pp. 1692–1700

    2021

  13. [22]

    Daunhawer, T

    I. Daunhawer, T. M. Sutter, R. Marcinkevičs, J. E. Vogt, Self-supervised DisentanglementofModality-SpecificandSharedFactorsImprovesMul- timodal Generative Models, in: Z. Akata, A. Geiger, T. Sattler (Eds.), Pattern Recognition, Lecture Notes in Computer Science, Springer Inter...

    2021

  14. [23]

    Vasco, H

    M. Vasco, H. Yin, F. S. Melo, A. Paiva, Leveraging hierarchy in multi- modal generative models for effective cross-modality inference, Neural Networks 146 (2022) 238–255.doi:10.1016/j.neunet.2021.11.019. URL https://linkinghub.elsevier.com/retrieve/pii/ S0893608021004470

    2022 doi

  15. [24]

    Palumbo, L

    E. Palumbo, L. Manduchi, S. Laguna, D. Chopard, J. E. Vogt, Deep gen- erative clustering with multimodal diffusion variational autoencoders, in: The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=k5THrhXDV3

    2024

  16. [25]

    Bounoua, G

    M. Bounoua, G. Franzese, P. Michiardi, Multi-modal latent diffusion, Entropy 26 (4) (2024).doi:10.3390/e26040320. URL https://www.mdpi.com/1099-4300/26/4/320

    2024 doi

  17. [26]

    Rezende, S

    D. Rezende, S. Mohamed, Variational inference with normalizing flows, in: F. Bach, D. Blei (Eds.), Proceedings of the 32nd International Con- ference on Machine Learning, Vol. 37 of Proceedings of Machine Learn- ing Research, PMLR, Lille, France, 2015, pp. 1530–1538

    2015

  18. [27]

    Papamakarios, T

    G. Papamakarios, T. Pavlakou, I. Murray, Masked Autoregressive Flow for Density Estimation, in: Advances in Neural Information Processing Systems, Vol. 30, Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper/2017/hash/ 6c1da886822c67822bcf3679d04369fa-Abstr...

    2017

  19. [28]

    doi:10.1201/b10905-10

    MCMC Using Hamiltonian Dynamics, Chapman and Hall/CRC, 2011, pages: 139-188 Publication Title: Handbook of Markov Chain Monte Carlo. doi:10.1201/b10905-10. URL https://www.taylorfrancis.com/chapters/edit/10.1201/ b10905-10/mcmc-using-hamiltonian-dynamics-radford-neal

    2011 doi

  20. [29]

    Betancourt, A conceptual introduction to hamiltonian monte carlo (2018)

    M. Betancourt, A conceptual introduction to hamiltonian monte carlo (2018). arXiv:1701.02434. URL https://arxiv.org/abs/1701.02434

    2018 arXiv

  21. [30]

    Lecun, L

    Y. Lecun, L. Bottou, Y. Bengio, P. Haffner, Gradient-based learning appliedtodocumentrecognition, ProceedingsoftheIEEE86(11)(1998) 2278–2324, conference Name: Proceedings of the IEEE.doi:10.1109/ 5.726791

    1998

  22. [31]

    Netzer, T

    Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, A. Ng, Reading Digits in Natural Images with Unsupervised Feature Learning, NIPS (Jan. 2011)

    2011

  23. [32]

    Andrew, R

    G. Andrew, R. Arora, J. Bilmes, K. Livescu, Deep Canonical Correla- tion Analysis, in: Proceedings of the 30th International Conference on Machine Learning, PMLR, 2013, pp. 1247–1255, iSSN: 1938-7228. URL https://proceedings.mlr.press/v28/andrew13.html

    2013

  24. [33]

    Poklukar, M

    P. Poklukar, M. Vasco, H. Yin, F. S. Melo, A. Paiva, D. Kragic, Geomet- ric Multimodal Contrastive Representation Learning, in: Proceedings of the 39th International Conference on Machine Learning, PMLR, 2022, pp. 17782–17800, iSSN: 2640-3498. URL https://proceedings.mlr.press...

    2022

  25. [34]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, I. Sutskever, Learning transferable visual models from natural language supervision, CoRR abs/2103.00020 (2021). arXiv:2103.00020. URL https://arxiv.org/ab...

    2021 arXiv

  26. [35]

    A. Abid, J. Zou, Contrastive Variational Autoencoder Enhances Salient Features, arXiv:1902.04601 [cs, stat] (Feb. 2019). URL http://arxiv.org/abs/1902.04601 45

    1902 arXiv

  27. [36]

    Y. Tian, D. Krishnan, P. Isola, Contrastive multiview coding, CoRR abs/1906.05849 (2019). arXiv:1906.05849. URL http://arxiv.org/abs/1906.05849

    2019 arXiv

  28. [37]

    Ramsauer, T

    M.Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, S.Hochreiter, Gans trained by a two time-scale update rule converge to a local nash equi- librium, in: Advances in Neural Information Processing Systems, 2017

    2017

  29. [38]

    Ghosh, M

    P. Ghosh, M. S. M. Sajjadi, A. Vergari, M. Black, B. Schölkopf, From variational to deterministic autoencoders (2020).arXiv:1903.12436. URL https://arxiv.org/abs/1903.12436

    2020 arXiv

  30. [39]

    arXiv:1505.05770

    D.J.Rezende, S.Mohamed, Variationalinferencewithnormalizingflows (2016). arXiv:1505.05770. URL https://arxiv.org/abs/1505.05770

    2016 arXiv

  31. [40]

    Tomczak, M

    J. Tomczak, M. Welling, Vae with a vampprior, in: A. Storkey, F. Perez- Cruz (Eds.), Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics, Vol. 84 of Proceedings of Ma- chine Learning Research, PMLR, 2018, pp. 1214–1223. URL https:...

    2018

  32. [41]

    D. P. Kingma, T. Salimans, R. Jozefowicz, X. Chen, I. Sutskever, M. Welling, Improving Variational Inference with Inverse Autoregres- sive Flow, number: arXiv:1606.04934 arXiv:1606.04934 [cs, stat] (Jan. 2017). URL http://arxiv.org/abs/1606.04934

    2017 arXiv

  33. [42]

    Tucker, D

    G. Tucker, D. Lawson, S. Gu, C. J. Maddison, Doubly reparameterized gradient estimators for monte carlo objectives (2018). arXiv:1810. 04152. URL https://arxiv.org/abs/1810.04152

    2018 arXiv

  34. [43]

    D. R. Hardoon, S. Szedmak, J. Shawe-Taylor, Canonical corre- lation analysis: an overview with application to learning meth- ods, Neural Computation 16 (12) (2004) 2639–2664. doi:10.1162/ 0899766042321814

    2004

  35. [44]

    N. Y. Bilenko, J. L. Gallant, Pyrcca: Regularized Kernel Canonical Cor- relation Analysis in Python and Its Applications to Neuroimaging, Fron- tiers in Neuroinformatics 10 (2016).doi:10.3389/fninf.2016.00049. 46 URL https://www.frontiersin.org/journals/neuroinformatics/ artic...

    2016

  36. [45]

    M. A. Alam, V. D. Calhoun, Y.-P. Wang, Identifying outliers using multiple kernel canonical correlation analysis with application to imaging genetics, Computational Statistics & Data Analysis 125 (2018) 70–85. doi:https://doi.org/10.1016/j.csda.2018.03.013. URL https://www.sci...

    2018 doi

  37. [46]

    Preechakul, N

    K. Preechakul, N. Chatthee, S. Wizadwongsa, S. Suwajanakorn, Diffu- sion autoencoders: Toward a meaningful and decodable representation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 10619–10629

    2022

  38. [47]

    C. I. Kanatsoulis, X. Fu, N. D. Sidiropoulos, M. Hong, Structured sumcor multiview canonical correlation analysis for large-scale data, IEEE Transactions on Signal Processing 67 (2) (2019) 306–319.doi: 10.1109/TSP.2018.2878544

    2019

  39. [48]

    Chadebec, L

    C. Chadebec, L. J. Vincent, S. Allassonniere, Pythae: Unifying Gener- ative Autoencoders in Python - A Benchmarking Use Case, 2022. URL https://openreview.net/forum?id=w7VPQWgnn3s

    2022

  40. [49]

    Javaloy, M

    A. Javaloy, M. Meghdadi, I. Valera, Mitigating Modality Collapse in Multimodal VAEs via Impartial Optimization, arXiv:2206.04496 [cs] (Jun. 2022). URL http://arxiv.org/abs/2206.04496

    2022 arXiv

  41. [50]

    K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015) 770–778. URL https://api.semanticscholar.org/CorpusID:206594692

    2015

  42. [51]

    Duane, A

    S. Duane, A. D. Kennedy, B. J. Pendleton, D. Roweth, Hy- brid Monte Carlo, Physics Letters B 195 (2) (1987) 216–222. doi:https://doi.org/10.1016/0370-2693(87)91197-X. URL https://www.sciencedirect.com/science/article/pii/ 037026938791197X 47

    1987 doi

  43. [52]

    Liu, Monte Carlo Strategies in Scientic Computing, 2009.doi:10

    J. Liu, Monte Carlo Strategies in Scientic Computing, 2009.doi:10. 1007/978-0-387-76371-2

    2009

  44. [53]

    Brooks, A

    S. Brooks, A. Gelman, G. Jones, X.-L. Meng, Handbook of Markov Chain Monte Carlo, Chapman and Hall/CRC, 2011. doi:10.1201/ b10905. URL http://dx.doi.org/10.1201/b10905

    2011 doi

  45. [54]

    Girolami, B

    M. Girolami, B. Calderhead, Riemann Manifold Langevin and Hamiltonian Monte Carlo Methods, Journal of the Royal Sta- tistical Society Series B: Statistical Methodology 73 (2) (2011) 123–214, _eprint: https://academic.oup.com/jrsssb/article- pdf/73/2/123/49162769/jrsssb_73_2_12...

    2011

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.