REVIEW 4 major objections 3 minor 15 references
Return of the Latent Space COWBOYS: Re-thinking the use of VAEs for Bayesian Optimisation of Structured Spaces
T0 review · 4 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Decoupling the VAE from the surrogate — fitting the GP on molecules with a Tanimoto kernel and sampling latent codes predicted to beat the best so far — finds better molecules than latent-space Bayesian optimisation.
desk verdict Useful empirical method with a sampler that targets the wrong distribution; fix the acceptance ratio and tone down the claims. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the Bayesian update $p(z \mid f(h_\theta(z)) > f^*, \mathcal{D}) \propto p(g_{\theta,z} > f^* \mid \mathcal{D})\, p(z)$: the VAE's Gaussian prior over latent codes is reweighted by the GP's closed-form probability-of-improvement likelihood, and each accepted code is passed through the deterministic most-likely decoder $h_\theta(z) = \arg\max_x p_\theta(x \mid z)$, so the whole scheme is posterior sampling under a Gaussian prior. That reduction matters because it is the setting the Preconditioned Crank-Nicolson MCMC algorithm was designed for: PCN concentrates on the annulus of high-probability mass instead of wandering through low-probability regions, which keeps it effective in the 128- and 256-dimensional latent spaces used here. The second supporting object is the Tanimoto kernel on count-based molecular fingerprints, which lets the GP measure similarity where chemistry is compared rather than inheriting the VAE's learned geometry.
What would settle it
Run the same optimisation loop on a synthetic structured problem where the latent space is aligned with the objective by construction — the most-likely decoding makes the objective a smooth function of z — while the Tanimoto fingerprints are perturbed so the structure-space GP is the weaker model; the paper's mechanism predicts COWBOYS should still match or beat a fine-tuning latent-space method, because the box-search critique applies regardless of alignment. A clear win for the latent-space method in that setting would show the reported gains depend on misaligned VAE spaces rather than on the decoupled design. A cheaper complementary check is to replace the 100-step non-reversible sampler with a converged sampler on one benchmark and measure how much performance changes.
Extended reading notes
Core claim
COWBOYS's central claim is that the two pathologies of latent-space Bayesian optimisation — GP surrogates that model the latent-to-objective map poorly, and box-clipped search regions that miss the shell of credible latent codes in high dimensions — can be avoided outright rather than patched. The method fits its GP in the original structured space, using a Tanimoto kernel over count-based molecular fingerprints, and never fits a model inside the latent space; the VAE contributes only through its generative distribution. Search proceeds by sampling latent codes from the posterior $p(z \mid f(h_\theta(z)) > f^*, \mathcal{D}) \propto p(f(h_\theta(z)) > f^* \mid \mathcal{D})\, p(z)$, where $h_\theta$ is the deterministic most-likely decoding of the VAE and the likelihood is the GP's closed-form probability of improvement over the best objective value $f^*$ seen so far. Because that likelihood is exactly a probability-of-improvement term, candidate generation reduces to classical posterior sampling under a Gaussian prior, which the paper carries out with a modified Preconditioned Crank-Nicolson MCMC sampler initialised from the current best latent code. The paper reports that this decoupled scheme improves sample efficiency in finding high-potential molecules across 16 molecular-design benchmarks, and that baselines do not catch up even when granted up to two orders of magnitude more evaluations.
Load-bearing premise
The load-bearing premise is that a short run of Markov-chain Monte Carlo — started at the best candidate found so far, run for roughly a hundred steps with an adaptively sized, deliberately non-reversible proposal — produces a set of candidate molecules varied and accurate enough to keep the search efficient, even though the paper's Section 5.2 concedes that this modification forfeits the standard sample-quality guarantees of the underlying algorithm.
Editorial extensions
If this is right
- Under the low evaluation budgets typical of lead optimisation, COWBOYS identifies better molecules than latent-space BO methods that fine-tune their VAE, without fine-tuning anything itself.
- Because the GP never sees the latent space, the method transfers to any structured domain with a suitable kernel — proteins, molecular graphs, code, 3D designs — carrying domain knowledge through the kernel rather than through the generative model.
- Arbitrarily large batches of candidates can be drawn in parallel from the conditioned distribution, making the scheme a natural fit for parallel evaluation settings.
- Exact GP inference in structure space currently limits COWBOYS to budgets of a few thousand evaluations; reaching very large budgets such as 80,000 is left as future work requiring scalable sparse GPs for discrete data.
- The paper's ablations indicate both components are needed: replacing the structure-space GP with a latent-space GP, or decoding stochastically at query time, each costs performance.
Reading between the lines
- The Gaussian-annulus critique is broader than the paper states: any generative model with a high-dimensional Gaussian prior concentrates its plausible samples in a shell, so COWBOYS-style sampling without box clipping should also be tried in GAN and diffusion-model latent spaces, where bounded search regions remain standard practice.
- A testable prediction of the paper's mechanism is that COWBOYS's advantage should grow as the objective's smoothness in structure space diverges from its smoothness in the latent space; on objectives that are smooth in z by construction, the decoupling gain should shrink, and that experiment is not in the paper.
- Because every MCMC chain starts at the current best latent code, the active mechanism may be close to local refinement under a global surrogate; a cheaper approximation — a few hundred GP-guided perturbations of the best molecule with no MCMC at all — would test whether the posterior-sampling formalism or the decoupled GP carries the gains.
- An extension the authors gesture at but do not test: averaging the GP likelihood over several stochastic decodings per latent code might recover the diversity lost by the deterministic-decoder approximation without re-introducing the alignment noise that hurt their stochastic-decoding ablation.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes COWBOYS, a latent-space Bayesian optimization method for structured (molecular) spaces that decouples a pretrained VAE from a Gaussian Process surrogate. Instead of fitting the GP in the VAE latent space and optimizing an acquisition function over a clipped box, COWBOYS fits the GP directly in structure space with a Tanimoto kernel and samples new candidates by drawing latent codes from the distribution p(z | f(h(z)) > f*, D), formed by Bayes rule as the VAE prior weighted by the GP's Probability-of-Improvement likelihood, then decoding each code through the most likely decoder output. The authors argue this avoids two LSBO pathologies: poor latent-space surrogate fit and the difficulty of choosing a bounded latent search region. Experiments cover 16 benchmark tasks and 24 baselines, including low-budget molecular MPO problems (Fig. 3), a high-dimensional discrete BO benchmark (Table 1), and a comparison with non-fine-tuning LSBO at much larger budgets (Fig. 4). The paper also includes an ablation study of MCMC chains/steps, a latent-GP variant, stochastic- versus deterministic-decoding, and degraded surrogate fingerprints. The central claim is improved sample efficiency under constrained evaluation budgets relative to existing LSBO methods.
Significance. If the empirical gains hold, the paper makes a useful conceptual contribution: it shows that a VAE can be used purely as a generator, with a structure-space GP doing the discrimination, and that a sampling-based combination can beat methods that fine-tune the VAE during BO. The empirical breadth is a genuine strength: 16 benchmarks, 24 baselines, comparisons against both fine-tuning and non-fine-tuning LSBO, and a clear ablation study. The released code is integrated into an established benchmarking suite, which should help follow-up work. However, the central theoretical framing has a load-bearing implementational flaw: the PCN acceptance ratio in Algorithm 4 does not target the posterior stated in Eq. (5). In addition, the claim of 'consistently outperforms' overstates Table 1, and the headline low-budget experiments depend on a privately supplied VAE. These issues do not necessarily invalidate the empirical findings, but they must be resolved before the paper's advertised mechanism can be taken at face value.
major comments (4)
- [Algorithm 4 / Eq. (5)] The acceptance ratio in Algorithm 4 is not the Metropolis-Hastings ratio for the target in Eq. (5). For the PCN proposal z' ~ N(sqrt(1-beta^2) z, beta^2 I), the proposal is reversible with respect to the standard normal prior, so q(z | z') / q(z' | z) = p(z) / p(z'). The correct acceptance probability for the target pi(z) proportional to L(z) p(z), with L(z) = p(g_z > f* | D), is min(1, L(z') / L(z)). Algorithm 4 instead uses min(1, L(z') p(z') / (L(z) p(z))), which omits the proposal-density ratio. As written, the chain over-accepts moves toward the origin and does not have the distribution in Eq. (5) as its stationary distribution. This is not a formal nicety: the paper's central mechanism is the Bayesian update (4)-(5), and the pseudocode implements a different target. Please (i) confirm whether the released code uses the printed ratio or the correct MH ratio, (ii) correct the pseudocode, and (iii) add a small diagnostic on a synthetic likelihood where the target is known, verifying that the stationary distribution of the implemented chain matches Eq. (4).
- [Section 7.2 / Table 1] The abstract and Section 7.2 say that COWBOYS 'consistently outperforms all' compared methods, but Table 1 contains counter-examples: on albuterol_similarity, HillClimbing reports 0.487 vs. COWBOYS's 0.472; on drd2_docking, RandomLineBO reports 0.346 vs. 0.342; and on rdkit_logp, both CMAES (14.474) and RandomLineBO (19.484) report higher means than COWBOYS (10.678). If the intended claim is 'best or within one standard deviation on most tasks', that should be stated and quantified with, for example, win rates or paired significance tests. As it stands, the unconditional phrasing overstates the evidence in the paper's own table.
- [Section 7.1] The headline low-budget results in Figure 3 use a SELFIES VAE 'provided via direct correspondence' from the authors of Maus et al. (2022a), rather than a publicly released checkpoint. This makes the main low-budget experiments unreproducible from the released repository and weakens the paper's otherwise strong reproducibility story. Please either release the exact VAE checkpoint used for Figure 3, or rerun those experiments with a publicly available VAE and report any differences.
- [Section 5.2 / Appendix A] The paper explicitly forfeits rigorous MCMC guarantees in Section 5.2, and Appendix A adds two further departures from exact sampling: adaptive step-size selection and qEI-based subset selection, which the authors note 'breaks MCMC's detailed balance'. These are reasonable engineering choices for BO, but combined with the acceptance-ratio issue in Algorithm 4, they mean the paper overstates the 'simple yet principled Bayesian update' framing. Please either provide evidence that the implemented sampler approximately targets the distribution in Eq. (4), or soften the claim and present COWBOYS as a heuristic biased sampling scheme whose empirical performance is the main contribution.
minor comments (3)
- [Figure 4 caption] The caption contains a typo: 'improvemnt' should be 'improvement'.
- [Table 2] The leftmost column of Table 2 is labelled 'Oracle' but the text never defines what this column represents; please clarify whether it is a no-MCMC oracle, a large-computation reference, or something else.
- [Section 4] The sentence 'We now summarise Categorical Optimisation With Belief Of underlYing Structures (COWBOYSs)' has an extra 's' in the acronym; it should read '(COWBOYS)'.
Circularity Check
No significant circularity: the COWBOYS target distribution is an explicitly constructed Bayes-rule posterior, and the paper's central claim is supported by external benchmarks rather than by the definition of the method.
full rationale
No circular step is load-bearing. Equation (5) rewrites the constructed condition p(z | g_{θ,z} > f*, D_n^X) as p(g_{θ,z} > f* | D_n^X) p(z), but the paper explicitly presents this as 'decomposing (4) via Bayes' rule' and uses it to design a sampler; it does not present the identity as an empirical prediction or fitted result. The central claim that COWBOYS improves identification of high-potential molecules is evaluated against external oracles (GuacaMol and PMO benchmarks) and 24 baselines, so the reported gains are not forced by the method's construction. Self-citations (e.g., Ober et al. 2021; Moss & Griffiths 2020; Griffiths et al. 2022) support background claims about GP/VAE limitations and are corroborated by external citations and by the paper's own ablations, so they are not load-bearing. One non-circular caveat: Algorithm 4's acceptance ratio includes p(zproposal)/p(zcurrent) even though PCN's proposal is reversible with respect to p(z); the stationary distribution therefore differs from Eq. (4), and Section 5.2 already concedes that the non-reversible modification 'forfeits the rigorous sample quality guarantees' of PCN. This is an internal sampler-validity/correctness concern, not a circularity, and does not affect the circularity score.
Assumptions & free parameters
free parameters (5)
- Tanimoto kernel output scale sigma^2 =
not specified (fitted via GP marginal likelihood)
- GP kernel lengthscale and observation noise =
BoTorch defaults
- PCN initial step size beta and target acceptance rate =
beta_0 = 0.1, target acceptance 0.243
- Number of MCMC chains C and steps S =
C=1, S=100 (batch size 1); C=50, S=100 (batch size >5)
- Latent dimension and architecture of VAE =
256-dim SELFIES VAE (7.1, from Maus et al.); 128-dim fully-connected VAE (7.2, new)
assumptions (7)
- domain assumption The VAE has a standard Gaussian prior p(z) = N(0, I) over latent codes.
- ad hoc to paper The stochastic decoder is replaced by the most likely decoding h(z) = argmax_x p(x|z).
- domain assumption The VAE generates valid, diverse molecular structures when decoding from its prior.
- domain assumption Tanimoto kernel on count-based molecular fingerprints is a suitable similarity measure for modeling objective functions over molecules.
- domain assumption The GP posterior predictive distribution provides calibrated probabilities p(g_{theta,z} > f* | D).
- standard math Gaussian Annulus Theorem applies to the latent codes of the VAE.
- standard math Bayes' rule and GP conditioning equations are used correctly.
Cite this review
Pith. "Pith review of Return of the Latent Space COWBOYS: Re-thinking the use of VAEs for Bayesian Optimisation of Structured Spaces." pith.science (2026). https://pith.science/paper/VKNXPM42
@misc{pith2026250703910,
author = {Pith},
title = {Pith review of: Return of the Latent Space COWBOYS: Re-thinking the use of VAEs for Bayesian Optimisation of Structured Spaces},
year = {2026},
howpublished = {\url{https://pith.science/paper/VKNXPM42}},
note = {Machine review of arXiv:2507.03910}
}
read the original abstract
Bayesian optimisation in the latent space of a Variational AutoEncoder (VAE) is a powerful framework for optimisation tasks over complex structured domains, such as the space of scientifically interesting molecules. However, existing approaches tightly couple the surrogate and generative models, which can lead to suboptimal performance when the latent space is not tailored to specific tasks, which in turn has led to the proposal of increasingly sophisticated algorithms. In this work, we explore a new direction, instead proposing a decoupled approach that trains a generative model and a Gaussian Process (GP) surrogate separately, then combines them via a simple yet principled Bayesian update rule. This separation allows each component to focus on its strengths -- structure generation from the VAE and predictive modelling by the GP. We show that our decoupled approach improves our ability to identify high-potential candidates in molecular optimisation problems under constrained evaluation budgets.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[2]
Average performance (± s.d.) over 5 repetitions of COWBOYS on the discrete BO benchmarking suite of (Gonz´alez-Duque et al., 2024). We stress best average scores achieved after 300 evaluations (dark) and scores within a single standard deviation of best (light). All runs of COWBOYS achieve roughly comparable results except the two on the far right, i.e, u...
work page 2024
-
[9]
Gaussian Process Molecule Property Prediction with FlowMO
Moss, H., Leslie, D., Beck, D., Gonzalez, J., and Rayson, P. Boss: Bayesian optimization over string spaces. Ad- vances in neural information processing systems, 2020a. Moss, H. B. and Griffiths, R.-R. Gaussian process molecule property prediction with flowmo.arXiv preprint arXiv:2010.01118,
work page Pith review arXiv 2010
-
[11]
Big Batch Bayesian Active Learning by Considering Predictive Probabilities
Ober, S. W., Power, S., Diethe, T., and Moss, H. B. Big batch Bayesian active learning by considering predictive probabilities. arXiv preprint arXiv:2501.08223,
-
[13]
White, T. Sampling generative networks. arXiv preprint arXiv:1609.04468,
-
[14]
as Algorithms 3 and 4 which we use to get a set of samples from COWBOYS sampling objective (4). Note that we keep track of the latent value that decoded to give the best structure so far zbest and use this to start our chains. We also employ an adaptive choice of PCN’s scale parameter β following Andrieu & Thoms (2008). Often, especially early in the opti...
work page 2008
-
[1985]
Denoising diffusion im- plicit models
Song, J., Meng, C., and Ermon, S. Denoising diffusion im- plicit models. arXiv preprint arXiv:2010.02502, 2020a. Song, Y ., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020b. Stanton, S., Maddox, W., Gruver, N., Maffetto...
arXiv 2010
-
[2011]
The Bayesian approach to global optimization
Mockus, J. The Bayesian approach to global optimization. In System Modeling and Optimization: Proceedings of the 10th IFIP Conference New York City, USA, August 31–September 4, 1981, pp. 473–481. Springer,
work page 1981
-
[2013]
πbo: Augmenting acquisition functions with user beliefs for bayesian optimization
Hvarfner, C., Stoll, D., Souza, A., Lindauer, M., Hutter, F., and Nardi, L. πbo: Augmenting acquisition functions with user beliefs for bayesian optimization. In Tenth International Conference of Learning Representations, ICLR 2022,
work page 2022
Show all 15 references
-
[2018]
A survey and benchmark of high-dimensional bayesian optimization of discrete sequences
Gonz´alez-Duque, M., Michael, R., Bartels, S., Zainchkovskyy, Y ., Hauberg, S., and Boomsma, W. A survey and benchmark of high-dimensional bayesian optimization of discrete sequences. arXiv preprint arXiv:2406.04739,
-
[2019]
E., Verma, P., John, S., Picheny, V ., Moss, H., and Solin, A
Chang, P. E., Verma, P., John, S., Picheny, V ., Moss, H., and Solin, A. Fantasizing with dual gps in bayesian optimization and active learning. arXiv preprint arXiv:2211.01053,
-
[2020]
J., Paige, B., and Hern ´andez-Lobato, J
Kusner, M. J., Paige, B., and Hern ´andez-Lobato, J. M. Grammar variational autoencoder. In International con- ference on machine learning , pp. 1945–1954. PMLR,
1945
-
[2021]
B., Ober, S
Moss, H. B., Ober, S. W., and Picheny, V . Information- theoretic inducing point placement for high-throughput bayesian optimisation. arXiv preprint arXiv:2206.02437,
-
[2022]
O., and Nardi, L
Hvarfner, C., Hellsten, E. O., and Nardi, L. Vanilla bayesian optimization performs great in high dimension. arXiv preprint arXiv:2402.02229,
-
[2024]
M., Griffiths, R.-R., Cowen-Rivers, A
Grosnit, A., Tutunov, R., Maraval, A. M., Griffiths, R.-R., Cowen-Rivers, A. I., Yang, L., Zhu, L., Lyu, W., Chen, Z., Wang, J., et al. High-dimensional Bayesian optimisation with variational autoencoders and deep metric learning. arXiv preprint arXiv:2106.03609,
-
[2025]
J., Bradshaw, J., and Gardner, J
Maus, N., Jones, H., Moore, J., Kusner, M. J., Bradshaw, J., and Gardner, J. Local latent space Bayesian optimization over structured inputs. Advances in Neural Information Processing Systems, 2022a. Maus, N., Wu, K., Eriksson, D., and Gardner, J. Discovering many diverse solu...
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.