REVIEW 3 major objections 4 minor 43 references
How to Spend Your Oracle Budget: Practical Guidance for Protein Structure Prediction Models
T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read The paper claims O3 is the best way to spend a small oracle budget when guiding Boltz-2 structure prediction, while FK-steering and DPO need larger budgets to pay off.
desk verdict First systematic oracle-budget benchmark for protein guidance, but O3's low-budget dominance is partly an artifact of per-budget tuning of its subspace dimension on the test target. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is O3's surrogate subspace $U$, constructed from the $d$ highest-scoring seed latents: a Knothe–Rosenblatt transform maps a point $u \in [0,1]^{d-1}$ to simplex weights, and a LOL projection $\ell(w,Z) = w^\top Z$ maps those weights back into the latent support of the Gaussian prior. Because Boltz-2 is made deterministic through its probability-flow formulation, each decoded structure can be scored by the oracle, and Bayesian optimisation with a Gaussian-process surrogate and Log Expected Improvement spends the remaining budget choosing new points in this low-dimensional subspace. This machinery turns a high-dimensional generation problem into a $d-1$ dimensional black-box optimisation problem that is cheap enough to navigate with a small number of oracle calls.
What would settle it
Re-run the same six budget configurations on a diverse set of protein targets using a genuinely expensive, non-differentiable oracle such as a short molecular-dynamics stability score; if at $N \leq 100$ Best K-of-N ties or beats O3 on most targets, the paper's low-budget recommendation fails.
Extended reading notes
Core claim
The paper claims to provide the first practical reference for oracle-budget-aware guidance in protein structure prediction, and the first application of the O3 framework to such a model. On calmodulin with the TM-score oracle, O3 achieves a mean-of-K TM-score that plateaus around 0.81 and outperforms all baselines at budgets up to 1000 calls, while Best K-of-N stays roughly flat at about 0.60; FK-steering rises from about 0.55 at 20 calls to about 0.73 at 1000, and DPO reaches about 0.71, with the online variant improving steadily and the offline variant flat regardless of budget. On E. coli aspartate transcarbamoylase with the reference-free MolProbity oracle under max-of-K, Best K-of-N is strongest at large budgets and O3 does not improve, showing that the ranking depends on the oracle and the metric. The paper concludes that no single guidance method dominates across all budgets and oracles, and that the practical recommendation is to use O3 at constrained oracle budgets.
Load-bearing premise
The recommendations assume that two cheap stand-in scores—TM-score, which needs the known reference structure, and MolProbity, which checks local geometry—behave like the expensive biological oracle calls the budget is meant to model, and that results on two test proteins generalise to other targets.
Editorial extensions
If this is right
- At oracle budgets of 100 calls or fewer, practitioners should prefer O3 over Best K-of-N, FK-steering, and DPO, since it is the only method that meaningfully improves on the simple sampling baseline.
- FK-steering can overtake Best K-of-N at budgets of a few hundred calls when the oracle rewards global fold similarity, but it should not be trusted when the oracle measures local geometry, because intermediate denoising steps then carry noisy reward signal.
- Online DPO is the only method that improves through parameter updates and should be reserved for budgets large enough to fine-tune a foundation-scale model; offline DPO plateaus and spends the same budget without comparable gains.
- Method selection should be made jointly on budget size and oracle behaviour rather than chosen once: the empirical ranking on TM-score is not the same as the ranking on MolProbity.
- Best K-of-N is a stronger baseline than its simplicity suggests, staying roughly flat across budgets and beating sophisticated methods in some large-budget, max-of-K settings.
Reading between the lines
- The paper's own ablation suggests that random search inside the O3 subspace already beats Best K-of-N, so a cheaper practical variant that skips Bayesian optimisation might preserve much of the low-budget advantage while spending even fewer calls on surrogate fitting.
- The divergent behaviour of FK-steering under TM-score versus MolProbity implies that an adaptive oracle-aware scheduler could switch guidance methods based on whether the oracle scores global shape or local geometry, something the paper does not evaluate.
- A testable extension is to replace TM-score with a reference-free oracle that still varies smoothly with global structure, such as an ensemble-consistency or model-confidence score; if O3's low-budget dominance persists, the advice would extend beyond settings where a ground-truth structure is available.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents an empirical comparison of four guidance methods—FK-steering, DPO, Best K-of-N sampling, and the recently proposed O3—for spending a fixed oracle budget when generating protein structures with Boltz-2. It evaluates mean- and max-of-K scores at six (N,K) budgets on calmodulin (1CLL) with a TM-score oracle and on E. coli aspartate transcarbamoylase (9EEH) with a MolProbity-based oracle. The headline finding is that O3 dominates at low-to-mid oracle budgets on the TM-score target, while FK-steering and DPO improve as the budget grows, leading to the practical recommendation to use O3 under constrained budgets. The paper includes ablations for O3 subspace dimension, seed selection, GP kernel, budget allocation, and FK-steering and DPO variants.
Significance. If the headline comparison were fully controlled, this would be a useful first practical reference for oracle-budget-aware guidance in protein structure prediction. The paper has real strengths: the methods are clearly defined, oracles are external benchmarks, results are accompanied by error bars and multiple seeds, and the appendices document compute usage and numerous ablations. The O3 application to Boltz-2 is new and the low-budget regime is under-explored, so the question is timely. However, two uncontrolled choices—per-budget selection of O3's subspace dimension on the test target, and the deterministic-sampler difference for O3—directly affect the headline ranking, and the evidence base is only two targets/oracles. The practical advice is therefore not yet supported at the strength with which it is stated.
major comments (3)
- [§5.2, Fig. 5, Table A.1] The headline O3 curves in Fig. 2 are computed with per-budget values of the subspace dimension d (and M) listed in Table A.1, and §5.2 shows that the best d changes with budget (d=6 at N=20 versus d=10 at N=200). Because d and M are selected using the evaluation oracle on the same test target (Fig. 5 sweeps d on 1CLL/TM-score), the Fig. 2 O3 results are an upper envelope over hyperparameters rather than the performance of a fully specified method. None of the baselines has an analogous per-budget free parameter tuned on the test target, so the comparison is asymmetric; moreover Fig. 5 uses only three seeds and the d values differ between targets (Table B.1). A validation-based rule for selecting d and M, or a sensitivity analysis showing that the low-budget dominance survives a fixed d, is required before the Section 6 recommendation can be accepted.
- [§3.1 footnote; §B.2] O3 requires a deterministic generative model and is therefore run with the probability-flow ODE, while FK-steering, DPO, and Best K-of-N use the stochastic Boltz-2 sampler. The paper's own discussion in Appendix B.2 attributes part of O3's 9EEH behavior to this sampler mismatch and notes that the stochastic sampler improves output diversity. This confounds the method comparison in Fig. 2: the differences between O3 and the baselines could be due to the guidance method or to the sampling scheme. To make the benchmark fair, the authors should either adapt O3 to the stochastic sampler or run the baselines under the same deterministic ODE, and report both settings.
- [§4, §6, Figs. B.1-B.2] The practical advice 'use O3 at constrained oracle budgets' is stated without sufficient qualification relative to the evidence. On 9EEH with MolProbity, O3 is not dominant under max-of-K (Best K-of-N is strongest at large budgets, Fig. B.2), and under mean-of-K the between-method differences are small with overlapping three-seed error bars; the authors themselves describe these as consistent trends rather than individually meaningful differences. The abstract's statement that no single method dominates across budgets and oracles is in tension with the unqualified 'O3 dominates' in Section 6. Furthermore, the TM-score oracle requires the known ground-truth reference, which is precisely what is unavailable in the design settings the paper motivates, while MolProbity has a narrow score range and, as the authors note, gives nearly uninformative intermediate signal to FK-steering. A two-target evidence base with these oracle properties is too narrow to support a general practitioner recommendation.
minor comments (4)
- [Tables A.1 and B.1] The rows are visually concatenated (e.g., 'n20 k220 6 10 5 8'), making N, K, M, d, and nrounds hard to read; please reformat as proper tables with clear column separation.
- [Figure A.10 caption] The caption says 'Bar are means' but should say 'Bars are means'.
- [§3.3] The DPO experiments never report β, learning rate, batch size, or number of epochs; for a benchmarking paper these hyperparameters should be specified in the text or appendix.
- [Figure 4 caption] The caption uses 'Best-of-N (mean=0.60, n=5)', where n appears to denote the number of seeds rather than the sampling budget; please clarify the notation to avoid confusion with N.
Circularity Check
O3's headline low-budget win uses per-budget subspace dimensions selected on the test oracle, making the comparison an upper envelope rather than a specified method.
-
fitted input called prediction
[Section 5.1 (Fig. 2), Section 5.2 (Fig. 5), Table A.1]
"O3 outperforms all baselines at low-mid budgets (N≤1000 ) with the mean value plateauing at∼0.81 ... In the experiment summarised by Fig. 5 we sweep across subspace dimensions d for different available budgets. The key takeaway is that best-performing d varies with the budget — different oracle-call regimes favour different subspace dimensions. For example, when N=20 the best-performing dimension is d=6; however, for N=200 dimension d=10 yields better predictions."
The headline comparison (Fig. 2) presents O3's mean-of-K TM-score at each budget, while Table A.1 fixes a single subspace dimension d per (N,K) configuration for those runs. Section 5.2 and Fig. 5 show that the best d is selected on the same budget, target, and oracle by sweeping d against the TM-score evaluation metric. The paper offers no validation-based rule for choosing d without accessing the test-target oracle, so a practitioner at an unknown budget cannot apply O3 as specified. The reported O3 performance is therefore the upper envelope of the d sweep, not the performance of a single well-defined method, while FK-steering, DPO, and Best K-of-N have no comparable per-budget free parameter fitted on the evaluation oracle.
full rationale
The empirical portion of the paper is not circular in the usual sense: TM-score and MolProbity are external oracles, the baseline algorithms (FK-steering, DPO, Best K-of-N) are named and implemented from the literature, and Boltz-2 is a third-party model. A benchmark against external ground truth cannot be 'equivalent to its inputs by construction' merely because the authors recommend a method from their own prior work. The self-citation to Willis et al. (2025) is real—Henry Moss appears on both papers—but O3's subspace construction is described concretely (Algorithm 1, Knothe–Rosenblatt plus LOL projection, BO in U) and its performance is measured, not asserted from the citation. The specific circularity is narrower and located in the O3 hyperparameter d: Section 5.2 sweeps d on the same 1CLL/TM-score problem used for the headline Fig. 2, and Table A.1 then fixes per-budget d values. Because no rule is given for setting d without the test-target oracle, Fig. 2's O3 bars are best-case selections over d, an upper envelope rather than a single specified method. This makes the low-budget dominance claim 'fitted input called prediction' in the sense of pattern 2: the method's headline performance is partly manufactured by choosing the free parameter on the evaluation oracle. It is partial rather than total circularity: O3 may still be competitive at many d values, and the 9EEH/MolProbity replication is an external check, but the headline quantitative claim is not a fair comparison as reported.
Assumptions & free parameters
free parameters (4)
- d (O3 subspace dimension) =
varies per budget: d=6 (N=20), d=10 (N=200), etc. (Table A.1)
- M (initial samples for O3 subspace) =
varies: M=10 (N=20), M=100 (N=1000), etc. (Table A.1)
- lambda (FK-steering reward scale) =
50
- DPO hyperparameters (beta, learning rate, batch size, epochs) =
not reported
assumptions (4)
- domain assumption The O3 framework (surrogate chart via Knothe-Rosenblatt and LOL projection) defines a valid low-dimensional subspace for optimization.
- domain assumption Boltz-2's stochastic generation can be replaced by its probability-flow ODE without losing the properties needed for guidance.
- domain assumption TM-score (against a known structure) and MolProbity (physical plausibility) are representative proxies for expensive biological oracles.
- domain assumption The two test proteins, 1CLL and 9EEH, represent typical protein structure prediction targets.
Cite this review
Pith. "Pith review of How to Spend Your Oracle Budget: Practical Guidance for Protein Structure Prediction Models." pith.science (2026). https://pith.science/paper/ECD6LOSX
@misc{pith2026260812192,
author = {Pith},
title = {Pith review of: How to Spend Your Oracle Budget: Practical Guidance for Protein Structure Prediction Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/ECD6LOSX}},
note = {Machine review of arXiv:2608.12192}
}
read the original abstract
Foundation models for protein structure prediction remain unreliable on certain targets. External oracles can flag and correct these failures, but biological oracles are expensive, making oracle budget a critical constraint. Existing guidance methods, such as FK-steering, DPO, and Best K-of-N sampling, differ in how they spend this budget, yet no systematic comparison exists to guide method selection. To bridge this gap, we benchmark these methods alongside the recently proposed Optimisation Over Outputs (O3), which applies off-the-shelf optimisers within a generative model's latent subspace. We extend the usage of O3 to protein structure prediction models. Overall, our work provides the first practical reference for oracle budget-aware guidance. Our evaluation on two protein targets, calmodulin (1CLL) and E. coli aspartate transcarbamoylase (9EEH), reveals that no single method consistently dominates across all budgets and oracles. Specifically, O3 proves most effective at low oracle budgets, while FK-steering and DPO demonstrate improved performance as the budget increases. We distil these findings into actionable recommendations for practitioners operating under real-world oracle-budget constraints.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
arXiv , author =:2509.23800 , primaryClass =
-
[2]
2023 , publisher=
Bayesian optimization , author=. 2023 , publisher=
2023
-
[3]
International Conference on Machine Learning , pages=
Return of the Latent Space COWBOYS: Re-thinking the use of VAEs for Bayesian Optimisation of Structured Spaces , author=. International Conference on Machine Learning , pages=. 2025 , organization=
work page 2025
-
[4]
arXiv , author =:2501.06848 , primaryClass =
-
[5]
arXiv , author =:2305.18290 , primaryClass =
-
[6]
and Bambrick, Joshua and Bodenstein, Sebastian W
Abramson, Josh and Adler, Jonas and Dunger, Jack and Evans, Richard and Green, Tim and Pritzel, Alexander and Ronneberger, Olaf and Willmore, Lindsay and Ballard, Andrew J. and Bambrick, Joshua and Bodenstein, Sebastian W. and Evans, David A. and Hung, Chia-Chun and O'Neill, Michael and Reiman, David and Tunyasuvunakool, Kathryn and Wu, Zachary and Žemgul...
-
[7]
Passaro, Saro and Corso, Gabriele and Wohlwend, Jeremy and Reveiz, Mateo and Thaler, Stephan and Somnath, Vignesh Ram and Getz, Noah and Portnoi, Tally and Roy, Julien and Stark, Hannes and Kwabi-Addo, David and Beaini, Dominique and Jaakkola, Tommi and Barzilay, Regina , month =
-
[8]
Buttenschoen, Martin and Morris, Garrett M. and Deane, Charlotte M. , issn =. Chemical Science , number =
Show all 43 references
-
[9]
and Ojoawo, Adedolapo and Otten, Renee and Apitz, Julia M
Wayment-Steele, Hannah K. and Ojoawo, Adedolapo and Otten, Renee and Apitz, Julia M. and Pitsawong, Warintra and Hömberger, Marc and Ovchinnikov, Sergey and Colwell, Lucy and Kern, Dorothee , issn =. Nature , month =
-
[10]
, issn =
Yin, Rui and Pierce, Brian G. , issn =. Protein Science , month =
-
[11]
and Dror, Ron O
Hollingsworth, Scott A. and Dror, Ron O. , issn =. Neuron , month =
-
[12]
Jumper, John and Evans, Richard and Pritzel, Alexander and Green, Tim and Figurnov, Michael and Ronneberger, Olaf and Tunyasuvunakool, Kathryn and Bates, Russ and Žídek, Augustin and Potapenko, Anna and Bridgland, Alex and Meyer, Clemens and Kohl, Simon A. A. and Ballard, Andr...
-
[13]
arXiv , author =:2408.08558 , primaryClass =
-
[14]
arXiv , author =:2006.09191 , primaryClass =
2006 arXiv
-
[15]
arXiv , author =:2201.11872 , primaryClass =
-
[16]
and Duvenaud, David and Hernández-Lobato, José Miguel and Sánchez-Lengeling, Benjamín and Sheberla, Dennis and Aguilera-Iparraguirre, Jorge and Hirzel, Timothy D
Gómez-Bombarelli, Rafael and Wei, Jennifer N. and Duvenaud, David and Hernández-Lobato, José Miguel and Sánchez-Lengeling, Benjamín and Sheberla, Dennis and Aguilera-Iparraguirre, Jorge and Hirzel, Timothy D. and Adams, Ryan P. and Aspuru-Guzik, Alán , issn =. ACS Central Scie...
-
[17]
arXiv , author =:2105.05233 , primaryClass =
-
[18]
arXiv , author =:2207.12598 , primaryClass =
-
[19]
arXiv , author =:2305.13301 , primaryClass =
-
[20]
arXiv , author =:2309.17400 , primaryClass =
-
[21]
arXiv , author =:2311.12908 , primaryClass =
-
[22]
arXiv , author =:2206.00364 , primaryClass =
-
[23]
and Daulton, Samuel and Letham, Benjamin and Wilson, Andrew Gordon and Bakshy, Eytan , booktitle =
Balandat, Maximilian and Karrer, Brian and Jiang, Daniel R. and Daulton, Samuel and Letham, Benjamin and Wilson, Andrew Gordon and Bakshy, Eytan , booktitle =
-
[24]
2006 , publisher=
Gaussian processes for machine learning , author=. 2006 , publisher=
2006
-
[25]
Advances in neural information processing systems , volume=
Unexpected improvements to expected improvement for bayesian optimization , author=. Advances in neural information processing systems , volume=
-
[26]
and Means, Anthony R
Chattopadhyaya, Rajagopal and Meador, William E. and Means, Anthony R. and Quiocho, Florante A. , issn =. Journal of Molecular Biology , month =
-
[27]
and Westbrook, John and Feng, Zukang and Gilliland, Gary and Bhat, T
Berman, Helen M. and Westbrook, John and Feng, Zukang and Gilliland, Gary and Bhat, T. N. and Weissig, Helge and Shindyalov, Ilya N. and Bourne, Philip E. , issn =. Nucleic Acids Research , month =
-
[28]
The Annals of Mathematical Statistics , month =
Rosenblatt, Murray , issn =. The Annals of Mathematical Statistics , month =
-
[29]
Michigan Mathematical Journal , month =
Knothe, Herbert , issn =. Michigan Mathematical Journal , month =
-
[30]
Proteins: Structure, Function, and Bioinformatics , month =
Zhang, Yang and Skolnick, Jeffrey , issn =. Proteins: Structure, Function, and Bioinformatics , month =
-
[31]
arXiv preprint arXiv:2011.13456 , year=
Score-based generative modeling through stochastic differential equations , author=. arXiv preprint arXiv:2011.13456 , year=
2011 arXiv
-
[32]
arXiv preprint arXiv:2405.08448 , year=
Understanding the performance gap between online and offline alignment algorithms , author=. arXiv preprint arXiv:2405.08448 , year=
-
[33]
and Arendall, W
Chen, Vincent B. and Arendall, W. Bryan and Headd, Jeffrey J. and Keedy, Daniel A. and Immormino, Robert M. and Kapral, Gary J. and Murray, Laura W. and Richardson, Jane S. and Richardson, David C. , journal =. 2010 , publisher =
2010
-
[34]
and Headd, Jeffrey J
Williams, Christopher J. and Headd, Jeffrey J. and Moriarty, Nigel W. and Prisant, Michael G. and Videau, Lizbeth L. and Deis, Lindsay N. and Verma, Vishal and Keedy, Daniel A. and Hintze, Bradley J. and Chen, Vincent B. and Jain, Swati and Lewis, Steven M. and Arendall, W. Br...
2018
-
[35]
and Patterson, Michael G
Miller, Robert C. and Patterson, Michael G. and Bhatt, Neti and Pei, Xiaokun and Ando, Nozomi , title =. Nature Communications , year =. doi:10.1038/s41467-026-70909-y , url =
-
[36]
Langley , title =
P. Langley , title =. Proceedings of the 17th International Conference on Machine Learning (ICML 2000) , address =. 2000 , pages =
2000
-
[37]
T. M. Mitchell. The Need for Biases in Learning Generalizations. 1980
1980
-
[38]
M. J. Kearns , title =
-
[39]
Machine Learning: An Artificial Intelligence Approach, Vol. I. 1983
1983
-
[40]
R. O. Duda and P. E. Hart and D. G. Stork. Pattern Classification. 2000
2000
-
[41]
Suppressed for Anonymity , author=
-
[42]
Newell and P
A. Newell and P. S. Rosenbloom. Mechanisms of Skill Acquisition and the Law of Practice. Cognitive Skills and Their Acquisition. 1981
1981
-
[43]
A. L. Samuel. Some Studies in Machine Learning Using the Game of Checkers. IBM Journal of Research and Development. 1959
1959
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.