{"id":"0a068245-5bc8-45e5-b816-fbffc85b0685","arxiv_id":"2608.12192","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Under a fixed oracle budget, latent-subspace optimization (O3) beats steering and fine-tuning baselines at low budgets, while FK-steering and DPO catch up as the budget grows.","lead":"This paper benchmarks four ways to spend a limited budget of oracle calls when guiding protein structure prediction models, and finds that the choice of method depends on budget size. It offers the first practical comparison of these guidance strategies, which matters because expensive biological oracles make budget-aware method selection a real constraint in drug discovery and design.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"O3's low-budget win depends on choosing its subspace dimension d per budget on the test target; without a validation-based d rule, the headline comparison is an upper envelope, not a fair benchmark.","rationale":"The reader's conditional verdict is reasonable, and my concern sharpens the internal-validity threat: the main experiment's key hyperparameter appears to be tuned on the evaluation target. This is more decisive than the oracle-generalization concern because it attacks whether Fig. 2 supports the claim even on its own terms. The paper has real strengths: multiple seeds, ablations, a second oracle, and clear error bars; the 9EEH/MolProbity result partially supports O3 on mean-of-K and partially contradicts on max-of-K (Appendix B.2), which reinforces the need to qualify the practical advice. I do not allege any bad faith; the omission is that d selection is not described as a validation procedure. If the authors can show d was chosen by a fixed rule or that the result is robust to d, the paper's conditional acceptance is appropriate. If not, the headline recommendation should be weakened. Agreement is partial because the reader's stated weakest assumption was oracle generalizability, whereas I focus on test-set hyperparameter selection; both point to the same overall conditional verdict.","tokens_in":16089,"tokens_out":10326,"duration_ms":93579,"concrete_test":"Use the released code (or re-implement) to re-run the N=20, 50, 100 1CLL/TM-score comparisons with (a) a single fixed d across all budgets, e.g. d=6, and (b) d selected from a validation split of the oracle budget, e.g. reserve half the seeds, choose d on them, then evaluate on the held-out half. If O3's mean-of-K no longer exceeds Best K-of-N by more than the reported +/-1 std error bars, the claim that O3 is the only method that meaningfully improves on Best K-of-N at low budgets depends on test-set hyperparameter tuning and should be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Fig. 2's headline result is not a fair method comparison unless O3's subspace dimension d is chosen without using the evaluation oracle on the test target. Section 5.2 shows the best d varies with budget (d=6 at N=20, d=10 at N=200), and Table A.1 lists per-budget d and M values used for the headline curves, while Fig. 5 sweeps d on the same 1CLL/TM-score problem. No rule is given for setting d that a practitioner could apply at unknown budget; if the reported curves use the best d per budget, O3's mean-of-K is an upper envelope over the swept hyperparameter rather than the performance of a specified method. The other baselines have no comparable per-budget free parameter, so the comparison is asymmetric. Because Fig. 5 uses only three seeds and the 9EEH table uses different d values, the tuning is also unstable across targets. If O3's margin at N<=100 disappears under validation-based d selection, the central advice fails.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an empirical comparison of four guidance methods—FK-steering, DPO, Best K-of-N sampling, and the recently proposed O3—for spending a fixed oracle budget when generating protein structures with Boltz-2. It evaluates mean- and max-of-K scores at six (N,K) budgets on calmodulin (1CLL) with a TM-score oracle and on E. coli aspartate transcarbamoylase (9EEH) with a MolProbity-based oracle. The headline finding is that O3 dominates at low-to-mid oracle budgets on the TM-score target, while FK-steering and DPO improve as the budget grows, leading to the practical recommendation to use O3 under constrained budgets. The paper includes ablations for O3 subspace dimension, seed selection, GP kernel, budget allocation, and FK-steering and DPO variants.","tokens_in":16240,"tokens_out":7625,"duration_ms":65229,"significance":"If the headline comparison were fully controlled, this would be a useful first practical reference for oracle-budget-aware guidance in protein structure prediction. The paper has real strengths: the methods are clearly defined, oracles are external benchmarks, results are accompanied by error bars and multiple seeds, and the appendices document compute usage and numerous ablations. The O3 application to Boltz-2 is new and the low-budget regime is under-explored, so the question is timely. However, two uncontrolled choices—per-budget selection of O3's subspace dimension on the test target, and the deterministic-sampler difference for O3—directly affect the headline ranking, and the evidence base is only two targets/oracles. The practical advice is therefore not yet supported at the strength with which it is stated.","major_comments":[{"comment":"The headline O3 curves in Fig. 2 are computed with per-budget values of the subspace dimension d (and M) listed in Table A.1, and §5.2 shows that the best d changes with budget (d=6 at N=20 versus d=10 at N=200). Because d and M are selected using the evaluation oracle on the same test target (Fig. 5 sweeps d on 1CLL/TM-score), the Fig. 2 O3 results are an upper envelope over hyperparameters rather than the performance of a fully specified method. None of the baselines has an analogous per-budget free parameter tuned on the test target, so the comparison is asymmetric; moreover Fig. 5 uses only three seeds and the d values differ between targets (Table B.1). A validation-based rule for selecting d and M, or a sensitivity analysis showing that the low-budget dominance survives a fixed d, is required before the Section 6 recommendation can be accepted.","section":"§5.2, Fig. 5, Table A.1"},{"comment":"O3 requires a deterministic generative model and is therefore run with the probability-flow ODE, while FK-steering, DPO, and Best K-of-N use the stochastic Boltz-2 sampler. The paper's own discussion in Appendix B.2 attributes part of O3's 9EEH behavior to this sampler mismatch and notes that the stochastic sampler improves output diversity. This confounds the method comparison in Fig. 2: the differences between O3 and the baselines could be due to the guidance method or to the sampling scheme. To make the benchmark fair, the authors should either adapt O3 to the stochastic sampler or run the baselines under the same deterministic ODE, and report both settings.","section":"§3.1 footnote; §B.2"},{"comment":"The practical advice 'use O3 at constrained oracle budgets' is stated without sufficient qualification relative to the evidence. On 9EEH with MolProbity, O3 is not dominant under max-of-K (Best K-of-N is strongest at large budgets, Fig. B.2), and under mean-of-K the between-method differences are small with overlapping three-seed error bars; the authors themselves describe these as consistent trends rather than individually meaningful differences. The abstract's statement that no single method dominates across budgets and oracles is in tension with the unqualified 'O3 dominates' in Section 6. Furthermore, the TM-score oracle requires the known ground-truth reference, which is precisely what is unavailable in the design settings the paper motivates, while MolProbity has a narrow score range and, as the authors note, gives nearly uninformative intermediate signal to FK-steering. A two-target evidence base with these oracle properties is too narrow to support a general practitioner recommendation.","section":"§4, §6, Figs. B.1-B.2"}],"minor_comments":[{"comment":"The rows are visually concatenated (e.g., 'n20 k220 6 10 5 8'), making N, K, M, d, and nrounds hard to read; please reformat as proper tables with clear column separation.","section":"Tables A.1 and B.1"},{"comment":"The caption says 'Bar are means' but should say 'Bars are means'.","section":"Figure A.10 caption"},{"comment":"The DPO experiments never report β, learning rate, batch size, or number of epochs; for a benchmarking paper these hyperparameters should be specified in the text or appendix.","section":"§3.3"},{"comment":"The caption uses 'Best-of-N (mean=0.60, n=5)', where n appears to denote the number of seeds rather than the sampling budget; please clarify the notation to avoid confusion with N.","section":"Figure 4 caption"}],"recommendation":"major_revision","confidential_remarks":"The main concern is whether the authors can re-run the comparison under a principled d-selection protocol; if they can, the paper is publishable. I also note that O3 is the authors' own prior method (Willis et al. 2025) with overlapping authorship, so editors should ensure the novelty and positioning are presented transparently. The current manuscript appears to be a workshop-length empirical study; the balance between the headline claim and the two-target evidence base should be checked during revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper gives the first systematic comparison of oracle-budget-aware guidance methods for protein structure prediction, and the first application of O3 to that domain. The headline claim—O3 dominates at low budgets—is not fully supported as stated, because the O3 subspace dimension d is chosen per budget on the test target, making the reported curves an upper envelope rather than a fair method comparison. Still, the benchmark is carefully run and worth refereeing.\n\nWhat is genuinely new: the budget comparison across FK-steering, DPO, Best K-of-N, and O3 on two proteins with two oracles, with multiple seeds, error bars, and decent ablations. The 9EEH/MolProbity result is a real asset: it shows an oracle-dependent reversal (FK-steering strong on TM-score, weak on MolProbity) and the authors discuss it openly rather than hiding it. That honesty carries through the appendix, where they acknowledge that O3 uses the probability-flow ODE while baselines use the stochastic sampler—a real confound but a transparent one.\n\nThe soft spots are concentrated where the reader flagged them. The d-tuning issue is load-bearing. Table A.1 lists per-budget M and d values used for the headline curves, and Fig. 5 sweeps d on the same 1CLL/TM-score problem. No validation-based rule is given for selecting d at an unknown budget, so a practitioner cannot reproduce the advertised gains without peeking at the test oracle. The 9EEH max-of-K result, where O3 lands below Best K-of-N, cuts against the abstract's 'O3 proves most effective at low budgets'—the abstract should have qualified that with the oracle and metric. The lack of released code is a smaller issue but matters for a paper positioned as a practical reference.\n\nI don't think the core advice is wrong: the ablations show the low-dimensional subspace itself carries value, and O3 with random sampling beats Best K-of-N at N=100. But the magnitude of the claimed dominance is likely inflated. The fix is feasible—report a d-selection rule based on the budget alone or a validation split—and the benchmark would be a useful baseline for the community regardless.\n\nVerdict: send to review. A serious referee would ask exactly the d-selection question and the sampler confound; the authors can address both. This is a solid, useful contribution with a fixable flaw, not a desk reject.","headline":"First systematic oracle-budget benchmark for protein guidance, but O3's low-budget dominance is partly an artifact of per-budget tuning of its subspace dimension on the test target.","tokens_in":16893,"tokens_out":4270,"would_cite":true,"duration_ms":33721,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims O3 is the best way to spend a small oracle budget when guiding Boltz-2 structure prediction, while FK-steering and DPO need larger budgets to pay off.","keywords":["protein structure prediction","oracle budget","inference-time guidance","O3","FK-steering","direct preference optimisation","Bayesian optimisation","latent subspace"],"falsifier":"Re-run the same six budget configurations on a diverse set of protein targets using a genuinely expensive, non-differentiable oracle such as a short molecular-dynamics stability score; if at $N \\leq 100$ Best K-of-N ties or beats O3 on most targets, the paper's low-budget recommendation fails.","tokens_in":15827,"feed_emoji":"🧬","tokens_out":10919,"duration_ms":82187,"temperature":0.7,"pith_summary":"The paper asks how a practitioner with a fixed, small number of expensive oracle calls should guide a protein-structure prediction model toward better outputs. It benchmarks four guidance strategies applied to Boltz-2 at budgets from 20 to 1000 oracle calls, scoring predictions with TM-score against a known structure and with MolProbity for physical plausibility. Its central finding is that no method dominates everywhere: O3, which optimises inside a low-dimensional latent subspace built from a few high-scoring seed generations, is the only method that meaningfully improves on simple Best K-of-N sampling when the budget is 100 calls or less, and it leads at low-to-mid budgets; FK-steering and online DPO improve as the budget grows. The paper distils this into practical advice: use O3 when oracle calls are constrained, consider FK-steering at moderate budgets, and reach for DPO when enough calls are available to fine-tune the model.","feed_headline":"O3 wins when oracle calls are scarce","feed_subtitle":"First Boltz-2 guidance benchmark: O3 leads at low budgets; FK-steering and DPO catch up as calls grow.","key_machinery":"The central object is O3's surrogate subspace $U$, constructed from the $d$ highest-scoring seed latents: a Knothe–Rosenblatt transform maps a point $u \\in [0,1]^{d-1}$ to simplex weights, and a LOL projection $\\ell(w,Z) = w^\\top Z$ maps those weights back into the latent support of the Gaussian prior. Because Boltz-2 is made deterministic through its probability-flow formulation, each decoded structure can be scored by the oracle, and Bayesian optimisation with a Gaussian-process surrogate and Log Expected Improvement spends the remaining budget choosing new points in this low-dimensional subspace. This machinery turns a high-dimensional generation problem into a $d-1$ dimensional black-box optimisation problem that is cheap enough to navigate with a small number of oracle calls.","core_discovery":"The paper claims to provide the first practical reference for oracle-budget-aware guidance in protein structure prediction, and the first application of the O3 framework to such a model. On calmodulin with the TM-score oracle, O3 achieves a mean-of-K TM-score that plateaus around 0.81 and outperforms all baselines at budgets up to 1000 calls, while Best K-of-N stays roughly flat at about 0.60; FK-steering rises from about 0.55 at 20 calls to about 0.73 at 1000, and DPO reaches about 0.71, with the online variant improving steadily and the offline variant flat regardless of budget. On E. coli aspartate transcarbamoylase with the reference-free MolProbity oracle under max-of-K, Best K-of-N is strongest at large budgets and O3 does not improve, showing that the ranking depends on the oracle and the metric. The paper concludes that no single guidance method dominates across all budgets and oracles, and that the practical recommendation is to use O3 at constrained oracle budgets.","pith_inferences":["The paper's own ablation suggests that random search inside the O3 subspace already beats Best K-of-N, so a cheaper practical variant that skips Bayesian optimisation might preserve much of the low-budget advantage while spending even fewer calls on surrogate fitting.","The divergent behaviour of FK-steering under TM-score versus MolProbity implies that an adaptive oracle-aware scheduler could switch guidance methods based on whether the oracle scores global shape or local geometry, something the paper does not evaluate.","A testable extension is to replace TM-score with a reference-free oracle that still varies smoothly with global structure, such as an ensemble-consistency or model-confidence score; if O3's low-budget dominance persists, the advice would extend beyond settings where a ground-truth structure is available."],"forward_implications":["At oracle budgets of 100 calls or fewer, practitioners should prefer O3 over Best K-of-N, FK-steering, and DPO, since it is the only method that meaningfully improves on the simple sampling baseline.","FK-steering can overtake Best K-of-N at budgets of a few hundred calls when the oracle rewards global fold similarity, but it should not be trusted when the oracle measures local geometry, because intermediate denoising steps then carry noisy reward signal.","Online DPO is the only method that improves through parameter updates and should be reserved for budgets large enough to fine-tune a foundation-scale model; offline DPO plateaus and spends the same budget without comparable gains.","Method selection should be made jointly on budget size and oracle behaviour rather than chosen once: the empirical ranking on TM-score is not the same as the ranking on MolProbity.","Best K-of-N is a stronger baseline than its simplicity suggests, staying roughly flat across budgets and beating sophisticated methods in some large-budget, max-of-K settings."],"supporting_citations":[{"why":"Supplies the O3 method that is the paper's main contribution, defining the example-based latent subspace and optimisation procedure.","marker":"Willis et al. (2025)"},{"why":"Defines FK-steering, the inference-time Sequential Monte Carlo baseline the paper compares against.","marker":"Singhal et al. (2025)"},{"why":"Defines DPO, the preference-optimisation objective used as the fine-tuning baseline.","marker":"Rafailov et al. (2023)"},{"why":"Provides the diffusion-DPO loss approximation used to apply DPO to Boltz-2.","marker":"Wallace et al. (2023)"},{"why":"Supplies Boltz-2, the frozen generative model all four guidance methods are applied to.","marker":"Passaro et al. (2025)"},{"why":"Defines the TM-score oracle used to score structures against the calmodulin ground truth.","marker":"Zhang & Skolnick (2004)"},{"why":"Provides the LOL projection used in O3 to map simplex weights back into the latent space.","marker":"Bodin et al. (2024)"},{"why":"Defines MolProbity, the reference-free physical-plausibility oracle used on the 9EEH target.","marker":"Chen et al. (2010); Williams et al. (2018)"},{"why":"Provides the probability-flow formulation that makes Boltz-2 deterministic, a requirement for O3.","marker":"Karras et al. (2022)"},{"why":"Supplies the Bayesian optimisation framework used to spend the remaining oracle budget in the O3 subspace.","marker":"Garnett (2023)"}],"fun_headline_variants":["O3 leads at low oracle budgets; FK-steering catches up as calls grow","First oracle-budget guidance benchmark: O3 at 20, FK at 1000","No oracle-guidance method dominates across all budgets and oracles","O3 best when oracle calls are scarce; FK-steering and DPO improve later","Budget-aware protein guidance: O3 for tight budgets, FK/DPO for generous"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The recommendations assume that two cheap stand-in scores—TM-score, which needs the known reference structure, and MolProbity, which checks local geometry—behave like the expensive biological oracle calls the budget is meant to model, and that results on two test proteins generalise to other targets.","fun_headline_variants_meta":{"raw":{"variants":["O3 leads at low oracle budgets; FK-steering catches up as calls grow","First oracle-budget guidance benchmark: O3 at 20, FK at 1000","No oracle-guidance method dominates across all budgets and oracles","O3 best when oracle calls are scarce; FK-steering and DPO improve later","Budget-aware protein guidance: O3 for tight budgets, FK/DPO for generous"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000188,"raw_usage":{"total_tokens":1345,"prompt_tokens":968,"completion_tokens":377,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":584,"completion_tokens_details":{"reasoning_tokens":270}},"tokens_in":584,"tokens_out":377,"duration_ms":3820,"temperature":1.0,"reasoning_tokens":270,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:12:39.077255+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same six budget configurations on a diverse set of protein targets using a genuinely expensive, non-differentiable oracle such as a short molecular-dynamics stability score; if at $N \\leq 100$ Best K-of-N ties or beats O3 on most targets, the paper's low-budget recommendation fails.","supporting_citations":[{"cited_title":"and Arendall, W","cited_arxiv_id":null,"evidence_quote":"Defines MolProbity, the reference-free physical-plausibility oracle used on the 9EEH target."}],"review_version":1}