{"id":"d586457e-943a-4af3-b975-73fd1324bf8b","arxiv_id":"2608.01378","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Safety in surrogate-driven design requires oracle-only certification; rank preservation is the exact criterion for oracle use, and selection-aware auditing is the cheapest way to certify it.","lead":"Machine-learning models that predict experimental outcomes can safely propose candidates, but any certified conclusion must be backed by a real measurement. The paper proves this separation rule, introduces a cheap champion audit that licenses when a model is trustworthy, and shows where audited surrogates actually cut certified evaluation cost.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 14's optimality claim is overbroad: the Ω(N/ρ0) lower bound holds only for the ordinal/regret guarantee, while the paper's own value-floor audit certifies a deployed guarantee in O(ρ^{-1} log(1/δ)) calls; the abstract's unqualified 'no audit design of any kind' is unsupported.","rationale":"The paper's formal theory appears sound: the separation theorem is a clean invariance argument on a pre-sampled noise array; the necessity construction is a valid deterministic self-confirmation; the audit lower bound is a correct needle-in-haystack coupling; and the cost dichotomy is properly conditioned. The reader's weakest assumption (one-shot i.i.d. pools, fixed D, pre-sampled noise) is an acknowledged limitation, not a hidden flaw. My concern is narrower but load-bearing because it touches the abstract's central promise: Theorem 14 is proved for the regret/ordinal guarantee, not for every notion of deployed selection quality. The paper's own value-floor license is asymptotically cheaper, so the unqualified 'no audit design of any kind' sentence is false. Since the reader already flagged the qualifier in the rationale and issued CONDITIONAL, my read does not change the verdict; it sharpens the required revision: state the ordinal/regret scope in the abstract and Theorem 14, and let Remark 15's distinction carry the rest. Agreement is partial because the reader's formal weakest_assumption concerns sequential/credit-assignment scope, whereas my concern is about the audit-object scope; both are scope conditions on the central claims.","tokens_in":39240,"tokens_out":23744,"duration_ms":222642,"concrete_test":"Run a side-by-side simulation on the hard family of SI S7.3 (X=[0,1], D uniform, fhat=-1 on [0,1/2], fhat=x on (1/2,1], K cells of mass 2ρ0/N, f0≡0, fj=B·1_{Cj}) at N=100, ρ0=0.01. Implement two auditors: the champion qsel audit (two oracle calls per round, needing qsel≤ρ0/N) and the value-floor audit of SI S7.2 (one oracle call per round, Ta=(2/ρ)log(1/δ)). Record oracle calls needed to (a) certify P(FAIL)≤ρ0 and (b) certify P(champion ε} to level ρ0 needs Ω(N/ρ0) oracle queries; this uses the completeness condition that the auditor pass a designated zero-regret instance f0 with probability at least 1/2. The abstract and the main-text statement of Theorem 14 drop the 'ordinal/regret' qualifier, asserting that no audit design of any kind can certify deployed selection quality with asymptotically fewer queries. That broader statement is contradicted by the paper's own selection-aware value-floor license (SI Thm. S7.2), which certifies a one-sided over-prediction/value guarantee with O(ρ^{-1} log(1/δ)) oracle calls, independent of N and far below N/ρ0. Remark 15 and Corollary S7.6 acknowledge that value licenses escape the N-floor because they certify a weaker object, so the formal results are internally consistent; the problem is the unqualified headline. Since optimality of the champion audit is the keystone of the 'price of trust' narrative, the overstatement needs correction before the paper is accepted as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper develops a formal framework for deciding when a machine-learned surrogate may replace a costly oracle in design campaigns. Its main theoretical contributions are: (i) an accuracy--decision separation showing that R^2-type metrics are structurally blind to argmax regret, with matching upper and lower bounds on the resulting 'selection tax'; (ii) a separation principle stating that certification statistics must be computed solely from oracle evaluations while proposal machinery is unrestricted (Theorem 4), together with a deterministic self-confirmation failure for contaminated protocols (Theorem 5); (iii) oracle-cost bounds for certified elimination search (Theorems 6--7); (iv) an ordinal admissibility criterion with an audit--deployment gap and selection-aware audits (Theorems 8--11); (v) a universal lower bound on audit query complexity for the ordinal/regret guarantee (Theorem 14 and SI Theorem S7.7); and (vi) a strict-improvement dichotomy for generative versus fixed-menu search (Theorem 12 and SI Theorems S7.1--S7.3). The claims are supported by experiments on three exhaustively ground-truthed tasks (17,353 real training outcomes), 432 surrogate fits, and an extensive open-source verification suite.","tokens_in":39463,"tokens_out":27884,"duration_ms":248837,"significance":"Should the results stand, this is a substantial contribution: it converts 'trust in surrogates' from an accuracy heuristic into a measurable, license-based protocol with formal guarantees, and it provides a query-complexity lower bound over arbitrary audit designs for the ordinal/regret guarantee. The paper's main strengths are the clean pre-sampled-array argument behind the oracle-only invariance theorem, the explicit self-confirmation construction, the tight Theta(N) versus Theta(N^2) audit-statistic gap, the coupling-based universal lower bound, and the unusually complete reproducibility apparatus (ground-truth tables, committed code, numerical verification of each theorem). The experiments sensibly distinguish calibration of the audit statistic from independent confirmation, and the collapse of R^2 as a deployment diagnostic on the combinatorial benchmark is a concrete, falsifiable observation. The principal weakness is public-facing overstatement of the optimality result's scope, which is correctable and is detailed below.","major_comments":[{"comment":"The abstract states that selection-aware audits are 'provably optimal in query complexity--no audit design of any kind can certify deployed selection quality with asymptotically fewer oracle queries.' As the SI itself makes precise, SI Theorem S7.7 proves an Omega(N/rho_0) lower bound only for the ordinal/regret event FAIL = {max_i f(X_i) - f(hat x) > epsilon}, under a nontriviality/completeness condition. The paper's own value-floor license (SI Theorem S7.2, Corollary S7.6) certifies the one-sided deployed guarantee P(f(champion) < v_hat) <= 4rho with O(rho^{-1} log(1/delta)) oracle calls, independent of N. Remark 15 and Corollary S7.6 explicitly acknowledge that value licenses escape the N-floor because they certify a different, weaker object. The formal theorems are internally consistent, but the unqualified headline claim is false as written and should be amended to 'for the ordinal/regret guarantee' in the abstract, in the Theorem 14 statement, and in the concluding 'price of trust' narrative. Because this optimality claim is a keystone contribution, the scope correction is necessary before publication.","section":"Abstract; Theorem 14; SI Thm. S7.7"},{"comment":"The claimed Theta(M/log(1/rho)) separation is presented as 'conditional on the audit passing,' but the proof in Theorem S7.2 splits the guarantee into two distinct high-probability events: the validity event P(f(champion) < v_hat) <= 4rho holds with probability at least 1 - delta, while the high-level event v_hat >= q_{1 - log(1/rho)/M} - epsilon (which is what makes the screening factor meaningful) holds only with probability at least 1 - delta^{2/3} under the stated Ta >= (2/rho) log(1/delta). The combined success probability is therefore at least 1 - delta - delta^{2/3}, not 1 - delta. This is fixable by setting Ta >= (3/rho) log(1/delta) or by explicitly defining 'audit passing' as the conjunction of both events, but as written the probability accounting in the dichotomy's cost claim is incomplete.","section":"Theorem 12(b); SI Cor. S7.3; SI Thm. S7.2"}],"minor_comments":[{"comment":"The main-text statement of Theorem 1 omits the qualifications in the SI: the near-perfect-R^2 construction requires even N with N >= 4(1+beta)^2/epsilon. Please add the qualification or explicitly defer to the SI.","section":"Theorem 1; SI Thm. S3.1"},{"comment":"The main-text margin condition says 'the fraction of candidates within u of optimal is bounded by Cu^alpha,' while the SI uses #{x : 0 < Delta_x <= u} <= N min(1, C u^alpha). The factor N and the min(1,.) should be reflected in the main text for consistency.","section":"Theorem 7; SI Thm. S5.5"},{"comment":"The sentence 'high correlation is calibration, not clairvoyance' is an important and honest caveat. Consider moving it to the start of Section 4.3 so that readers do not mistake the 0.89--0.99 correlations for independent confirmation of the theory.","section":"Section 4.3; SI S8.2"},{"comment":"The final-paragraph statement that the policy-selection form 'follows from the pool theorems' should cite Corollary S5.7 by name; the main text currently refers only to 'SI Section S5.1', which is less precise.","section":"Section 5; SI Cor. S5.7"},{"comment":"In Equation (1), the notation N qbar_delta(events, T_a) uses the word 'events' as a variable; define it in the main text as the number of regret events K_T, as in SI Proposition S8.1, to avoid confusion.","section":"Equation (1); SI Prop. S8.1"}],"recommendation":"major_revision","confidential_remarks":"The formal SI is largely self-consistent, and the proofs are credible; the main risk for the journal is that the abstract and Theorem 14's unqualified optimality sentence will be quoted out of context. I would request the scope correction before acceptance. The paper's self-review and reproducibility apparatus is unusually thorough, which strengthens my confidence in the formal results."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—\n\nThis is a serious paper and should be read. The headline result is a separation principle for surrogate-driven design: you may let predictions propose anything, but every certified conclusion must be computed from true evaluations only. Sufficiency is a clean union-bound argument (SI S5.1); necessity is a deterministic self-confirmation construction with probability-one failure for contaminated certification. That pair, plus the matching upper/lower bounds on the selection tax (Theorem 3), and the universal audit lower bound (Theorem 14/S7.7) make the paper a real contribution. The audit statistic qsel tracks deployed regret at 0.89–0.99 Spearman across 432 fits on three ground-truthed tasks, while R2 can be 0.33. The experimental program is unusually honest: every oracle call is a real training run, and the SI explicitly attributes prior results and flags limitations.\n\nThe soft spot is the scoping of the optimality claim. As stated in the abstract, 'no audit design of any kind can certify deployed selection quality with asymptotically fewer oracle queries,' and the main-text Theorem 14 is only slightly more careful. The formal lower bound is for the ordinal/regret guarantee—FAIL = {regret > ε}—and it uses a completeness condition (the auditor must pass a designated safe instance with probability at least 1/2). The paper's own value-floor license (SI Thm. S7.2) certifies a one-sided over-prediction guarantee with O(ρ^{-1} log(1/δ)) oracle calls, independent of N, which is far below N/ρ0. Remark 15 and Corollary S7.6 explicitly acknowledge that value licenses certify a weaker object, so the math is internally consistent. The problem is the unqualified headline. A reader will walk away thinking the champion audit is optimal for every notion of deployed quality, which is false. That should be fixed with a qualifier in the abstract and in Theorem 14; it is not fatal, but it is the kind of overstatement that invites a correct rebuttal.\n\nOther soft spots are minor and acknowledged: the α-margin law in Theorem 7 is an assumption, not a consequence; the guarantees are single-decision (pool/generative) and do not cover sequential credit assignment; and the 25× cost reduction is amortized and conditional on the audit passing. None of these undermines the core.\n\nVerdict: send it to review. I would accept it conditional on the optimality-claim correction. This deserves a serious referee—the theory is original, the experiments are reproducible, and the framing is useful to practitioners.","headline":"A serious, careful paper on when surrogates can substitute for expensive evaluations; the optimality claim is slightly oversold but the core theory and experiments are solid and peer-review-worthy.","tokens_in":40033,"tokens_out":2917,"would_cite":true,"duration_ms":25816,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A surrogate may propose candidates freely, but every certified conclusion must rest on true evaluations alone—a separation sufficient for safety with no surrogate assumptions and necessary, since treating predictions as measurements…","keywords":["surrogate-driven design","certification","selection-aware auditing","separation principle","selection tax","oracle complexity","rank preservation","licenses"],"falsifier":"Run the paper's contaminated-certification construction on a two-candidate noiseless task: the surrogate's predictions are reversed, accepted pseudo-labels enter the certification intervals with the standing of measurements, and no oracle is consulted. Theorem 5 predicts the wrong candidate is certified with probability one once the intervals narrow below half the value gap; if any contaminated protocol of this class certifies correctly, or if the self-confirming intervals fail to close, the necessity half of the separation principle is refuted.","tokens_in":38989,"feed_emoji":"🧪","tokens_out":12239,"duration_ms":100842,"temperature":0.7,"pith_summary":"Design campaigns in chemistry, materials, and machine learning share a bottleneck: true evaluation is expensive, and surrogates that predict it are increasingly used to grade candidates and feed their own predictions back into the search. This paper asks when that substitution is safe, what a certificate of safety must cost, and when it pays. Its central answer is an architectural rule: surrogate predictions may propose candidates and train models without restriction, but every certified conclusion—intervals, eliminations, stopping rules, reported values—must be computed from true evaluations alone. The paper proves this separation is sufficient with no assumptions on the surrogate and necessary, because admitting predictions as measurements opens a deterministic self-confirmation failure; it also proves that accuracy metrics cannot anchor trust, that trust must be purchased through selection-aware audits, and that audited surrogates cut certified oracle cost for generative search but never for fixed-menu selection.","feed_headline":"Separation rule: surrogates may propose, only oracles may certify","feed_subtitle":"Proof that certified design conclusions must rest on true evaluations—and when model-driven search pays.","key_machinery":"The load-bearing object is the two-pipe information architecture: an unrestricted proposal pipe that may consume surrogate outputs, pseudo-labels, and arbitrary retraining, and a certification pipe that consumes oracle evaluations only. It is made concrete by a $\\delta$-admissible confidence system built on a pre-sampled oracle noise array, so the coverage event lives on the array and pseudo-labels never touch it; that is what makes validity uniform over surrogate behavior. The companion mechanism is the selection-aware 'champion' audit, which draws a pool, lets the surrogate select its champion, and verifies only that champion with the oracle; this samples the deployment workflow rather than the candidate distribution, turning a passive audit's $\\Theta(\\min(1,Np))$ risk into a directly certified deployed failure rate at $O(\\rho^{-1}\\log(1/\\delta))$ oracle calls. The cost dichotomy is carried by a quantile license for generative search, which certifies designs at a quality bar for $O(1)$ oracle evaluations each once the audit passes, against an $\\Omega(1/p_v)$ oracle-only baseline.","core_discovery":"The paper's core discovery is a separation principle with matching necessity. Any design loop may let the surrogate do whatever it likes on the proposal side—select candidates, train on pseudo-labels, retrain freely—provided the certification side computes its confidence intervals, eliminations, stopping rules, and reported values from oracle evaluations alone. Under that architecture, with probability at least $1-\\delta$ every certified conclusion is correct uniformly over all surrogate behaviors, including surrogates constructed adversarially with knowledge of the true objective. The converse is a deterministic failure construction: if accepted pseudo-labels enter certification statistics with the standing of measurements, there is a two-candidate task and a surrogate for which the protocol certifies the wrong candidate with probability one at zero oracle cost, its confidence generated entirely by the model's agreement with itself. Around this core the paper establishes supporting results: $R^2$ is neither necessary nor sufficient for decision quality, and screening $N$ candidates inflates expected over-prediction at the champion by at most $2\\sigma\\sqrt{\\ln N}$ with a matching lower bound; the exact criterion for a model to act as an oracle is rank preservation on pairs separated by more than a tolerance, not accuracy; the only audit design that certifies deployed selection quality at a profit is selection-aware champion auditing, which is optimal among all audit designs up to a log factor; and for costs there is a strict-improvement dichotomy—certified fixed-menu selection never benefits from pseudo-labels in the tabular model, while certified generative search is cheaper by a factor $\\Theta(M/\\log(1/\\rho))$ conditional on a passing audit. Across 432 surrogate fits on three ground-truthed tasks, the audit statistic tracks deployed regret at Spearman correlation 0.80–0.99 while $R^2$ falls as low as 0.33, and audited screening reduces certified oracle cost by a measured factor of 25.","pith_inferences":["If the separation principle transfers to multi-step design loops, the paper's own boundary suggests that any campaign reusing model outputs to drive learning—replay buffers, policy improvement, model-based rollouts—should be treated as unlicensed until per-step certificates exist; the paper leaves credit assignment open.","The per-fit bimodality documented here (identical recipes with near-identical audited rates deploying at failure 0 versus 1) implies that model-class reputation and training-regime reputation cannot substitute for per-fit auditing in practice, a conclusion that extends beyond the three tested tasks.","The selection-tax bound yields a cheap engineering heuristic the paper does not spell out: for large screens, discount any surrogate-reported champion value by roughly $\\sigma\\sqrt{2\\ln N}$ unless a license says otherwise, since even an unbiased surrogate over-promises at the top by that amount.","The cost dichotomy suggests a practical division of labor for design campaigns: exhaustively certify fixed menus by direct oracle elimination, but invest in selection-aware audits for generative campaigns; the budget-plane results imply the training-label/audit-round split is the main strategic dial."],"forward_implications":["If the separation principle is right, then reporting $R^2$, MSE, or any marginal accuracy metric as evidence of deployment reliability is unsafe: those metrics are structurally blind to the quality of the single selected candidate, and screening adds a quantified selection tax of order $\\sqrt{\\ln N}$ on expected over-prediction.","Any campaign that issues confidence intervals, eliminations, stopping decisions, or reported values must compute them from true evaluations alone; doing so preserves validity even if the surrogate is adversarially wrong, at the price of wasting oracle budget but never corrupting a certificate.","Certification of a surrogate must be performed by an audit that imitates deployment—draw pools, let the surrogate pick champions, verify only champions—because passive random-sample audits can both reject safe surrogates and approve dangerously aligned ones.","In generative search toward a quality bar, a surrogate holding a selection-aware license cuts certified oracle cost by roughly the screening factor (measured 25× on a standard architecture benchmark), while in tabular fixed-menu selection no use of pseudo-labels can reduce the certified cost.","Licenses bind to one trained model and one candidate distribution; retraining or distribution drift voids them, and renewal costs the same audit budget as the original, so re-auditing rather than transfer is the operational rule."],"supporting_citations":[{"why":"Supplies the conservative-surrogate modeling tradition that the one-sided no-disappointment license grounds in a checkable certificate.","marker":"1"},{"why":"Supplies the optimizer's-curse baseline: the selected maximum of a noisy predictor is biased upward, which Theorems 1–3 formalize as a selection tax.","marker":"6"},{"why":"Supplies the winner's-curse framing for over-prediction at the selected candidate, the phenomenon the selection-tax bounds quantify.","marker":"7"},{"why":"Provides the exhaustive neural-architecture ground truth (15,625 trained cells) on which the audit statistic's deployment correlation is validated.","marker":"8"},{"why":"Defines the marginal split-conformal floor baseline whose champion-level violation the selection-aware license is compared against.","marker":"12"},{"why":"Supplies selection-conditional conformal machinery, the sharper instrument where exchangeability holds and the gap the champion audit fills.","marker":"13"},{"why":"Provides the classical quantile-bandit lower bound $\\Omega(1/p_v)$ that forms the oracle-only baseline in the strict-improvement dichotomy.","marker":"16"},{"why":"Provides the action-elimination interval certificates that the certified-elimination protocol combines with the separation principle.","marker":"39"}],"fun_headline_variants":["Surrogates propose, oracles certify—proven separation rule","R² can't anchor trust: audits do in surrogate design","Let models propose, but certify only on true evaluations","Rank preservation, not accuracy, makes a model an oracle","Selection tax: why screening inflates over-prediction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The guarantees cover one decision made once—choosing from a fixed pool or producing one generated design—with candidates drawn independently from a fixed distribution and the surrogate's outputs not reused to drive later learning. If a campaign lets the model's own predictions steer a multi-step search, the separation guarantee, the license, and the audit-cost floor do not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Surrogates propose, oracles certify—proven separation rule","R² can't anchor trust: audits do in surrogate design","Let models propose, but certify only on true evaluations","Rank preservation, not accuracy, makes a model an oracle","Selection tax: why screening inflates over-prediction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000252,"raw_usage":{"total_tokens":1721,"prompt_tokens":1266,"completion_tokens":455,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":882,"completion_tokens_details":{"reasoning_tokens":372}},"tokens_in":882,"tokens_out":455,"duration_ms":4041,"temperature":1.0,"reasoning_tokens":372,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:08:42.407564+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's contaminated-certification construction on a two-candidate noiseless task: the surrogate's predictions are reversed, accepted pseudo-labels enter the certification intervals with the standing of measurements, and no oracle is consulted. Theorem 5 predicts the wrong candidate is certified with probability one once the intervals narrow below half the value gap; if any contaminated protocol of this class certifies correctly, or if the self-confirming intervals fail to close, the necessity half of the separation principle is refuted.","supporting_citations":[{"cited_title":"Conservative Objective Models for Effective Offline Model-Based Optimization.Proc","cited_arxiv_id":null,"evidence_quote":"Supplies the conservative-surrogate modeling tradition that the one-sided no-disappointment license grounds in a checkable certificate."},{"cited_title":"E.; Winkler, R","cited_arxiv_id":null,"evidence_quote":"Supplies the optimizer's-curse baseline: the selected maximum of a noisy predictor is biased upward, which Theorems 1–3 formalize as a selection tax."},{"cited_title":"C.; Clapp, R","cited_arxiv_id":null,"evidence_quote":"Supplies the winner's-curse framing for over-prediction at the selected candidate, the phenomenon the selection-tax bounds quantify."},{"cited_title":"NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search","cited_arxiv_id":null,"evidence_quote":"Provides the exhaustive neural-architecture ground truth (15,625 trained cells) on which the audit statistic's deployment correlation is validated."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the marginal split-conformal floor baseline whose champion-level violation the selection-aware license is compared against."},{"cited_title":"Confidence on the Focal: Conformal Prediction with Selection-Conditional Coverage.J","cited_arxiv_id":null,"evidence_quote":"Supplies selection-conditional conformal machinery, the sharper instrument where exchangeability holds and the gap the champion audit fills."},{"cited_title":"Pure Exploration in Infinitely-Armed Bandit Models with Fixed-Confidence.Proc","cited_arxiv_id":null,"evidence_quote":"Provides the classical quantile-bandit lower bound $\\Omega(1/p_v)$ that forms the oracle-only baseline in the strict-improvement dichotomy."},{"cited_title":"Even-Dar, S","cited_arxiv_id":null,"evidence_quote":"Provides the action-elimination interval certificates that the certified-elimination protocol combines with the separation principle."}],"review_version":2}