{"id":"f3142a60-2433-41b5-a676-3ea7c825db9d","arxiv_id":"2607.05620","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Conformal intervals for counterfactual baseline outcomes can be nested inside online conformal SafeOpt to keep safety-constraint violations at or below a user-specified rate under weighted exchangeability.","lead":"The paper shows how to run safe Bayesian optimization when the safety bar is a counterfactual baseline policy whose outcomes are never observed. It uses conformal prediction to build valid uncertainty intervals for those missing outcomes and folds them into SafeOpt so constraint violations stay at or below a user-chosen rate.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Theorem 4.1's probability rests on modeling excess indicators as i.i.d. Bernoulli(ε′), but adaptive selection of xt induces dependence that is never justified.","rationale":"The reader correctly flags the “unproved binomial excess model” as a limitation that keeps the paper at CONDITIONAL, yet lists weighted exchangeability itself as the single weakest assumption. The more immediate load-bearing gap is internal to the proof of Thm. 4.1: even after weighted exchangeability is granted, the passage from marginal coverage to the exact binomial CDF still requires an independence claim that the adaptive nature of SafeOpt violates and that the authors never establish. Because the reader already discounted the paper for this very gap, the overall verdict remains CONDITIONAL; no stronger or weaker recommendation is warranted. The proposed simulator experiment would settle the issue cleanly, since the chemical setting supplies the ground-truth excesses that MovieLens cannot.","tokens_in":18075,"tokens_out":575,"duration_ms":36635,"concrete_test":"With the chemical-reaction simulator (true counterfactuals known), execute 500 independent runs of SafeOpt-CPC (T=100, ε=0.01), record the empirical histogram of the number of excesses, and perform a Kolmogorov–Smirnov test against Binomial(100,0.01). If the p-value falls below 0.01 or the upper tail is materially heavier, the binomial probability claimed in Thm. 4.1 overstates the guarantee.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Appx. A.1 the proof first obtains marginal conformal coverage P(Y(xsoc,t) ≤ Ût) ≥ 1-ε′ under weighted exchangeability, then asserts that the number of excesses is distributed exactly as Binomial(T,ε′) so that the safety event holds with the binomial CDF probability stated in Thm. 4.1. That distributional claim requires the indicators 1{Y(xsoc,t)>Ût} to be independent. Because xt is chosen by the adaptive rule (Eq. 4) that depends on the entire history Ot (previous outcomes, perceived q-values, and GP posteriors), the sequence of test points is dependent; the excess indicators therefore form a dependent (generally non-exchangeable) sequence. The paper only remarks that independence is “reasonable \\ldots given no dependence of the xsoc values across time,” without a martingale argument, mixing condition, or any other concentration inequality that survives adaptivity. Consequently the precise high-probability statement of the central theorem is not rigorously established even under the theorem’s own weighted-exchangeability hypothesis.","agreement_with_reader":"partial"},"referee_report":{"model":"grok-4.5","summary":"The paper introduces SafeOpt-CPC, which integrates split conformal prediction for unobserved counterfactual outcomes of a known baseline (standard-of-care) policy into online conformal safe Bayesian optimization. Safety constraints of the form q(x) involving Y(x)−Y(xsoc)+ω (or multi-baseline and minimization variants) are replaced by conformal upper/lower bounds so that the perceived constraint can be fed to the SafeOpt machinery of Zhang et al. (2024). Under weighted exchangeability the authors prove (Theorem 4.1) that the realized violation rate stays ≤α with probability given by a binomial CDF on the number of “excess” conformal failures; they also supply re-weighting schemes for propensity and covariate shift, non-stationarity algorithms, a coverage-gap analysis, experiments on a chemical-reaction simulator (true counterfactuals available) and MovieLens, and a sensitivity study.","tokens_in":18448,"tokens_out":1337,"duration_ms":18511,"significance":"Relative safety with respect to an unobserved baseline is a practically important constraint class (clinical trials, recommendation, process control). Nesting offline conformal intervals inside online conformal SafeOpt is a clean technical contribution; the paper supplies an explicit safety proof, handles several covariate-shift regimes, and evaluates on a simulator that grants ground-truth counterfactuals. The sensitivity analysis and non-stationarity extensions further strengthen the empirical package. If the high-probability claim can be placed on a fully rigorous footing, the method would be a useful tool for high-stakes sequential decision making.","major_comments":[{"comment":"Theorem 4.1 / Appendix A.1: the high-probability statement rests on modeling the excess indicators 1{Y(xsoc,t)>Ût} as i.i.d. Bernoulli(ε′), so that their sum is exactly Binomial(T,ε′). Because xt is chosen by the adaptive rule (Eq. 4) that depends on the entire history Ot (GP posteriors, previous perceived q-values, and the conformal bounds themselves), the sequence of test points—and therefore the excess indicators—is dependent. The only justification offered is that independence is “reasonable \times given no dependence of the xsoc values across time.” This is insufficient for the precise binomial-CDF probability claimed in the theorem. Either supply a martingale / mixing / Azuma-style concentration argument that survives adaptivity, or weaken the guarantee to an expectation or Markov bound that does not require independence.","section":"Theorem 4.1, Appendix A.1"},{"comment":"Section 4.1.2 and Eq. (26): the adjustment α′=(α−ε′)/(1−ε′) correctly accounts for the expected fraction of excesses, but the subsequent online-conformal update (Eq. 11 rewritten with α′) still treats every perceived violation as an independent Bernoulli trial for the purpose of the β-adaptation schedule. When excesses are dependent (see previous comment) the excess-violation process that drives βt is no longer a simple martingale difference sequence; the interaction between the offline conformal excesses and the online β update is not analyzed. A short argument showing that the Zhang et al. (2024) guarantee continues to hold under the nested, possibly dependent excess process would close the gap.","section":"Section 4.1.2, Eq. (26)"},{"comment":"MovieLens experimental design (Section 5): the evaluation assumes that the counterfactual rating under the standard-of-care movie equals the user’s observed rating for that movie. While this is a convenient proxy when true counterfactuals are unavailable, it is a strong modeling assumption that is never stress-tested (e.g., by injecting noise into the proxy or by comparing against a held-out rating). Because the paper’s central empirical claim is that true violation rates stay below α, the MovieLens results should be clearly labeled as relying on this proxy, and at least one sensitivity check should be reported.","section":"Section 5, MovieLens paragraph"}],"minor_comments":[{"comment":"Figure 1 caption and surrounding text: the distinction between “potential/perceived” and “true” violations is central, yet the figure legend and main-text description are terse. A one-sentence definition in the caption would help readers.","section":"Figure 1"},{"comment":"Notation: Ûsoc_t(xsoc,t) versus Ûsoc_t(xt) appears inconsistently in Eqs. (13)–(16) and (24). Standardize the argument of the conformal bound.","section":"Section 4.1"},{"comment":"Appendix C algorithms: Alg. 3 and Alg. 4 are useful, but the detection criteria for “changepoint” and “drift” are left unspecified. A brief pointer to a concrete detector (or a statement that detection is assumed perfect) would improve reproducibility.","section":"Appendix C"},{"comment":"Hyper-parameter choices (η=0.2, α=0.1, ε=0.01) are stated in Appendix D without a short ablation; a one-paragraph sensitivity note in the main text would be helpful.","section":"Appendix D"},{"comment":"Typographical: “violate rate” → “violation rate” (Introduction); “100ϵ′ of the timesteps” → “100ϵ′% of the timesteps” (proof sketch).","section":"Introduction / Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The independence gap identified by the skeptic is real and load-bearing for the exact statement of Theorem 4.1; it is fixable by either a concentration argument or a modest weakening of the claim, so major revision rather than reject is appropriate. The paper is a solid incremental contribution on top of Zhang et al. (2024) and Tibshirani et al. (2019); novelty is adequate for a specialized ML venue but the authors should be asked to clarify the precise technical delta in the camera-ready version."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful thing here is a concrete algorithm (SafeOpt-CPC) that lets SafeOpt treat a relative safety constraint whose baseline is never observed. They nest split conformal intervals for the counterfactual (Lei & Candès style, with propensity or density-ratio weights) inside Zhang et al.’s online conformal SafeOpt, adjust the target rate to α′ = (α−ε)/(1−ε), and give changepoint/drift reweighting variants. That combination, plus the multi-baseline union-bound form, is new and immediately usable for clinical-style or process-control sequential design.\n\nWhat they do well: the proof structure is transparent under weighted exchangeability (Tibshirani + Barber coverage-gap bounds + Zhang’s online guarantee). Experiments are the right ones—reaction simulator with true counterfactuals, MovieLens with an explicit (if strong) assumption, synthetic sensitivity on bias/variance/noise/weight misspecification—and they show the method stays under α while ordinary BO does not. Non-stationarity appendices are practical rather than hand-wavy.\n\nSoft spots, in proportion. The stress-test is right that Appx. A.1 treats the excess indicators as i.i.d. Bernoulli(ε′) so the safety event holds with the exact binomial CDF; adaptive choice of xt via the history-dependent SafeOpt rule makes that distributional claim unjustified. They only say independence is “reasonable.” That weakens the precise high-probability statement of Thm. 4.1, though a looser concentration or martingale bound would almost certainly still give a usable guarantee. MovieLens counterfactuals are assumed equal to observed ratings; code is redacted. Neither sinks the paper. Free parameters (η, α, ε, ω) are user-facing and standard for the literature.\n\nThis is for people already working on safe BO, constrained bandits, or high-stakes sequential design who need relative rather than absolute constraints. It deserves a serious referee; the central idea is sound and the experiments support the practical claim even if the binomial tail needs tightening. I would engage with it and expect it to be cited inside the subfield.","headline":"Solid, usable extension of conformal SafeOpt to counterfactual baselines; the binomial-tail claim is slightly overstated under adaptivity, but the core method and experiments hold up.","tokens_in":19028,"tokens_out":531,"would_cite":true,"duration_ms":5207,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Conformal intervals let safe Bayesian optimization treat unobserved baseline policies as safety constraints without exceeding a user-set violation rate.","keywords":["safe Bayesian optimization","conformal prediction","counterfactual inference","covariate shift","safety constraints","standard of care"],"falsifier":"Run the chemical-reaction simulator with deliberately misspecified propensity weights or an ignored changepoint and check whether the empirical true violation rate exceeds α more often than the binomial tail bound of Theorem 4.1 permits.","tokens_in":18971,"feed_emoji":"🛡️","tokens_out":806,"duration_ms":15554,"temperature":0.7,"pith_summary":"Many high-stakes decisions require that a new action never perform much worse than an established baseline policy. The baseline outcome is counterfactual and therefore unobserved once a different action is taken. This paper shows how to keep Bayesian optimization safe under exactly those relative constraints. It estimates the missing baseline outcomes with split conformal prediction, substitutes the resulting one-sided bounds into the safety constraint, and tightens the allowed violation rate so that true violations stay at or below a user-specified α. Experiments on a chemical-reaction simulator and MovieLens recommendations confirm that the method respects the rate while still improving the objective, and that the same conformal machinery can be re-weighted for common forms of covariate shift.","feed_headline":"Safe BO stays under α when baselines are counterfactual","feed_subtitle":"Conformal intervals turn unobserved standard-of-care outcomes into guarantees that violations stay below a set rate.","key_machinery":"SafeOpt-CPC: split conformal prediction of the counterfactual baseline outcome (with propensity or density-ratio weights restoring exchangeability), whose one-sided quantiles are substituted into the SafeOpt safety constraint and whose coverage failure probability is folded into a reduced online violation budget α′.","core_discovery":"Under weighted exchangeability of calibration and test scores, SafeOpt-CPC satisfies the online safety constraint that the fraction of timesteps violating q(x) ≥ 0 is at most α, with probability equal to the binomial CDF that the number of conformal excesses stays ≤ Tε′. The algorithm replaces the unobserved baseline outcome by a conformal upper or lower bound whose failure probability ε is absorbed into a tightened target α′, then runs online conformal SafeOpt on the resulting surrogate constraint.","pith_inferences":["The same conformal wrapper could be dropped into any other constrained optimizer that only needs a conservative estimate of an unobserved safety signal.","Because the coverage gap under weight misspecification is explicit, one could monitor unlabeled data online and pause optimization when the estimated gap threatens the α budget.","Clinical-trial designers could treat the tolerance ω and the two rates α and ε as transparent, auditable knobs."],"forward_implications":["Practitioners can optimize treatments or recommendations while guaranteeing that outcomes fall below the standard-of-care by at most a tolerance ω on all but an α-fraction of steps.","The same construction extends to constraints containing several estimated counterfactuals via a union bound on the conformal failure probabilities.","When covariate shift or non-stationarity is detected, re-weighting or time-decayed calibration preserves the formal safety guarantee.","The method remains valid under noisy or biased point estimators of the baseline; only the width of the conformal intervals (and therefore the number of fallback steps) increases."],"fun_headline_variants":["Safe BO with conformal bounds on counterfactual baselines","Conformal intervals keep SafeOpt violations under α","SafeOpt-CPC: α-safety when baselines stay unobserved","Counterfactual policies in safe Bayesian optimization","Weighted exchangeability yields α-controlled SafeOpt"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The weights that rebalance the calibration set must correctly describe how the test covariates relate to the baseline policy; if those weights are wrong, the conformal intervals lose coverage and the safety proof collapses.","fun_headline_variants_meta":{"raw":{"variants":["Safe BO with conformal bounds on counterfactual baselines","Conformal intervals keep SafeOpt violations under α","SafeOpt-CPC: α-safety when baselines stay unobserved","Counterfactual policies in safe Bayesian optimization","Weighted exchangeability yields α-controlled SafeOpt"]},"model":"grok-4.5","effort":"low","cost_usd":0.0046,"raw_usage":{"total_tokens":1318,"prompt_tokens":731,"num_sources_used":0,"completion_tokens":56,"cost_in_usd_ticks":46000000,"prompt_tokens_details":{"text_tokens":731,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":531,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":731,"tokens_out":56,"duration_ms":5028,"temperature":1.0,"reasoning_tokens":531,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-11T04:49:16.603471+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Run the chemical-reaction simulator with deliberately misspecified propensity weights or an ignored changepoint and check whether the empirical true violation rate exceeds α more often than the binomial tail bound of Theorem 4.1 permits.","supporting_citations":[],"review_version":1}