{"id":"7252aa31-69ad-42ff-985f-bab873698ff5","arxiv_id":"2607.06879","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":1,"one_line_summary":"The PROBE algorithm provably reduces the sample complexity of best-arm identification by using cheap proxy scores as control variates, achieving oracle-level savings even when the reward-proxy correlation is unknown.","lead":"This paper introduces PROBE, an algorithm that uses cheap AI-generated proxy scores to cut the number of expensive real-world experiments needed to identify the best option in a bandit setting. It matters because it provides mathematical guarantees that these savings are safe even when the proxy's accuracy is unknown and must be learned on the fly.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"Gaussian assumption is load-bearing for the chi-square certificate, but is a transparent and standard modeling choice that does not undermine the contribution.","rationale":"The reader's identification of the Gaussian assumption as the weakest link is correct and well-targeted. I verified the proof chain: the δ-PAC guarantee flows through the clean event E, which depends on the calibration event (Lemma 3, uses chi-square tails), the leverage event (Lemma 2, uses Gaussian tail bounds), the variance-certificate event (uses chi-square lower tail via Laurent-Massart), and the mean-estimation event (uses conditional Gaussianity of OLS intercept). All four components require Gaussianity. However, this is a transparent modeling assumption common in the BAI literature, the paper is clear about it, and the experiments on non-Gaussian (Bernoulli) data suggest practical robustness even where the theory doesn't directly apply. The proof structure is sound within its stated assumptions, the sample complexity bound follows correctly from the geometric elimination argument, and the algorithmic design (one-round lag, monotone updates, residual-variance certificate) is well-motivated and internally consistent. No hidden assumptions or internal inconsistencies were found. The concern is real but does not change the verdict from ACCEPT.","tokens_in":24352,"tokens_out":2903,"duration_ms":85409,"concrete_test":"Run a simulation with a heavy-tailed reward distribution (e.g., reward = p_i + Student-t(3) noise, proxy correlated via a Gaussian copula) and measure the empirical error rate of PROBE at δ=0.05 across 10,000 repetitions. If the error rate exceeds 5% by a meaningful margin, the Gaussian assumption is not merely a theoretical convenience but a practical correctness risk. If the error rate stays near 5%, the chi-square certificate is robust enough that the concern is primarily theoretical.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reader correctly identifies the bivariate Gaussian model (Eq. 2) as the weakest assumption. The entire correctness argument rests on three distributional facts that all require Gaussianity: (1) the exact chi-square law (s−2)v̂/v ~ χ²_{s−2} (Proposition 2), which makes the upper certificate U(s,v̂,η) in Eq. 7 a valid one-sided bound; (2) the conditional Gaussianity of the OLS intercept p̂_i − p_i ~ N(0, v_i h_i) (Proposition 2), which drives the mean-estimation event E^p_{i,r}; and (3) the leverage bound h_i ≤ 2/s (Lemma 2), which uses Gaussian tail bounds on W̄ and Σ(W_l−W̄)². If rewards or proxies are non-Gaussian, all three pillars lose their exact forms, and the δ-PAC guarantee no longer follows from the current proof. The paper's phrase 'whose exact χ² law keeps the certificate valid regardless of the unknown correlation' is technically correct but could be read as implying distribution-free validity, which it is not. The auto-loan experiment uses Bernoulli rewards, where the theory does not directly apply, though empirically the algorithm still works. This is a real gap between theory and practice, but it is a standard and transparent modeling limitation in the bandit literature, not an internal inconsistency or a hidden flaw.","agreement_with_reader":"agree"},"referee_report":{"model":"glm-5.2","summary":"This paper studies fixed-confidence best-arm identification (BAI) in a setting where each costly reward pull is paired with a cheap proxy score (e.g., from an LLM or predictive model). The proxy mean is assumed known (estimated offline cheaply), but the reward-proxy correlation is unknown and must be learned online. The authors show that a control-variate adjustment reduces the problem to a heteroscedastic BAI problem with residual variance 1−ρ². The main algorithmic challenge is that a plug-in estimate of the residual variance is anti-conservative, risking incorrect eliminations. The authors propose PROBE, a phase-elimination algorithm that maintains a one-sided upper certificate on the residual variance via an OLS fit, exploiting the exact chi-square law of the residual variance estimator. PROBE is proven to be δ-PAC and achieves the known-correlation oracle sample complexity up to a multiplicative factor (1+κ) and an additive calibration cost of order Õ(K/κ²). The guarantee extends to the (ε,δ)-PAC setting. Experiments on synthetic data and an auto-loan pricing replay confirm the predicted variance reduction.","tokens_in":24603,"tokens_out":1173,"duration_ms":190824,"significance":"The paper addresses a timely and practically important problem: leveraging cheap but noisy AI/LLM-generated proxy scores to reduce the number of costly online experiments needed for decision-making. The core technical insight—using the exact chi-square law of the OLS residual variance to construct a valid upper certificate, rather than a plug-in correlation estimate—is elegant and well-motivated. The theoretical guarantee is clean, with an explicit price parameter κ for not knowing the correlation. The auto-loan replay experiment with LLM and tabular proxies demonstrates substantial practical gains (up to 61% sample savings), directly validating the theoretical predictions. The proof strategy via a 'clean event' is standard but correctly executed and self-contained.","major_comments":[{"comment":"The bivariate Gaussian observation model (Equation 2) is load-bearing for the entire theoretical framework. The exact chi-square law of the OLS residual variance estimator (Proposition 2), which underpins the upper certificate U(s, v̂, η) in Equation 7, requires Gaussianity. The conditional Gaussianity of the OLS intercept and the leverage bound (Lemma 2) also rely on Gaussian tail bounds. While this is a transparent and standard modeling choice in the bandit literature, the paper's phrasing in the abstract and Section 3.1—'whose exact χ² law keeps the certificate valid regardless of the unknown correlation'—could be misread as implying distribution-free validity. The authors should explicitly state in the main text (not just implicitly via the model definition) that the δ-PAC guarantee is contingent on the Gaussian assumption, and briefly discuss the gap between this theory and the Bern","section":null},{"comment":"The auto-loan replay experiment (Section 6) uses Bernoulli acceptance rewards, which violate the Gaussian assumption central to the theory. While the empirical results are compelling and show the algorithm works well in practice, the paper would benefit from a brief discussion of why PROBE remains robust in this non-Gaussian setting (e.g., asymptotic normality of OLS, or conservative nature of the certificate). This would strengthen the bridge between theory and practice.","section":null}],"minor_comments":[{"comment":"Section 1.1: The phrase 'whose exact χ² law keeps the certificate valid regardless of the unknown correlation' should be qualified to note it also requires the Gaussian observation model.","section":null},{"comment":"Algorithm 1: The notation s_{i,r} is used for the cumulative sample count, but it is initially set to 1 + t_cal and then updated as s_{i,r+1} = s_{i,r} + t_{i,r}. This is slightly confusing since t_{i,r} is the batch size for round r. Consider clarifying that s_{i,r} tracks the starting index of the round-r batch.","section":null},{"comment":"Table 1: The column 'Correct' reports 1.000 for all methods, which is good but could be more precise (e.g., 3000/3000).","section":null},{"comment":"Section 5.1: The proxy means are described as 'deliberately ordered in the opposite direction from the reward means.' It would help to explicitly state that this demonstrates the proxy's utility comes from within-arm covariance, not mean ranking, which is a key point.","section":null},{"comment":"Typo in Section 1.1: 'eO(K/κ²)' should be 'Õ(K/κ²)' for consistency with the rest of the paper.","section":null},{"comment":"The paper cites 'GPT-5.5 (OpenAI 2026)' and 'Qwen2.5-7B (Qwen et al. 2025)'. Ensure these references are complete and accurate for the camera-ready version.","section":null}],"recommendation":"minor_revision","confidential_remarks":"The paper is solid and the central claim is well-supported. The Gaussian assumption is the main theoretical limitation, but it is transparent and standard. The authors should address the presentation issues and add a brief discussion of the theory-practice gap in the experiments. The self-citation to Ji et al. (2025) is minor and does not create circularity; the core algorithmic design (OLS upper certificate, one-round lag) is novel and self-contained."},"author_rebuttal":null,"desk_editor":{"model":"glm-5.2","letter":"Here's the short version: this paper gives a clean, provably correct algorithm (PROBE) for best-arm identification when you have a cheap proxy correlated with an expensive reward. The core trick is using the exact chi-square law of the OLS residual variance to build a one-sided upper certificate on residual variance, sidestepping the anti-conservativeness you'd get from plugging in a correlation estimate. The PAC guarantee matches the known-correlation oracle up to a 1+κ factor and an additive calibration cost. It's a real result and worth taking seriously.","headline":"Solid contribution to proxy-augmented BAI; the Gaussian assumption is the main limitation but is transparent and standard.","tokens_in":25057,"tokens_out":177,"would_cite":true,"duration_ms":82185,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Cheap AI Predictions Cut Costly Experiments by Up to 61%","keywords":[],"falsifier":"A direct falsifier would be an instance where the reward-proxy pairs follow the specified bivariate Gaussian model, the correlation rho is large (say 0.8 or higher), and PROBE either fails to identify the best arm with probability at least 1-delta, or requires a number of samples that grows faster than the oracle benchmark by more than the stated constant factor. Another falsifier would be a demonstration that the OLS residual variance estimator's distribution deviates significantly from chi-square under the model, breaking the upper certificate and causing premature arm elimination.","tokens_in":24702,"feed_emoji":"🎯","tokens_out":788,"duration_ms":156718,"temperature":0.7,"pith_summary":"This paper proves that a cheap but imperfect AI-generated proxy score can be safely used to reduce the number of expensive real-world experiments needed to identify the best option among a finite set. The core mechanism is a control-variate adjustment: if one knows the correlation between the proxy and the true reward, one can subtract the predictable part of the reward noise, reducing the effective variance of each observation from 1 to 1 - rho^2. The difficulty is that this correlation is unknown and must be learned from the same expensive samples one is trying to save. The paper shows that naively plugging in an estimated correlation is dangerous because it can underestimate the residual variance, leading to premature and incorrect decisions. Instead, the proposed PROBE algorithm uses ordinary least squares to maintain a one-sided upper certificate on the residual variance, exploiting the exact chi-square law of the OLS residual to guarantee correctness regardless of the true unknown correlation. The result is a method that achieves the same sample efficiency as an oracle that knows the correlation in advance, up to a constant factor and a fixed calibration cost.","feed_headline":"Cheap AI Predictions Cut Costly Experiments by Up to 61%","feed_subtitle":"An algorithm called PROBE uses imperfect AI proxy scores to safely slash the number of expensive real-world tests needed to find the best决策,","key_machinery":"The key machinery is the residual-variance upper certificate U(s, v_hat, eta) derived from the chi-square law of the OLS residual variance estimator. Given a batch of s observations, the statistic (s-2) * v_hat / v follows a chi-square distribution with s-2 degrees of freedom, regardless of the unknown correlation rho. This scale-free property allows PROBE to construct a valid upper bound on v = 1 - rho^2 that holds with high probability, which then determines the batch size for the next elimination round. A calibration stage initializes the certificate within a factor (1 + kappa) of the true variance, and a monotone minimum update preserves this sandwich bound throughout the algorithm's run","core_discovery":"The central discovery is that the exact chi-square distribution of the OLS residual variance estimator provides a correlation-free, one-sided upper certificate on the true residual variance. This certificate can be used to size future sampling batches without ever needing to directly estimate or bound the reward-proxy correlation rho, thereby sidestepping the anti-conservatism and divergence problems that plague plug-in or conservative confidence bound approaches to rho. By maintaining this certificate through a monotone update and decoupling mean estimation from variance calibration via a one-round lag, the algorithm achieves oracle-level sample complexity with provable correctness.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Proxy Scores Cut Real-World Tests for Best-Arm Identification","PROBE Matches Oracle Sample Complexity With Imperfect AI Proxies","Chi-Square Certificate Enables Cost-Optimal Best-Arm Exploration","Cheap Proxies Reduce Costly Reward Pulls Without Knowing Correlation","Generative Proxy Scores Slash Sample Costs in Auto-Loan Pricing Replay"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The entire mathematical guarantee rests on the bivariate Gaussian observation model, where both the reward and the proxy score are assumed to be jointly normally distributed with unit variances. The exact chi-square law of the OLS residual variance, which makes the upper certificate valid and correlation-free, depends on this Gaussian assumption. If the reward or proxy distribution is non-Gaussian, the chi-square distribution no longer holds exactly, and the correctness proof","fun_headline_variants_meta":{"raw":{"variants":["Proxy Scores Cut Real-World Tests for Best-Arm Identification","PROBE Matches Oracle Sample Complexity With Imperfect AI Proxies","Chi-Square Certificate Enables Cost-Optimal Best-Arm Exploration","Cheap Proxies Reduce Costly Reward Pulls Without Knowing Correlation","Generative Proxy Scores Slash Sample Costs in Auto-Loan Pricing Replay"]},"model":"glm-5.2","effort":"low","cost_usd":0.0,"raw_usage":{"total_tokens":739,"prompt_tokens":651,"completion_tokens":88,"prompt_tokens_details":null},"tokens_in":651,"tokens_out":88,"duration_ms":70794,"temperature":1.0,"reasoning_tokens":null,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-09T23:37:49.454935+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"A direct falsifier would be an instance where the reward-proxy pairs follow the specified bivariate Gaussian model, the correlation rho is large (say 0.8 or higher), and PROBE either fails to identify the best arm with probability at least 1-delta, or requires a number of samples that grows faster than the oracle benchmark by more than the stated constant factor. Another falsifier would be a demonstration that the OLS residual variance estimator's distribution deviates significantly from chi-square under the model, breaking the upper certificate and causing premature arm elimination.","supporting_citations":[],"review_version":1}