{"id":"aabb1477-710d-47e8-80cb-42f4cbc97735","arxiv_id":"2502.02363","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"FAB-PPI applies frequentist-assisted-by-Bayes confidence regions to the PPI rectifier, yielding shorter asymptotic confidence intervals when predictions are accurate and reverting to standard PPI under a horseshoe prior when they are not.","lead":"This paper combines prediction-powered inference with Bayes-assisted confidence regions called FAB, producing shorter valid confidence intervals when machine-learning predictions are good and automatic fallback to standard intervals when predictions are poor. It is a methods paper for semi-supervised statistical estimation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central coverage guarantee is asymptotic and depends on plugging the estimated standard error into the FAB prior; no finite-sample coverage control is shown for the small-n regime that motivates PPI.","rationale":"The reader's CONDITIONAL verdict is appropriate. I agree that Assumption 3.1 is load-bearing: the entire theory in Section 4 and the supplementary lemmas relies on a CLT and on variance estimators converging almost surely, and the FAB construction loses its exact coverage guarantee when sigma is replaced by a data-dependent estimate. The paper does establish asymptotic coverage correctly, and the horseshoe prior's reversion property is plausible; I found no internal mathematical contradiction that would require rejection. The main gap is the mismatch between the abstract's unconditional 'correct coverage for any prior' and the theorem's asymptotic, CLT-dependent statement, plus the absence of any finite-sample or slow-CLT evaluation. My proposed test would settle whether the practical guarantee holds in the small-n regime that PPI is designed for. This does not change the reader's conditional verdict.","tokens_in":29121,"tokens_out":18340,"duration_ms":196124,"concrete_test":"Run the mean-estimation experiment of Section 5.1 with n = 50, N = 10^6, and heavy-tailed label noise (e.g., a t-distribution with 4 degrees of freedom) so the CLT is a poor approximation; compute the empirical coverage of the FAB-PPI (HS) 90% interval over 10^4 repetitions, alongside PPI++. Also compare FAB-PPI using the estimated bsigma_theta against an oracle version using the true sigma. If the empirical coverage is below 0.88, or the estimated-sigma version undercovers the oracle version by more than 1 percentage point, the asymptotic guarantee does not transfer to the finite-sample sparse-label regime that motivates the method.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Theorem 4.1 (Section 4.4) states liminf coverage under Assumption 3.1, which requires CLTs for bm_theta and bDelta_theta with variance estimators converging almost surely. The proof (Lemmas S3.1 and S3.2) treats bsigma_theta as a plug-in for the known FAB scale sigma; exact coverage of the FAB construction (Definition 3.2) is only guaranteed for a fixed prior and known sigma. Setting the prior scale tau_n = bsigma_theta makes the prior data-dependent, so the abstract's claim of 'correct coverage for any prior' is an overstatement: only asymptotic coverage is established, for two specific priors, and only when the CLT and variance estimates are accurate. Since the method is explicitly motivated by scarce labelled data (small n), the practical validity of FAB-PPI in the advertised regime is unsupported by the theory. The experiments average over 1000 repetitions with no reported error bars and do not probe a slow-CLT or small-n regime, so the gap between asymptotic guarantee and finite-sample behaviour is not empirically addressed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FAB-PPI, a modification of prediction-powered inference (PPI) that constructs confidence regions for the rectifier Δθ using the frequentist-assisted-by-Bayes (FAB) framework with a prior on Δθ. The method replaces the standard CLT-based confidence interval for Δθ in the PPI procedure with an FAB confidence region, and the PPI estimator with a Tweedie-style posterior-mean shrinkage estimator. The authors consider a Gaussian prior and a horseshoe prior, with the prior scale set to the estimated standard error of the rectifier estimator. They prove asymptotic coverage of the resulting PPI confidence region under a CLT assumption (Theorem 4.1), a reversion-to-PPI property under the horseshoe prior for large estimated rectifiers (Proposition 4.2), and consistency of the horseshoe-based mean estimator as opposed to the Gaussian-based one (Proposition 4.3). Experiments on synthetic and real data show that FAB-PPI with the horseshoe prior can produce shorter confidence intervals than PPI when predictions are good, while matching PPI when predictions are poor.","tokens_in":29294,"tokens_out":5959,"duration_ms":62818,"significance":"If the theoretical and empirical claims hold, FAB-PPI is a practically useful extension of PPI: it provides a principled way to shrink the rectifier when predictions are accurate, with an automatic fallback to standard PPI under heavy-tailed priors. The paper's main strengths are its clear derivation, the closed-form marginal likelihood for the horseshoe prior, the modularity with PPI and PPI++, the formal asymptotic proofs in the appendix, and the released code. The conceptual contribution—using the FAB framework to inject prior knowledge about prediction quality into PPI—is novel and well-motivated. However, the practical significance is tempered by the fact that the central coverage guarantee is asymptotic, while the method is explicitly motivated by settings with scarce labelled data. The experiments do not directly address the small-sample regime where the CLT and variance-estimator consistency might fail, and they omit error bars on the reported performance metrics.","major_comments":[{"comment":"The prior scale is set to the estimated standard error, τ_n = bσ_θ (Section 4.3), making the prior data-dependent. The exact-coverage property of the FAB construction described in Section 3.3 (Definition 3.2) applies only to a fixed prior and known scale σ. Theorem 4.1 therefore establishes only asymptotic coverage, under Assumption 3.1's CLT and almost-sure variance-estimator consistency. The abstract and the bullet list in Section 1, however, state that FAB-PPI 'maintains its frequentist guarantees' and has 'correct coverage for any choice of prior' without the asymptotic qualifier. Since the method is motivated by scarce labelled data (small n), the gap between the asymptotic guarantee and the advertised regime is load-bearing. The authors should either provide finite-sample coverage bounds or a Berry–Esseen-type refinement, or clearly restrict all coverage claims to the asymptotic regime and add diagnostics for checking Assumption 3.1 in practice.","section":"§4.3, Theorem 4.1, and Abstract"},{"comment":"The experimental evaluation reports only point estimates of MSE, interval volume, and coverage, averaged over 1000 (or, in Figure S10, 100) repetitions, with no standard errors or confidence intervals. Coverage is a random quantity over simulation repetitions; without error bars or a table of worst-case coverages, the plots cannot distinguish nominal coverage from meaningful undercoverage, especially at the smaller sample sizes (n = 50, 100, 200) that are central to the method's motivation. Furthermore, no experiment probes a setting where Assumption 3.1 is questionable—e.g., where n is too small for the sample-variance estimators to be accurate or where the rectifier distribution is skewed or heavy-tailed—so the finite-sample validity claim is not empirically supported. The authors should report standard errors or confidence bands for all performance metrics and include at least one small-n or non-CLT scenario.","section":"§5, Figures 1–3 and S6–S13"},{"comment":"Proposition 4.3 states that the FAB-PPI mean estimator under the horseshoe prior is consistent while the Gaussian-prior estimator is not. The proof in Appendix S3.3 relies on the assertion that ℓ'_HS(y; 1, 1) is bounded and on the almost-sure convergence of (bσ^PP)^2 ℓ'(bΔ^PP; bσ^PP, bσ^PP) to 0. While the boundedness is plausible from the asymptotic expansion in S1.1, it is not demonstrated in the main text or in the proof itself, and the Gaussian-prior inconsistency is shown only under the specific choice τ_n = bσ_θ. The paper would be strengthened by stating the precise regularity conditions on the prior and the estimator needed for the limit (S41)–(S42) to hold uniformly, rather than as a one-line assertion.","section":"§4.5 and Proposition 4.3"}],"minor_comments":[{"comment":"The notation N(Δθ; 0, bσθ) in Equation (22) is inconsistent with the variance-parameter convention used elsewhere in the paper (e.g., Section 3.3 writes N(β, σ²)). Please clarify whether bσθ denotes a standard deviation or a variance; the downstream formulas in S3.3 imply the prior variance is bσθ².","section":"Equation (22)"},{"comment":"Figure S10 reports results 'over 100 repetitions' while all other experimental figures use 1000 repetitions; the authors should state why this figure uses fewer repetitions or increase the number to match the rest of the paper.","section":"Figure S10 caption"},{"comment":"The biased-predictions experiment assumes N is infinite; while this simplifies the analysis of the rectifier, it is an idealization that is not achievable in practice. A brief remark on how the results would change for finite N would improve the interpretation.","section":"Section 5.1, biased predictions"},{"comment":"The proposition is labeled 'informal' in the main text and the formal statement appears only in Appendix S3.2. Since the formal result is available, the authors should state the proposition with its precise assumptions (e.g., uniform convergence of the spending function) directly in the main text, or at least give a precise reference to Corollary S3.5 and its conditions.","section":"Section 4.4, Proposition 4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a methodologically sound extension of PPI with a clear contribution, and the asymptotic theory appears internally consistent. The main risk is the mismatch between the abstract's unqualified coverage claims and the actual asymptotic guarantee under Assumption 3.1. The authors' reliance on their own prior work (Cortinovis & Caron 2024) is appropriate because they build on those results. A major revision that either sharpens the finite-sample theory or recalibrates the claims and experimental evidence would make the paper suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is new and worth engaging with: wrap the PPI rectifier in a FAB confidence region with a horseshoe prior, so good predictions give shorter intervals and bad predictions revert to standard PPI. The theory is the main event and it largely holds up. Theorem 4.1's asymptotic coverage follows by Boole's inequality from CLT assumptions, Lemma S3.2 cleanly shows the horseshoe's reversion to the classical z-interval, and Proposition 4.3 nicely exposes why the Gaussian prior is inconsistent for mean estimation. The appendix does the work; I checked the steps and they are coherent. The parameter-free default (prior scale equal to the estimated standard error) is pragmatic, and the code being public is a plus.\n\nThe soft spot is exactly what the stress-test note flags. The abstract and introduction say FAB-PPI has 'correct coverage for any choice of prior.' That is true in the abstract FAB setup with fixed prior and known scale, but here the prior scale is the estimated standard error. Coverage is asymptotic only, and the small-n regime that motivates PPI has no finite-sample control. The paper should replace 'correct coverage' with 'asymptotically correct coverage' and state explicitly that the data-dependent prior scale is justified by the CLT and variance consistency. This is a presentation flaw, not a mathematical one, but it matters because readers will take the abstract at face value.\n\nThe experiments are adequate but not polished: averages over 1000 repetitions with no error bars, and some coverage plots (e.g., forest at small n) dip below nominal without comment. Since the method's selling point is shorter intervals at a given coverage, seeing the spread of coverage across repetitions would help. The real-data results do show the expected pattern, and the biased-predictions simulation clearly demonstrates the horseshoe's robustness advantage over the Gaussian.\n\nOverall, this is a serious contribution from people who know the FAB and PPI literatures. The theory is solid, the prior contrast is instructive, and the method is likely to be used. It deserves a careful referee round, with the main requests being an honest coverage statement and a bit more transparency in the experiments.\n\nMy recommendation: send it to review, and expect it to be accepted after those revisions.","headline":"FAB-PPI is a genuinely new and mostly sound extension of prediction-powered inference; the main fix needed is to align the abstract's coverage claim with the asymptotic, plug-in-scale reality.","tokens_in":29849,"tokens_out":1571,"would_cite":true,"duration_ms":19969,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62F25","62F15","62G20"],"pacs":[],"model":"deepseek-v4-flash","headline":"FAB-PPI: a prior on prediction quality shortens valid prediction-powered confidence intervals.","keywords":["prediction-powered inference","frequentist-assisted by Bayes","FAB confidence regions","horseshoe prior","semi-supervised inference","confidence intervals","machine learning predictions"],"falsifier":"Run the biased-predictor simulation from the paper with $n=N=10^4$ and a rectifier several standard errors from zero; if the horseshoe FAB-PPI interval's endpoints do not match the PPI interval's endpoints to within Monte Carlo error, the reversion property fails. Conversely, if coverage is below $1-\\alpha$ in a setting where Assumption 3.1 is satisfied, the coverage theorem fails.","tokens_in":28883,"feed_emoji":"🎯","tokens_out":7717,"duration_ms":70867,"temperature":0.7,"pith_summary":"FAB-PPI is a method for prediction-powered inference (PPI) that treats the quality of machine-learning predictions as something we have prior beliefs about, and then uses those beliefs to build confidence intervals that are still valid in the frequentist sense. The paper's central claim is that by placing a horseshoe prior on the rectifier — the expected gap between the predictions and the truth used to correct PPI — the resulting intervals are shorter than standard PPI intervals when predictions are good, and revert to standard PPI intervals when predictions are bad. The authors prove asymptotic coverage at level $1-\\alpha$, show the reversion-to-PPI property formally, and prove that the mean estimator is consistent under the horseshoe prior but not under a Gaussian prior. A sympathetic reader would care because this gives a parameter-free way to make PPI adaptive to prediction quality without sacrificing its guarantees.","feed_headline":"Horseshoe prior tightens prediction-powered confidence intervals","feed_subtitle":"FAB-PPI keeps nominal coverage while shrinking intervals when predictions are good, and reverts to PPI when they are not.","key_machinery":"The load-bearing object is the FAB confidence region, which for a scalar normal observation $W\\sim N(\\beta,\\sigma^2)$ solves for the region with exact $(1-\\delta)$ coverage that minimises expected volume under a prior $\\pi_0(\\beta)$; by the Neyman-Pearson lemma it is the acceptance region of a most powerful test, computable from the marginal likelihood. FAB-PPI applies this to the rectifier estimator $\\widehat{\\Delta}_\\theta$. The horseshoe prior, a scale mixture of normals with a half-Cauchy on the scale, contributes an infinitely tall spike at zero (strong shrinkage when predictions are good) and power-law tails (so the region reverts to the classical interval when $|\\widehat{\\Delta}_\\theta|$ is large). Tweedie's formula turns the derivative of the log-marginal likelihood into a Bayes-assisted point estimator $\\widehat{\\Delta}_\\theta+\\widehat\\sigma_\\theta^2\\ell'(\\widehat{\\Delta}_\\theta;\\widehat\\sigma_\\theta,\\tau_n)$.","core_discovery":"Under Assumption 3.1, FAB-PPI constructs confidence regions for $\\theta^\\star$ by combining a standard confidence interval for the fit term $m_\\theta$ with a FAB confidence region for the rectifier $\\Delta_\\theta$, where the FAB region is chosen to minimise expected volume under a prior whose scale is the estimated standard error. Theorem 4.1 states that these regions have correct asymptotic coverage for both the Gaussian and horseshoe priors. Proposition 4.2 states that under the horseshoe prior, when the estimated rectifier is large, the FAB-PPI region converges to the standard CLT-based PPI region; the Gaussian version instead grows without bound. Proposition 4.3 states that the FAB-PPI mean estimator is consistent under the horseshoe prior and inconsistent under the Gaussian prior. The paper presents these results as the theoretical basis for using the horseshoe as the default prior in FAB-PPI.","pith_inferences":["The paper leaves implicit that the horseshoe's reversion property makes FAB-PPI a shrinkage estimator whose effective prior influence automatically vanishes when data disagree; this suggests it could serve as a general adaptive bias-correction layer for semi-supervised estimators beyond PPI.","A testable extension is to replace the horseshoe with other power-law-tailed scale mixtures (e.g. beta-prime priors) and check whether the same coverage, shrinkage, and reversion properties hold with cheaper computation.","The prior scale $\\tau_n=\\widehat\\sigma_\\theta$ is justified asymptotically; in small samples, users may need to calibrate the scale by simulation, since the coverage theorem does not quantify the finite-sample error.","Although the paper focuses on mean and M-estimation, the same construction applies to any parameter defined by a convex loss whose subgradient admits the $m_\\theta+\\Delta_\\theta$ decomposition, so the result should transfer to quantile and logistic objectives."],"forward_implications":["FAB-PPI intervals are shorter than standard PPI intervals when the predictor is accurate, with the same nominal coverage, as shown on the AlphaFold, Galaxies, Forest, and Healthcare datasets.","If the predictor is heavily biased, the horseshoe version stops shrinking and matches standard PPI, while the Gaussian version's interval grows without bound; this makes horseshoe FAB-PPI robust to prior misspecification.","The mean estimator under the horseshoe prior is consistent as $n,N\\to\\infty$; under the Gaussian prior it is not, so the horseshoe is the theoretically supported default.","FAB-PPI can be combined with power tuning (PPI++) and extends to multivariate parameters through independent priors and a union bound.","The method inherits PPI's computational requirement of a grid search over $\\theta$ in general convex problems, except in special cases like mean estimation and linear regression."],"supporting_citations":[{"why":"Defines prediction-powered inference and the $m_\\theta+\\Delta_\\theta$ decomposition that FAB-PPI builds on.","marker":"Angelopoulos et al. (2023a)"},{"why":"Introduces the PPI++ power-tuned estimator and the control-variate form that FAB-PPI++ adapts.","marker":"Angelopoulos et al. (2023b)"},{"why":"Provides the frequentist-assisted-by-Bayes construction used to build the rectifier's confidence region.","marker":"Yu & Hoff (2018)"},{"why":"Establishes that minimising expected confidence-region volume is equivalent to pointwise most-powerful tests.","marker":"Pratt (1961; 1963)"},{"why":"Supplies the horseshoe prior whose spike-and-heavy-tail shape drives shrinkage and reversion.","marker":"Carvalho et al. (2010)"},{"why":"Proves that FAB regions with power-law-tailed priors revert to classical intervals for large observations.","marker":"Cortinovis & Caron (2024)"},{"why":"Tweedie's formula links the log-marginal-likelihood derivative to the posterior mean used in the estimator.","marker":"Efron (2011)"},{"why":"Motivates the choice of the estimated standard error as the horseshoe prior scale.","marker":"Piironen & Vehtari (2017)"}],"fun_headline_variants":["Horseshoe prior tightens prediction-powered intervals","FAB-PPI: Bayesian prior sharpens PPI inference","Adaptive PPI: horseshoe prior reverts when predictions mislead","Tighter PPI intervals from Bayesian prior, with same coverage"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is Assumption 3.1: the estimators of the fit term and the rectifier are asymptotically normal with consistently estimated variances; if n and N are too small or the variance estimators are poor, the coverage and efficiency guarantees have no support.","fun_headline_variants_meta":{"raw":{"variants":["Horseshoe prior tightens prediction-powered intervals","FAB-PPI: Bayesian prior sharpens PPI inference","Adaptive PPI: horseshoe prior reverts when predictions mislead","Tighter PPI intervals from Bayesian prior, with same coverage"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000595,"raw_usage":{"total_tokens":2745,"prompt_tokens":865,"completion_tokens":1880,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":481,"completion_tokens_details":{"reasoning_tokens":1810}},"tokens_in":481,"tokens_out":1880,"duration_ms":13131,"temperature":1.0,"reasoning_tokens":1810,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T12:24:54.163361+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the biased-predictor simulation from the paper with $n=N=10^4$ and a rectifier several standard errors from zero; if the horseshoe FAB-PPI interval's endpoints do not match the PPI interval's endpoints to within Monte Carlo error, the reversion property fails. Conversely, if coverage is below $1-\\alpha$ in a setting where Assumption 3.1 is satisfied, the coverage theorem fails.","supporting_citations":[],"review_version":1}