{"id":"64feb419-d0eb-48ed-9700-9efe5c180bc4","arxiv_id":"2506.03670","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Bayesian updating is reframed as an optimization problem without inherent uncertainty quantification, and a PAC-style calibration step is proposed to give predictive intervals frequentist coverage.","lead":"This position paper argues that Bayesian posteriors should not be read as uncertainty quantification, only as ensemble weightings, because Bayesian updating is equivalent to a regularized optimization problem. The authors propose frequentist coverage measures for priors and a calibration step that turns Bayesian predictive intervals into intervals with claimed frequentist validity.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 9 is asserted, not established: without a finite-sample PAC bound the calibrated-interval guarantee is unsupported, and the paper's own misspecification study shows the guarantee can fail in practice.","rationale":"The reader's weakest assumption is the correct one. The paper's constructive claim is the calibration algorithm of Section 4.2, and Eq. 9 is its only formal support. Since C(epsilon) is never given and the text defers the generalization gap to future work, the claim 'leads to valid frequentist intervals with high probability' is currently an assertion. The support assumption is also load-bearing, and the beta20=3 result shows a regime in which the calibrated interval fails; calling this a numerical issue does not change the fact that the guarantee is not delivered there. The Section 5.2 statement that no explicit assumptions were made is contradicted by the support assumption and should be corrected. I also considered whether the Section 3 inference (Eq. 1 lacking an explicit uncertainty term implies the posterior does not encode uncertainty) is the more fundamental flaw, since UQ can be an implicit property of an optimization solution; however, the paper's own Bernstein-von Mises discussion concedes that posterior UQ can be frequentist-valid in some regimes, and the position is best read as a qualified challenge. The missing Eq. 9 bound is therefore the decisive technical gap. The CONDITIONAL verdict remains appropriate: the paper should either derive the bound or appropriately soften the guarantee claim.","tokens_in":9869,"tokens_out":10120,"duration_ms":99444,"concrete_test":"Derive an explicit finite-sample bound for sup_q |R_emp(q)-R(q)| in the scalar-q interval family used by Section 4.2, using only assumptions stated there (i.i.d. quantile-estimation data, support inclusion, 0-1 loss). If the derivation requires additional assumptions such as bounded predictive density, Lipschitz quantile functions, or discretization of q, then Eq. 9 is not supported as written. Cross-check the resulting C(delta,m) numerically in the Study 1 setup at m=30 and alpha=0.1 by estimating the frequency of R(test,q_hat) <= alpha+C over 10^4 data splits; if no small C gives the claimed 1-delta frequency, the guarantee fails in the paper's own simulation regime.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing gap is in the constructive half of the paper. Section 4.2's claim that the calibration algorithm leads to valid frequentist intervals with high probability is stated through Eq. 9, but Eq. 9 is never proved. C(epsilon) is left unspecified, the uniformity argument over q is omitted, and the text explicitly defers the generalization gap to future work, naming no bound. The support assumption before Eq. 6 ('the support of the data is a subset of the support of the predictive distribution') is the condition that makes some q drive empirical risk to alpha, and it is not guaranteed for misspecified or overparameterized models. The beta20=3 experiment in Section 5.2 is precisely a failure case: calibrated coverage stays below 0.9. The paper attributes this to numerical precision (q of order 1e-40), but a frequentist guarantee that is not realizable in finite precision is not a working guarantee. The Section 5.2 sentence claiming no explicit assumptions were made contradicts the support assumption stated one page earlier and is an overclaim. The negative position about Bayesian UQ is arguable, but the paper's own positive contribution remains conditional on a bound that has not been provided.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that Bayesian posterior distributions, when viewed as solutions to the optimization problem in Eq. (1), do not by themselves provide uncertainty quantification, because no explicit coverage or accuracy objective appears in that problem. It proposes three frequentist quality measures for priors (Eqs. 2, 4, and 5) and a calibration algorithm (Sec. 4.2) that adjusts the central quantile q of the predictive distribution on a held-out set, claiming PAC-style frequentist coverage of the resulting intervals (Eq. 9). Two simulation studies on Bayesian linear regression, including a misspecified model, compare calibrated and naive intervals. The paper concludes that Bayesian UQ requires frequentist validation and calibration before use.","tokens_in":10032,"tokens_out":12328,"duration_ms":136225,"significance":"The paper addresses an important and actively debated question, and its central negative claim is coherent as a position. The proposed quality measures are reasonable formalizations of prior performance, and the calibration algorithm is concrete and honestly evaluated on a separate test set, so the empirical demonstration is not circular. The paper also states several of its limitations explicitly, including the unproved generalization bound and the numerical precision issue. If the missing PAC bound is supplied, or if the claims are appropriately weakened to a conjecture, the paper could be a useful contribution to the debate on Bayesian UQ in machine learning. As it stands, however, the positive contribution rests on an asserted rather than proved guarantee, and the simulations contain a clear failure case.","major_comments":[{"comment":"The central claim that the calibration algorithm \"leads to valid frequentist intervals with high probability\" is stated without proof. The bound is announced as achievable \"using the PAC framework,\" but C(epsilon) is never specified, the sample-size dependence is unclear (epsilon(n) despite the validation set having size m), and the need for uniformity over the grid-searched q is not addressed. The paper itself defers the generalization gap to future work, and the low-complexity claim is supported only by the heuristic that \"only one parameter is estimated.\" This is load-bearing because the paper's constructive claim depends on Eq. (9).","section":"Sec. 4.2, Eq. (9)"},{"comment":"The argument that some q drives the empirical risk to alpha relies on the support of the data being a subset of the support of the predictive distribution. This condition is not guaranteed for misspecified or overparameterized models, and no diagnostic is offered. The beta20=3 experiment in Sec. 5.2 is a concrete failure: calibrated coverage remains below the 0.9 target, and the paper's explanation in terms of numerical precision (q of order 1e-40) does not amount to a working guarantee. The paper also does not specify what the algorithm does when no grid point achieves empirical risk <= alpha, which is exactly the situation in that experiment.","section":"Sec. 4.2, support assumption before Eq. (6)"},{"comment":"The sentence \"since no explicit assumptions were made in the presentation in section 4.2, there are not any limitations of the utility of the method on more complex model classes\" contradicts the support assumption stated in Sec. 4.2 and overlooks the dependence on the unproved PAC bound in Eq. (9). This overclaim should be removed or substantially qualified.","section":"Sec. 5.2"},{"comment":"The empirical evaluation averages over only 10 seeds and reports no standard errors or confidence bands for coverage or width. Because Eq. (9) is unproved, the simulations carry much of the evidence for the algorithm's practical validity. Without variability measures or a description of the grid resolution and stopping rule, the reader cannot assess the stability or reproducibility of the reported differences, nor whether the beta20=3 failure is an artifact of the particular grid choice.","section":"Secs. 5.1-5.2"}],"minor_comments":[{"comment":"The word \"pesimisstic\" should be \"pessimistic,\" and \"can be as a softer version\" should be \"can be seen as a softer version.\"","section":"Sec. 4.1"},{"comment":"The word \"achiving\" should be \"achieving.\"","section":"Sec. 5.2"},{"comment":"The reference to Theodoridis [2015] appears twice in the introductory paragraph; please consolidate.","section":"Introduction"},{"comment":"The notation P(Y^{v,*,i} | X^v_i) is not defined explicitly; a short definition of the predictive distribution on the quantile estimation set would help the reader.","section":"Sec. 4.2"},{"comment":"The grid-search resolution, the stopping tolerance, and the behavior when no q satisfies the empirical risk condition are not specified; these details are needed for reproducibility.","section":"Sec. 4.2"},{"comment":"The heading \"Optimization does not give UQ\" states a categorical conclusion, while the text offers an interpretive argument; consider softening the wording to avoid inviting a formal disproof that the paper does not provide.","section":"Sec. 3"},{"comment":"The paper does not discuss the close relationship to conformal prediction, which gives finite-sample coverage under exchangeability; a brief comparison would clarify the contribution.","section":"References"},{"comment":"The sentence \"the solution of a well specified problem, as discussed in 4.2, always exists\" is too strong because the existence of a suitable q depends on the support assumption and on numerical realizability; please qualify.","section":"Sec. 6"}],"recommendation":"major_revision","confidential_remarks":"The paper is a position paper, so the standard for formal proof may be more relaxed than for a full theoretical contribution. However, the abstract and conclusions assert a working frequentist guarantee, and the constructive half of the paper depends on Eq. (9), which is explicitly deferred to future work. This is a load-bearing gap rather than a presentation issue. If the authors reframe the algorithm as a heuristic with formal analysis as a conjecture, or supply a proof of the bound, the paper would be much stronger. The comparison with conformal prediction is also worth requesting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the paper gives a clean formulation of Bayesian updating as regularized ensemble optimization (from Knoblauch et al.) and argues from there that posterior spread is not automatically uncertainty quantification. That argument is sane, and the paper is honest that its positive half—a calibration algorithm with a PAC-style guarantee—is not yet a theorem. Second, the calibration algorithm is conceptually split conformal prediction specialized to predictive distributions; the authors cite Park et al. but not the conformal literature, and the stated guarantee in Eq. 9 is an assertion, not a derivation.\n\nWhat is genuinely useful is the way it lays out prior quality measures (average, worst-case, probabilistic) as frequentist targets. Those definitions are clean and could be operationalized. The simulation study is transparent: well-specified cases calibrate fine, extreme priors break down, and the misspecified beta20=3 case fails to reach 90% coverage. The authors attribute that to numerical precision, which is plausible but also an admission that the guarantee is not realizable in finite precision.\n\nThe biggest soft spot is Eq. 9. The PAC bound is stated without proof, C(epsilon) is never given, and the text explicitly defers the generalization gap to future work. That is load-bearing because the paper's positive claim is valid frequentist intervals with high probability. Also, Section 5.2's sentence that 'no explicit assumptions were made' conflicts with the support assumption before Eq. 6. That overclaim should be fixed. The simulations average only 10 seeds with no error bars, so the empirical claims are weaker than they look. The missing split-conformal citation is minor but real.\n\nThis paper is for people working on Bayesian deep learning UQ who need a reminder that the posterior is a model average, not a certificate. A rigorous referee should see it, but the verdict should be conditional: either prove the bound, add a finite-sample version, or soften the claim.\n\nI would send it to review; the position is important and the defects are fixable. But I would not accept it with Eq. 9 as is.","headline":"A defensible position paper that sharpens the Bayesian-UQ-as-ensemble view, but its central calibration guarantee is a stated hope rather than a proven theorem.","tokens_in":10625,"tokens_out":2124,"would_cite":true,"duration_ms":22602,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62F15","62G15","68T05"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that Bayesian posteriors do not meaningfully encode uncertainty, so predictive intervals must be calibrated by frequentist means to be valid.","keywords":["Bayesian uncertainty quantification","frequentist coverage","PAC-Bayes","predictive intervals","posterior calibration","ensemble learning","prior quality","optimization-based Bayesian inference"],"falsifier":"Run the Section 4.2 calibration on many independent simulated training/calibration/test splits with a misspecified model and a prior mean far from the truth, then record the fraction of splits in which the calibrated interval achieves the nominal $1-\\alpha$ coverage; if that fraction does not reach the claimed high-probability level once an explicit $C(\\epsilon)$ is supplied, Eq. 9 is false as stated.","tokens_in":9558,"feed_emoji":"📊","tokens_out":8264,"duration_ms":78197,"temperature":0.7,"pith_summary":"Bayesian updating in machine learning, the paper argues, does not deliver uncertainty quantification for free: the posterior is the solution to an optimization problem that contains no explicit uncertainty objective, so interpreting it as UQ is not justified. A posterior is better understood as an optimally weighted ensemble of models, and any Bayesian credible or predictive interval must be checked by frequentist criteria before it supports an uncertainty claim. To make that check concrete, the paper defines average, worst-case, and probabilistic measures of a prior's quality, and proposes a calibration algorithm that adjusts the quantiles of the Bayesian predictive distribution until empirical coverage on a separate dataset reaches a target level. A PAC-style bound is stated as giving valid frequentist coverage with high probability, and simulations show the calibrated intervals reaching nominal coverage where naive Bayesian intervals do not, except under severe misspecification or prior-data conflict where numerical precision fails. The position, if accepted, turns Bayesian UQ into a two-stage procedure: first build the posterior ensemble, then learn valid intervals from data.","feed_headline":"Bayesian UQ is not free: calibration must guarantee coverage","feed_subtitle":"Posteriors are optimal ensembles, not uncertainty statements; calibration gives frequentist validity.","key_machinery":"The load-bearing object is the optimization-based representation of Bayes' rule (Eq. 1), which converts posterior inference into a regularized optimization over probability measures, with the prior as 'centroid' and the divergence as regularizer; the paper reads this representation as evidence that posterior uncertainty is a by-product, not an objective. The second piece is the Section 4.2 calibration routine: predictive intervals are defined by quantiles $c_1(q,x)$ and $c_2(q,x)$ of the predictive distribution, a quantile-estimation dataset is used to compute empirical risk as a 0-1 loss over $q$, and grid search finds the smallest symmetric $q$ with empirical risk at most $\\alpha$. The PAC bound of Eq. 9 is the mechanism that supposedly converts this empirical risk into a population coverage guarantee up to slack $C(\\epsilon)$.","core_discovery":"The paper's central claim is that Bayes' rule is an optimization scheme in disguise: choosing the posterior $p$ by minimizing $\\mathbb{E}_{f\\sim p}[l(f(x),y)] + D(p\\|\\pi_0)$ over all probability measures, with negative log-likelihood loss and KL divergence as $D$, reproduces the Bayes update. Since that objective contains no term that asks for calibrated or valid uncertainty statements, the paper concludes that posterior or predictive distributions carry no built-in uncertainty guarantee. It therefore reinterprets the posterior as an optimal ensemble relative to a prior, and treats the predictive distribution as an ensemble summary rather than a UQ device. It then supplies frequentist tools: quality measures $Q(\\pi_0)$, $Q'(\\pi_0)$, and $Q''(\\pi_0)$ for evaluating priors by average, worst-case, and probabilistic coverage, and a calibration procedure that chooses a symmetric quantile $\\hat{q}$ by binary classification on a held-out quantile-estimation set. The claimed warrant for the procedure is the PAC bound $P(R((X,Y),\\hat{q})\\le \\alpha + C(\\epsilon))\\ge 1-\\epsilon(n)$, giving valid frequentist intervals with high probability.","pith_inferences":["A natural extension the authors leave implicit is to benchmark the calibrated Bayesian predictive intervals against other distribution-free interval constructions on the same regression and classification tasks.","The prior-quality measures $Q$, $Q'$, and $Q''$ suggest an operational prior-selection rule: choose the prior that maximizes estimated probabilistic coverage $Q''(\\pi_0)$, turning subjective prior choice into a tuned hyperparameter.","If the missing generalization gap $C(\\epsilon)$ turns out to be large, the paper's suggestion to replace $\\alpha$ by a smaller $\\alpha'$ would make the method conservative at the cost of wider intervals; this trade-off is a testable prediction of the framework.","The documented failure at $\\beta_{20}=3$ points to an improvement: a coverage-aware fallback that expands intervals whenever the predictive density at observed points falls below numerical precision, rather than relying on grid-search quantiles."],"forward_implications":["Bayesian credible and predictive intervals in machine learning would no longer be reported as uncertainty quantifiers until their frequentist coverage has been checked.","Posteriors from Bayesian neural networks become ensemble weightings, and valid UQ becomes a separate calibration step on held-out data.","Prior choice becomes an empirical problem: priors can be compared by their average, worst-case, or probabilistic coverage, giving Bayesian inference an external validity criterion.","The one-parameter quantile calibration offers a simple, distribution-free route to predictive intervals with approximately nominal coverage, requiring only grid search on a quantile-estimation dataset.","The paper's own simulations indicate that under strong prior-data conflict or severe misspecification, even calibrated intervals can fail because the predictive distribution assigns numerically negligible probability to observed points."],"supporting_citations":[{"why":"Supplies the optimization-centric view of Bayes' rule (Eq. 1) that the paper's argument starts from.","marker":"Knoblauch et al. [2022]"},{"why":"Establishes that the Bayes update is the optimal solution to an information-processing objective, which the paper cites as the basis of the optimization representation.","marker":"Zellner [1988]"},{"why":"Introduces PAC confidence sets via calibrated prediction, which the Section 4.2 algorithm adapts to predictive intervals.","marker":"Park et al. [2019]"},{"why":"Extends PAC confidence predictions to classifiers, the second source for the calibration-as-classification approach.","marker":"Park et al. [2020]"},{"why":"Provides PAC-Bayesian guarantees for randomized estimators that motivate treating the posterior as an ensemble.","marker":"McAllester [1998]"},{"why":"Supplies PAC-Bayesian margin bounds used to frame generalization guarantees for stochastic predictions.","marker":"McAllester [2003a]"},{"why":"Documents that Bayesian neural network posteriors improve predictive performance when the likelihood is overcounted, evidence that standard posterior interpretation is fragile.","marker":"Wenzel et al. [2020]"},{"why":"Supplies the Bernstein-von Mises theorem used to delimit when Bayesian credible intervals are asymptotically valid.","marker":"Van der Vaart [2000]"}],"fun_headline_variants":["Bayes rule is optimization, not uncertainty quantification","Posteriors are ensembles, not uncertainty statements","No free lunch: Bayesian UQ needs calibration","Bayesian UQ is optimization in disguise, requires calibration","Bayes posteriors: optimal ensembles, not calibrated UQ"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire high-probability validity claim rests on an unproved PAC bound: the paper assumes that a low empirical error on the quantile-estimation dataset implies a low population error with unspecified slack $C(\\epsilon)$, and even acknowledges that the generalization gap remains to be shown.","fun_headline_variants_meta":{"raw":{"variants":["Bayes rule is optimization, not uncertainty quantification","Posteriors are ensembles, not uncertainty statements","No free lunch: Bayesian UQ needs calibration","Bayesian UQ is optimization in disguise, requires calibration","Bayes posteriors: optimal ensembles, not calibrated UQ"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000438,"raw_usage":{"total_tokens":2183,"prompt_tokens":863,"completion_tokens":1320,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":479,"completion_tokens_details":{"reasoning_tokens":1244}},"tokens_in":479,"tokens_out":1320,"duration_ms":9997,"temperature":1.0,"reasoning_tokens":1244,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:57:36.608780+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the Section 4.2 calibration on many independent simulated training/calibration/test splits with a misspecified model and a prior mean far from the truth, then record the fraction of splits in which the calibrated interval achieves the nominal $1-\\alpha$ coverage; if that fraction does not reach the claimed high-probability level once an explicit $C(\\epsilon)$ is supplied, Eq. 9 is false as stated.","supporting_citations":[{"cited_title":"An optimization-centric view on bayes' rule: Reviewing and generalizing variational inference","cited_arxiv_id":null,"evidence_quote":"Supplies the optimization-centric view of Bayes' rule (Eq. 1) that the paper's argument starts from."},{"cited_title":"Optimal information processing and bayes's theorem","cited_arxiv_id":null,"evidence_quote":"Establishes that the Bayes update is the optimal solution to an information-processing objective, which the paper cites as the basis of the optimization representation."},{"cited_title":"Some pac-bayesian theorems","cited_arxiv_id":null,"evidence_quote":"Provides PAC-Bayesian guarantees for randomized estimators that motivate treating the posterior as an ensemble."}],"review_version":1}