{"id":"50aca7c6-e332-4ebf-b050-e5959ba72b3b","arxiv_id":"2505.13585","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"SBMC samples from a MAP-anchored approximate posterior with parallel MCMC/SMC chains, delivering deep-ensemble-level accuracy with better calibrated epistemic uncertainty at similar cost.","lead":"SBMC is a Bayesian deep learning method that interpolates between a single point estimate and the full posterior by shrinking the prior around the MAP, then runs many short parallel MCMC or SMC chains. It matches deep-ensemble accuracy at similar cost while producing epistemic uncertainty estimates that improve OOD detection and abstention decisions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SBMC's 'same wall-clock as DE' claim is unsupported: Table 1 counts worker-epochs, not wall-clock, and SBMC adds a serial 160-epoch MAP phase before sampling.","rationale":"The reader's weakest assumption is the anchored-prior heuristic, which is real but secondary: the method's practical utility is demonstrated empirically, and the paper itself frames Appendix C as a sketch. A more load-bearing issue is the cost metric, because the abstract's headline claim is explicitly about wall-clock parity. The paper defines epochs as a proxy for wall-clock, but counts N parallel chains as N× the cost while also adding a serial MAP phase. The table's total-cost column equalizes total worker-epochs, not wall-clock; by the paper's own per-chain time-cost definition, SBMC's critical path is roughly double DE's. This is an internal inconsistency, not a disagreement with consensus. The check is concrete and mechanical: recompute Table 1 as a schedule. No code is provided, but this test can be done from the paper's own entries. The UQ improvements are plausible and supported by multiple tables; the concern is not that the method is useless, but that the advertised cost advantage is not established. The reader noted Table 1 as internally inconsistent in the rationale, so there is partial agreement, but the weakest_assumption field points elsewhere. Since the issue is addressable and the empirical UQ evidence is substantive, the verdict stays CONDITIONAL rather than escalating to REJECT.","tokens_in":45883,"tokens_out":9418,"duration_ms":92132,"concrete_test":"Recompute every entry of Table 1 using wall-clock critical path: start time 0, MAP phase if required, then sampling/ensemble phase with N/P workers. Compare S-SMC∥ and S-HMC∥ against Deep Ens at equal worker counts, and also rerun the UQ metrics with an equal-wall-clock budget (e.g., allow DE 320 epochs per member, or reduce SBMC sampling to ~96 epochs after a 160-epoch MAP). If SBMC's accuracy/UQ advantage persists at equal wall-clock, the claim stands; if it vanishes or requires the 2× serial MAP phase, the central claim should be revised to 'comparable total compute, not wall-clock.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that SBMC matches deep ensembles in wall-clock cost. This is not what Table 1 measures. Section 2 defines time cost as epochs per chain/particle, with N chains run in parallel; but the 'Total Cost' column multiplies by N: Deep Ens P=1 is 176×10=1760, and S-HMC∥ P=1 is 160×10+160=1760. The +160 is a separate, serial MAP-training phase that must finish before sampling begins. On identical parallel hardware, the wall-clock critical path is therefore 160 (MAP) + 160 (chains) = 320 epochs for S-HMC∥ versus 176 epochs for DE (if DE members are also run in parallel), a 1.8× overhead. The text even concedes 'SBMC methods require the MAP estimator, so their total time cost is roughly double'—which contradicts the abstract's 'same wall-clock time' phrasing. The UQ results may survive, but the headline comparison is currently an accounting artifact: worker-epochs are equalized while wall-clock is not.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Scalable Bayesian Monte Carlo (SBMC), a method for Bayesian deep learning that combines a MAP-anchored Gaussian prior with an interpolation parameter s, and parallel SMC or MCMC samplers that target the resulting anchored posterior. The authors claim that, for the same wall-clock time as deep ensembles (DE), SBMC delivers comparable or better accuracy and substantially improved epistemic uncertainty quantification, demonstrated on MNIST7, IMDb, CIFAR-10, and a preliminary GPT-2 experiment. The paper includes extensive experiments with 5 realizations, ablations over s and P, and comparisons with MAP, DE, MC Dropout, Laplace approximation, and SWA, as well as long-run HMC as a gold standard.","tokens_in":46080,"tokens_out":5088,"duration_ms":45464,"significance":"If the headline claim were fully supported, SBMC would be a practically valuable contribution: it offers a sampling-based method with formal consistency for its target, while providing epistemic UQ at a cost comparable to a popular heuristic (DE). The empirical study is unusually thorough for this line of work: standard errors from 5 realizations are reported, many baselines are included, and the sensitivity to the tuning parameters s and P is examined. The authors are also transparent about the approximate nature of the target and explicitly flag the heuristic character of the theoretical sketch. However, the central 'same wall-clock time' claim is not supported by the paper's own cost accounting, which uses worker-epochs rather than wall-clock time. This is a load-bearing issue that must be resolved before the paper's main contribution can be accepted as stated.","major_comments":[{"comment":"The claim that SBMC matches deep ensembles 'for the same wall-clock time' is contradicted by the paper's own accounting. Section 2 defines time cost as 'simulation time per chain/particle' in epochs, a hardware-agnostic proxy, and the Total Cost column in Table 1 counts MAP epochs plus (sampling epochs × N × P). On identical parallel hardware, the wall-clock critical path for S-HMC∥ (P=1, N=10) is 160 epochs for the MAP phase plus 160 epochs for the parallel sampling phase, i.e. 320 epochs, versus 176 epochs for a deep ensemble whose members run in parallel. The text near Table 1 even concedes 'SBMC methods require the MAP estimator, so their total time cost is roughly double,' which directly contradicts the abstract's phrasing. The comparison should be re-framed as equal worker-epochs or total compute, and the abstract and title-level claims should be adjusted accordingly.","section":"Section 2, Table 1, Abstract"},{"comment":"The NLL for the 200,000-epoch HMC gold standard is listed as 1.94±0.004, whereas Table 8 reports 1.942e-1 (0.1942) for the same setting. The decimal error in Table 1 makes the gold-standard look dramatically worse and is inconsistent with the supporting tables. This should be corrected, and the surrounding text should be checked for similar transcription errors.","section":"Table 1, HMC (GS) row"},{"comment":"The theoretical justification is explicitly heuristic: Appendix C states 'let's suppose we wave our hands' in deriving the link to a Laplace approximation, and Section 5 recommends treating convergence heuristically because the target is not the posterior for s<1. The empirical UQ gain is thus an empirical observation for an approximate, data-dependent target, not a consequence of a formal theory. The paper should state this limitation more prominently in the abstract and introduction, so that readers do not infer a principled derivation of the reported UQ improvement.","section":"Appendix C and Section 5"}],"minor_comments":[{"comment":"The symbol π0 is used both for the original prior N(0,V) and for the anchored prior N(α θ_MAP, Σ(s)), which causes confusion. Please use distinct notation, e.g. π_0^orig and π_0^anch.","section":"Section 2, Eq. (2)"},{"comment":"The caption says 'every method runs for 25 epochs except the Gold-Standard,' which is inconsistent with Table 1 where MNIST7 runs are 160–178 epochs. Please clarify the dataset or setting to which the 25-epoch statement applies.","section":"Figure 1 caption"},{"comment":"The phrase 'bold the first 30% data in mean' (e.g. in Tables 4–7) is unclear. Please explain what the bolding indicates and why the 30% cutoff is used.","section":"Tables with bolding"},{"comment":"Some references are incomplete, e.g. 'Qiu et al.' appears without a year or venue. Please normalize all citations.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The wall-clock versus worker-epochs discrepancy is the central issue. If the authors can re-frame the claim honestly and show that the UQ improvement persists under a fair wall-clock comparison (or at equal total compute), the paper would be a solid empirical contribution. The current abstract overstates the result, and the internal contradiction is likely to draw criticism from reviewers and readers. The NLL typo in Table 1 should also be fixed before resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dave,\n\nQuick take: the SBMC paper is a decent empirical contribution that overstates its headline cost claim. The anchored posterior idea—MAP-centered Gaussian prior with scalar s—is a new construction, and the paper distinguishes it cleanly from SWA-anchored ULA (Paulin et al.) and cold posteriors. The parallel SMC/MCMC implementation is sensible, and the experiments are thorough: five seeds, standard errors, ablations over s and P, and OOD detection via a meta-classifier. On IMDb the epistemic UQ gains over deep ensembles are real; on CIFAR-10 they're modest but present.\n\nThe biggest problem is the 'same wall-clock time' claim in the abstract. Table 1 counts worker-epochs, not wall-clock. SBMC adds a serial 160-epoch MAP phase before sampling, so the critical path is around 320 epochs versus 176 for parallel deep ensembles. The text even admits 'total time cost is roughly double.' That doesn't kill the method—it still delivers better UQ than DE at roughly 1.8× wall-clock, which is a reasonable trade—but the abstract as written is misleading.\n\nThe theory is explicitly hand-wavy (Appendix C literally says 'let's suppose we wave our hands'), and the anchored posterior is data-dependent, so the epistemic entropy is not for the true posterior. The authors are transparent about this, and the sampler is consistent for its target. There's also a table inconsistency: HMC (GS) at 200k epochs shows NLL 1.94, worse than at 20k epochs (0.222), which looks like a typo. No code is provided, which hurts reproducibility.\n\nTarget readers are practitioners who want cheap epistemic UQ and don't mind working with an approximate posterior. The method is plausible, the citation pattern is fine, and the limitations are honestly stated. I'd send this to review—it needs a revision, not a rejection.","headline":"Solid empirical contribution with a misleading wall-clock claim: SBMC's cost is roughly double deep ensembles, but the UQ gains on IMDb are real.","tokens_in":46677,"tokens_out":3332,"would_cite":true,"duration_ms":29046,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62F15","65C05","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"At the same wall-clock cost as a deep ensemble, SBMC matches its accuracy and supplies epistemic uncertainty that deep ensembles miss.","keywords":["Bayesian deep learning","uncertainty quantification","sequential Monte Carlo samplers","Hamiltonian Monte Carlo","anchored posterior","deep ensembles","epistemic uncertainty","out-of-distribution detection"],"falsifier":"On a small problem where a long gold-standard HMC run is feasible, compute the rank correlation between SBMC's per-input epistemic entropy at the recommended default $s=0.1$ and the gold-standard posterior's over correct, incorrect, and OOD inputs; a weak or negative correlation on the incorrect/OOD subset would falsify the claim that the anchored posterior recovers useful epistemic uncertainty.","tokens_in":45679,"feed_emoji":"🎲","tokens_out":8025,"duration_ms":69374,"temperature":0.7,"pith_summary":"The paper introduces Scalable Bayesian Monte Carlo (SBMC), a recipe for Bayesian uncertainty in deep networks at the same wall-clock budget as a deep ensemble. Instead of sampling the true posterior, SBMC targets an anchored distribution pulled toward the maximum-a-posteriori weights by a scalar parameter $s$, which makes the target easy to sample when $s$ is small while preserving accuracy. In experiments on MNIST, CIFAR, and IMDb, short parallel MCMC and SMC chains on this anchored target match or beat deep-ensemble accuracy and produce epistemic-entropy scores that separate correct from incorrect and out-of-domain predictions. The paper thereby argues that the gap between principled but slow samplers and fast but heuristic uncertainty estimates can be closed.","feed_headline":"A MAP-anchored sampler matches deep ensembles and flags bad predictions","feed_subtitle":"It runs at the same epoch budget and adds the epistemic uncertainty deep ensembles miss.","key_machinery":"The load-bearing object is the anchored posterior $\\pi_s(\\theta) \\propto L(\\theta)\\,\\mathcal{N}(\\theta;\\theta_{\\mathrm{MAP}}, s v I_d)$, with scalar interpolation parameter $s\\in[0,1]$ (the paper uses $\\alpha(s)=\\mathbf{1}_{s<1/2}$ and $\\Sigma(s)=s v I_d$ in the isotropic case). It turns an intractable posterior into a family of targets interpolating between a point mass at the MAP and the full posterior; small $s$ flattens the likelihood relative to the anchored prior, improves Hessian conditioning, and lets short parallel chains mix. The companion machinery is the parallel BMC layer: $P$ independent executions of SMC$_{\\parallel}$ or MCMC$_{\\parallel}$, each with $N$ particles or chains, combined through a weighted ratio estimator without inter-process communication.","core_discovery":"The paper's central claim is that, for a fixed time budget measured in epochs, targeting the MAP-anchored posterior $\\pi_s(\\theta) \\propto L(\\theta)\\mathcal{N}(\\theta;\\theta_{\\mathrm{MAP}}, s v I_d)$ with parallel SMC or MCMC recovers essentially the accuracy of the point estimator while supplying an epistemic-uncertainty signal that deep ensembles systematically underestimate. In the reported numbers, SBMC at roughly 160 to 200 epochs reaches accuracy comparable to deep ensembles and, when its posterior features are fed to a confidence meta-classifier, achieves higher F1 and AUC-ROC for detecting misclassifications and out-of-distribution inputs; the long-run gold-standard HMC posterior remains the stated ceiling. The interpolation parameter $s$ is the control knob: $s=0$ collapses to the MAP, $s=1$ is the true posterior, and intermediate values trade a small bias for much faster mixing.","pith_inferences":["The paper's own experiments tune $s$ per dataset (0.05 to 0.35), so a fully automatic selection rule for $s$, for example empirical Bayes on the marginal likelihood or a cross-validated grid, is the natural next step; without it, the scalable claim carries an extra tuning cost.","The anchored-posterior trick is a post-hoc add-on: once a MAP or deep ensemble exists, short chains on the last layer or on LoRA adapters could give the same UQ benefit to frozen pretrained models, which is the direction the GPT-2 appendix gestures at.","A minimal deployment test would ask whether a single epistemic-entropy threshold, without the seven-feature meta-classifier, already separates OOD from in-domain inputs; if it does, SBMC's practical overhead drops further."],"forward_implications":["At equal per-chain epoch cost, SBMC matches deep-ensemble accuracy on MNIST7, IMDb, and CIFAR10 while delivering markedly higher epistemic entropy, especially on out-of-distribution inputs.","Epistemic entropy from the SBMC posterior ranks individual predictions: incorrect and OOD examples receive higher scores, so the signal can drive abstention decisions via a confidence meta-classifier.","Because the sampler is consistent for the anchored target, any Bayesian Monte Carlo algorithm, such as SMC$_{\\parallel}$, HMC$_{\\parallel}$, or future stochastic-gradient variants, can be dropped into SBMC without changing the estimator.","Scaling the number of parallel processes $P$ improves uncertainty estimates, with S-SMC$_{\\parallel}$ improving more sharply than S-HMC$_{\\parallel}$ as $P$ grows."],"supporting_citations":[{"why":"Defines the deep-ensemble baseline whose accuracy and cost SBMC is benchmarked against.","marker":"[Lakshminarayanan et al., 2017]"},{"why":"Supplies MC Dropout, the fast UQ baseline that also passes the epistemic-entropy tolerance on MNIST7.","marker":"[Gal and Ghahramani, 2016]"},{"why":"Provides the SMC sampler framework that S-SMC$_{\\parallel}$ instantiates with adaptive tempering.","marker":"[Del Moral et al., 2006]"},{"why":"Supplies the HMC kernel used in S-HMC$_{\\parallel}$ and in the gold-standard long runs.","marker":"[Neal et al., 2011]"},{"why":"Establishes the near-linear parallel speedup of SMC$_{\\parallel}$ and MCMC$_{\\parallel}$ that the parallel layer relies on.","marker":"[Liang et al., 2025]"},{"why":"Provides the Laplace-approximation competitor and the Kronecker-factored Hessian used in the benchmarks.","marker":"[Daxberger et al., 2021]"},{"why":"Gives the SWA averaging baseline compared in the study.","marker":"[Izmailov et al., 2018]"},{"why":"Gives the SWAG uncertainty baseline compared in the study.","marker":"[Maddox et al., 2019]"}],"fun_headline_variants":["Same epoch budget, sharper uncertainty than deep ensembles","MAP-anchored sampler matches deep nets, flags bad predictions","Scalable Bayesian MC: uncertainty deep ensembles miss","Fast Bayesian deep learning: uncertainty without ensemble cost"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a single scalar anchoring strength, chosen per dataset and justified only by a heuristic sketch, makes the approximate posterior's epistemic-entropy ranking of mistakes and out-of-distribution inputs trustworthy.","fun_headline_variants_meta":{"raw":{"variants":["Same epoch budget, sharper uncertainty than deep ensembles","MAP-anchored sampler matches deep nets, flags bad predictions","Scalable Bayesian MC: uncertainty deep ensembles miss","Fast Bayesian deep learning: uncertainty without ensemble cost"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000196,"raw_usage":{"total_tokens":1340,"prompt_tokens":906,"completion_tokens":434,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":522,"completion_tokens_details":{"reasoning_tokens":372}},"tokens_in":522,"tokens_out":434,"duration_ms":4678,"temperature":1.0,"reasoning_tokens":372,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:13:09.289665+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a small problem where a long gold-standard HMC run is feasible, compute the rank correlation between SBMC's per-input epistemic entropy at the recommended default $s=0.1$ and the gold-standard posterior's over correct, incorrect, and OOD inputs; a weak or negative correlation on the incorrect/OOD subset would falsify the claim that the anchored posterior recovers useful epistemic uncertainty.","supporting_citations":[{"cited_title":"Averaging weights leads to wider optima and better generalization","cited_arxiv_id":null,"evidence_quote":"Gives the SWA averaging baseline compared in the study."}],"review_version":1}