{"id":"6ca9b725-5bca-45d8-aac3-dba0bfa52a7d","arxiv_id":"2507.14652","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A hybrid VI-HMC method runs expensive Hamiltonian Monte Carlo only on the neural network parameters most responsible for predictive uncertainty, cutting sampling cost while approximating full HMC posteriors.","lead":"This paper combines two standard Bayesian methods, quick approximate inference and slow but accurate Hamiltonian Monte Carlo, to sample neural network weights more cheaply. It identifies which network parameters matter for uncertainty, runs the expensive accurate sampler only on those, and shows the shortcut on small nets and physics-based operator networks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Freezing low-sensitivity parameters at VI means can bias predictions and understate uncertainty; the paper provides no calibration evidence against full HMC.","rationale":"The reader's weakest assumption correctly identifies the freezing of insensitive parameters at VI means as the load-bearing approximation. My reading agrees: the method's accuracy claims require that this approximation preserve the posterior predictive distribution, but the paper provides only qualitative comparisons and explicitly demonstrates in Appendix E that uncertainty shrinks as τ decreases. In addition, the sensitivity ranking is based on a first-order expansion at the VI mean and on VI variances, which are unreliable when VI is inaccurate—the very scenario the paper motivates. No calibration or coverage metric is reported, so the severity of the bias is unknown. These points do not invalidate the method outright; they justify conditional acceptance pending a quantitative test against full HMC. Since the reader already reaches the same conclusion, the verdict remains CONDITIONAL and no adjustment is needed.","tokens_in":512,"tokens_out":2956,"duration_ms":89417,"concrete_test":"Run the full HMC and VI-HMC chains already used for Case II (141 parameters, 10 chains, 10^4 samples each) and compute the empirical coverage of the central 90% posterior predictive interval on the 300-point validation set, as well as the mean absolute error of the posterior mean predictions. If the VI-HMC coverage is markedly below 90% or the posterior mean differs substantially from full HMC in the interpolation/extrapolation regions, the freezing approximation is not valid and the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that HMC can be run only on the most sensitivity-important parameters while freezing the rest at their VI means relies on the assumption that the frozen parameters do not materially affect the posterior predictive distribution. This assumption is not adequately supported. The sensitivity measure in Eq. (17) is local: it uses first-order derivatives evaluated at the VI mean and variances from the VI posterior. VI is known to be inaccurate for BNN posteriors, so parameters with small VI variance or small gradient at the VI mean may still have substantial posterior uncertainty or nonlinear effects. Moreover, even if a parameter contributes little to the output variance individually, it may be strongly correlated with sensitive parameters; fixing it at its VI mean can collapse joint uncertainty and alter the predictive distribution. The paper itself shows in Appendix E that as the threshold τ is reduced, the reported uncertainty shrinks (Figure E.18), confirming that the freezing step discards genuine posterior uncertainty. Additionally, freezing parameters at VI means can bias the mean prediction, as acknowledged in Case II: the VI means may not coincide with the true posterior means, so the hybrid prediction is a mixture of VI and HMC predictions. In Case I, for example, the frozen parameter ω2 has VI mean 2.94 while the true value is -3 and full HMC recovers -3; fixing it at the VI mean introduces a systematic error. Because no coverage or calibration metrics are reported, the magnitude of this bias and undercoverage is unknown. The method's practical usefulness hinges on whether the freezing approximation preserves calibrated predictive uncertainties; the current evidence does not establish this.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a hybrid VI-HMC method for Bayesian neural networks and neural operators. The method first trains a mean-field variational posterior on the full network, then ranks parameters by a local first-order sensitivity score S_i^2 = sigma_i^2 times the average squared gradient at the VI mean. Parameters whose cumulative sensitivity reaches a threshold tau are sampled with HMC, while the remaining parameters are fixed at their VI means. The authors report experiments on two small BNN benchmarks, a Burgers-equation DeepONet, and a hypersonic-flow cone DeepONet, and claim that the reduced-space HMC closely reproduces full-HMC posterior predictive uncertainties at substantially lower cost.","tokens_in":20230,"tokens_out":8323,"duration_ms":101880,"significance":"If the central approximation is valid, the contribution is significant: it would make full-batch HMC feasible for networks with tens to hundreds of thousands of parameters, and the application to neural operators for hypersonic flow is timely and practically motivated. The paper is commendably explicit about its approximations, provides code, and includes a limitations section. However, the current validation is mostly visual and qualitative, the core freezing assumption is not supported by a calibration test or error bound, and the operator-network examples lack any full-HMC benchmark. The idea is promising, but the evidence as presented is not yet sufficient for the strength of the claims made.","major_comments":[{"comment":"The definition of the reduced parameter count N_hat is inconsistent with the stated goal. The text says tau is the proportion of variance to be captured and the experiments describe selecting the first parameters whose cumulative sensitivity reaches tau. The displayed equation uses 'max' and '<= tau', which for a nondecreasing cumulative ratio selects the largest N_T with cumulative ratio no greater than tau and would typically pick a set capturing slightly less than tau, not the four parameters claimed for case I. The inequality direction and the optimization should be reversed: one needs the smallest N_T such that the cumulative ratio is at least tau.","section":"Section 3.1, Eq. (18)"},{"comment":"The central approximation—fixing low-sensitivity parameters at their VI means—is not supported by a quantitative error bound or calibration test. The sensitivity score in Eq. (17) is local and first-order, evaluated at the VI mean, so parameters with small VI variance, small gradient, or strong correlation with sensitive parameters can still materially affect the posterior predictive distribution when the full posterior is considered. Appendix E explicitly shows that the reported uncertainty shrinks as tau decreases, confirming that the freezing step discards genuine posterior uncertainty, and no criterion for choosing tau or measuring the induced bias is provided.","section":"Section 3.2, Eq. (20), Algorithm 2, Appendix E"},{"comment":"The main claim that VI-HMC reproduces HMC uncertainty is not quantitatively validated for the operator problems. For Burgers, full HMC is not run and the only reported accuracy numbers are mean relative L2 errors: VI 3.8% versus VI-HMC 7.52%, so the VI-HMC mean prediction is worse than VI. No coverage, interval score, or other calibration metric is reported. For the cone problem there is no full-HMC baseline at all, so statements such as 'VI overestimates uncertainties when compared to the hybrid VI-HMC approach' compare two approximations rather than validating against the target posterior.","section":"Sections 4.2.1 and 4.2.2, Table 2"},{"comment":"The statement that 'the parameters sampled through VI-HMC closely match those obtained using the full HMC approach' is not supported by Table 1. The frozen parameters omega_2 and phi_1 are fixed at VI means 2.94 and -0.072, whereas full HMC gives -3.00 and 6.29, and VI-HMC samples around a different mode (-4.00) for omega_1 than full HMC (4.00). The predictive agreement in Figure 3 appears to rely on the functional symmetry of the sinusoidal representation, not on the method recovering the full posterior. The text should state clearly that VI-HMC targets a conditional posterior anchored at the VI mode and justify under what conditions predictive quantities, rather than parameter values, are the objects of inference.","section":"Section 4.1.1, Table 1"}],"minor_comments":[{"comment":"The 'Required step size' column mixes values for HMC and VI-HMC with footnotes in a way that is hard to read; please reformat and also report effective sample sizes in addition to acceptance rates so that the computational gains are measured per effective sample.","section":"Section 4.2.3, Table 2"},{"comment":"The statement in Appendix C that 'Both the prior and likelihood are assumed to be the same for VI, HMC, and VI-HMC approaches' is difficult to reconcile with the discussion in Section 5, which describes using a small likelihood standard deviation for VI and a large one for HMC in Case II; please clarify whether the final reported experiments use identical likelihoods for all three methods.","section":"Section 5 and Appendix C"},{"comment":"The caption lists 'samples, mean prediction, true solution, initial condition' without explaining which quantity each curve or band represents or how the 'best' and 'worst' cases are defined; please clarify.","section":"Figure 10 caption"},{"comment":"The sentence 'Cobb and Jalaian [25] proposed a new integration scheme by splitting the data into batches that preserve the Hamiltonian' appears to misstate the method; the cited work uses a symmetric splitting of the Hamiltonian dynamics, not a splitting of data batches. Please verify and correct the wording.","section":"Introduction, Section 1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a reasonable candidate for major revision if the authors can add quantitative calibration or coverage experiments against full HMC on problems where full HMC is feasible, clarify the threshold selection issue, and correct Eq. (18). I do not see a fundamental circularity or a novelty problem, but the current evidence does not justify the strength of the central accuracy claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you work on scalable Bayesian inference for neural operators. The idea is simple: train VI on the full network, rank parameters by their contribution to predictive variance via a first-order sensitivity analysis, freeze the low-ranking ones at VI means, and run HMC only on the small active set. They demonstrate this on BNNs and on DeepONets for Burgers and hypersonic cone flow, with code and a cost comparison showing roughly 2x speedups and much better acceptance rates than full HMC. The exposition is clear, and the sensitivity ranking is verified against analytic derivatives in the small Case I. That is real work and it ships code, which I credit.\n\nThe soft spot is the freezing step. The paper shows visual agreement with full HMC only in the two small BNN examples. In the larger DeepONet problems, full HMC is not run at all, so there is no calibration or coverage check. Appendix E shows that lowering the threshold tau shrinks reported uncertainty, which is expected if you are discarding variance by construction, but the paper does not quantify how much of that discarded variance matters for the predictive distribution. The stress-test note about Case I is fair: the frozen parameter omega2 has VI mean 2.94 while the true value is -3, and full HMC recovers -3. The predictive curve may still look acceptable because VI means as a set reparameterize the function, but that is exactly why freezing can bias predictions. Also, in the Burgers example the mean relative L2 error for VI-HMC is 7.52% versus 3.8% for VI, which is a red flag for the claim that the hybrid improves mean predictions. Section 5 does acknowledge some limitations, but the central assumption deserves more scrutiny than it gets.\n\nThat said, the paper does not oversell the method; it frames the sparsity as a hypothesis. The contribution is a useful heuristic with a clear computational benefit. Whether it produces calibrated uncertainties is an empirical question the paper does not yet answer. This is a solid candidate for peer review, not a desk reject. I would ask the authors to add a medium-scale comparison with full HMC (coverage of predictive intervals, not just visual agreement), to report calibration metrics on all examples, and to state more precisely under what conditions freezing low-sensitivity parameters is valid. As it stands, I would not cite it as a validated method, but I would send it to review and would be interested to see the revised version.","headline":"A practical VI-HMC hybrid with a clear computational benefit, but the core freezing approximation needs calibration evidence before I would trust its uncertainty estimates.","tokens_in":20797,"tokens_out":2304,"would_cite":false,"duration_ms":29854,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62F15","65C05","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"A hybrid VI-HMC method makes full-batch HMC tractable for neural networks by sampling only the parameters that drive predictive uncertainty.","keywords":["Hamiltonian Monte Carlo","variational inference","Bayesian neural networks","neural operators","uncertainty quantification","sensitivity analysis","DeepONet","dimensionality reduction"],"falsifier":"Run full HMC and VI-HMC on a linear-Gaussian regression network where the posterior is analytically known and compute the exact predictive variance; if fixing the low-sensitivity parameters at their VI means changes the predictive variance by more than the tolerance implied by the chosen threshold $\\tau$, the central assumption behind the method is false.","tokens_in":19761,"feed_emoji":"🧠","tokens_out":5082,"duration_ms":58513,"temperature":0.7,"pith_summary":"The paper tries to establish that Hamiltonian Monte Carlo, normally too expensive for large Bayesian neural networks, can be made practical by first running cheap variational inference and then measuring how much each network parameter contributes to predictive variance. It claims that this contribution is sparse: a small fraction of parameters carries most of the output uncertainty, so HMC only needs to sample those, while the rest are frozen at their variational means. If true, the method would give HMC-grade posterior samples and uncertainty estimates at a fraction of the cost, on networks with tens to hundreds of thousands of parameters. The paper demonstrates the idea on small Bayesian networks and on DeepONet surrogates for Burgers' equation and for wall-pressure in hypersonic flow over a cone.","feed_headline":"Sampling only key parameters makes HMC Bayesian inference 19x cheaper","feed_subtitle":"First VI ranks which weights drive predictive variance, then HMC samples only those few.","key_machinery":"The carrying object is the per-parameter sensitivity score $S_i^2$, which multiplies the variational posterior variance $\\sigma_i^2$ by the average squared gradient of the network output with respect to parameter $\\theta_i$, evaluated at the VI mean. A threshold $\\tau$ on the cumulative fraction of explained variance selects the active subset $\\Theta_s$; the conditional posterior $P(\\Theta_s \\mid D, \\Theta_{\\sim s} = \\mu_{\\sim s})$ is then sampled by HMC. The computational argument is that HMC cost scales roughly as $O(D^{5/4})$ in parameter dimension, so removing inert coordinates reduces leapfrog error accumulation, raises acceptance rates, and permits larger step sizes.","core_discovery":"The central claim is that posterior predictive uncertainty in a Bayesian neural network or neural operator can be accurately recovered by sampling, with HMC, only the parameter subset with the largest sensitivity scores $S_i^2$, defined from a first-order Taylor expansion of the network output around the variational mean as $S_i^2 = \\frac{\\sigma_i^2}{N_d}\\sum_{j=1}^{N_d}\\left(\\frac{\\partial F_\\mu(x_j)}{\\partial \\theta_i}\\right)^2$. The remaining parameters are conditioned on their variational means, so HMC draws from $P(\\Theta_s \\mid D, \\Theta_{\\sim s} = \\mu_{\\sim s})$ instead of the full posterior. The paper reports that this reduced-space HMC matches full HMC uncertainty estimates on small networks where both can be run, captures parameter correlations that mean-field VI misses, and makes inference feasible for DeepONets with 16,321 to 172,401 parameters.","pith_inferences":["A natural stress test the paper does not run is to compare VI-HMC predictive intervals against full HMC on a moderately sized network across many datasets and thresholds, checking coverage rather than visual agreement.","Because Appendix E shows reported uncertainty shrinks as the threshold $\\tau$ decreases, a practitioner should treat the threshold as a tunable calibration knob and validate interval coverage on held-out data.","The sensitivity ranking is computed on training inputs averaged over the data set; out-of-distribution or extrapolation queries could reorder the ranking, so the active subset may not transfer to new input regions.","The same sensitivity screen could be replaced by second-order variance propagation or by gradients evaluated at several posterior draws, which would test whether the local first-order ranking at the VI mean is stable."],"forward_implications":["Full-batch HMC becomes applicable to networks with tens to hundreds of thousands of parameters, not just the small networks where it was previously feasible.","With a fixed step size, reduced-space HMC achieves substantially higher acceptance rates than full HMC, and with acceptance fixed at 80 percent it tolerates step sizes up to several orders of magnitude larger.","VI-HMC uncertainty estimates track HMC more closely than VI alone does, including the correlation structure between paired parameters that mean-field VI discards.","The same recipe extends from functions to operators, producing uncertainty estimates for DeepONet surrogates of PDE solutions, including Burgers' equation and hypersonic cone flow.","The computational advantage should grow with network size because the cost saving from dimension reduction scales with the exponent in the $O(D^{5/4})$ HMC cost estimate."],"supporting_citations":[{"why":"Establishes HMC as an accurate sampler for Bayesian neural networks and supplies the dimension-dependent cost scaling that motivates the reduction.","marker":"[7]"},{"why":"Provides the symmetric splitting HMC formulation used in the implementation to avoid gradient pathologies in full-batch sampling.","marker":"[25]"},{"why":"Uses HMC to characterize what Bayesian neural network posteriors really look like, motivating the need for accurate posterior sampling beyond VI.","marker":"[24]"},{"why":"Proposes low-dimensional subspace inference for Bayesian deep learning, a related dimensionality-reduction strategy this work builds on and contrasts with.","marker":"[47]"},{"why":"Selects subnetworks to preserve predictive uncertainty, the closest selection-based alternative that the proposed sensitivity ranking is compared against.","marker":"[48]"},{"why":"Supplies the step-size adaptation algorithm used for the fixed-acceptance-rate cost comparison between HMC and VI-HMC.","marker":"[58]"},{"why":"Defines the DeepONet architecture used for the neural operator experiments.","marker":"[61]"},{"why":"Provides the hypersonic cone wall-pressure data set used in the final operator learning demonstration.","marker":"[71]"}],"fun_headline_variants":["Sensitivity-ranked HMC cuts Bayesian inference cost 19x","Hybrid VI+HMC speeds Bayesian nets by sampling only key weights","Forget full HMC: sample only uncertainty-driving parameters","Cheaper Bayesian nets: HMC only on high-sensitivity weights"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that parameters judged insensitive by a local first-order sensitivity analysis at the variational mean can be frozen at their VI means without materially changing the posterior predictive distribution, an assumption the paper does not bound quantitatively.","fun_headline_variants_meta":{"raw":{"variants":["Sensitivity-ranked HMC cuts Bayesian inference cost 19x","Hybrid VI+HMC speeds Bayesian nets by sampling only key weights","Forget full HMC: sample only uncertainty-driving parameters","Cheaper Bayesian nets: HMC only on high-sensitivity weights"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000304,"raw_usage":{"total_tokens":1783,"prompt_tokens":1017,"completion_tokens":766,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":633,"completion_tokens_details":{"reasoning_tokens":694}},"tokens_in":633,"tokens_out":766,"duration_ms":8674,"temperature":1.0,"reasoning_tokens":694,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:51:14.659465+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run full HMC and VI-HMC on a linear-Gaussian regression network where the posterior is analytically known and compute the exact predictive variance; if fixing the low-sensitivity parameters at their VI means changes the predictive variance by more than the tolerance implied by the chosen threshold $\\tau$, the central assumption behind the method is false.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes HMC as an accurate sampler for Bayesian neural networks and supplies the dimension-dependent cost scaling that motivates the reduction."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the symmetric splitting HMC formulation used in the implementation to avoid gradient pathologies in full-batch sampling."},{"cited_title":"Izmailov, S","cited_arxiv_id":null,"evidence_quote":"Uses HMC to characterize what Bayesian neural network posteriors really look like, motivating the need for accurate posterior sampling beyond VI."},{"cited_title":"Izmailov, W","cited_arxiv_id":null,"evidence_quote":"Proposes low-dimensional subspace inference for Bayesian deep learning, a related dimensionality-reduction strategy this work builds on and contrasts with."},{"cited_title":"Daxberger, E","cited_arxiv_id":null,"evidence_quote":"Selects subnetworks to preserve predictive uncertainty, the closest selection-based alternative that the proposed sensitivity ranking is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the step-size adaptation algorithm used for the fixed-acceptance-rate cost comparison between HMC and VI-HMC."},{"cited_title":"Morra, C","cited_arxiv_id":null,"evidence_quote":"Provides the hypersonic cone wall-pressure data set used in the final operator learning demonstration."}],"review_version":1}