{"id":"2ab3a4d6-d02c-4dab-be9a-95c6a51848c8","arxiv_id":"2501.04259","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"DF-GMVI approximates Bayesian posteriors with Gaussian mixtures using derivative-free quadrature that preserves covariance positivity and affine invariance.","lead":"Bayesian inverse problems often need many expensive forward-model evaluations and lack gradients. This paper introduces DF-GMVI, a derivative-free variational method that combines Fisher-Rao natural gradients with specialized quadrature, and shows it handles multimodal posteriors in up to 100 dimensions and a Navier-Stokes recovery task.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The positivity guarantee in Prop. 5.1 is bought by dropping 2Diag(A^T c) and column-interaction terms from the Hessian quadrature (4.5); this can bias the stationary covariance, so stability alone does not establish posterior accuracy.","rationale":"The paper's own strongest claim is conditional: positivity is guaranteed, but the abstract packages this as stable and accurate posterior approximation. The proof of Proposition 5.1 is valid given the quadrature; my concern is that the construction deliberately discards the indefinite parts of the exact Hessian expectation to make that proof work. This is not merely a missing convergence theorem; it is a concrete bias mechanism. The derivation after (4.5) states the exact quadratic-model Hessian has D+2Diag(A^T c)+B^T B, and (4.5) retains only the positive-semidefinite part. In a scalar quadratic residual, the omitted term 2ac enters the stationary covariance equation directly; when the residual at the mode is large in magnitude, the bias is first-order in that residual. The numerical experiments (one-dimensional bimodal, Rosenbrock, Navier-Stokes) happen to have small residuals at the converged modes or symmetric A-column structure, so they may not expose this; a targeted experiment with a misspecified forward model would. The reader identified the same assumption, accuracy of the surrogate Hessian and block-diagonal Fisher information, as weakest; I agree and sharpen it by naming the dropped term. I do not see a reason to change the verdict: the paper should be published only with an explicit statement and numerical check of this bias, plus the reader's requested error bars and full-dimensional evaluation.","tokens_in":28889,"tokens_out":9578,"duration_ms":94710,"concrete_test":"Run DF-GMVI on the scalar problem F(θ)=aθ²+bθ+c with a Gaussian prior and with ac chosen so that |2ac| is comparable to b²+6a² (e.g., a=1, c=-4), and compare the stationary mean and covariance against the exact natural-gradient flow (2.3) evaluated with the true expected Hessian. If the stationary covariance differs by more than 10%, or the total-variation error to the reference posterior is visibly worse than a version that includes 2Diag(A^T c) (using a Levenberg-Marquardt shift only when needed for positivity), then the truncation in (4.5) is load-bearing. As a second check, repeat the one-dimensional bimodal Case D from Section 6.1 with an artificially shifted residual to increase A^T c.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that DF-GMVI is a stable and accurate derivative-free approximator of the posterior. Proposition 5.1 proves only the stability half: for Δt∈(0,1), C_k stays positive definite. This is achieved by design in Definition 4.1: in the quadratic surrogate F(θ)=A(θ⊙θ)+Bθ+c, the exact expected Hessian contains the term 2Diag(A^T c) plus column-interaction terms 2Σ_{l≠j} a_j^T a_l; equation (4.5) replaces these with 6Diag(A^T A)+B^T B, projecting onto a positive-semidefinite surrogate. The discarded terms are not small in general: for a scalar model F(θ)=aθ²+bθ+c, (4.5) gives b²+6a², while the exact expected Hessian is b²+6a²+2ac; when the residual c at the mode is large and negative, the surrogate overestimates curvature, shrinking the stationary C_k and making the posterior approximation overconfident. A similar bias occurs when the columns of A are correlated. Because the fixed point of (4.9) is defined by these quadrature-approximated gradients and Hessians, a biased Hessian changes the fixed point itself; Section 7 explicitly defers convergence analysis. Thus the abstract's claim that DF-GMVI captures complex posteriors rests on the unverified accuracy of the PSD-projected Hessian, not on Proposition 5.1. This concern compounds the inherited block-diagonal Fisher information approximation (C.16).","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper develops Derivative Free Gaussian Mixture Variational Inference (DF-GMVI), a variational method for Bayesian inverse problems where the forward model is expensive and gradients are unavailable. The method discretizes a Fisher-Rao natural gradient flow over Gaussian mixture variational families, using two derivative-free quadrature rules: one for the least-squares potential ΦR(θ)=1/2 F(θ)^T F(θ) (Definition 4.1) and one for the Gaussian-mixture entropy terms (Definition 4.3). The main theoretical claims are covariance positivity for time steps Δt∈(0,1) (Proposition 5.1) and affine invariance, restricted to lower-triangular linear transformations when the Cholesky square root is used (Proposition 5.2). Numerical experiments cover 1D and 2D targets, 100-dimensional versions of those targets, comparisons with other variational and MCMC methods, and a 128-parameter Navier-Stokes initial-condition recovery problem with a bimodal posterior.","tokens_in":29243,"tokens_out":6782,"duration_ms":71878,"significance":"If the accuracy of the quadrature approximation can be established, DF-GMVI would be a practically valuable contribution: it combines derivative-free updates, a deliberate treatment of entropy and cross-entropy consistency, a simple stability guarantee, and a large-scale PDE application. The paper has clear strengths: Proposition 5.1 is elementary and machine-checkable; the affine-invariance analysis is explicit; the numerical study is broad and includes comparisons with BBVI, natural-gradient and Wasserstein-gradient methods, and affine-invariant MCMC; and the code is publicly available. The principal gap is that the stability guarantee (Proposition 5.1) is not accompanied by an accuracy statement for the surrogate Hessian, so the evidence for the central claim of accurate posterior approximation is currently empirical and partly heuristic.","major_comments":[{"comment":"The PSD projection in the Hessian quadrature is load-bearing for posterior accuracy, but no accuracy estimate is provided. The exact expected Hessian of the quadratic surrogate contains the terms 2Diag(A^T c) and the off-diagonal column interactions 2Σ_{l≠j} a_j^T a_l; Eq. (4.5) replaces these with 6Diag(A^T A)+B^T B. These dropped terms are not negligible in general: for a scalar quadratic F(θ)=aθ²+bθ+c, the omitted term is 2ac, which can be large and negative when the residual at the mode is large, biasing the stationary covariance toward overconfidence. Because the fixed point of (4.9) is defined through the quadrature-approximated gradients and Hessians, a biased Hessian changes the fixed point itself, not merely the convergence path. Proposition 5.1 guarantees only positivity, and Section 7 explicitly defers convergence and accuracy analysis for general posteriors. The abstract's accuracy claim therefore rests on the heuristic PSD projection in Definition 4.1, compounded by the unquantified block-diagonal Fisher information approximation (C.16) introduced in Section 2.1. Please add an error bound for the surrogate Hessian under stated smoothness assumptions, or an explicit quantitative check that the discarded curvature terms are controlled on the problem class considered, or soften the accuracy claim to match the proven stability guarantee.","section":"§4, Definition 4.1 and Eq. (4.5)"},{"comment":"The claim that DF-GMVI remains accurate in 100 dimensions is supported only by two-dimensional marginal densities of the first two coordinates. The remaining 98 coordinates θc are coupled to θ through the matrix K and contribute to the posterior, but no reference comparison is reported for them. Total variation on the 2D marginal cannot detect errors in the other 98 dimensions, so the statements that performance is 'robust across different dimensionalities' and that DF-GMVI avoids the deterioration seen for BBVI/MCMC in 100D are not fully supported. A comparison in at least one additional coordinate direction, a one-dimensional slice or marginal of θc, or a summary statistic such as the marginal variance or RMSE in θc would make the high-dimensional claim credible.","section":"§6.2, 100-dimensional tests (text near Figure 4)"}],"minor_comments":[{"comment":"The abstract states that DF-GMVI 'guarantees covariance positivity and affine invariance' without qualification, while Proposition 5.2 establishes affine invariance only for invertible lower-triangular transformations when the Cholesky square root is used. The abstract should be qualified to match this restriction.","section":"Abstract and Proposition 5.2"},{"comment":"The text says the positive definite part of the expectation is retained, but 6Diag(A^T A)+B^T B is only positive semidefinite in general; 'positive semidefinite part' would be more accurate.","section":"§4, after Eq. (4.5)"},{"comment":"The caption of Figure 2 lists the bottom-row noise variance as 1.5², while the text defines Case D as η∼N(0, 2.0²). The caption should be corrected.","section":"§6.1 and Figure 2"},{"comment":"The contribution statement 'requires no hyperparameter tuning' is too strong given that α, Δt, K, and w_min are user-set parameters. Appendix B studies sensitivity, but the paper does not show that the method is insensitive for all problem classes; please soften the wording.","section":"§1.2 and §6"},{"comment":"The text says Figure 5 shows results over 500 iterations, while Section 6 states that all tests run for 200 iterations; this inconsistency in reported iteration counts should be reconciled so the comparison is unambiguous.","section":"§6.2, comparison setup"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about the PSD-projected Hessian lands: Definition 4.1 is where the central accuracy claim is made, and the paper proves stability rather than accuracy. The manuscript is not fatally flawed—the numerical study is strong and the stability proof is sound—but the load-bearing accuracy question needs to be addressed either by analysis or by carefully restricting the claims. I would not require a full convergence proof, but the current abstract overstates what the theory and experiments establish."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"DF-GMVI is a real algorithmic addition to derivative-free variational inference: it combines Fisher-Rao natural gradient flow on Gaussian mixtures with new quadrature rules for the potential and its gradient/Hessian, and adds a positivity guarantee for the covariance updates. The numerical study is broad (1D through 100D, plus a Navier-Stokes inverse problem) and the code is available.\n\nThe genuinely new pieces are the Hessian quadrature (4.5) with the PSD correction, the matching quadrature for the log-mixture Hessian (4.8), and Proposition 5.1, which gives a clean condition (Δt<1) under which C_k stays positive definite. The consistency guideline—treating entropy and cross-entropy integrals with the same quadrature—is sensible and appears to fix the overlap-region bias that GMKI suffered from. Experiments support the claim that DF-GMVI handles multimodal, curved, and infinite-mode targets better than BBVI and affine-invariant MCMC on the tested problems.\n\nThe main soft spot is the gap between stability and accuracy. Proposition 5.1 proves positivity because the quadrature is deliberately projected onto a positive-semidefinite surrogate: it drops the 2Diag(A^T c) term and the off-diagonal column interactions from the exact expected Hessian of the quadratic model. Those terms can be large and have either sign, so the surrogate Hessian can change the stationary covariance. The paper gives no accuracy bound for this approximation and explicitly defers convergence analysis. So the abstract's wording—'guarantees covariance positivity and affine invariance, offering a stable and efficient framework'—is accurate as far as it goes, but it should not be read as a guarantee of posterior accuracy. The inherited block-diagonal Fisher information approximation (C.16) compounds this concern.\n\nTwo lesser issues. The 'no hyperparameter tuning' claim overreaches; α, Δt, K, and the weight floor are hyperparameters, even if the sensitivity study shows robustness. The numerics are single-seed runs without error bars, and the 100D evaluation is only on 2D marginals, so the conjecture about dimensionality is not fully supported.\n\nOverall: a solid, useful paper that deserves a serious referee. I would accept conditionally, asking for a bound or quantification of the Hessian approximation error and repeated-seed error bars. I would cite it.","headline":"DF-GMVI is a genuine algorithmic contribution—stable derivative-free mixture VI—but the stability guarantee does not establish posterior accuracy, and the Hessian quadrature's PSD projection can bias the fixed point.","tokens_in":29789,"tokens_out":3983,"would_cite":true,"duration_ms":36531,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62F15","65M32","90C56"],"pacs":[],"model":"deepseek-v4-flash","headline":"DF-GMVI makes derivative-free Bayesian inference stable by keeping Gaussian-mixture covariances positive and captures multimodal, curved posteriors.","keywords":["Bayesian inverse problems","variational inference","Gaussian mixture","derivative-free methods","Fisher-Rao gradient flow","natural gradient","covariance positivity","multimodal posteriors"],"falsifier":"Set a target posterior with known non-Gaussian form in which the dropped cross term $2\\operatorname{Diag}(A^T c)$ in the Hessian expectation is large, for example by giving $F$ a strong quadratic interaction along a single coordinate. If DF-GMVI's converged density differs from the best $K$-component Gaussian mixture fit by a total-variation distance much larger than the same distance for a gradient-based natural-gradient flow using the exact Hessian, then the truncation in Definition 4.1 is the limiting error; repeating over targets with increasing dropped-term magnitude would map the accuracy boundary of the method.","tokens_in":28668,"feed_emoji":"🎲","tokens_out":9703,"duration_ms":86184,"temperature":0.7,"pith_summary":"This paper introduces DF-GMVI, a derivative-free variational inference algorithm that approximates Bayesian posteriors with Gaussian mixtures when the forward model is expensive and gradients are unavailable. The method discretizes the Fisher-Rao natural gradient flow using two specialized quadrature rules: one evaluates the least-squares potential and its gradient and Hessian through finite differences at $\\sigma$ points, and one treats the mixture entropy terms consistently. The paper's central claim is that the resulting updates keep every mixture covariance positive definite whenever the time step satisfies $0<\\Delta t<1$, and that the scheme is affine invariant under triangular coordinate transformations. Because of this stability, the algorithm can use large time steps without covariance breakdown and, in the reported experiments, captures multiple, overlapping, infinitely many, and curved modes in up to 100 dimensions, including a Navier-Stokes initial-condition recovery. The practical motivation is that many large-scale inverse problems have black-box or nondifferentiable forward models, so a stable derivative-free posterior approximation matters.","feed_headline":"Derivative-free sampler keeps covariances positive, finds curved modes","feed_subtitle":"New DF-GMVI method needs no gradients or tuning to resolve overlapping, curved, high-dimensional Bayesian posteriors.","key_machinery":"The machinery is the Fisher-Rao (natural) gradient flow restricted to Gaussian mixture variational families, together with a pair of derivative-free quadrature rules. Definition 4.1 generates $2N_\\theta+1$ $\\sigma$ points per Gaussian component and approximates the expectation of $\\Phi_R$ and its gradient by mean-point and finite-difference formulas; the Hessian expectation is kept in the positive semidefinite form $\\sqrt{C}^{-T}(6\\operatorname{Diag}(A^T A)+B^T B)\\sqrt{C}^{-1}$. Definition 4.3 approximates the entropy terms consistently at the same mean points and replaces the negative-definite part of the log-mixture Hessian by $-C_k^{-1}$, which is exact in weighted average. The load-bearing identity is the covariance update $C_k^{-1}(t+\\Delta t)=(1-\\Delta t)C_k^{-1}(t)+\\Delta t$ times a positive semidefinite matrix, which is what yields Proposition 5.1 and enables $O(1)$ time steps.","core_discovery":"The central discovery is that the instability of Gaussian-mixture variational inference, where covariance matrices lose positive definiteness under large time steps, can be removed by choosing consistent and curvature-aware quadrature rules. In Definition 4.1 the paper approximates expectations of $\\Phi_R(\\theta)=\\tfrac12 F(\\theta)^T F(\\theta)$ at $2N_\\theta+1$ $\\sigma$ points per Gaussian component, using finite differences to form matrices $B$ and $A$. The Hessian expectation is approximated as $\\sqrt{C}^{-T}(6\\operatorname{Diag}(A^T A)+B^T B)\\sqrt{C}^{-1}$, keeping only the positive semidefinite part of the exact Gaussian-Newton-type expression. Combining this with a Hessian correction for the log-mixture term in Definition 4.3 makes the covariance update $C_k^{-1}(t+\\Delta t)=(1-\\Delta t)C_k^{-1}(t)+\\Delta t\\cdot(\\text{positive semidefinite terms})$, so Proposition 5.1 gives $C_k$ positive definite for every $0<\\Delta t<1$. Proposition 5.2 establishes affine invariance of the continuous natural gradient flow and a restricted (lower-triangular) affine invariance for the discretized algorithm when Cholesky square roots are used. The paper's claim is that this stability and invariance, together with linear cost in the dimension for forward-model evaluations, makes DF-GMVI a reliable derivative-free tool for multimodal Bayesian inverse problems.","pith_inferences":["A testable extension of Proposition 5.1 is that any quadrature whose Hessian surrogate is positive semidefinite and whose entropy Hessian term is replaced by the exact weighted average $-C_k^{-1}$ would enjoy the same stability guarantee, since the proof uses only positivity, not the particular finite-difference formula.","The stability guarantee does not by itself control the gap between the stationary point of (4.9) and the true posterior; accuracy still rests on the surrogate Hessian's fidelity, so the most informative future experiment would compare DF-GMVI's stationary KL divergence against the best achievable $K$-component mixture approximation on targets with controlled curvature.","The block-diagonal Fisher information approximation inherited from earlier work assumes well-separated modes; for strongly overlapping components this approximation may be the limiting error even though covariance positivity and consistent entropy treatment fix the overlap artifacts observed in the earlier method.","Because the algorithm is affine invariant under triangular transformations, its empirical convergence rates on anisotropic posteriors should persist under linear reparameterizations, which suggests the reported banana-shaped-case results are not artifacts of a particular coordinate system."],"forward_implications":["Users can set $\\Delta t=0.5$ and run hundreds of iterations with no time-step tuning while preserving covariance positivity, as the paper does in all reported experiments.","Derivative-free posterior approximation costs only $(2N_\\theta+1)K$ forward-model evaluations per iteration, and those evaluations are embarrassingly parallel, making high-dimensional inverse problems feasible.","Because the entropy and cross-entropy terms are treated with consistent quadrature, overlapping modes are resolved in a way that the earlier Kalman-based derivative-free method could not, according to the paper's experiments.","The method captures posterior shapes that typically break gradient-free samplers: narrow curved banana valleys, circular ridges with infinitely many modes, and multimodal 100-dimensional marginals.","The Navier-Stokes experiment indicates that a 128-parameter PDE inverse problem with a symmetric bimodal posterior can be solved by a derivative-free variational method that converges in fewer than 50 iterations."],"supporting_citations":[{"why":"Supplies the Gaussian mixture Kalman inversion baseline, the block-diagonal Fisher information approximation, and the 1D bimodal and circular test problems that DF-GMVI improves.","marker":"[15]"},{"why":"Derives natural-gradient variational inference with mixture-of-exponential-family approximations, the source of the flow (2.3) and its block-diagonal Fisher information approximation.","marker":"[50]"},{"why":"Introduces affine-invariant ensemble sampling, the invariance concept and stretch-move baseline used in Section 6.","marker":"[31]"},{"why":"Provides the unscented transformation whose sigma-point idea Definition 4.1 extends to Hessian-aware derivative-free quadrature.","marker":"[44]"},{"why":"Black-box variational inference is the derivative-free Monte Carlo baseline whose high variance motivates the quadrature design.","marker":"[63]"},{"why":"Wasserstein gradient flow variational inference with Gaussian mixtures is a comparator whose covariance update rule is used for time-step selection in Section 6.","marker":"[48]"},{"why":"Gives the Fisher-Rao gradient-flow perspective and the affine-invariance and exponential-convergence results that justify the natural-gradient flow (2.3).","marker":"[14]"}],"fun_headline_variants":["Derivative-free variational inference stabilizes Gaussian mixtures","No gradients? No problem: stable Gaussian mixture posterior fits","Curvature-aware quadrature keeps Gaussian mixture covariances positive","Affine-invariant, gradient-free Bayesian inference for 100-D problems","DF-GMVI: stable, tune-free posterior sampling for curved 100-D modes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the curvature kept by the derivative-free quadrature is close enough to the true curvature that the updates settle near the true posterior; the paper proves the updates stay numerically stable, not that they land on the right answer.","fun_headline_variants_meta":{"raw":{"variants":["Derivative-free variational inference stabilizes Gaussian mixtures","No gradients? No problem: stable Gaussian mixture posterior fits","Curvature-aware quadrature keeps Gaussian mixture covariances positive","Affine-invariant, gradient-free Bayesian inference for 100-D problems","DF-GMVI: stable, tune-free posterior sampling for curved 100-D modes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000637,"raw_usage":{"total_tokens":2974,"prompt_tokens":1023,"completion_tokens":1951,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":639,"completion_tokens_details":{"reasoning_tokens":1861}},"tokens_in":639,"tokens_out":1951,"duration_ms":14037,"temperature":1.0,"reasoning_tokens":1861,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:39:26.704223+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Set a target posterior with known non-Gaussian form in which the dropped cross term $2\\operatorname{Diag}(A^T c)$ in the Hessian expectation is large, for example by giving $F$ a strong quadratic interaction along a single coordinate. If DF-GMVI's converged density differs from the best $K$-component Gaussian mixture fit by a total-variation distance much larger than the same distance for a gradient-based natural-gradient flow using the exact Hessian, then the truncation in Definition 4.1 is the limiting error; repeating over targets with increasing dropped-term magnitude would map the accuracy boundary of the method.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Gaussian mixture Kalman inversion baseline, the block-diagonal Fisher information approximation, and the 1D bimodal and circular test problems that DF-GMVI improves."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Derives natural-gradient variational inference with mixture-of-exponential-family approximations, the source of the flow (2.3) and its block-diagonal Fisher information approximation."},{"cited_title":"Goodman and J","cited_arxiv_id":null,"evidence_quote":"Introduces affine-invariant ensemble sampling, the invariance concept and stretch-move baseline used in Section 6."},{"cited_title":"Ranganath, S","cited_arxiv_id":null,"evidence_quote":"Black-box variational inference is the derivative-free Monte Carlo baseline whose high variance motivates the quadrature design."}],"review_version":1}