SOAP-Bubbles: Structured Weight Uncertainty for Neural Networks

Adrian Robert Minut; Marco Miani; Mohammad Emtiyaz Khan; Nico Daheim; Thomas M\"ollenhoff; Wu Lin

SOAP-Bubbles turn a cheap diagonal uncertainty method into structured weight posteriors by running it inside SOAP’s eigenspace.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 12:42 UTC pith:J6GNMOOS

load-bearing objection Abstract-only: a clean SOAP+IVON construction for structured weight uncertainty that looks useful if the exact-recovery and LM claims hold, but neither is checkable yet. the 3 major comments →

arxiv 2606.23357 v2 pith:J6GNMOOS submitted 2026-06-22 cs.LG

SOAP-Bubbles: Structured Weight Uncertainty for Neural Networks

Adrian Robert Minut , Nico Daheim , Marco Miani , Mohammad Emtiyaz Khan , Wu Lin , Thomas M\"ollenhoff This is my paper

classification cs.LG

keywords structured weight uncertaintySOAP optimizerIVONvariational inferenceposterior covariancelanguage-model pretrainingEVONeigenspace

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Structured uncertainty over neural-network weights can improve calibration, generalization, and decision-making, but full covariances are usually too expensive and too invasive to compute. This paper claims that the SOAP optimizer already maintains a useful eigenbasis of the curvature, so one can simply run a standard diagonal variational method (IVON) inside that basis and map the result back with SOAP’s preconditioner. The resulting posteriors, called SOAP-Bubbles, cost roughly the same as ordinary SOAP training and need no major pipeline changes. The authors show that the construction recovers the exact Gaussian covariance on logistic regression and, on language-model pretraining, substantially outperforms existing diagonal-covariance baselines. If the claim holds, practitioners can obtain richer weight uncertainty at the cost of an optimizer they may already be using.

Core claim

Running the diagonal variational method IVON in the eigenspace of SOAP’s preconditioner, then transforming the diagonal covariance by that same preconditioner, yields structured Gaussian posteriors (SOAP-Bubbles) whose estimation cost is comparable to SOAP itself and that recover the exact covariance for logistic regression while outperforming diagonal methods on language-model pretraining.

What carries the argument

EVON (Eigenspace-VON): IVON’s diagonal covariance is estimated after rotating parameters into SOAP’s eigenbasis; the SOAP preconditioner then maps that diagonal matrix into a non-diagonal structured covariance in the original parameter space.

Load-bearing premise

That the eigenbasis already maintained by SOAP is a sufficiently faithful coordinate system for the dominant posterior correlations of deep networks, so a diagonal approximation performed there becomes a useful structured posterior after transformation.

What would settle it

On a logistic-regression problem of moderate dimension, check whether the EVON covariance matrix matches the exact closed-form Gaussian posterior covariance (within numerical tolerance); any systematic mismatch falsifies the recovery claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Abstract-only: a clean SOAP+IVON construction for structured weight uncertainty that looks useful if the exact-recovery and LM claims hold, but neither is checkable yet.

read the letter

Punchline: this is a constructive synthesis—run IVON in SOAP’s eigenspace, transform the diagonal back with the preconditioner—and it names EVON / SOAP-Bubbles as the result. If the two headline claims are real, it is a practical step for scalable structured weight uncertainty. We only have the abstract, so treat everything below as provisional.

What is new and what they do well: the construction itself is clear and non-circular. SOAP already gives a useful geometry for optimization; IVON already gives a cheap diagonal variational posterior. Combining them so the diagonal lives in the preconditioner’s eigenbasis and is mapped back to a non-diagonal covariance is a concrete engineering idea, not a rehash of either paper alone. They claim exact Gaussian covariance recovery for logistic regression (a clean sanity check if true) and better LM pretraining than diagonal methods at SOAP-like cost with no pipeline rewrite. That is the right target: calibration, continual learning, and uncertainty-aware fine-tuning of large models care about structure, and cost parity with SOAP is the bar that matters.

Soft spots, in proportion: the load-bearing premise is that SOAP’s eigenspace is a faithful enough basis for the dominant posterior correlations. The abstract does not justify that independently; it is assumed by the construction. Free parameters (IVON variational knobs, SOAP rank/schedule) are still there. More importantly, without the full text we cannot audit the exact-recovery derivation, the experimental protocol, metrics, ablations, or whether the LM gains are controlled for compute and hyper-parameters. Soundness is therefore uncheckable, not disproven. The stress-test note is right that both central claims are currently unauditable; it is not inventing a flaw, just stating the information gap.

Who it is for: people already using or considering SOAP/IVON and anyone who needs structured weight uncertainty without a full second-order pipeline. It deserves a serious referee once the full paper is available—exact recovery plus controlled LM results would make it citeable; missing proofs or confounded baselines would not. I would not desk-reject on the abstract alone. Bring it to reading group only after the PDF lands; until then, flag it as a method to watch rather than something to build on.

Referee Report

3 major / 2 minor

Summary. The manuscript proposes EVON (Eigenspace-VON) and the associated SOAP-Bubbles posteriors: run the diagonal variational method IVON in the eigenspace of SOAP’s preconditioner, then transform the resulting diagonal covariance by that preconditioner to obtain a structured (non-diagonal) weight posterior. The abstract asserts two central results: (i) for logistic regression, EVON recovers the exact Gaussian covariance; (ii) for language-model pretraining, EVON yields significantly better results than existing diagonal-covariance methods, at costs comparable to SOAP and without major pipeline changes. The claimed contribution is therefore a practical route to more expressive Bayesian posteriors at deep-learning scale.

Significance. If the exact-recovery claim and the controlled LM gains hold, the work would materially lower the barrier to structured weight uncertainty: it reuses an already-deployed optimizer geometry, keeps wall-clock cost near SOAP, and requires no drastic training-pipeline redesign. That combination would be of clear interest to Bayesian deep learning and large-scale uncertainty quantification. The constructive nature of the method (diagonal IVON in a fixed eigenspace, then a linear transform) is also a strength if the mathematics and experiments are sound. These strengths cannot yet be credited, because only the abstract is available.

major comments (3)

Abstract, exact-recovery claim: the statement that EVON “recovers the exact Gaussian covariance” for logistic regression is load-bearing for the paper’s theoretical contribution. With only the abstract, there is no theorem statement, derivation, or proof sketch. The claim cannot be audited; a full derivation (or a clear statement of the precise sense of “exact”) is required before the result can be accepted.
Abstract, empirical claim: the assertion of “significantly better results” than diagonal-covariance methods on language-model pretraining is the other load-bearing claim. No tables, metrics, error bars, compute-matched baselines, or ablations are available. Without those, superiority cannot be assessed and may be confounded by hyper-parameter or compute differences.
Abstract, geometric premise: the construction assumes that SOAP’s preconditioner eigenspace is a sufficiently faithful basis for the dominant posterior correlations, so that a diagonal IVON estimate transformed by the same preconditioner yields a valid structured posterior. This premise underpins both the exact-recovery claim and the claimed gains over diagonal methods; it is not independently justified in the abstract and must be stated and tested in the full manuscript.

minor comments (2)

Abstract only: notation for the transformed covariance, the precise role of IVON’s variational hyperparameters, and the SOAP rank/update schedule are not specified; these should be made explicit once the full text is available.
The name “SOAP-Bubbles” is introduced without a short formal definition of the object (e.g., the family of covariances obtained by the EVON transform); a one-line definition would aid readers.

Circularity Check

0 steps flagged

No circularity found: abstract presents a constructive method (IVON in SOAP eigenspace + transform) without definitional loops or fitted-as-prediction reductions.

full rationale

Only the abstract is available. It describes EVON as running the existing diagonal method IVON in the eigenspace of SOAP’s preconditioner and transforming the diagonal estimate back into a structured covariance. The logistic-regression claim is an exact-recovery statement for a known Gaussian posterior; the LM claim is an empirical comparison to diagonal baselines. Neither claim, as stated, reduces by construction to a fitted target, a self-definition, or a load-bearing self-citation uniqueness theorem. No equations, uniqueness results, or self-citation chains appear in the provided text that would allow exhibiting a specific circular reduction. Residual risk that SOAP’s optimization geometry correlates with reported gains is not circularity under the stated criteria. Per hard rules, absence of quotable reduction implies score 0 and empty steps.

Axiom & Free-Parameter Ledger

2 free parameters · 3 axioms · 1 invented entities

Abstract-only review: free parameters, background axioms, and invented entities cannot be exhaustively audited from equations or experiments. The ledger records what the abstract itself makes load-bearing: variational mean-field/diagonal assumptions inside the SOAP basis, the validity of transforming that diagonal estimate by the SOAP preconditioner, and standard deep-learning training assumptions. No new physical entities are postulated.

free parameters (2)

IVON variational hyperparameters (learning rates, prior precision, temperature, etc.)
Diagonal variational methods such as IVON require scale and prior choices; the abstract does not state whether these are fixed, tuned, or derived. Any such knobs are free parameters of the reported LM results.
SOAP preconditioner rank / update schedule
SOAP’s geometry is controlled by implementation choices (e.g., how often the eigenspace is refreshed). Those choices affect both cost and the quality of the transformed covariance.

axioms (3)

ad hoc to paper A diagonal variational approximation (IVON) run in SOAP’s eigenspace, after transformation by the preconditioner, yields a useful structured posterior for deep networks.
This is the central modeling leap of the abstract; it is not a standard theorem of variational inference and is not independently justified here.
domain assumption SOAP’s preconditioner provides a stable, optimizable change of basis during training.
Inherited from the SOAP literature; assumed to remain valid when the same basis is used for uncertainty estimation.
standard math Standard variational inference and Gaussian posterior assumptions for logistic regression.
Needed for the claim that EVON recovers the exact Gaussian covariance in the logistic-regression case.

invented entities (1)

SOAP-Bubbles (named structured posteriors) no independent evidence
purpose: Label for the non-diagonal covariances obtained by transforming IVON’s diagonal estimate with SOAP’s preconditioner.
Naming device for the method’s output; not a new physical particle or force. Independent evidence would be external predictive checks (e.g., calibration or OOD metrics) not visible in the abstract.

pith-pipeline@v1.1.0-grok45 · 6063 in / 2604 out tokens · 39137 ms · 2026-07-12T12:42:45.039567+00:00 · methodology

0 comments

read the original abstract

Structured weight-uncertainty can improve many aspects of deep learning, but it remains costly to estimate and difficult to implement. Here, we show that these issues can be addressed by adapting the SOAP optimizer. Our key idea is to run IVON, an existing diagonal-covariance variational method, in the eigenspace of SOAP's preconditioner and then use the preconditioner to transform the diagonal estimate into a non-diagonal covariance. The resulting method has costs similar to those of SOAP and requires no drastic changes to training pipelines. We call the posteriors obtained in this way SOAP-Bubbles and our new optimizer Eigenspace-VON (EVON). We show that, for logistic regression, EVON recovers the exact Gaussian covariance and that, for language model pretraining, it yields significantly better results than existing diagonal-covariance methods. Our work makes it easier to estimate more expressive posterior distributions for deep learning at scale.