Pith. sign in

REVIEW 3 major objections 6 minor 34 references

A covariance penalty on last-layer heads restores the function-space diversity that weight orthonormality cannot, recovering much of a deep ensemble’s calibration at one backbone cost.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-30 13:54 UTC pith:PIFCESGB

load-bearing objection Clean mechanism paper: activation covariance really does restore last-layer diversity that weight orthonormality misses, with honest feature-quality gating—and one real tuning soft spot on the headline OOD numbers. the 3 major comments →

arxiv 2607.23856 v2 pith:PIFCESGB submitted 2026-07-26 cs.LG

Covariance Last-Layer Ensembles: Function-Space Diversity for Efficient Uncertainty Quantification

H. Martin Gillis , Isaac Xu , Gabriel Spadon , Thomas Trappenberg This is my paper
classification cs.LG
keywords uncertainty quantificationdeep ensemblelast-layer ensemblelast-layer diversityout-of-distributionorthonormal certificatesfunction-space diversitycalibration
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Last-layer ensembles put many cheap linear heads on one frozen backbone so a single forward pass can estimate epistemic uncertainty from member disagreement. Shared gradients pull those heads toward the same function, so the disagreement signal collapses. Weight orthonormality (as in Orthonormal Certificates) only decorrelates weights and often fails to diversify predictions. This paper shows that a direct covariance penalty on member activations restores function-space diversity, lifts in-distribution prediction variance and calibration toward deep-ensemble levels at 1× backbone cost, and does so without hurting accuracy. The same last-layer view yields a two-axis taxonomy of detectors and a scale-invariant direction score that repairs the label-free near-OOD failure of the usual certificate-norm score on every backbone tested. Detection gains still depend on strong frozen features: diversity is restored regardless, but OOD ROC AUC rises mainly when the representation is already good.

Core claim

Targeting collapse directly in function space with a covariance penalty on member activations (cov-LLE) restores the diversity that weight-orthonormality cannot. At matched K it recovers a large fraction of a deep ensemble’s in-distribution prediction variance and calibration at 1× backbone cost with no accuracy loss, and it significantly beats the weight-orthonormal last-layer ensemble on far-OOD on strong frozen backbones. Treating Orthonormal Certificates as a last-layer ensemble also motivates a scale-invariant direction score that adds roughly 0.16–0.18 ROC AUC on near-OOD for every backbone.

What carries the argument

Covariance Last-Layer Ensemble (cov-LLE): K heads on a frozen feature map, each with a small embedding, trained with cross-entropy plus a Barlow-Twins/VICReg-style covariance penalty on the centered concatenated activations so members are decorrelated in function space rather than only in weight space.

Load-bearing premise

Restored head diversity turns into better out-of-distribution detection only when the frozen backbone already has strong features; weak representations keep the diversity gain from becoming a detection gain.

What would settle it

On a strong frozen backbone (e.g. ResNet-18 or WRN-28-10 on CIFAR-10), replace the covariance penalty with weight orthonormality or no penalty at matched K and check whether far-OOD ROC AUC and in-distribution prediction variance/ECE still match the paper’s reported lifts; if cov-LLE does not beat ortho-LLE on far-OOD and sit between ortho-LLE and a deep ensemble on variance and ECE, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Single-pass last-layer ensembles can approach deep-ensemble calibration and disagreement without K independent networks when diversity is enforced on activations.
  • Weight-space orthonormality is an incomplete fix; function-space decorrelation is the direct remedy for shared-backbone collapse.
  • Label-free OC near-OOD failures are largely scoring artifacts: direction (angle) scoring recovers signal the certificate norm discards.
  • OOD gains from last-layer diversity remain gated by feature quality, not merely bi-Lipschitz conditioning or adversarial robustness.
  • A 2×2 taxonomy (supervised vs unsupervised members × norm vs disagreement scoring) organizes when each detector cell works or inverts.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Any shared-backbone multi-head method (not only LLEs) may need an activation-level diversity term once heads sit above a frozen or slowly changing trunk.
  • Max-fusing magnitude and direction OC scores is a practical default when far-OOD sometimes lives in scale and near-OOD in angle.
  • If feature quality is the gate, investing in the backbone representation may buy more OOD detection than further head-diversity tricks on weak features.
  • The same covariance objective could be tested as a cheap drop-in for other explicit-member efficient ensembles that still share most of the network.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper studies diversity collapse in last-layer ensembles (LLEs): K linear heads on a shared frozen backbone tend to converge to the same function, destroying the disagreement signal used for epistemic uncertainty and OOD detection. It makes three contributions. First, a diagnostic 2×2 taxonomy (member objective × scoring rule) that places Orthonormal Certificates (OC) as the weight-orthonormal, unsupervised-norm corner of the LLE design space, and shows the label-free OC norm score is gated by backbone feature quality rather than bi-Lipschitz conditioning (supported by RobustBench and spectral-normalization ablations). Second, cov-LLE: a Barlow-Twins/VICReg-style covariance penalty on member activations that decorrelates heads in function space rather than weight space, shown at matched K to recover a substantial fraction of deep-ensemble prediction variance and calibration at 1× backbone cost, and to significantly beat the weight-orthonormal LLE on far-OOD on two strong CIFAR backbones. Third, a scale-invariant direction score S_D = ||C^T φ(x)||/||φ(x)|| = sin θ for OC certificates that repairs the near-OOD failure of the magnitude score (+0.16 to +0.18 ROC AUC on every backbone) and de-inverts the SVHN far-OOD failure on three of four backbones, with a label-free max-fusion as a safe default. Experiments use 10 seeds with paired t-tests, multiple backbones (CIFARNet, WRNs, ResNet-18, RobustBench checkpoints, CIFAR-100 VGG, DistilBERT), and the negative results (cov-LLE

Significance. If the results hold, this is a useful contribution to efficient uncertainty quantification. The function-space/weight-space distinction for ensemble diversity penalties is conceptually clean and the paper demonstrates concretely where weight-orthonormality fails (the two-layer-head result in SI S2, where the ortho penalty is functionally inert, δ_f = 0.015, while the covariance penalty lifts δ_f to 0.52, is a crisp mechanistic demonstration). The direction score is essentially parameter-free, label-free, backed by a simple geometric identity (sin θ), and delivers a large, consistent near-OOD gain on every backbone tested — this is the strongest and most portable result in the paper. The honest reporting of boundary conditions (feature-quality gating, the WRN-10-4 far-OOD inversion of the direction score, adversarial robustness degrading detection) and the consistent 10-seed paired-testing protocol raise confidence. The authors are also appropriately modest about raw performance, acknowledging KNN leads on ROC AUC and framing the contribution as mechanism plus diagnosis rather than state-of-the-art. The main weakness is a hyperparameter-selection protocol that partially undermines t

major comments (3)
  1. [§4.2, Figure 3, Table 2 (also SI S2)] The headline cov-LLE result (Table 2: ResNet-18 far-OOD 0.892→0.912, t≈8; WRN-28-10 0.922→0.946, t≈11) is reported at λ_cov=0.5, which Figure 3 shows was selected by sweeping λ_cov ∈ {0.1, 0.5, 2.0, 4.0} on the same near-OOD AUC, far-OOD AUC, and ECE metrics that Table 2 then reports. In OOD detection this is a load-bearing methodological issue: the premise of the task is that OOD data is unavailable at tuning time, so a hyperparameter chosen on test OOD AUC inflates the realized gap in a way that cannot be reproduced in deployment. The comparison is also asymmetric — λ_ortho=3.0 is 'used throughout' with no stated selection procedure — so the paired-t advantage partly measures tuned-vs-untuned rather than function-space-vs-weight-space. The sensitivity is not benign: at λ_cov=4 detection collapses (0.765/0.833, ECE 0.206), so the reported point sits at a selected interior optimum of a c
  2. [Abstract; §4.2; Table 3; SI S6] The abstract states cov-LLE 'recovers much of the diversity and calibration of a deep ensemble... at no cost to accuracy' and that it outperforms the weight-space penalty, but does not state the feature-quality gate that the body of the paper itself establishes. On the weak CIFARNet (Table 3), cov-LLE is significantly *worse* than ortho-LLE on detection (near 0.644→0.591, far 0.679→0.605, BALD), and on raw DistilBERT features (SI S6) it is significantly worse on near-OOD (0.773→0.752, t≈−14). The Discussion handles this honestly ('what is gated is the detection gain, not the collapse mitigation'), but the abstract — the part most readers will cite — presents the gain without the condition under which it holds. Since the gating finding is itself one of the paper's stated contributions (contribution 1), the abstract should say explicitly that the detection advantage of cov-LLE requires str
  3. [§4.2, Table 3] Two aspects of the deep-ensemble recovery claim need attention. (i) The cov-LLE prediction variance on CIFAR is 9.27±4.535 ×10⁻³ — a standard deviation of nearly 50% of the mean across 10 seeds, versus 22.14±0.79 for the deep ensemble. This suggests the covariance penalty introduces substantial seed-level instability in the very quantity the paper uses as its diversity evidence; the authors should comment on the source (e.g., which seeds, whether it correlates with detection) and preferably report medians or per-seed values. (ii) The calibration recovery is backbone-dependent in sign: on MNIST (Table 3) cov-LLE ECE is 0.009 vs 0.001 for ortho-LLE (a 9× regression), and on WRN-28-10 (Table 2) ECE marginally worsens (0.029→0.034). The text's phrase 'improves calibration where gains exist' acknowledges this only obliquely; the ECE regression on MNIST and WRN-28-10 should be stated plainly a
minor comments (6)
  1. [Table 6] Several FPR@95 entries are exactly 1.000 (van-LLE EPKL/VGMU near and far; ortho-LLE VGMU near). An FPR of 1.0 at 95% TPR is worse than chance and presumably reflects score inversion on some component dataset; a footnote explaining this would help, especially since the corresponding AUCs are ~0.83–0.87.
  2. [§3, Eq. (7)] Ω_cov penalizes ||(1/N)Z^TZ − I||²_F, which includes the diagonal (unit-variance) terms as in VICReg/Barlow-Twins. Please state explicitly whether the variance-to-one constraint on each embedding dimension is intended (it also acts as an anti-collapse/anti-shrinkage term, which may itself contribute to the effect), and whether an off-diagonal-only variant was tried.
  3. [Figure 3] The x-axis tick labels are garbled in the rendered figure ('0.1 0.1 0.1 0.10.5 0.5 0.5 0.52.0...'), apparently from overlapping per-panel tick labels. Please fix for readability.
  4. [SI S4] Typo: 'far = SVH' should read 'far = SVHN'.
  5. [§4.4 / SI S3] The max-fusion rule (Eq. S1) standardizes each score with training-set statistics before taking the max — this is a reasonable label-free default, but it would help to state in the main text that fusion does not match the best single score on either near- or far-OOD (e.g., ResNet-18 near: direction 0.778 vs fusion 0.759), so readers do not read 'never inverts' as 'dominates'.
  6. [§2 / §3] VGMU is used as a scoring rule throughout but is only defined by citation to the authors' prior work [6]; a one-line definition would make the paper more self-contained.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a careful and fair report. The three major comments all identify genuine weaknesses in presentation or protocol, and we will revise to address each. On the λ_cov selection issue, the criticism is correct in principle: selecting the penalty weight on the reported test OOD metrics is methodologically inappropriate for OOD detection, and we will re-select λ_cov using only ID data (ID prediction variance / ID validation calibration) and re-report Table 2 at that value. Preliminary checks indicate the ID-data-selected optimum lands at or near λ_cov=0.5, but we will report whatever the honest number is. On the abstract, we will add the feature-quality gate explicitly. On Table 3, we will report per-seed/median statistics for the prediction variance and state the MNIST and WRN-28-10 ECE regressions plainly.

read point-by-point responses

  1. Referee: Table 2's headline cov-LLE results are reported at λ_cov=0.5, selected by sweeping λ_cov on the same near-OOD AUC, far-OOD AUC, and ECE metrics that Table 2 reports. OOD data is unavailable at tuning time, so this inflates the realized gap; the comparison is also asymmetric because λ_ortho=3.0 has no stated selection procedure, and λ_cov=0.5 sits at a selected interior optimum of a curve that collapses at λ_cov=4.

    Authors: The referee is right, and this is the most consequential comment in the report. The premise of OOD detection is that OOD data is unavailable at tuning time, so selecting λ_cov on test OOD AUC is not a defensible deployment protocol. We will revise as follows. (i) Re-select λ_cov using only in-distribution quantities: an ID validation split, choosing the value that maximizes function-space diversity / ID prediction variance subject to an ID ECE constraint. In our existing sweeps, ID prediction variance rises monotonically with λ_cov while ID ECE begins degrading around λ_cov≈0.5–1.0, so an ID-only criterion lands near 0.5; we will report the exact selected value and, if it differs from 0.5, re-run Table 2 at that value with the paired tests recomputed. (ii) State the λ_ortho selection procedure symmetrically: λ_ortho=3.0 was chosen on MNIST ID-diversity/accuracy criteria before any OOD evaluation, following the OC paper's observation that detection is insensitive above threshold; we will document this and, for full symmetry, re-select λ_ortho under the identical ID-only protocol and report both. (iii) Reframe Figure 3 explicitly as a sensitivity analysis (showing the interior optimum and the over-decorrelation collapse), not a selection curve, and note that the qualitative conclusion (function-space beats weight-space at any non-degenerate λ_cov) holds across the plateau, not only at the peak. We note the far-OOD gain replicates on the untuned CIFAR-100 VGG (SI S7, 0.745→0.761) at the same λ_cov, which is evidence the optimum transfers, but we agree this does not substitute for a clean protocol. revision: yes

  2. Referee: The abstract presents the cov-LLE diversity/calibration gain without the feature-quality gate that the body of the paper itself establishes (cov-LLE is significantly worse than ortho-LLE on CIFARNet detection and on raw DistilBERT near-OOD). Since gating is contribution 1, the abstract should state the condition under which the detection advantage holds.

    Authors: We agree entirely. The abstract reports the recovery result and 'at no cost to accuracy' but omits the condition the paper itself identifies as a contribution: the detection gain is feature-quality gated, while the collapse mitigation (restored function-space diversity and calibration) is not. On the weak CIFARNet cov-LLE is significantly below ortho-LLE on detection, and on raw DistilBERT features it is significantly worse on near-OOD; a reader of the abstract alone would not learn this. We will revise the abstract to state explicitly that cov-LLE restores function-space diversity and calibration on all backbones tested, but that the conversion of that diversity into a detection advantage requires strong feature representations, and that on weak features the weight-orthonormal LLE remains the better detector. We will also align the conclusion similarly. revision: yes

  3. Referee: (i) cov-LLE prediction variance on CIFAR is 9.27±4.535×10⁻³ — SD nearly 50% of the mean across seeds, vs 22.14±0.79 for the deep ensemble — suggesting seed-level instability in the diversity evidence; report the source, medians, or per-seed values. (ii) The calibration recovery is backbone-dependent in sign: MNIST ECE regresses 9× (0.001→0.009) and WRN-28-10 marginally worsens (0.029→0.034); these regressions should be stated plainly rather than obliquely.

    Authors: Both points are fair. (i) The large SD comes from a subset of seeds in which a few members over-decorrelate, inflating the across-member variance; it is not detection-relevant noise (detection AUC is stable across the same seeds, ±0.003–0.007), but the referee is correct that a 50% relative SD undermines the variance-as-evidence presentation. We will report per-seed values and medians for prediction variance in Table 3 and SI, check and report the correlation between per-seed variance and per-seed detection, and soften the variance framing accordingly (the bounded δ_f = 1−CKA measure, which is better behaved, will be presented alongside). (ii) We will state both regressions explicitly in the text: on MNIST cov-LLE ECE is 0.009 vs 0.001 for ortho-LLE (a real regression, though small in absolute terms and far below the single-network/ortho values on the harder CIFAR task), and on WRN-28-10 ECE marginally worsens (0.029→0.034). The calibration-recovery claim will be scoped to where it holds — the weak-backbone CIFAR setting and the ResNet-18 — rather than implied generally. revision: yes

Circularity Check

1 steps flagged

Empirical methods paper: evaluation metrics are external to the training penalty; no derivation reduces to its inputs by construction.

specific steps
  1. self citation load bearing [Section 2 / Table 6; Ref. [6]]
    "Every OOD score, namely BALD, Expected Pairwise KL divergence (EPKL), Variance-Gated Margin Uncertainty (VGMU), and their cov-LLE variants, comes from a single shared implementation"

    VGMU is taken from the authors' own concurrent TMLR paper. This is ordinary self-citation of a scoring variant, not a uniqueness theorem or premise that forces the main result; BALD/EPKL alone already support the cov-LLE comparisons. Listed only as a minor, non-load-bearing instance.

full rationale

The central claims are empirical comparisons (cov-LLE vs ortho-LLE vs deep ensemble on prediction variance, CKA, ECE, and ROC AUC), not first-principles derivations. The covariance penalty (Eq. 7) encourages decorrelated member activations; function-space diversity and OOD scores are separately measured quantities that need not improve (and on weak backbones sometimes do not). That gap shows the claim is not tautological with the objective. The direction score SD=sinθ is an algebraic re-expression of the OC certificates under orthonormality, presented as a scoring fix rather than a predicted discovery forced by a fit. Self-citation of the authors' VGMU score [6] is peripheral—one of several scoring rules beside standard BALD/EPKL—and is not load-bearing for the cov-LLE or direction-score results. Hyperparameter choice of λ_cov≈0.5 via sweeps that include evaluation OOD metrics is a methodological soft spot (test-regime selection), but it is not circularity in the analyzer's sense: the paper does not fit a parameter and then rename that fit as an independent prediction. Against external benchmarks (OpenOOD baselines, RobustBench, deep ensembles) the work is self-contained. Score 1 only for the minor non-load-bearing self-citation.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 3 invented entities

Load-bearing content is empirical method design plus standard UQ/OOD assumptions. Free parameters are penalty weights and ensemble size chosen by ablation. Axioms are domain conventions (BALD as epistemic, frozen φ, held-out-class OOD). Invented entities are named methods/scores, not physical posits; independent evidence is the reported experiments only.

free parameters (5)
  • λ_cov (covariance penalty weight) = 0.5
    Detection peaks near 0.5 and collapses at high values (Fig. 3); fixed at 0.5 after ablation on MNIST/CIFAR evaluation tasks.
  • λ_ortho (weight-orthonormality penalty) = 3.0
    Fixed throughout for ortho-LLE comparisons; controls weight-space diversity baseline.
  • λ_oc (OC orthonormality strength) = 5.0
    Taken from OC literature sensitivity; fixed at 5.0 with K=32 certificates.
  • K (number of heads/certificates) = 32 (default); 10 for deep-ensemble match
    Default 32 (≈d/4 on MNIST); deep-ensemble match uses K=10. OC collapses as K→d.
  • Per-head embedding dimension D = 32 (MNIST study)
    Small embedding φ→z_k before logits for cov-LLE; set to 32 in MNIST two-layer study.
axioms (5)
  • domain assumption Epistemic uncertainty for OOD is well captured by ensemble disagreement (BALD / mutual information residual).
    Eq. 1 and §2; standard in deep ensembles but not uniquely correct among UQ formalisms.
  • domain assumption A frozen pretrained/feature backbone φ is an adequate substrate; head diversity cannot undo feature collapse in φ.
    Stated throughout §§1–4; gates whether diversity becomes detection.
  • domain assumption Held-out-class and semantically distant datasets are valid near/far OOD proxies.
    §3 protocol (CIFAR split, SVHN, OpenOOD, 20 Newsgroups).
  • domain assumption Linear CKA and across-member prediction variance are valid function-space diversity measures.
    §2–4.2 definitions used to claim collapse mitigation.
  • standard math Standard optimization and generalization facts for Adam-trained linear heads on frozen features.
    Training setup in §3; no exotic analysis.
invented entities (3)
  • Covariance Last-Layer Ensemble (cov-LLE) independent evidence
    purpose: Direct function-space diversity via covariance penalty on concatenated member activations/embeddings.
    Core method; Ω_cov in Eq. 7 added to cross-entropy.
  • OC direction score S_D = ||C^T φ(x)|| / ||φ(x)|| independent evidence
    purpose: Scale-invariant, label-free alternative to certificate magnitude for near-OOD.
    §4.4; sinθ geometry relative to certificate span.
  • 2×2 detector taxonomy (member objective × scoring rule) independent evidence
    purpose: Organize OC, LLE, and hybrid cells; diagnose supervised×norm inversion.
    §3–4.1 unifying frame; not a physical entity but a paper-introduced organizing construct.

pith-pipeline@v1.2.0-grok45-kimik3 · 27512 in / 3707 out tokens · 62010 ms · 2026-07-30T13:54:20.707571+00:00 · methodology

0 comments
read the original abstract

A Last-Layer Ensemble (LLE), $K$ linear units on one shared frozen feature map, is an efficient single-pass approach to the disagreement-based epistemic uncertainty for out-of-distribution (OOD) detection. Its weakness is that members share the backbone gradient and can converge toward the same function, collapsing the inter-member diversity the signal depends on. Whether last-layer diversity can be restored, and what mitigates the collapse, is an open question. The weight-orthonormality defining Orthonormal Certificates (OC), the weight-orthonormal special case of the LLE, is only an indirect correction; it decorrelates the weights of the members, not their predictions. Here, we instead target the collapse directly in function space, with a Covariance Last-Layer Ensemble (cov-LLE) that places a direct covariance penalty on member activations. Cov-LLE restores the function-space diversity that weight-orthonormality cannot, and at matched $K$ recovers much of the diversity and calibration of a deep ensemble at $1\times$ backbone cost (in-distribution prediction variance $0.05\!\to\!9.3$ vs. $22.1$ ($\times10^{-3}$), and ECE $0.135\!\to\!0.090$ vs. $0.035$, for a $K\times$-cost deep ensemble), at no cost to accuracy. Viewing OC as a last-layer ensemble also organizes detectors into a two-axis taxonomy (by how their units are trained and how their outputs are scored) and exposes the OC score as a magnitude, motivating a scale-invariant, label-free direction score that repairs its near-OOD failure, adding $+0.16$ to $+0.18$ ROC AUC on every backbone.

Figures

Figures reproduced from arXiv: 2607.23856 by Gabriel Spadon, H. Martin Gillis, Isaac Xu, Thomas Trappenberg.

Figure 1
Figure 1. Figure 1: The 2 × 2 structure holds on natural images; its label-free corner is feature-quality gated. Member objective (unsu￾pervised/supervised) × scoring rule (norm/disagreement) for the CIFARNet and WRN-10-4; near-OOD = held-out CIFAR 5–9, far-OOD = SVHN (ROC AUC). The supervised × norm cell inverts on both, while the other three detect; the label-free OC norm (unsupervised × norm) is near chance on the CIFARNet… view at source ↗
Figure 2
Figure 2. Figure 2: Cov-LLE is between the weight-orthonormal LLE [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Detection peaks at an interior 𝝀cov≈0.5 and collapses as the penalty over-decorrelates, while function-space diversity keeps rising. Covariance-penalty strength 𝝀cov on two frozen backbones, ResNet-18 and WRN-28-10, showing near-OOD, far-OOD, in-distribution ECE, and diversity 𝜹f . ID = CIFAR-10, OpenOOD protocol (SI S1) [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The direction score measures the angle a feature [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

34 extracted references · 4 linked inside Pith

  1. [1]

    Bardes, J

    A. Bardes, J. Ponce, and Y. LeCun. 2022. VICReg: Variance-invariance-covariance Regularization for self-supervised learning. InICLR. OpenReview.net, Online

    2022

  2. [2]

    Croce, M

    F. Croce, M. Andriushchenko, V. Sehwag, E. Debenedetti, N. Flammarion, M. Chiang, P. Mittal, and M. Hein. 2021. RobustBench: A standardized adversarial robustness benchmark. InNeurIPS. Curran Associates Inc., Red Hook, NY, USA

    2021

  3. [3]

    Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. 2021. Laplace redux – effortless Bayesian deep learning. InNeurIPS. Curran Associates Inc., Red Hook, NY, USA

    2021

  4. [4]

    Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. InICML. PMLR, Online

    2016

  5. [5]

    González Ballester

    Adrian Galdran, Johan Verjans, Gustavo Carneiro, and Miguel A. González Ballester. 2023. Multi-head multi-loss model calibration. InMICCAI, Vol. 14222. Springer Nature Switzerland, Cham, Switzerland, 108–117. doi:10.1007/978-3- 031-43898-1_11

    doi:10.1007/978-3- 2023

  6. [6]

    Martin Gillis, Isaac Xu, and Thomas Trappenberg

    H. Martin Gillis, Isaac Xu, and Thomas Trappenberg. 2026. Variance-gated ensembles: An epistemic-aware framework for uncertainty estimation.TMLR6 (2026)

    2026

  7. [7]

    James Harrison, John Willes, and Jasper Snoek. 2024. Variational Bayesian last layers. InICLR. OpenReview.net, Online

    2024

  8. [8]

    Dai, and Dustin Tran

    Marton Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snoek, Balaji Lakshminarayanan, Andrew M. Dai, and Dustin Tran. 2021. Train- ing independent subnetworks for robust prediction. InICLR. OpenReview.net, Online. 8

    2021

  9. [9]

    Dan Hendrycks and Kevin Gimpel. 2017. A baseline for detecting misclassified and out-of-distribution examples in neural networks. InICLR. OpenReview.net, Online

    2017

  10. [10]

    Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. 2011. Bayesian active learning for classification and preference learning. arXiv preprint. doi:10.48550/arXiv.1112.5745

    Pith/arXiv arXiv doi:10.48550/arxiv.1112.5745 2011

  11. [11]

    Andreas Kirsch. 2025. (implicit) ensembles of ensembles: Epistemic uncertainty collapse in large models.TMLR5 (2025), 1–16. doi:10.48550/arXiv.2409.02628

    Pith/arXiv arXiv doi:10.48550/arxiv.2409.02628 2025

  12. [12]

    Kornblith, M

    S. Kornblith, M. Norouzi, H. Lee, and G. Hinton. 2019. Similarity of neural network representations revisited. InICML. PMLR, Online

    2019

  13. [13]

    Agustinus Kristiadi, Matthias Hein, and Philipp Hennig. 2020. Being Bayesian, even just a bit, fixes overconfidence in ReLU networks. InICML. PMLR, Online. doi:10.48550/arXiv.2002.10118

    Pith/arXiv arXiv doi:10.48550/arxiv.2002.10118 2020

  14. [14]

    Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017. Simple and scalable predictive uncertainty estimation using deep ensembles. InNeurIPS. Curran Associates Inc., Red Hook, NY, USA

    2017

  15. [15]

    Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. 2018. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In NeurIPS. Curran Associates Inc., Red Hook, NY, USA

    2018

  16. [16]

    Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David Crandall, and Dhruv Batra. 2015. Why M heads are better than one: Training a diverse ensemble of deep networks. arXiv preprint. doi:10.48550/arXiv.1511.06314

    Pith/arXiv arXiv doi:10.48550/arxiv.1511.06314 2015

  17. [17]

    Shiyu Liang, Yixuan Li, and R. Srikant. 2018. Enhancing the reliability of out- of-distribution image detection in neural networks. InICLR. OpenReview.net, Online

    2018

  18. [18]

    Jeremiah Zhe Liu, Shreyas Padhy, Jie Ren, Zi Lin, Yeming Wen, Ghassen Jer- fel, Zachary Nado, Jasper Snoek, Dustin Tran, and Balaji Lakshminarayanan

  19. [19]

    Owens, and Yixuan Li

    Weitang Liu, Xiaoyun Wang, John D. Owens, and Yixuan Li. 2020. Energy-based out-of-distribution detection. InNeurIPS. Curran Associates Inc., Red Hook, NY, USA

    2020

  20. [20]

    Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. 2018. Spectral normalization for generative adversarial networks. InICLR. OpenRe- view.net, Online

    2018

  21. [21]

    Nalisnick, A

    E. Nalisnick, A. Matsukawa, Y. W. Teh, D. Gorur, and B. Lakshminarayanan. 2019. Do deep generative models know what they don’t know?. InICLR. OpenRe- view.net, Online

    2019

  22. [22]

    Ian Osband, Zheng Wen, Seyed Mohammad Asghari, Vikranth Dwaracherla, Morteza Ibrahimi, Xiuyuan Lu, and Benjamin Van Roy. 2023. Epistemic neural networks. InNeurIPS. Curran Associates Inc., Red Hook, NY, USA

    2023

  23. [23]

    Y. Sun, C. Guo, and Y. Li. 2021. ReAct: Out-of-distribution detection with rectified activations. InNeurIPS. Curran Associates Inc., Red Hook, NY, USA

    2021

  24. [24]

    Y. Sun, Y. Ming, X. Zhu, and Y. Li. 2022. Out-of-distribution detection with deep nearest neighbors. InICML. PMLR, Online

    2022

  25. [25]

    Natasa Tagasovska and David Lopez-Paz. 2019. Single-Model Uncertainties for Deep Learning. InNeurIPS. Curran Associates Inc., Red Hook, NY, USA

    2019

  26. [26]

    Joost van Amersfoort, Lewis Smith, Andrew Jesson, Oscar Key, and Yarin Gal

  27. [27]

    Joost van Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal. 2020. Un- certainty estimation using a single deep deterministic neural network. InICML. PMLR, Online

    2020

  28. [28]

    H. Wang, Z. Li, L. Feng, and W. Zhang. 2022. ViM: Out-of-distribution with virtual-logit matching. InCVPR. IEEE, Los Alamitos, CA, USA

    2022

  29. [29]

    Yeming Wen, Dustin Tran, and Jimmy Ba. 2020. BatchEnsemble: An alternative approach to efficient ensemble and lifelong learning. InICLR. OpenReview.net, Online

    2020

  30. [30]

    Jingkang Yang, Pengyun Wang, Dejian Zou, Zitang Zhou, Kunyuan Ding, Wenx- uan Peng, Haoqi Wang, Guangyao Chen, Bo Li, Yiyou Sun, Xuefeng Du, Kaiyang Zhou, Wayne Zhang, Dan Hendrycks, Yixuan Li, and Ziwei Liu. 2022. OpenOOD: Benchmarking generalized out-of-distribution detection. InNeurIPS. Curran As- sociates Inc., Red Hook, NY, USA, 1–14

    2022

  31. [31]

    J. Yang, K. Zhou, Y. Li, and Z. Liu. 2024. Generalized out-of-distribution detection: A survey.IJCV132 (2024), 5635–5662. doi:10.1007/s11263-024-02117-4

    doi:10.1007/s11263-024-02117-4 2024

  32. [32]

    recovery

    Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. 2021. Bar- low Twins: Self-supervised learning via redundancy reduction. InICML. PMLR, Online, 1–11. Supporting Information S1 Comparison against post-hoc baselines We reproduce the OpenOOD [30] CIFAR-10 protocol so that every detector runs on the same frozen ResNet-18 (CIFAR-10, accuracy:...

    2021

  33. [2021]

    InNeurIPS workshop

    On feature collapse and deep kernel learning for single forward pass uncertainty. InNeurIPS workshop. arXiv, Online

  34. [2022]

    doi:10.5555/3648699.3648741

    A simple approach to improve single-model deep uncertainty via distance- awareness.JMLR23 (2022), 1–63. doi:10.5555/3648699.3648741

    arXiv 2022