REVIEW 5 cited by
Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks
read the original abstract
The point estimates of ReLU classification networks---arguably the most widely used neural network architecture---have been shown to yield arbitrarily high confidence far away from the training data. This architecture, in conjunction with a maximum a posteriori estimation scheme, is thus not calibrated nor robust. Approximate Bayesian inference has been empirically demonstrated to improve predictive uncertainty in neural networks, although the theoretical analysis of such Bayesian approximations is limited. We theoretically analyze approximate Gaussian distributions on the weights of ReLU networks and show that they fix the overconfidence problem. Furthermore, we show that even a simplistic, thus cheap, Bayesian approximation, also fixes these issues. This indicates that a sufficient condition for a calibrated uncertainty on a ReLU network is "to be a bit Bayesian". These theoretical results validate the usage of last-layer Bayesian approximation and motivate a range of a fidelity-cost trade-off. We further validate these findings empirically via various standard experiments using common deep ReLU networks and Laplace approximations.
Forward citations
Cited by 5 Pith papers
-
A 65 nm Multi-Modal Bayesian Inference Engine with 16.3 fJ/Sample Calibration-Free GRNG for Risk-Aware At-Home Skin Lesion Screening
A 65 nm compute-in-memory multimodal Bayesian inference engine with calibration-free GRNG achieves 1.4x equal-risk coverage, >1.5x perturbation robustness, 5.5x process-variation resilience, and 1.8% higher balanced a...
-
Covariance Last-Layer Ensembles: Function-Space Diversity for Efficient Uncertainty Quantification
Cov-LLE decorrelates member activations in function space, recovering substantial deep-ensemble diversity and calibration at 1× backbone cost, and a direction score repairs OC near-OOD detection.
-
AGNFormer I: Reconstruction of AGN spectra using a probabilistic transformer model
An uncertainty-aware transformer reconstructs masked AGN broad lines and spectral halves with 4-16% flux errors and beats eleven purpose-built Lyα-reconstruction algorithms on a blind benchmark.
-
Low Rank Based Subspace Inference for the Laplace Approximation of Bayesian Neural Networks
Derives optimal low-rank subspace for Laplace approx in BNNs, provides scalable outperforming version, and new comparison metric.
-
A 65 nm Multi-Modal Bayesian Inference Engine with 16.3 fJ/Sample Calibration-Free GRNG for Risk-Aware At-Home Skin Lesion Screening
A 65 nm compute-in-memory chip implements multi-modal Bayesian neural networks with a calibration-free GRNG to deliver risk-aware skin lesion screening with reported gains in coverage, robustness, and efficiency over ...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.