Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Models

Dinh Phung; Ngoc-Quan Pham; Quyen Tran; Tan Nguyen; Trung Le; Tuan Truong

REVIEW 3 major objections 6 minor 80 references

Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Models

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that Bayesian fine-tuning of foundation models can be improved by making the K sampled model particles interact during training rather than drawing them independently from the posterior, and that adding a divergence loss…

desk verdict IBDR's empirical recipe is plausible and worth trying, but the theoretical guarantee does not survive contact with Eq. (5) and the KL formula in Corollary 4.3 is wrong. read the letter →

arxiv 2506.07247 v1 pith:FNTBNMJX submitted 2025-06-08 cs.LG

Ngoc-Quan Pham , Tuan Truong , Quyen Tran , Tan Nguyen , Dinh Phung , Trung Le This is my paper

classification cs.LG

keywords Bayesiandeeplearningensemblediversitydistributionalrobustnessparameter-efficientfine-tuningLoRAsharpness-awareminimizationparticlesamplingfoundationmodel

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that Bayesian fine-tuning of foundation models can be improved by making the sampled model particles interact during training instead of drawing them independently from a posterior. It introduces IBDR, a framework in which $K$ particles are drawn from a joint distribution $Q^K$, trained with a divergence loss that spreads their non-maximal predictions, and kept robust through a Wasserstein distributional-robustness term. Empirically, four such particles beat every compared baseline on the VTAB-1K image benchmark and on the commonsense reasoning tasks, while also giving the best expected-calibration error in most settings. This matters because it suggests a cheap way to get better accuracy and calibration from an ensemble of fine-tuned models without changing inference.

What carries the argument

The mechanism is the joint approximate posterior $Q^K = Q \odot \cdots \odot Q$ over $K$ particles together with a divergence loss $l_{\mathrm{div}}$ defined as the determinant of the Gram matrix of normalized non-maximal prediction vectors, so that maximizing the determinant maximizes the volume those vectors span. Theorem 4.1 is the load-bearing identity: it upper-bounds the population loss $L_D(Q^K)$ by $\min_{\lambda\ge 0}\{\lambda\rho + \mathbb{E}_{\theta\sim Q^K}[\max_{\theta'}\{L_S(\theta')-\lambda c_K(\theta,\theta')\}]\}$ plus a term $L\sqrt{(K\,D_{KL}(Q\|P)+\log(1/\delta))/(2N)}$, connecting Wasserstein distributional robustness, sharpness-aware minimization, and the KL divergence between approximate and prior. The practical algorithm replaces the joint maximizer with independent per-particle ascent steps and a fixed $\beta$ regularizer; the paper calls this a minor relaxation.

What would settle it

On a fixed VTAB-1K task, train IBDR and estimate the held-out population loss $L_D(Q^K)$; then evaluate the right-hand side of Theorem 4.1 using the same $\rho$, $\alpha$, $\beta$, and $\sigma$ used in training. If the held-out loss exceeds the bound's right-hand side, then the relaxed objective in Eq. (5) is not covered by the theorem.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that a Bayesian ensemble's quality is limited by the lack of explicit interaction between particles, and that this interaction can be added in a principled way. The approximate posterior is defined over the product space $\Theta^K$ of $K$ models, with loss $\ell(\theta_{1:K};x,y)=\frac{1}{K}\sum_i l(\theta_i;x,y)+\alpha\,l_{\mathrm{div}}(\theta_{1:K};x,y)$, where $l_{\mathrm{div}}$ maximizes the volume spanned by normalized non-maximal prediction vectors across particles, written as a determinant in the spirit of determinantal point processes. Theorem 4.1 bounds the population loss $L_D(Q^K)$ by a Wasserstein-DRO adversarial term plus a KL-divergence term, and the resulting algorithm alternates a SAM-style ascent to adversarial $\theta'_i$ with gradient descent on the particle means $\mu_i$ and the robustness radius $\lambda$. The claim is that this simultaneously enforces low loss, low sharpness, and diversity, and the experiments report consistent gains, for example 73.6% average on VTAB-1K versus 70.9% for SVGD, and 79.66% versus 78.75% for BLoB on commonsense reasoning.

Load-bearing premise

The load-bearing premise is that the theoretical guarantee proven for a joint adversarial objective over all particles still holds for the implemented objective, which replaces that joint step with independent per-particle updates and a fixed regularizer; no proof in the paper shows the bound applies to the objective actually minimized.

Editorial extensions

If this is right

If the reported numbers hold, adding IBDR's interaction term to LoRA-based fine-tuning gives a consistent accuracy lift over plain LoRA, SAM, SA-BNN, SGLD, Deep Ensembles, BayesTune, and SVGD on small-data vision tasks with only one thousand training examples.
Because particles are sampled independently at inference, the diversity cost is paid only during training; the deployed ensemble costs no more than a standard deep ensemble.
If Theorem 4.1 covers the relaxed objective, the method combines a distributional-robustness guarantee with a diversity mechanism, making sharpness-aware Bayesian methods a special case without interactions.
The same four-particle recipe transfers from vision to language, from ViT-B/16 to LLaMA-2-7B, so the interaction term is not architecture-specific.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A direct way to test the paper's mechanism is to measure pairwise prediction disagreement across $\alpha$ values; the volume-based argument predicts that disagreement among non-maximal predictions, not just accuracy, should rise with $\alpha$ up to the stability limit.
If the relaxation gap between Corollary 4.3 and Eq. (5) were closed with a proof, the empirical gains would become certified; that is the main theoretical step the paper leaves open.
The divergence loss uses only predictor probabilities on the training batch, so one could plausibly replace the DPP determinant with other diversity measures, such as cosine repulsion or Hilbert-Schmidt independence, and check whether the same accuracy-calibration trade-off appears.
The paper's out-of-distribution experiments suggest an implicit consequence: the diversity-forcing term may act as an OOD detector by keeping particles' non-maximal predictions spread rather than confidently aligned on spurious classes.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The short version: IBDR is a useful empirical recipe — DPP-based diversity on non-maximal predictions, SAM-style adversarial perturbation, and a mixture-of-Gaussians posterior over four LoRA particles — and the gains over strong baselines on VTAB-1K and commonsense reasoning are consistent, though modest. The paper is pleasant to read and the ablations are honest. But the theoretical claim, as written, does not hold up.

The step from Corollary 4.3 to Eq. (5) is labeled 'a minor relaxation' and it is not. Corollary 4.3 bounds the population loss by a quantity involving a joint maximization over the adversarial tuple θ'_{1:K}, with the DPP divergence term evaluated on those adversarial particles. Eq. (5) replaces that with a sum of independent per-particle maxima, evaluating the divergence on the original particles θ_{−i}. These are not equal and no inequality in either direction is given. If the Eq. (5) value falls below the Corollary bound, minimizing it does not minimize a proved upper bound. So the claimed distributional robustness guarantee does not apply to the implemented objective without an additional argument.

There is also a concrete error in the KL divergence formula in the proof of Corollary 4.3: for N(μ, σ²I) versus N(0,I) the correct KL involves d(σ² − 1 − 2 log σ), not d(σ − log σ). That looks like a typo rather than a conceptual break, but it needs fixing.

The empirical side is more solid. They report mean and std for IBDR but not for the baselines, which weakens the significance of the 2.7-point VTAB average gain. Still, the improvement pattern is consistent across 19 datasets and 6 reasoning tasks, and the ablation on α and ρ shows the method is not tearing at the seams.

Who should read it? Anyone building uncertainty-aware PEFT methods; there are good ideas here for how to decouple particle predictions. The theory section should be read as motivation, not proof. As a reviewer, I would send it back for major revision: fix the KL formula, either prove the relaxation or explicitly state that Eq. (5) is a heuristic and remove the guarantee language, and add error bars for baselines. Given the novelty and the breadth of experiments, it deserves serious referee time.

Best.

Referee Report

3 major / 6 minor

Summary. The paper proposes Interactive Bayesian Distributional Robustness (IBDR), a method for fine-tuning foundation models as an ensemble of K particles. It defines a joint loss over the product posterior Q^K that combines the average per-particle task loss with a DPP-based divergence loss, and uses Wasserstein distributionally robust optimization to derive Theorem 4.1, an upper bound on the population loss. Corollaries specialize the bound to a mixture-of-Gaussians posterior (Corollary 4.3), and Eq. (5) relaxes the bound into the training objective used in Algorithm 1. The method is evaluated on VTAB-1K with ViT-B/16 and on six commonsense-reasoning tasks with LLaMA-2, reporting accuracy and ECE improvements over baselines.

Significance. If the theoretical grounding held, IBDR would be a principled way to inject diversity into Bayesian ensembles while retaining a distributional-robustness guarantee, generalizing SA-BNN to product distributions. The empirical evaluation covers two distinct domains and shows consistent average improvements, and the ablation studies (Tables 4-7) provide useful sensitivity information. However, the connection between the proved bound and the implemented objective is not established, and the reported KL divergence is wrong; these issues undermine the paper's central theoretical claim. The empirical claims also lack error bars for baselines. The contribution is therefore best viewed as an empirically motivated ensemble-diversity heuristic whose theoretical framing requires substantial revision.

major comments (3)

[§4.3, Eq. (5); Appendix C.3] The 'minor relaxation' from Corollary 4.3 to Eq. (5) is not justified. Corollary 4.3 bounds L_D(Q^K) by an expression containing E_{θ1:K∼Q}[ max_{θ'_{1:K}} { (1/K)Σ_i l(θ'_i) + α l_div(θ'_{1:K}) − (λ/K)Σ_i c(θ_i,θ'_i) } ]. Eq. (5) replaces this joint maximization with (1/K) Σ_i max_{θ'_i} [ l(θ'_i) + α l_div(θ'_i, θ_{−i}) − λ c(θ_i,θ'_i) ], where the DPP determinant in l_div is evaluated with the original particles θ_{−i} rather than the adversarial θ'_{−i}. The two quantities are not equal in general, and no inequality in either direction is supplied. Consequently, Theorem 4.1 does not provide an upper bound on the objective actually minimized by IBDR, and the stated distributional-robustness guarantee does not apply to the implemented algorithm. This is the load-bearing step connecting the theory to the method, so it must be repaired or the theory must be explicitly repositioned as heuristic motivation.
[Appendix C.3, Corollary 4.3] The KL divergence formula is incorrect. For Q = N(μ, σ²I) and P = N(0,I), D_KL(Q∥P) = 1/2(∥μ∥² + d(σ² − 1 − 2 log σ)), not 1/2(∥μ∥² + dσ − d log σ) as stated in the proof of Corollary 4.3. The displayed expression is dimensionally inconsistent (σ and σ² appear in the same sum) and omits the −d and 2 log σ terms. This error propagates to the bound in Corollary 4.3 and to the regularizer in Eq. (5). If β is a purely tunable hyperparameter, the paper should state that explicitly and stop presenting the regularizer as derived from the PAC-Bayes KL term.
[§5.1, Tables 1 and 3] The empirical claim that 'IBDR outperforms all baselines by large margins' is not supported by the reported variability. In Table 1, standard deviations are given only for IBDR (in parentheses), while no uncertainty is reported for any baseline; Table 3 reports no error bars for any method. Several head-to-head differences are small relative to typical run-to-run noise (e.g., DTD: 71.7 for IBDR vs 71.4 for SAM; ARC-E in Table 3: 86.95 vs 86.68 for BLoB). The authors should provide error bars over multiple seeds for all methods, or a paired significance test, before claiming a large-margin improvement.

minor comments (6)

[Algorithm 1] In Algorithm 1, the update line reads 'Compute µ_i ← λ − α_µ ∇_{µ_i} L(...)'; this should almost certainly be 'µ_i ← µ_i − α_µ ∇_{µ_i} L(...)'. Please fix the typo.
[§5.1, Table 2] The BayesTune ECE for Caltech101 is reported as 0.93, which is an order of magnitude larger than the other entries in that column (around 0.08–0.25). Please verify this value; it may be a decimal-point error (0.093).
[§5.1] The text lists 'AdamW' among the baselines, but Table 1 has no row labeled AdamW; presumably LoRA trained with AdamW is intended. Please clarify the naming.
[Eq. (5)] In Eq. (5), the expectation E_{θ1:K∼Q} is placed outside the sum of per-particle maxima, but each term in the sum depends on θ_{−i} inside l_div(θ'_i, θ_{−i}). Please clarify the order of sampling and maximization, and whether θ_{−i} is held fixed when optimizing over θ'_i.
[Abstract and §5.2] The abstract refers to 'the common reasoning language task', while the body uses 'commonsense reasoning'; please unify the terminology.
[General] No code repository or reproducibility statement is provided. For a journal submission, I would expect either a link to the code or a detailed statement of the computational environment and hyperparameter search.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: IBDR's theoretical bound is adapted from external PAC-Bayes and Wasserstein-DRO results, its objectives are not fitted to the reported test numbers, and self-citations are related work rather than load-bearing premises; the unproved Corollary 4.3-to-Eq.(5) relaxation is a soundness gap, not a circular reduction.

full rationale

The claimed derivation chain is not equivalent to its inputs by construction. Theorem 4.1 (Sec. 4.2; Appendix C.1) is proved by combining the external PAC-Bayes bound of Alquier et al. (2016) with Wasserstein-DRO duality from Blanchet & Murthy (2019); the KL-decomposition step and the mixture-of-Gaussians KL estimate in Corollary 4.3 are standard computations, and the paper does not define the posterior or loss in terms of the VTAB-1K or commonsense accuracies it reports. The empirical claims are evaluated on external benchmarks (VTAB-1K, six commonsense tasks) against external baselines, so no fitted parameter is renamed as a prediction. Self-citations to SA-BNN (Nguyen et al., 2023b) and Truong et al. (2025b) appear in related-work context and as background for the 'special case' remark; they are not the support for Theorem 4.1. One genuine rigor concern is flagged: Section 4.3 says 'we apply a minor relaxation to the right-hand side' when moving from Corollary 4.3 to Eq. (5), replacing the joint maximization over θ'_{1:K} by per-particle maximizations with ldiv(θ'_i, θ_{-i}) and replacing the KL square-root term by β(Σ||μ_i||²+d(σ-logσ)); no inequality is proved to show Eq. (5) still upper-bounds L_D(Q^K). This is a correctness/justification gap, not a circular definition or a fitted-value-as-prediction, so under the hard rules it does not raise the circularity score.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

No new physical or ontological entities are introduced. The method modifies the training objective and posterior parameterization, but these are standard ML constructs (a loss term and a variational distribution), not newly postulated entities.

free parameters (5)

alpha (diversity trade-off) = 0.02
Scales the divergence loss l_div in Eq. (5); set to 0.02 for all datasets based on validation, reported in Section 5.1.
beta (regularization coefficient) = 1e-4
Multiplies the norm-based regularizer in Eq. (5); chosen by hand and fixed across datasets.
rho (Wasserstein ball radius / SAM step size) = tuned from {0.01, 0.05, 0.1} (main) or {0.01, 0.03, 0.05} (Appendix B)
Controls the adversarial perturbation size; tuned per dataset using the validation set (Section 5.1).
sigma (posterior standard deviation) = 0.1
Fixed for all particles; not learned (Section 4.3).
K (number of particles) = 4
Chosen as a trade-off between accuracy and runtime based on ablation (Appendix A.1).

assumptions (5)

domain assumption The loss function ℓ(θ;x,y) is bounded above by L > 0.
Required for the PAC-Bayes bound in Theorem 4.1; not proven, but CE loss with softmax and the bounded DPP determinant make it plausible.
standard math The Wasserstein DRO dual of Blanchet & Murthy (2019) applies to Q^K with cost c_K.
The paper invokes this dual without verifying upper semi-continuity and integrability of L_S(θ') (Appendix C.1).
ad hoc to paper The posterior Q is a mixture of K isotropic Gaussians N(μ_i, σ²I).
A modeling choice to make reparameterization and the KL term tractable; no justification for this family (Section 4.3).
ad hoc to paper The DPP-based divergence loss l_div on non-maximal prediction probabilities promotes useful ensemble diversity.
Adopted from Pang et al. (2019); the link between maximizing the spanned volume and ensemble accuracy is empirical, not proven (Section 4.3).
ad hoc to paper The relaxation in Eq. (5) preserves the theoretical bound's role as a training objective.
Called 'a minor relaxation'; no proof that per-particle adversarial maximization and fixed beta maintain the upper bound (Section 4.3).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Models." pith.science (2026). https://pith.science/paper/FNTBNMJX

@misc{pith2026250607247,
  author       = {Pith},
  title        = {Pith review of: Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FNTBNMJX}},
  note         = {Machine review of arXiv:2506.07247}
}

read the original abstract

We introduce Interactive Bayesian Distributional Robustness (IBDR), a novel Bayesian inference framework that allows modeling the interactions between particles, thereby enhancing ensemble quality through increased particle diversity. IBDR is grounded in a generalized theoretical framework that connects the distributional population loss with the approximate posterior, motivating a practical dual optimization procedure that enforces distributional robustness while fostering particle diversity. We evaluate IBDR's performance against various baseline methods using the VTAB-1K benchmark and the common reasoning language task. The results consistently show that IBDR outperforms these baselines, underscoring its effectiveness in real-world applications.

Figures

Figures reproduced from arXiv: 2506.07247 by the authors.

**Figure 1.** Percentage of OOD samples at different thresholds. Left: model trained on CIFAR-100 and tested on SVHN. Right: trained and tested on SVHN. diversity, these repulsive forces may compromise model robustness. However, our approach mitigates this issue by incorporating distributional robustness into the interactive framework. To evaluate the robustness of IBDR, we report the Expected Calibration Error (ECE) in [PITH_F… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

80 extracted references · 59 canonical work pages

[1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
[2]

Sharp-maml: Sharpness-aware model-agnostic meta learning

Abbas, M., Xiao, Q., Chen, L., Chen, P.-Y., and Chen, T. Sharp-maml: Sharpness-aware model-agnostic meta learning. In International conference on machine learning, pp.\ 10--32. PMLR, 2022

work page 2022
[3]

J., and Mandt, S

Alexos, A., Boyd, A. J., and Mandt, S. Structured stochastic gradient mcmc. In International Conference on Machine Learning, pp.\ 414--434. PMLR, 2022

work page 2022
[4]

On the properties of variational approximations of gibbs posteriors

Alquier, P., Ridgway, J., and Chopin, N. On the properties of variational approximations of gibbs posteriors. The Journal of Machine Learning Research, 17 0 (1): 0 8374--8414, 2016

work page 2016
[5]

Sharpness-aware minimization improves language model generalization

Bahri, D., Mobahi, H., and Tay, Y. Sharpness-aware minimization improves language model generalization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 7360--7371, 2022

work page 2022
[6]

Bernstein, D. S. Matrix mathematics: theory, facts, and formulas. Princeton university press, 2009

work page 2009
[7]

and Murthy, K

Blanchet, J. and Murthy, K. Quantifying distributional model risk via optimal transport. Mathematics of Operations Research, 44 0 (2): 0 565--600, 2019

work page 2019
[8]

Robust wasserstein profile inference and applications to machine learning

Blanchet, J., Kang, Y., and Murthy, K. Robust wasserstein profile inference and applications to machine learning. Journal of Applied Probability, 56 0 (3): 0 830--857, 2019

work page 2019

Show all 80 references

[9]

Weight uncertainty in neural network

Blundell, C., Cornebise, J., Kavukcuoglu, K., and Wierstra, D. Weight uncertainty in neural network. In International conference on machine learning, pp.\ 1613--1622. PMLR, 2015

2015
[10]

Improving generalization in federated learning by seeking flat minima

Caldarola, D., Caputo, B., and Ciccone, M. Improving generalization in federated learning by seeking flat minima. In European Conference on Computer Vision, pp.\ 654--672. Springer, 2022

2022
[11]

Variational inference with continuously-indexed normalizing flows

Caterini, A., Cornish, R., Sejdinovic, D., and Doucet, A. Variational inference with continuously-indexed normalizing flows. In Uncertainty in Artificial Intelligence, pp.\ 44--53. PMLR, 2021

2021
[12]

Swad: Domain generalization by seeking flat minima

Cha, J., Chun, S., Lee, K., Cho, H.-C., Park, S., Lee, Y., and Park, S. Swad: Domain generalization by seeking flat minima. Advances in Neural Information Processing Systems, 34: 0 22405--22418, 2021

2021
[13]

Stochastic gradient hamiltonian monte carlo

Chen, T., Fox, E., and Guestrin, C. Stochastic gradient hamiltonian monte carlo. In International conference on machine learning, pp.\ 1683--1691. PMLR, 2014

2014
[14]

When vision transformers outperform resnets without pre-training or strong data augmentations

Chen, X., Hsieh, C.-J., and Gong, B. When vision transformers outperform resnets without pre-training or strong data augmentations. In International Conference on Learning Representations, 2022

2022
[15]

Boolq: Exploring the surprising difficulty of natural yes/no questions

Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human...

2019
[16]

Think you have solved question answering? try arc, the ai2 reasoning challenge

Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

2018 arXiv
[17]

Imagenet: A large-scale hierarchical image database

Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.\ 248--255. Ieee, 2009

2009
[18]

Sharp minima can generalize for deep nets

Dinh, L., Pascanu, R., Bengio, S., and Bengio, Y. Sharp minima can generalize for deep nets. In International Conference on Machine Learning, pp.\ 1019--1028. PMLR, 2017

2017
[19]

G., Shamsi, A., Guo, X.-Y., Mohammadi, A., Alinejad-Rokny, H., Sejdinovic, D., Teney, D., Ranasinghe, D

Doan, B. G., Shamsi, A., Guo, X.-Y., Mohammadi, A., Alinejad-Rokny, H., Sejdinovic, D., Teney, D., Ranasinghe, D. C., and Abbasnejad, E. Bayesian low-rank learning (bella): A practical approach to bayesian neural networks. In Proceedings of the AAAI Conference on Artificial In...

2025
[20]

An image is worth 16x16 words: Transformers for image recognition at scale

Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020

2020
[21]

Efficient and scalable bayesian neural nets with rank-1 factors

Dusenberry, M., Jerfel, G., Wen, Y., Ma, Y., Snoek, J., Heller, K., Lakshminarayanan, B., and Tran, D. Efficient and scalable bayesian neural nets with rank-1 factors. In International conference on machine learning, pp.\ 2782--2792. PMLR, 2020

2020
[22]

Kronecker-factored approximate curvature for modern neural network architectures

Eschenhagen, R., Immer, A., Turner, R., Schneider, F., and Hennig, P. Kronecker-factored approximate curvature for modern neural network architectures. Advances in Neural Information Processing Systems, 36: 0 33624--33655, 2023

2023
[23]

Sharpness-aware minimization for efficiently improving generalization

Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations, 2021

2021
[24]

and Ganguli, S

Fort, S. and Ganguli, S. Emergent properties of the local geometry of neural loss landscapes. arXiv preprint arXiv:1910.05929, 2019

1910 arXiv
[25]

and Ghahramani, Z

Gal, Y. and Ghahramani, Z. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp.\ 1050--1059. PMLR, 2016

2016
[26]

and Kleywegt, A

Gao, R. and Kleywegt, A. Distributionally robust stochastic optimization with wasserstein distance. Mathematics of Operations Research, 48 0 (2): 0 603--655, 2023

2023
[27]

Gao, R., Chen, X., and Kleywegt, A. J. Wasserstein distributional robustness and regularization in statistical learning. arXiv e-prints, pp.\ arXiv--1712, 2017

2017
[28]

Structured variational learning of bayesian neural networks with horseshoe priors

Ghosh, S., Yao, J., and Doshi-Velez, F. Structured variational learning of bayesian neural networks with horseshoe priors. In International Conference on Machine Learning, pp.\ 1744--1753. PMLR, 2018

2018
[29]

Practical variational inference for neural networks

Graves, A. Practical variational inference for neural networks. Advances in neural information processing systems, 24, 2011

2011
[30]

Gupta, A. K. and Nagar, D. K. Matrix variate distributions. Chapman and Hall/CRC, 2018

2018
[31]

and Schmidhuber, J

Hochreiter, S. and Schmidhuber, J. Simplifying neural nets by discovering flat minima. Advances in neural information processing systems, 7, 1994

1994
[32]

Parameter-efficient transfer learning for nlp

Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pp.\ 2790--2799. PMLR, 2019

2019
[33]

J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al

Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al. Lora: Low-rank adaptation of large language models. ICLR, 1 0 (2): 0 3, 2022

2022
[34]

Hu, Z., Wang, L., Lan, Y., Xu, W., Lim, E.-P., Bing, L., Xu, X., Poria, S., and Lee, R. K.-W. Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. arXiv preprint arXiv:2304.01933, 2023

2023 arXiv
[35]

P., and Wilson, A

Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D. P., and Wilson, A. G. Averaging weights leads to wider optima and better generalization. In UAI , pp.\ 876--885. AUAI Press, 2018

2018
[36]

Jastrzebski, S., Kenton, Z., Arpit, D., Ballas, N., Fischer, A., Bengio, Y., and Storkey, A. J. Three factors influencing minima in sgd. ArXiv, abs/1711.04623, 2017

2017 arXiv
[37]

Fantastic generalization measures and where to find them

Jiang, Y., Neyshabur, B., Mobahi, H., Krishnan, D., and Bengio, S. Fantastic generalization measures and where to find them. In ICLR . OpenReview.net, 2020

2020
[38]

S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang, P

Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang, P. T. P. On large-batch training for deep learning: Generalization gap and sharp minima. In ICLR . OpenReview.net, 2017

2017
[39]

and Hospedales, T

Kim, M. and Hospedales, T. Bayestune: Bayesian sparse deep model fine-tuning. Advances in Neural Information Processing Systems, 36: 0 65317--65365, 2023

2023
[40]

Kingma, D. P. and Welling, M. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013

2013 arXiv
[41]

P., Salimans, T., and Welling, M

Kingma, D. P., Salimans, T., and Welling, M. Variational dropout and the local reparameterization trick. Advances in neural information processing systems, 28, 2015

2015
[42]

Determinantal point processes for machine learning

Kulesza, A., Taskar, B., et al. Determinantal point processes for machine learning. Foundations and Trends in Machine Learning , 5 0 (2--3): 0 123--286, 2012

2012
[43]

Simple and scalable predictive uncertainty estimation using deep ensembles

Lakshminarayanan, B., Pritzel, A., and Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017

2017
[44]

The power of scale for parameter-efficient prompt tuning

Lester, B., Al-Rfou, R., and Constant, N. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp.\ 3045--3059, 2021

2021
[45]

and Wang, D

Liu, Q. and Wang, D. Stein variational gradient descent: A general purpose bayesian inference algorithm. Advances in neural information processing systems, 29, 2016

2016
[46]

F., Cheng, K.-T., and Chen, M.-H

Liu, S.-Y., Wang, C.-Y., Yin, H., Molchanov, P., Wang, Y.-C. F., Cheng, K.-T., and Chen, M.-H. Dora: Weight-decomposed low-rank adaptation, 2024. URL https://arxiv.org/abs/2402.09353

2024 arXiv
[47]

P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks

Liu, X., Ji, K., Fu, Y., Tam, W., Du, Z., Yang, Z., and Tang, J. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp.\ 61--68, 2022

2022
[48]

and Welling, M

Louizos, C. and Welling, M. Multiplicative normalizing flows for variational bayesian neural networks. In International Conference on Machine Learning, pp.\ 2218--2227. PMLR, 2017

2017
[49]

Can a suit of armor conduct electricity? a new dataset for open book question answering

Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789, 2018

2018 arXiv
[50]

Variational inference with gaussian score matching

Modi, C., Gower, R., Margossian, C., Yao, Y., Blei, D., and Saul, L. Variational inference with gaussian score matching. Advances in Neural Information Processing Systems, 36: 0 29935--29950, 2023

2023
[51]

and Khan, M

M \"o llenhoff, T. and Khan, M. E. SAM as an optimal relaxation of bayes. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=k4fevFqSQcX

2023
[52]

Neal, R. M. Bayesian Learning for Neural Networks. Springer-Verlag, Berlin, Heidelberg, 1996. ISBN 0387947248

1996
[53]

Exploring generalization in deep learning

Neyshabur, B., Bhojanapalli, S., McAllester, D., and Srebro, N. Exploring generalization in deep learning. Advances in neural information processing systems, 30, 2017

2017
[54]

Optimal transport model distributional robustness

Nguyen, V.-A., Le, T., Bui, A., Do, T.-T., and Phung, D. Optimal transport model distributional robustness. In Advances in Neural Information Processing Systems, 2023 a

2023
[55]

Flat seeking bayesian neural networks

Nguyen, V.-A., Vuong, T.-L., Phan, H., Do, T.-T., Phung, D., and Le, T. Flat seeking bayesian neural networks. Advances in Neural Information Processing Systems, 2023 b

2023
[56]

Gaussian stochastic weight averaging for bayesian low-rank adaptation of large language models

Onal, E., Fl \"o ge, K., Caldwell, E., Sheverdin, A., and Fortuin, V. Gaussian stochastic weight averaging for bayesian low-rank adaptation of large language models. arXiv preprint arXiv:2405.03425, 2024

2024 arXiv
[57]

Improving adversarial robustness via promoting ensemble diversity

Pang, T., Xu, K., Du, C., Chen, N., and Zhu, J. Improving adversarial robustness via promoting ensemble diversity. In International Conference on Machine Learning, pp.\ 4970--4979, 2019

2019
[58]

Regularizing neural networks by penalizing confident output distributions

Pereyra, G., Tucker, G., Chorowski, J., Kaiser, ., and Hinton, G. Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv:1701.06548, 2017

2017 arXiv
[59]

Relative flatness and generalization

Petzka, H., Kamp, M., Adilova, L., Sminchisescu, C., and Boley, M. Relative flatness and generalization. In NeurIPS, pp.\ 18420--18432, 2021

2021
[60]

Generalized federated learning via sharpness aware minimization

Qu, Z., Li, X., Duan, R., Liu, Y., Tang, B., and Lu, Z. Generalized federated learning via sharpness aware minimization. In International conference on machine learning, pp.\ 18250--18280. PMLR, 2022

2022
[61]

Walsh-hadamard variational inference for bayesian deep learning

Rossi, S., Marmin, S., and Filippone, M. Walsh-hadamard variational inference for bayesian deep learning. Advances in Neural Information Processing Systems, 33: 0 9674--9686, 2020

2020
[62]

L., Bhagavatula, C., and Choi, Y

Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64 0 (9): 0 99--106, 2021

2021
[63]

Certifying some distributional robustness with principled adversarial training

Sinha, A., Namkoong, H., and Duchi, J. Certifying some distributional robustness with principled adversarial training. In International Conference on Learning Representations, 2018

2018
[64]

Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks

Sung, Y.-L., Cho, J., and Bansal, M. Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 5227--5237, 2022

2022
[65]

Llama 2: Open foundation and fine-tuned chat models, 2023

Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Couairon, R., and Authors, M. Llama 2: Open foundation and fine-tuned chat models, 2023

2023
[66]

Replora: Reparameterizing low-rank adaptation via the perspective of mixture of experts

Truong, T., Nguyen, C., Nguyen, H., Le, M., Le, T., and Ho, N. Replora: Reparameterizing low-rank adaptation via the perspective of mixture of experts. In Proceedings of the 42st International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research, Van...

2025
[67]

Improving generalization with flat hilbert bayesian inference

Truong, T., Tran, Q., Pham-Ngoc, Q., Ho, N., Phung, D., and Le, T. Improving generalization with flat hilbert bayesian inference. In Proceedings of the 42st International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research, Vancouver, Canada, 2025 b . PMLR

2025
[68]

Optimal Transport: Old and New

Villani, C. Optimal Transport: Old and New. Grundlehren der mathematischen Wissenschaften. Springer Berlin Heidelberg, 2008. ISBN 9783540710493. URL https://books.google.com.vn/books?id=NZXiNAEACAAJ

2008
[69]

Sharpness-aware gradient matching for domain generalization

Wang, P., Zhang, Z., Lei, Z., and Zhang, L. Sharpness-aware gradient matching for domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 3769--3778, 2023

2023
[70]

Blob: Bayesian low-rank adaptation by backpropagation for large language models

Wang, Y., Shi, H., Han, L., Metaxas, D., and Wang, H. Blob: Bayesian low-rank adaptation by backpropagation for large language models. arXiv preprint arXiv:2406.11675, 2024

2024 arXiv
[71]

The implicit and explicit regularization effects of dropout

Wei, C., Kakade, S., and Ma, T. The implicit and explicit regularization effects of dropout. In International conference on machine learning, pp.\ 10181--10192. PMLR, 2020

2020
[72]

and Teh, Y

Welling, M. and Teh, Y. W. Bayesian learning via stochastic gradient L angevin dynamics. In Proceedings of the 28th International Conference on International Conference on Machine Learning, ICML'11, pp.\ 681–688, Madison, WI, USA, 2011 a . Omnipress. ISBN 9781450306195

2011
[73]

and Teh, Y

Welling, M. and Teh, Y. W. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pp.\ 681--688, 2011 b

2011
[74]

X., Robeyns, M., Wang, X., and Aitchison, L

Yang, A. X., Robeyns, M., Wang, X., and Aitchison, L. Bayesian low-rank adaptation for large language models. arXiv preprint arXiv:2308.13111, 2023

2023 arXiv
[75]

S., Neumann, M., Dosovitskiy, A., Beyer, L., Bachem, O., Tschannen, M., Michalski, M., Bousquet, O., Gelly, S., and Houlsby, N

Zhai, X., Puigcerver, J., Kolesnikov, A., Ruyssen, P., Riquelme, C., Lucic, M., Djolonga, J., Pinto, A. S., Neumann, M., Dosovitskiy, A., Beyer, L., Bachem, O., Tschannen, M., Michalski, M., Bousquet, O., Gelly, S., and Houlsby, N. A large-scale study of representation learnin...

2020 arXiv
[76]

Be your own teacher: Improve the performance of convolutional neural networks via self distillation

Zhang, L., Song, J., Gao, A., Chen, J., Bao, C., and Ma, K. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 3713--3722, 2019

2019
[77]

Llama-adapter: Efficient fine-tuning of large language models with zero-initialized attention

Zhang, R., Han, J., Liu, C., Zhou, A., Lu, P., Qiao, Y., Li, H., and Gao, P. Llama-adapter: Efficient fine-tuning of large language models with zero-initialized attention. In The Twelfth International Conference on Learning Representations, 2024

2024
[78]

Bayesian attention belief networks

Zhang, S., Fan, X., Chen, B., and Zhou, M. Bayesian attention belief networks. In International Conference on Machine Learning, pp.\ 12413--12426. PMLR, 2021

2021
[79]

Flatness-aware minimization for domain generalization

Zhang, X., Xu, R., Yu, H., Dong, Y., Tian, P., and Cui, P. Flatness-aware minimization for domain generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 5189--5202, 2023

2023
[80]

M., and Lu, H

Zhang, Y., Xiang, T., Hospedales, T. M., and Lu, H. Deep mutual learning. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 4320--4328, 2018

2018

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

[2] [2]

Sharp-maml: Sharpness-aware model-agnostic meta learning

Abbas, M., Xiao, Q., Chen, L., Chen, P.-Y., and Chen, T. Sharp-maml: Sharpness-aware model-agnostic meta learning. In International conference on machine learning, pp.\ 10--32. PMLR, 2022

work page 2022

[3] [3]

J., and Mandt, S

Alexos, A., Boyd, A. J., and Mandt, S. Structured stochastic gradient mcmc. In International Conference on Machine Learning, pp.\ 414--434. PMLR, 2022

work page 2022

[4] [4]

On the properties of variational approximations of gibbs posteriors

Alquier, P., Ridgway, J., and Chopin, N. On the properties of variational approximations of gibbs posteriors. The Journal of Machine Learning Research, 17 0 (1): 0 8374--8414, 2016

work page 2016

[5] [5]

Sharpness-aware minimization improves language model generalization

Bahri, D., Mobahi, H., and Tay, Y. Sharpness-aware minimization improves language model generalization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 7360--7371, 2022

work page 2022

[6] [6]

Bernstein, D. S. Matrix mathematics: theory, facts, and formulas. Princeton university press, 2009

work page 2009

[7] [7]

and Murthy, K

Blanchet, J. and Murthy, K. Quantifying distributional model risk via optimal transport. Mathematics of Operations Research, 44 0 (2): 0 565--600, 2019

work page 2019

[8] [8]

Robust wasserstein profile inference and applications to machine learning

Blanchet, J., Kang, Y., and Murthy, K. Robust wasserstein profile inference and applications to machine learning. Journal of Applied Probability, 56 0 (3): 0 830--857, 2019

work page 2019