Pith. sign in

REVIEW 2 major objections 5 minor 27 references

A unifying Bayesian framework for adversarial robustness

T0 review · 2 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that adversarial robustness is Bayesian inference through a stochastic adversarial channel, uniting adversarial training, adversarial purification, and randomized smoothing as special cases—and extending robustness to full

desk verdict A useful Bayesian unification of AT, purification, and RS with a clean BAL-consistency proof; the proactive defense's Jensen surrogate needs tighter scrutiny before 'fully Bayesian' claims stick. read the letter →

arxiv 2510.09288 v2 pith:NE6SSVNS submitted 2025-10-10 stat.ML cs.LG

Pablo G. Arce , Roi Naveiro , David Ríos Insua This is my paper
classification stat.MLcs.LG MSC 62F1568T07
keywords adversarialrobustnessBayesianinferencetrainingpurificationrandomizedsmoothingstochasticchannelvariationalneuralnetworks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to give adversarial robustness a single statistical foundation. It models the attacker's perturbation as a random transformation, the 'adversarial channel', whose distribution encodes uncertainty about the adversary. From this channel it derives two defenses: a reactive one for test time (purification) and a proactive one for training time (generalized adversarial training). The paper shows that well-known defenses—adversarial training and randomized smoothing—are limiting cases of the framework, and that the proactive defense robustifies full Bayesian predictive distributions in both classification and regression. The payoff, if correct: defenses that were viewed as competing heuristics become two ends of one probabilistic design space, with explicit assumptions.

What carries the argument

The central object is the adversarial channel, a conditional probability distribution p(x'|x,θ) that describes how a clean input x becomes an observed corrupted input x' as a random function of the model parameters θ. In the reactive model, the channel is used at test time to infer the latent clean input; in the proactive model, the channel is integrated into the training likelihood, so the robust posterior is proportional to p(θ)∏ E_{x'|x_i,θ}[p(y_i|x'_i,θ)]. The channel's flexibility—fixed noise, probabilistic iterative attacks, mixtures of attacks, and learned neural generators—is what lets the framework interpolate between and subsume existing defenses.

What would settle it

Train the proactive defense on a small regression or classification problem where the true robust posterior can be computed by exhaustive MCMC, then compare the variational posterior from the Jensen-bound objective against the true posterior on predictive NLL under held-out attacks; if the variational posterior diverges materially or loses adversarial robustness relative to the true posterior, the unification claim for the proactive defense fails.

Watch

Extended reading notes

Core claim

The central claim is that adversarial robustness is best understood as inference through a stochastic adversarial channel p(x'|x,θ). Modeling the channel explicitly yields two Bayesian strategies: a reactive defense, which infers the latent clean input at test time and marginalizes over the model posterior, and a proactive defense, which integrates the channel into the training likelihood to produce a robust posterior. The paper proves that standard adversarial training is a deterministic, point-estimate limit of the proactive defense (Dirac-delta channel plus MAP estimation), and that randomized smoothing is recovered from the reactive defense under a Gaussian channel, MAP parameters, and a

Load-bearing premise

The load-bearing premise is that maximizing the Jensen lower bound on the robust ELBO produces a variational posterior close enough to the stated robust posterior that the defense is genuinely Bayesian; if that surrogate is loose, the 'fully Bayesian' label for the proactive defense does not follow.

Editorial extensions

If this is right

  • Adversarial training becomes a special case of the proactive defense, so stochastic or distributional training is a strict generalization of deterministic minimax training.
  • Randomized smoothing becomes a special case of the reactive defense, giving purification and smoothing a shared probabilistic interpretation.
  • The proactive method can protect full predictive distributions, not just point predictions, covering both classification and regression under a single variational objective.
  • Training against a distribution of attacks (mixture or learned channels) confers robustness against attack types not seen during training, a benefit beyond deterministic adversarial training.
  • The reactive defense exposes the oversmoothing and calibration trade-offs of purification, explaining why simplified purification can degrade clean accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the unification holds, one could design a continuum of defenses by tuning the channel: a single training procedure could sweep from deterministic adversarial training to randomized smoothing, letting practitioners choose the operating point on the robustness–clean-accuracy curve.
  • The use of a Jensen lower bound suggests a practical diagnostic: monitor the gap between the surrogate and an unbiased ELBO estimate during training; if the gap is large, the fitted variational posterior may not reflect the stated robust model, weakening the 'fully Bayesian' interpretation.
  • The framework implies a principled way to select the channel via Bayesian model comparison across candidate channel families, treating the adversary's strategy as a model parameter to be inferred rather than assumed.
  • Beyond prediction, the robust predictive distributions produced by the proactive defense could be plugged directly into decision-theoretic pipelines, making robust decisions rather than robust point forecasts.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes a Bayesian framework for adversarial robustness in which the adversary is modeled by a stochastic channel p(x'|x,θ). Two inference strategies are derived: a reactive defense applied at test time (adversarial purification) and a proactive defense applied during training (adversarial training). The paper proves that adversarial training and randomized smoothing arise as limiting cases, and that a previously proposed Bayesian adversarial learning sampler is probabilistically inconsistent. Empirical validation is carried out on MNIST classification and three regression datasets using Bayesian neural networks, with the proactive defense as the main focus. The central formal derivations are mostly sound, but the proactive defense replaces the stated ELBO with a Jensen lower bound whose tightness is never assessed; this is the key weakness in the paper's 'fully Bayesian' and empirical claims.

Significance. If the framework is taken as stated, it offers a useful conceptual unification of adversarial training, randomized smoothing, and adversarial purification under a single Bayesian formalism, and extends robustness to Bayesian predictive distributions in both classification and regression. The SM proofs—of the inconsistency of BAL, of the distinctness of reactive and proactive predictive distributions, and of the AT/RS limiting cases—are clear and, to my reading, correct. The empirical study is broad and includes both point metrics and proper scoring rules. However, the main load-bearing empirical objective is a Jensen lower bound rather than the actual ELBO; without an assessment of the gap, the empirical success cannot be attributed to the proposed robust posterior. This tempers the strength of the contribution and requires revision.

major comments (2)
  1. [§3.3, Eq. (7) and following] The paper states that the intractable inner log-expectation in the ELBO (7) is bypassed by Jensen's inequality, yielding a lower bound L̃, and that L̃ is then maximized. This surrogate is exactly the ELBO of a different model in which the adversarial latent x' is integrated out with its prior, i.e. stochastic data augmentation. The gap L − L̃ = E_qΣ_i[log E_{x'}p(y|x',θ) − E_{x'}log p(y|x',θ)] is never quantified. For neural likelihoods under adversarial perturbations this gap can be large. Since all proactive experiments in Tables 1–4 use this surrogate, the headline empirical claims validate the surrogate, not the robust posterior in Eq. (6). The authors should either use the SM F estimator for the true ELBO gradient, provide a bound or empirical estimate of the gap, or explicitly reframe the contribution as an approximate Bayesian method with a separate justification for the surrogate
  2. [Abstract & §4.2 (NN50) and §5] The abstract and conclusion describe the method as a 'fully Bayesian framework' and Section 4.2 calls NN50 'specification-free'. This overstates what is implemented. The proactive objective actually trained is a Jensen lower bound, not the posterior corresponding to Eq. (6); the reactive defense uses an empirical distribution in place of a learned generative model; and the 'learned adversarial channel' for NN50 is a GAN-style min-max adversary with its own specification and training procedure. The prose should carefully separate the formal framework from the implemented approximations, and avoid claiming that the empirical benefits arise specifically from Bayesian inference over the robust posterior.
minor comments (5)
  1. [§3.3, Eq. (5)] Equation (5) writes p(y_i|x_i,θ,ϕ) = ∫ p(y_i|x_i,θ)p(x'_i|x_i,θ) dx'_i. The first term in the integrand should be p(y_i|x'_i,θ), since y_i is stated to be conditionally independent of x_i given x'_i and θ. Equation (6) uses the correct form, so this is a typo, but it should be fixed.
  2. [§3.2] The text refers twice to 'the inner expectation in (12)' and 'inner expectation (12)', but no equation (12) appears in the paper; the intended reference seems to be Eq. (2). Please correct the cross-reference.
  3. [§4.2] The sentence 'NN50—trained exclusively against a learned NN adversary...' contradicts Section 4.1, where NN50 is described as using a 50/50 mix of adversarial and clean data. Please rephrase to avoid ambiguity.
  4. [§4.1 and Tables 1–4] The 'AT' baseline is implemented as a one-step PGD augmentation with a 50/50 clean/adversarial mix. This is not the standard multi-step adversarial training of Madry et al. (2018), and it performs unusually poorly (e.g., 0.14 accuracy under PGD on MNIST). Please either rename the baseline to 'one-step adversarial training' or use a stronger AT implementation, so the comparison with OS50 is interpreted fairly.
  5. [§3.1] Typo: 'within within the setting' should be 'within the setting'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: AT/RS recoveries are explicit special cases; the Jensen-bound training is an approximation, not a self-referential fit.

full rationale

The paper's central derivations are self-contained. The proactive defense starts from a model-based robust posterior (Eq. 6) and derives an ELBO (Eq. 7); the Jensen-lower-bound objective in Sec. 3.3 is a genuine (if unquantified) approximation of that ELBO, not a restatement of the target. The AT and RS recovery propositions in SM E and SM C explicitly impose simplifying channel and estimation choices (Dirac worst-case channel plus MAP; isotropic Gaussian channel plus uniform latent prior plus MAP) and then compute the resulting predictor; these are special-case derivations, not definitions chosen to mirror the known methods. The empirical comparisons are against fixed external attacks and baselines, with no fitted parameter relabeled as a prediction. Self-citations (Arce et al., 2025; Gallego et al., 2024; Insua et al., 2023) are contextual and not load-bearing for the framework's validity. The only notable rigor gap—the unexplored Jensen gap between Eq. (7) and the optimized surrogate—is an approximation error concern, not circularity.

Assumptions & free parameters 4 free parameters · 7 assumptions · 1 invented entities

The framework requires several modeling choices: a channel distribution, a generative model for covariates, a posterior approximation, and tractability assumptions. The paper states most of these explicitly, but the Jensen-bound surrogate, the offline approximation, and the hand-set channel parameters are load-bearing. The empirical results therefore validate the chosen instances of the framework, not the framework itself, and the paper does not quantify the gap between the true ELBO and the optimized lower bound.

free parameters (4)
  • epsilon (attack radius) = 1 for MIX training; 2 for evaluation
    The adversarial budget is fixed by hand and is the central threat-model parameter throughout the experiments.
  • channel noise standard deviation sigma = not stated in the main text; stored in the code repository
    The Gaussian noise added to one-step PGD and other channels controls the stochasticity of the channel and is chosen by hand.
  • MIX mixture weights = 0.4 identity; 0.2 for each of CE, CW, and entropy one-step adversaries
    The mixture channel probabilities are chosen a priori to create diversity, not derived from the framework.
  • reactive sample counts S and N = S=5 posterior samples, N=100 training points
    The empirical approximation of the reactive PPD uses these hand-set sample counts; they control cost and approximation quality.
assumptions (7)
  • domain assumption Conditional independence: y_j is independent of x'_j given x_j and theta, and x'_j is independent of phi given x_j and theta.
    Defines the reactive PPD in Eq. (1). If an adversary uses label information or otherwise violates this graph structure, the posterior is misspecified.
  • domain assumption The offline approximation p(theta, phi | x'_j, D) ≈ p(theta, phi | D).
    This decouples parameter and latent-input inference in Eq. (2). The paper calls it a key simplifying assumption, and it is necessary for tractability.
  • ad hoc to paper The empirical distribution of training data substitutes for the generative model p(x|phi).
    Makes Eqs. (3) and (4) feasible, but forces the reactive defense to store the training set and creates a biased plug-in ratio estimator in the online defense.
  • ad hoc to paper A Jensen lower bound on the ELBO is a valid surrogate objective for the proactive posterior.
    Section 3.3 maximizes L_tilde(psi) instead of the true ELBO L(psi). Unless the bound is tight, the learned variational posterior is not the VI optimum for the stated robust posterior.
  • standard math Posterior and adversarial channels are reparameterizable.
    Required for the unbiased stochastic-gradient estimator of L_tilde; not satisfied by discrete or black-box attack channels.
  • ad hoc to paper AT recovery requires a Dirac-delta worst-case channel and a MAP point estimate.
    SM E shows that this limit reproduces minimax AT, but only when a true worst-case adversary is available and uncertainty about the adversary is ignored.
  • ad hoc to paper RS recovery requires MAP estimates, a uniform prior over x, and an isotropic Gaussian channel.
    SM C forces p(x|x') to be N(x'; sigma^2 I), but the resulting smoothed classifier uses soft class probabilities under the MAP model, not the hard-label majority vote of Cohen et al.'s randomized smoothing.
invented entities (1)
  • stochastic adversarial channel p(x'|x, theta)
    purpose: Formalizes uncertainty about the adversary and is the central object integrated out by both reactive and proactive defenses.
    This is a statistical modeling construct, not an independently measured entity. Its form is chosen by the modeler (Gaussian noise, PGD+noise, mixture, learned NN), and the paper provides no external falsifiable handle for the channel itself.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A unifying Bayesian framework for adversarial robustness." pith.science (2026). https://pith.science/paper/NE6SSVNS

@misc{pith2026251009288,
  author       = {Pith},
  title        = {Pith review of: A unifying Bayesian framework for adversarial robustness},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NE6SSVNS}},
  note         = {Machine review of arXiv:2510.09288}
}
read the original abstract

The vulnerability of machine learning models to adversarial attacks remains a critical societal security challenge. Traditional defenses, such as adversarial training, typically robustify models by minimizing a worst-case loss. These deterministic approaches do not account for uncertainty in the adversary's attack. While stochastic defenses placing a probability distribution on the adversary exist, they often lack statistical rigor and fail to make explicit their underlying assumptions. To resolve these issues, we introduce a formal Bayesian framework that models adversarial uncertainty through a stochastic channel, articulating all probabilistic assumptions. This yields two robustification strategies: a proactive defense enacted during training, aligned with adversarial training, and a reactive defense enacted during operations, aligned with adversarial purification. Several state-of-the-art defenses can be recovered as limiting cases of our model. We empirically validate our methodology, showcasing the benefits of explicitly modeling adversarial uncertainty.

Figures

Figures reproduced from arXiv: 2510.09288 by the authors.

Figure 1
Figure 1. , depicting a standard training phase on clean data {(xi , yi)} N i=1 used to learn the posterior over ϕ and θ. At test time, the defense is enacted upon ob￾ϕ θ xi yi i = 1, . . . , N xj x ′ j yj j = 1, . . . , M [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Model for proactive defense, where the train [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Accuracy and NLL against PGD50 attack [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Accuracy and NLL against PGD+ attack. 4.2 Case Study: Classification Evaluating Defenses We evaluate our proposed methodology against strong white-box attacks on MNIST. Figures 3 and 4 present performance met￾rics against attack intensity for PGD and the multi￾objectiv…
Figure 6
Figure 6. Figure 6: shows results consistent with those reported in [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]
Figure 7
Figure 7. Figure 7: Accuracy and NLL against ENT attack. top outputs. In contrast, the MIX model shows clear resilience to these attacks, maintaining reliable predictions. Notably, after being attacked, the BL model’s probability mass becomes more dispersed across classes, reflecting degr…
Figure 8
Figure 8. Figure 8: Attacks to BL model [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]
Figure 9
Figure 9. Figure 9: Attacks to MIX model [PITH_FULL_IMAGE:figures/full_fig_p021_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

27 extracted references · 10 linked inside Pith

  1. [1]

    G., Naveiro, R., and Insua, D

    Arce, P. G., Naveiro, R., and Insua, D. R. (2025). Evasion attacks against bayesian predictive models. In Chiappa, S. and Magliacane, S., editors, Proceedings of the Forty-first Conference on Uncertainty in Artificial Intelligence , volume 286 of Proceedings of Machine Learning Research , pages 184--202. PMLR

    2025

  2. [2]

    Balaji, Y., Goldstein, T., and Hoffman, J. (2019). Instance adaptive adversarial training: Improved accuracy tradeoffs in neural nets. arXiv preprint arXiv:1910.08051

    arXiv 2019

  3. [3]

    M., Kucukelbir, A., and McAuliffe, J

    Blei, D. M., Kucukelbir, A., and McAuliffe, J. D. (2017). Variational inference: A review for statisticians. Journal of the American statistical Association , 112(518):859--877

    2017

  4. [4]

    Cai, Q.-Z., Du, M., Liu, C., and Song, D. (2018). Curriculum adversarial training. arXiv preprint arXiv:1805.04807

    arXiv 2018

  5. [5]

    Carreau, M., Naveiro, R., and Caballero, W. N. (2025). Poisoning bayesian inference via data deletion and replication. arXiv preprint arXiv:2503.04480

    arXiv 2025

  6. [6]

    Cohen, J., Rosenfeld, E., and Kolter, Z. (2019). Certified adversarial robustness via randomized smoothing. In international conference on machine learning , pages 1310--1320. PMLR

    2019

  7. [7]

    Cortez, P., Cerdeira, A., Almeida, F., Matos, T., and Reis, J. (2009). Wine Quality . UCI Machine Learning Repository

    2009

  8. [8]

    Dalvi, N., Domingos, P., Mausam, Sumit, S., and Verma, D. (2004). Adversarial classification . In Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , KDD '04, pages 99--108

    2004

Show all 27 references
  1. [9]

    De Palma, G., Kiani, B., and Lloyd, S. (2021). Adversarial robustness guarantees for random deep neural networks. In International Conference on Machine Learning , pages 2522--2534. PMLR

    2021

  2. [10]

    J., and Ong, Y.-S

    Dong, J., Qu, X., Wang, Z. J., and Ong, Y.-S. (2024). Enhancing adversarial robustness via uncertainty-aware distributional adversarial training. arXiv preprint arXiv:2411.02871

    2024 arXiv

  3. [11]

    G., Tsilivis, N., and Kempe, J

    Feng, Y., Rudner, T. G., Tsilivis, N., and Kempe, J. (2024). Attacking bayes: On the adversarial robustness of bayesian neural networks. arXiv preprint arXiv:2404.19640

    2024 arXiv

  4. [12]

    Gallego, V., Naveiro, R., and Insua, D. R. (2019). Reinforcement learning under threats. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 9939--9940

    2019

  5. [13]

    Gallego, V., Naveiro, R., Redondo, A., R \' os Insua, D., and Ruggeri, F. (2024). Protecting classifiers from attacks. Statistical Science , 39(3):449--468

    2024

  6. [14]

    J., Shlens, J., and Szegedy, C

    Goodfellow, I. J., Shlens, J., and Szegedy, C. (2014). Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572

    2014 arXiv

  7. [15]

    R., Naveiro, R., Gallego, V., and Poulos, J

    Insua, D. R., Naveiro, R., Gallego, V., and Poulos, J. (2023). Adversarial machine learning: Bayesian perspectives. Journal of the American Statistical Association , 118(543):2195--2206

    2023

  8. [16]

    Joseph, A., Melson, B., Rubisntein, B., and Tygar, J. (2019). Adversarial Machine Learning . Cambridge University Press

    2019

  9. [17]

    Kannan, H., Kurakin, A., and Goodfellow, I. (2018). Adversarial logit pairing. arXiv preprint arXiv:1803.06373

    2018 arXiv

  10. [18]

    LeCun, Y., Cortes, C., and Burges, C. (1998). THE MNIST DATABASE of handwritten digits . http://yann.lecun.com/exdb/mnist/

    1998

  11. [19]

    Lin, G., Li, C., Zhang, J., Tanaka, T., and Zhao, Q. (2024). Adversarial training on purification (atop): Advancing both robustness and generalization. arXiv preprint arXiv:2401.16352

    2024 arXiv

  12. [20]

    Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. (2018). Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations

    2018

  13. [21]

    Nie, W., Guo, B., Huang, Y., Xiao, C., Vahdat, A., and Anandkumar, A. (2022). Diffusion models for adversarial purification. arXiv preprint arXiv:2205.07460

    2022 arXiv

  14. [22]

    Papamakarios, G., Sterratt, D., and Murray, I. (2019). Sequential neural likelihood: Fast likelihood-free inference with autoregressive flows. In The 22nd international conference on artificial intelligence and statistics , pages 837--848. PMLR

    2019

  15. [23]

    and Xifara, A

    Tsanas, A. and Xifara, A. (2012). Energy efficiency. UCI Machine Learning Repository

    2012

  16. [24]

    and Kantarcioglu, M

    Vorobeichyk, Y. and Kantarcioglu, M. (2019). Adversarial Machine Learning . Morgan & Claypool

    2019

  17. [25]

    and Zhu, Z

    Ye, N. and Zhu, Z. (2018). Bayesian adversarial learning. In Proceedings of the 32nd International Conference on Neural Information Processing Systems , pages 6892--6901. Curran Associates Inc

    2018

  18. [26]

    Zhang, H., Yu, Y., Jiao, J., Xing, E., El Ghaoui, L., and Jordan, M. (2019). Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning , pages 7472--7482. PMLR

    2019

  19. [27]

    X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al

    Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. (2023). A survey of large language models. arXiv preprint arXiv:2303.18223 , 1(2)

    2023 arXiv

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.