REVIEW 3 major objections 5 minor 22 references
Robust image classification with multi-modal large language models
T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read A CLIP agreement check turns a 0% robust CIFAR-10 model into a 91.8% one.
desk verdict A simple, plausible rejection defense whose non-adaptive gains are real, but the adaptive-attack evaluation is underspecified and the worst-case claims are overstated. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the rejection score $R(x)$, an absolute difference between CLIP's highest text-image alignment score and its alignment score for the class predicted by the image classifier. It turns the defense into a semantic consistency check across two independently trained representation spaces: when the classifier's top class is not the class CLIP considers most aligned, Multi-Shield abstains. The same score also defines the adaptive attack's extra constraint $R(x')\le 0$, which is what makes the attacker's optimization harder.
What would settle it
Re-run the adaptive attack while explicitly enforcing R(x')≤0 at every optimization step and verify that the final adversarial candidates satisfy the constraint; if robust accuracy drops materially below the reported adaptive-attack numbers, the defense is weaker than claimed.
Extended reading notes
Core claim
The paper's central claim is that semantic consistency between an image classifier and a multimodal vision-language model is a powerful rejection signal. Multi-Shield decides by comparing the image classifier's predicted class with CLIP's best-aligned class prompt: if $R(x)=\left|\max_i h(x,P_i)-h(x,P_{f(x)})\right|$ is positive, the system abstains, where $h$ is CLIP's cosine similarity between the image and a textual prompt. Clean inputs tend to keep both models in agreement, while adversarial perturbations break the alignment, so abstention blocks most attacks. The paper further claims that this holds even against an adaptive attacker who knows the whole defense, because the attacker must simultaneously fool both models while keeping the rejection score non-positive.
Load-bearing premise
The adaptive-attack robustness numbers depend on the custom attack actually solving the constrained optimization it defines, including the rejection constraint R(x')≤0; the paper does not state how that constraint is enforced, so the reported values could be optimistic if the optimizer silently fails to satisfy it.
Editorial extensions
If this is right
- A non-robust CIFAR-10 model (C1) that AutoAttack defeats completely (0.0% robust accuracy) rises to 91.8% robust accuracy when Multi-Shield abstains on disagreement.
- Robust models gain on average 32 percentage points on CIFAR-10 and 65 on ImageNet under the non-adaptive attack, while clean accuracy falls only about 1.8% and 6% respectively.
- An adaptive attacker who knows the defense is still partially held back: the non-robust CIFAR-10 model retains 20.4% robust accuracy, and robust models gain 3.3% on CIFAR-10 and 5% on ImageNet on average.
- Relative to two existing rejection defenses, Multi-Shield achieves higher robust accuracy on CIFAR-10 and avoids training one-vs-all SVMs per class, making it cheaper to deploy on larger datasets.
- The rejection ratio rises with attack strength, meaning the defense increasingly abstains as perturbations grow larger.
Reading between the lines
- Because the defense depends on class names carrying semantic meaning, datasets with abstract or arbitrary labels (such as 'Class 1') would likely defeat it; automatically generating descriptive prompts for those classes is a natural extension the paper does not test.
- The paper notes that attacking either underlying CLIP model alone drives robust accuracy to 0%, so an adversary who optimizes directly against CLIP's alignment scores may open the rejection gate even when the image classifier is not fooled; this makes the defense's real-world strength partly dependent on the attacker's choice of objective.
- The same agreement score could be reused as a general uncertainty signal for selective prediction, flagging near-decision-boundary images for human review in non-adversarial settings; this use is left implicit in the paper.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Multi-Shield, a rejection-based defense that wraps any image classifier with a CLIP zero-shot classifier. At inference, it computes the agreement between the image classifier's prediction and CLIP's highest-scoring class (Eq. 6) and abstains when they disagree (Eq. 5). The authors evaluate Multi-Shield on CIFAR-10, ImageNet, Caltech-101, Food-101, Oxford-IIIT Pets, and STL-10, using both non-robust and RobustBench adversarially trained models. Under non-adaptive AutoAttack, they report large robust-accuracy gains (e.g., C1 from 0.0% to 91.8% on CIFAR-10), and they also report results for an 'adaptive variant' of AutoAttack that is formalized as the constrained program in Eqs. (8)-(11). The central claim is that Multi-Shield remains partially effective even under a worst-case adaptive attacker.
Significance. If the adaptive-robustness claims are substantiated, Multi-Shield is an attractive model-agnostic defense: it requires no retraining, it is simple to describe, and the clean-accuracy trade-off is modest. The paper has notable strengths: it uses standard RobustBench models, a standard attack (AutoAttack) at the conventional eps=8/255, it reports rejection ratios alongside robust accuracy, and it compares against two established rejection defenses (NR and DNR). The authors also honestly state a limitation (abstract class labels) and disclose in the Table I caption that attacking either CLIP model alone yields 0% robust accuracy. The main weakness is that the adaptive-attack evaluation is not described algorithmically, so the worst-case robustness claim rests on an underspecified experiment. The work is not circular: the method has no fitted parameters, and the rejection rule is a hard agreement check. The practical significance will depend on whether the adaptive evaluation can be reported transparently and reproduced.
major comments (3)
- [§III.C and §IV.A] Eqs. (8)-(11) define the adaptive attack as a constrained minimization with the rejection constraint R(x')<=0, but the paper never specifies how this constraint is enforced. Section IV.A only says that 'an adaptive variant, which takes into account Multi-Shield's rejection mechanism' is used; no loss function, projection step, Lagrange multiplier, or modification of AutoAttack's components (APGD, FAB, Square) is described. Without this information, the 'Multi-Shield - Adaptive Attack' column of Table I cannot be verified, and the conclusion that Multi-Shield 'remains effective even under worst-case adaptive attacks' is unsupported. The authors should provide the exact algorithm, including how Eq. (11) is handled during optimization and the fraction of final adversarial examples that actually satisfy R(x')<=0.
- [Table I caption and §III.C] The caption states that 'performing the attack on either of the underlying clip models yields a robust accuracy of 0%.' This shows that the CLIP component is itself highly vulnerable, so a truly adaptive attacker should be able to exploit CLIP and then only needs to align the image classifier to the same wrong class. The paper's adaptive attack is only described as a constrained program, and the reported adaptive robust accuracies (e.g., C1 20.4%) are consistent with the possibility that the optimizer fails to find points satisfying Eq. (11) rather than with a genuinely strong joint attack. The authors should demonstrate that the adaptive attack actually optimizes against both the image classifier and the CLIP model jointly, for example by reporting attack success rates on the CLIP component separately.
- [§IV.A, 'Evaluation Metrics' and Table I] Robust accuracy counts abstentions as successes. For the non-adaptive Multi-Shield row on C1, robust accuracy equals the rejection ratio (91.8%), so the headline improvement from 0.0% to 91.8% is entirely due to rejection rather than correct classification of adversarial inputs. This is a legitimate design choice for a rejection-based defense, but the paper should state it explicitly in the main text and report the accuracy of non-abstained inputs as a separate quantity. Currently the text says robust accuracy 'accounts for correct predictions and cases where it abstains,' which is easy to overlook and makes the reported improvements difficult to interpret.
minor comments (5)
- [§IV.A, 'Multishield Construction'] For CIFAR-10, the paper says a 'ViT-B visual encoder fine-tuned for CIFAR-10 classification [Tang et al., 2024]' is used, but it does not specify which CLIP checkpoint this comes from or which text encoder is paired with it. Please give the exact model identifiers so the experiments can be reproduced.
- [§IV.B and Figure 2] The text says 'in all three plots' when Figure 2 contains four subplots; also, the top two plots and bottom two plots differ in the models shown, which should be stated in the figure caption.
- [Table II] The lower part of Table II appears malformed: the row for C3 and C4 lacks the separate NR/DNR/Ours labels that appear for the other models, making it hard to read which defense corresponds to which number.
- [Eq. (6)] Since max_i h(x,P_i) >= h(x,P_j), the absolute value in Eq. (6) is redundant; if the authors intended to capture disagreement, the formula could be simplified or the sign convention clarified.
- [Section V] The limitation about abstract or meaningless labels is acknowledged, but the paper does not quantify how much robustness drops in that setting; a small experiment on a label-renamed dataset would strengthen the discussion.
Circularity Check
No circularity: Multi-Shield's rejection rule is a hard agreement check with no fitted parameters; the adaptive-attack implementation gap is an evaluation-transparency concern, not a circular derivation.
full rationale
Multi-Shield is defined by Eq. (5)-(7) as a hard agreement check between the image classifier's predicted class and the CLIP zero-shot top class, with a fixed threshold R(x)<=0. No parameter is learned from or fitted to the evaluation data, and no benchmark result is used to set the rejection rule. Under non-adaptive AutoAttack, the reported robust-accuracy gains for non-robust models are the direct and intended consequence of the attack targeting only the image classifier f while CLIP remains unperturbed; adversarial inputs then trigger R>0 and are abstained. The paper explicitly states that Robust Accuracy counts abstention as success ('accounting for correct predictions and cases where it abstains from making a prediction'), so this is a transparent metric definition for a rejection defense, not a hidden reduction of a predicted quantity to a fitted input. Comparisons with NR and DNR are external, and self-citations (Biggio and Roli 2018; Sotgiu et al. 2020; Cina et al. 2025) appear as background or baseline references, not as load-bearing justification of the defense's effectiveness. The genuine weakness is in Section III.C and IV.A: the adaptive attack is formulated with the constraint R(x')<=0 (Eq. 11), but the text never specifies how this constraint is enforced during AutoAttack, so the 'Multi-Shield - Adaptive Attack' column of Table I cannot be independently verified. That is an omitted-implementation/correctness-risk issue, not circularity: the defense itself does not reduce by construction to its inputs, and no fitted parameter is renamed as a prediction. The stated limitation about abstract or meaningless labels is likewise an acknowledged scope condition, not a circular step.
Assumptions & free parameters
assumptions (4)
- domain assumption CLIP zero-shot alignment scores h(x, P_i) are a valid measure of semantic agreement between an image and its class prompt.
- domain assumption The visual encoder used for CIFAR-10 (a ViT-B 'fine-tuned for CIFAR-10 classification' from FusionBench) and the HuggingFace text encoder produce embeddings in a shared space.
- ad hoc to paper The constrained optimization in Eq. (8)-(11) is solved effectively by the undisclosed adaptive AutoAttack variant.
- domain assumption Adversarial perturbations within epsilon=8/255 preserve enough semantic content that CLIP's alignment with the true class remains competitive.
Cite this review
Pith. "Pith review of Robust image classification with multi-modal large language models." pith.science (2026). https://pith.science/paper/76XNSF5S
@misc{pith2026241210353,
author = {Pith},
title = {Pith review of: Robust image classification with multi-modal large language models},
year = {2026},
howpublished = {\url{https://pith.science/paper/76XNSF5S}},
note = {Machine review of arXiv:2412.10353}
}
read the original abstract
Deep Neural Networks are vulnerable to adversarial examples, i.e., carefully crafted input samples that can cause models to make incorrect predictions with high confidence. To mitigate these vulnerabilities, adversarial training and detection-based defenses have been proposed to strengthen models in advance. However, most of these approaches focus on a single data modality, overlooking the relationships between visual patterns and textual descriptions of the input. In this paper, we propose a novel defense, MultiShield, designed to combine and complement these defenses with multi-modal information to further enhance their robustness. MultiShield leverages multi-modal large language models to detect adversarial examples and abstain from uncertain classifications when there is no alignment between textual and visual representations of the input. Extensive evaluations on CIFAR-10 and ImageNet datasets, using robust and non-robust image classification models, demonstrate that MultiShield can be easily integrated to detect and reject adversarial examples, outperforming the original defenses.
Figures
Reference graph
Works this paper leans on
-
[7]
Deep neural rejection against adver- sarial examples,
A. Sotgiu, A. Demontis, M. Melis, B. Biggio, G. Fumera, X. Feng, and F. Roli, “Deep neural rejection against adver- sarial examples,” EURASIP Journal on Information Secu- rity, vol. 2020, pp. 1–10,
work page 2020
-
[12]
Image-text retrieval: A survey on recent research and development,
M. Cao, S. Li, J. Li, L. Nie, and M. Zhang, “Image-text retrieval: A survey on recent research and development,” arXiv:2203.14713,
-
[14]
Visionllama: A unified llama interface for vision tasks,
X. Chu, J. Su, B. Zhang, and C. Shen, “Visionllama: A unified llama interface for vision tasks,” arXiv:2403.00522,
-
[15]
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems, 2017, pp. 5998–6008. N. Carlini and D. Wagner, “Adversarial examples are not easily detected: Bypassing ten detection methods,” in ACM workshop on artificial intelligence a...
arXiv 2017
-
[17]
An analysis of single-layer networks in unsupervised feature learning,
A. Coates, A. Ng, and H. Lee, “An analysis of single-layer networks in unsupervised feature learning,” in International conference on artificial intelligence and statistics , 2011, pp. 215–223. F. Croce, M. Andriushchenko, V . Sehwag, E. Debenedetti, N. Flammarion, M. Chiang, P. Mittal, and M. Hein, “Ro- bustbench: a standardized adversarial robustness be...
work page 2011
-
[18]
Improving robustness using generated data,
S. Gowal, S. Rebuffi, O. Wiles, F. Stimberg, D. A. Calian, and T. A. Mann, “Improving robustness using generated data,” in Advances in Neural Information Processing Systems , 2021, pp. 4218–4233. S. Addepalli, S. Jain, and V . B. R., “Efficient and effective augmentation strategy for adversarial training,” in Advances in Neural Information Processing Systems ,
work page 2021
-
[19]
Adversarial robustness: From self-supervised pre-training to fine-tuning,
T. Chen, S. Liu, S. Chang, Y . Cheng, L. Amini, and Z. Wang, “Adversarial robustness: From self-supervised pre-training to fine-tuning,” in IEEE Conference on Computer Vision and Pattern Recognition,, 2020, pp. 696–705. Y . Xu, Y . Sun, M. Goldblum, T. Goldstein, and F. Huang, “Exploring and exploiting decision boundary dynamics for adversarial robustness...
work page 2020
-
[20]
Fusionbench: A comprehensive benchmark of deep model fusion,
A. Tang, L. Shen, Y . Luo, H. Hu, B. Do, and D. Tao, “Fusionbench: A comprehensive benchmark of deep model fusion,” arXiv preprint arXiv:2406.03280 ,
Show all 22 references
-
[21]
Clipa-v2: Scaling CLIP training with 81.1% zero-shot imagenet accuracy within a $10, 000 budget; an extra $4, 000 unlocks 81.8% accuracy,
X. Li, Z. Wang, and C. Xie, “Clipa-v2: Scaling CLIP training with 81.1% zero-shot imagenet accuracy within a $10, 000 budget; an extra $4, 000 unlocks 81.8% accuracy,” arXiv, vol. abs/2306.15658,
-
[22]
Huggingface’s transformers: State-of-the-art natu- ral language processing,
T. Wolf, L. Debut, V . Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, and J. Brew, “Huggingface’s transformers: State-of-the-art natu- ral language processing,” arXiv, vol. abs/1910.03771, 2019
1910 arXiv
-
[2012]
Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories,
L. Fei-Fei, R. Fergus, and P. Perona, “Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories,” in Conference on computer vision and pattern recognition workshop, 2004, pp. 178–178. L. Bossard, M. Guillaum...
2004
-
[2014]
Towards evaluating the robustness of neural networks,
N. Carlini and D. Wagner, “Towards evaluating the robustness of neural networks,” in IEEE symposium on security and privacy, 2017, pp. 39–57. F. Croce and M. Hein, “Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free at- tacks,” in Internat...
2017
-
[2015]
Towards deep learning models resistant to adver- sarial attacks,
A. Madry, “Towards deep learning models resistant to adver- sarial attacks,” arXiv:1706.06083,
-
[2017]
Safetynet: Detecting and rejecting adversarial examples robustly,
J. Lu, T. Issaranon, and D. Forsyth, “Safetynet: Detecting and rejecting adversarial examples robustly,” in IEEE Interna- tional Conference on computer vision , 2017, pp. 446–454. M. Melis, A. Demontis, B. Biggio, G. Brown, G. Fumera, and F. Roli, “Is deep learning safe for ro...
2017
-
[2018]
A survey on multimodal large language models,
S. Yin, C. Fu, S. Zhao, K. Li, X. Sun, T. Xu, and E. Chen, “A survey on multimodal large language models,” arXiv:2306.13549,
-
[2019]
Learning transferable visual models from natural language supervision,
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International Conference on machine learn- ing, 2021, pp. 8748–8763. A. Madry, A. Ma...
2021
-
[2020]
Evasion attacks against 8 machine learning at test time,
B. Biggio, I. Corona, D. Maiorca, B. Nelson, N. Srndic, P. Laskov, G. Giacinto, and F. Roli, “Evasion attacks against 8 machine learning at test time,” in ECML PKDD, vol. 8190, 2013, pp. 387–402. C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. J. Goodfellow, and R...
2013
-
[2021]
Wild patterns: Ten years after the rise of adversarial machine learning,
B. Biggio and F. Roli, “Wild patterns: Ten years after the rise of adversarial machine learning,” in ACM SIGSAC Conf. on Computer and Communications Security , 2018, pp. 2154–
2018
-
[2022]
Visualbert: A simple and performant baseline for vision and language,
L. H. Li, M. Yatskar, D. Yin, C.-J. Hsieh, and K.-W. Chang, “Visualbert: A simple and performant baseline for vision and language,” arXiv:1908.03557,
1908 arXiv
-
[2023]
A survey of vision- language pre-trained models,
Y . Du, Z. Liu, J. Li, and W. X. Zhao, “A survey of vision- language pre-trained models,” arXiv:2202.10936,
-
[2024]
Towards open set deep networks,
A. Bendale and T. E. Boult, “Towards open set deep networks,” in IEEE Conf. on computer vision and pattern recognition , 2016, pp. 1563–1572. N. Papernot and P. McDaniel, “Deep k-nearest neighbors: Towards confident, interpretable and robust deep learning,” arXiv:1803.04765,
2016 arXiv
-
[2025]
Attackbench: Evaluating gradient-based attacks for adversarial examples,
A. E. Cin `a, J. Rony, M. Pintor, L. Demetrio, A. Demontis, B. Biggio, I. B. Ayed, and F. Roli, “Attackbench: Evaluating gradient-based attacks for adversarial examples,” in Pro- ceedings of the AAAI Conference on Artificial Intelligence , vol. 39, no. 3, 2025, pp. 2600–2608. ...
2025
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.