{"id":"0f03d3fb-1ec4-44b6-8e5f-8d7b96dc9bc8","arxiv_id":"2412.10353","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Multi-Shield rejects adversarial images by abstaining whenever a standard image classifier and a CLIP zero-shot classifier disagree, which raises robust accuracy noticeably under ordinary attacks and modestly under adaptive attacks.","lead":"Multi-Shield is an added safety layer that makes an image classifier refuse to answer when its prediction does not match a CLIP vision-language model's interpretation of the image. Tests on six datasets show it can block many adversarial attacks, at the cost of a small drop in accuracy on clean images.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Adaptive-attack evaluation is underspecified: no mechanism for enforcing R(x')≤0 in Eq. (11) is described, so the worst-case robustness claims are unsupported.","rationale":"The paper's proposed defense is a rejection layer: it abstains whenever the image classifier and CLIP disagree. The central validity question is whether the evaluation under an adaptive adversary is a faithful worst-case evaluation. The non-adaptive column is not load-bearing by itself, because any rejection mechanism trivially inflates robust accuracy when the attack is not optimized against the defense. The strongest claim in the paper explicitly says 'partially effective under a fully adaptive attacker,' so the adaptive column carries the weight. The paper provides no implementation details for the adaptive attack, particularly no explanation of how Eq. (11) is enforced. This is an internal reporting gap, not a disagreement with outside consensus, and it directly affects the soundness of the headline robustness numbers. The concrete test described above would settle whether the constraint is actually satisfied and whether the adaptive results are trustworthy. I find no independent evidence mitigating this gap: there is no released code, no formal verification, and the comparison in Table II is also based on the same underspecified adaptive attack. The reader's weakest assumption identified exactly this issue, and I agree with that assessment. My read does not change the reader's conditional verdict; it strengthens the need for the requested adaptive-attack details and reproducible artifacts before the strongest claims are accepted.","tokens_in":13446,"tokens_out":3905,"duration_ms":41821,"concrete_test":"Re-run Table I with a fully specified adaptive attack that solves Eq. (8)-(11), for example projected PGD on the combined objective L(x',y)+λR(x') with λ increased until every reported adversarial example satisfies R(x')≤0 exactly (or use an augmented-Lagrangian / Frank-Wolfe method). Report both the robust accuracy and the fraction of final examples satisfying the constraint. Also run a direct PGD attack on the CLIP encoder only (e.g., maximizing a wrong-class text-image alignment) and report robust accuracy; if C1's robust accuracy drops from 20.4% to near 0% under either test, the claim of partial effectiveness under adaptive attacks fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that Multi-Shield 'remains effective even under worst-case adaptive attacks' rests entirely on the 'Multi-Shield - Adaptive Attack' column of Table I. However, the paper never states how the constraint R(x')≤0 in Eq. (11) is enforced during optimization. Section IV.A only says 'an adaptive variant, which takes into account Multi-Shield's rejection mechanism'; no details are given about the loss function, projection step, Lagrange multiplier, or whether the final adversarial examples actually satisfy Eq. (11). Without this, the adaptive results could simply be standard AutoAttack on the image classifier followed by rejection of all examples that violate Eq. (11). That would not be an adaptive attack, and it would make the reported robust accuracies optimistic. The caption of Table I itself notes that performing the attack on either underlying CLIP model yields 0% robust accuracy, showing that the defense is bypassable if CLIP is fooled; a true adaptive attack should target CLIP jointly with the image classifier. Because the headline number '0% to 91.8%' is obtained under a non-adaptive AutoAttack that does not optimize against the defense, the only evidence for worst-case robustness is this underspecified adaptive attack. Thus the conclusion that Multi-Shield is partially effective under adaptive attacks is not yet established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Multi-Shield, a rejection-based defense that wraps any image classifier with a CLIP zero-shot classifier. At inference, it computes the agreement between the image classifier's prediction and CLIP's highest-scoring class (Eq. 6) and abstains when they disagree (Eq. 5). The authors evaluate Multi-Shield on CIFAR-10, ImageNet, Caltech-101, Food-101, Oxford-IIIT Pets, and STL-10, using both non-robust and RobustBench adversarially trained models. Under non-adaptive AutoAttack, they report large robust-accuracy gains (e.g., C1 from 0.0% to 91.8% on CIFAR-10), and they also report results for an 'adaptive variant' of AutoAttack that is formalized as the constrained program in Eqs. (8)-(11). The central claim is that Multi-Shield remains partially effective even under a worst-case adaptive attacker.","tokens_in":13692,"tokens_out":4161,"duration_ms":40850,"significance":"If the adaptive-robustness claims are substantiated, Multi-Shield is an attractive model-agnostic defense: it requires no retraining, it is simple to describe, and the clean-accuracy trade-off is modest. The paper has notable strengths: it uses standard RobustBench models, a standard attack (AutoAttack) at the conventional eps=8/255, it reports rejection ratios alongside robust accuracy, and it compares against two established rejection defenses (NR and DNR). The authors also honestly state a limitation (abstract class labels) and disclose in the Table I caption that attacking either CLIP model alone yields 0% robust accuracy. The main weakness is that the adaptive-attack evaluation is not described algorithmically, so the worst-case robustness claim rests on an underspecified experiment. The work is not circular: the method has no fitted parameters, and the rejection rule is a hard agreement check. The practical significance will depend on whether the adaptive evaluation can be reported transparently and reproduced.","major_comments":[{"comment":"Eqs. (8)-(11) define the adaptive attack as a constrained minimization with the rejection constraint R(x')<=0, but the paper never specifies how this constraint is enforced. Section IV.A only says that 'an adaptive variant, which takes into account Multi-Shield's rejection mechanism' is used; no loss function, projection step, Lagrange multiplier, or modification of AutoAttack's components (APGD, FAB, Square) is described. Without this information, the 'Multi-Shield - Adaptive Attack' column of Table I cannot be verified, and the conclusion that Multi-Shield 'remains effective even under worst-case adaptive attacks' is unsupported. The authors should provide the exact algorithm, including how Eq. (11) is handled during optimization and the fraction of final adversarial examples that actually satisfy R(x')<=0.","section":"§III.C and §IV.A"},{"comment":"The caption states that 'performing the attack on either of the underlying clip models yields a robust accuracy of 0%.' This shows that the CLIP component is itself highly vulnerable, so a truly adaptive attacker should be able to exploit CLIP and then only needs to align the image classifier to the same wrong class. The paper's adaptive attack is only described as a constrained program, and the reported adaptive robust accuracies (e.g., C1 20.4%) are consistent with the possibility that the optimizer fails to find points satisfying Eq. (11) rather than with a genuinely strong joint attack. The authors should demonstrate that the adaptive attack actually optimizes against both the image classifier and the CLIP model jointly, for example by reporting attack success rates on the CLIP component separately.","section":"Table I caption and §III.C"},{"comment":"Robust accuracy counts abstentions as successes. For the non-adaptive Multi-Shield row on C1, robust accuracy equals the rejection ratio (91.8%), so the headline improvement from 0.0% to 91.8% is entirely due to rejection rather than correct classification of adversarial inputs. This is a legitimate design choice for a rejection-based defense, but the paper should state it explicitly in the main text and report the accuracy of non-abstained inputs as a separate quantity. Currently the text says robust accuracy 'accounts for correct predictions and cases where it abstains,' which is easy to overlook and makes the reported improvements difficult to interpret.","section":"§IV.A, 'Evaluation Metrics' and Table I"}],"minor_comments":[{"comment":"For CIFAR-10, the paper says a 'ViT-B visual encoder fine-tuned for CIFAR-10 classification [Tang et al., 2024]' is used, but it does not specify which CLIP checkpoint this comes from or which text encoder is paired with it. Please give the exact model identifiers so the experiments can be reproduced.","section":"§IV.A, 'Multishield Construction'"},{"comment":"The text says 'in all three plots' when Figure 2 contains four subplots; also, the top two plots and bottom two plots differ in the models shown, which should be stated in the figure caption.","section":"§IV.B and Figure 2"},{"comment":"The lower part of Table II appears malformed: the row for C3 and C4 lacks the separate NR/DNR/Ours labels that appear for the other models, making it hard to read which defense corresponds to which number.","section":"Table II"},{"comment":"Since max_i h(x,P_i) >= h(x,P_j), the absolute value in Eq. (6) is redundant; if the authors intended to capture disagreement, the formula could be simplified or the sign convention clarified.","section":"Eq. (6)"},{"comment":"The limitation about abstract or meaningless labels is acknowledged, but the paper does not quantify how much robustness drops in that setting; a small experiment on a label-renamed dataset would strengthen the discussion.","section":"Section V"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a reasonable candidate for the journal, but the adaptive-attack evaluation is the load-bearing piece for the worst-case robustness claim and is currently a black box. The non-adaptive results and the comparison with NR/DNR are valuable regardless, so the paper is likely salvageable with a detailed description of the adaptive attack and clearer reporting of the rejection-based metric."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nMulti-Shield is a rejection layer that abstains whenever the image classifier and CLIP's zero-shot prediction disagree. The idea is simple and the non-adaptive results are striking: CIFAR-10 non-robust models go from ~0% to ~92% robust accuracy under AutoAttack, and robust models get a 32-65% average boost. Those numbers are plausible for an agreement check, and the clean-accuracy cost is modest.\n\nWhat's new is the specific combination—CLIP as a second opinion for adversarial rejection—rather than a new attack or training method. The paper runs six datasets, thirteen models, and compares against NR and DNR, which is a solid empirical contribution. It is training-free and has no fitted parameters, so there is no circularity in the benchmarks.\n\nThe soft spots are in the adaptive-attack evaluation. The paper formulates the adaptive attacker as a constrained program (Eq. 8-11) but never says how the constraint R(x')≤0 is enforced. No algorithm, no loss modification, no projection. That makes the 'Adaptive Attack' column hard to interpret. The stress-test worry that they simply rejected non-adaptive attacks does not hold up, because the adaptive numbers are much lower than the non-adaptive ones (C1 drops from 91.8% to 20.4%). So something stronger happened, but exactly what is undocumented. Also, robust accuracy counts abstentions as successes. For C1 under the adaptive attack, the rejection ratio is 0.4%, so almost all the 20.4% is true classification, not rejection—that is a weak outcome for that model and undercuts the claim that Multi-Shield 'remains effective' under adaptive attack. For robust models, adaptive gains are a few points: modest but real.\n\nTwo additional issues: the CLIP visual encoder for CIFAR-10 is described as a ViT-B fine-tuned for classification, but it is not clear that it preserves the vision-language alignment needed for meaningful agreement scores. And no code or data are released, which is a problem for a defense paper.\n\nOverall, this is a useful, straightforward study of an agreement-based defense with a promising non-adaptive profile. The adaptive story is plausible but not yet demonstrated. I would send it to review, asking for a precise adaptive-attack description, clarification of the CLIP setup, and code. It deserves a serious referee, but the worst-case claims need support.","headline":"A simple, plausible rejection defense whose non-adaptive gains are real, but the adaptive-attack evaluation is underspecified and the worst-case claims are overstated.","tokens_in":14194,"tokens_out":4280,"would_cite":false,"duration_ms":38861,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A CLIP agreement check turns a 0% robust CIFAR-10 model into a 91.8% one.","keywords":["adversarial examples","adversarial robustness","multi-modal large language models","CLIP","rejection mechanism","abstention","AutoAttack","vision-language models"],"falsifier":"Re-run the adaptive attack while explicitly enforcing R(x')≤0 at every optimization step and verify that the final adversarial candidates satisfy the constraint; if robust accuracy drops materially below the reported adaptive-attack numbers, the defense is weaker than claimed.","tokens_in":13270,"feed_emoji":"🛡️","tokens_out":8418,"duration_ms":67297,"temperature":0.7,"pith_summary":"The paper argues that a rejection layer based on multimodal agreement can make image classifiers far harder to attack, without retraining them. Multi-Shield wraps any image classifier with a CLIP zero-shot classifier; when the two models disagree about an image's class, the system abstains instead of predicting. Under the AutoAttack benchmark at epsilon=8/255, this lifts a non-robust CIFAR-10 model from 0.0% to 91.8% robust accuracy, and improves robust models by an average of 32 percentage points on CIFAR-10 and 65 on ImageNet. The price is a small clean-accuracy drop from rejecting some borderline clean images. The claim matters because it offers a plug-in robustness layer that combines existing adversarial training with a multimodal consistency check.","feed_headline":"A CLIP agreement check takes CIFAR-10 robustness from 0% to 92%","feed_subtitle":"When image classifier and CLIP disagree, abstaining blocks most adversarial attacks.","key_machinery":"The load-bearing mechanism is the rejection score $R(x)$, an absolute difference between CLIP's highest text-image alignment score and its alignment score for the class predicted by the image classifier. It turns the defense into a semantic consistency check across two independently trained representation spaces: when the classifier's top class is not the class CLIP considers most aligned, Multi-Shield abstains. The same score also defines the adaptive attack's extra constraint $R(x')\\le 0$, which is what makes the attacker's optimization harder.","core_discovery":"The paper's central claim is that semantic consistency between an image classifier and a multimodal vision-language model is a powerful rejection signal. Multi-Shield decides by comparing the image classifier's predicted class with CLIP's best-aligned class prompt: if $R(x)=\\left|\\max_i h(x,P_i)-h(x,P_{f(x)})\\right|$ is positive, the system abstains, where $h$ is CLIP's cosine similarity between the image and a textual prompt. Clean inputs tend to keep both models in agreement, while adversarial perturbations break the alignment, so abstention blocks most attacks. The paper further claims that this holds even against an adaptive attacker who knows the whole defense, because the attacker must simultaneously fool both models while keeping the rejection score non-positive.","pith_inferences":["Because the defense depends on class names carrying semantic meaning, datasets with abstract or arbitrary labels (such as 'Class 1') would likely defeat it; automatically generating descriptive prompts for those classes is a natural extension the paper does not test.","The paper notes that attacking either underlying CLIP model alone drives robust accuracy to 0%, so an adversary who optimizes directly against CLIP's alignment scores may open the rejection gate even when the image classifier is not fooled; this makes the defense's real-world strength partly dependent on the attacker's choice of objective.","The same agreement score could be reused as a general uncertainty signal for selective prediction, flagging near-decision-boundary images for human review in non-adversarial settings; this use is left implicit in the paper."],"forward_implications":["A non-robust CIFAR-10 model (C1) that AutoAttack defeats completely (0.0% robust accuracy) rises to 91.8% robust accuracy when Multi-Shield abstains on disagreement.","Robust models gain on average 32 percentage points on CIFAR-10 and 65 on ImageNet under the non-adaptive attack, while clean accuracy falls only about 1.8% and 6% respectively.","An adaptive attacker who knows the defense is still partially held back: the non-robust CIFAR-10 model retains 20.4% robust accuracy, and robust models gain 3.3% on CIFAR-10 and 5% on ImageNet on average.","Relative to two existing rejection defenses, Multi-Shield achieves higher robust accuracy on CIFAR-10 and avoids training one-vs-all SVMs per class, making it cheaper to deploy on larger datasets.","The rejection ratio rises with attack strength, meaning the defense increasingly abstains as perturbations grow larger."],"supporting_citations":[{"why":"Supplies the CLIP zero-shot classifier that Multi-Shield uses to compute text-image alignment scores.","marker":"[Radford et al., 2021]"},{"why":"Supplies AutoAttack, the attack used to measure both baseline and Multi-Shield robust accuracy.","marker":"[Croce and Hein, 2020]"},{"why":"Supplies the standardized benchmark and the pretrained CIFAR-10 and ImageNet models used in the experiments.","marker":"[Croce et al., 2021]"},{"why":"One of the adversarially trained baselines (C2) that Multi-Shield is applied to and improves.","marker":"[Carmon et al., 2019]"},{"why":"Motivates the adaptive attack protocol used for the worst-case robustness evaluation.","marker":"[Carlini et al., 2019]"},{"why":"Provides the Neural Rejection baseline that Multi-Shield is compared against.","marker":"[Melis et al., 2017]"},{"why":"Provides the Deep Neural Rejection baseline that Multi-Shield is compared against.","marker":"[Sotgiu et al., 2020]"}],"fun_headline_variants":["MultiShield abstains when CLIP and classifier disagree to block attacks","Using CLIP to reject adversarial inputs: MultiShield boosts robustness","Abstention on text-visual mismatch defeats most adversarial examples","CLIP disagreement signals attack: MultiShield rejects adversarial images","MultiShield: robust classification by abstaining on CLIP mismatch"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The adaptive-attack robustness numbers depend on the custom attack actually solving the constrained optimization it defines, including the rejection constraint R(x')≤0; the paper does not state how that constraint is enforced, so the reported values could be optimistic if the optimizer silently fails to satisfy it.","fun_headline_variants_meta":{"raw":{"variants":["MultiShield abstains when CLIP and classifier disagree to block attacks","Using CLIP to reject adversarial inputs: MultiShield boosts robustness","Abstention on text-visual mismatch defeats most adversarial examples","CLIP disagreement signals attack: MultiShield rejects adversarial images","MultiShield: robust classification by abstaining on CLIP mismatch"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000796,"raw_usage":{"total_tokens":3454,"prompt_tokens":848,"completion_tokens":2606,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":464,"completion_tokens_details":{"reasoning_tokens":2526}},"tokens_in":464,"tokens_out":2606,"duration_ms":15421,"temperature":1.0,"reasoning_tokens":2526,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:55:44.526165+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the adaptive attack while explicitly enforcing R(x')≤0 at every optimization step and verify that the final adversarial candidates satisfy the constraint; if robust accuracy drops materially below the reported adaptive-attack numbers, the defense is weaker than claimed.","supporting_citations":[],"review_version":1}