{"id":"afadfe36-456a-441f-9555-598c34339d03","arxiv_id":"2501.09446","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Adversarial training at both the CLIP pre-training stage and the LLaVA instruction-tuning stage produces vision-language models with state-of-the-art robustness and near-baseline clean performance.","lead":"This paper adds adversarial training to both stages of building a vision-language model: the initial image-text pre-training and the later visual instruction tuning, producing robust models called ΔCLIP and Δ²LLaVA. The authors report much higher resistance to adversarial image perturbations, with clean-image performance close to standard non-robust models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline ~20% SOTA gain is not attributable to double-defense training: ΔCLIP-H/14-336 (DataComp-1B, 336px) is compared against TeCoA/FARE L/14 (WIT-400M+ImageNet, 224px) on non-shared random subsets, with no same-setting ablation.","rationale":"The paper has real strengths: it reports a broad evaluation across 20+ datasets and four settings, gives training details such as PGD steps, radii, data, and compute, and honestly flags that its random subsets differ from prior work and that adversarial instruction tuning on vanilla CLIP crashed. The CLIP attack evaluation uses AutoAttack, which reduces concern about gradient masking. However, the central quantitative claim is a SOTA margin, and that margin is currently computed against baselines that differ in architecture, data, resolution, and evaluation subset. A method paper whose headline number changes when the comparison is made on common ground cannot support the new state-of-the-art claim; it would still demonstrate that large-scale adversarial pre-training is feasible and yields robust models, but not that it outperforms post-hoc fine-tuning at matched scale. This is fixable by releasing checkpoints, running a controlled same-backbone/same-data ablation, and sharing the evaluation subsets. Because the paper's own text acknowledges the non-comparability, the reader's conditional verdict is the right one; I see no reason to harden or soften it. The concern is about attribution and reproducibility, not about internal consistency or author conduct.","tokens_in":19756,"tokens_out":5519,"duration_ms":59578,"concrete_test":"Obtain or re-train with the released recipe ΔCLIP-H/14-336 and apply the TeCoA or FARE post-hoc adversarial fine-tuning procedure to a clean-pretrained H/14-336 CLIP trained on the same DataComp-1B data; evaluate both, plus the published TeCoA4/FARE4 checkpoints, on the same fixed 1000-sample ImageNet subset using the same AutoAttack pipeline and same seed. If the clean-pretrained plus post-hoc model closes the robust-accuracy gap to less than ~20 points, or beats ΔCLIP, the double-defense attribution fails. Also report full-dataset or seeded shared-subset numbers to make the SOTA claim reproducible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that adversarial pre-training plus instruction tuning yields a ~20% ImageNet robustness SOTA depends on a confounded comparison. In Tables 1-2, ΔCLIP-H/14-336 is trained on DataComp-1B with Recap synthetic captions and an added CoCa captioning loss, while TeCoA and FARE are L/14 models fine-tuned from OpenAI CLIP on WIT-400M + ImageNet at 224px. Architecture (H/14 vs L/14), training data (1B vs 400M), resolution (336 vs 224), and auxiliary losses all differ simultaneously, so the margin cannot be attributed specifically to adversarial pre-training. The paper itself states in Sections 4.1 and 4.2 that its random 1000/500-sample evaluation subsets differ from Schlarmann et al. and that results are not directly comparable; no shared subsets or seeds are provided. No same-architecture/same-data ablation of adversarial vs non-adversarial pre-training is reported, and no code or weights are currently available to re-run the comparison. As a result, the abstract's ~20% improvement over previous best is a cross-paper comparison on non-identical evaluation sets, not a controlled head-to-head.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-stage adversarial training recipe for vision-language models: (i) adversarial contrastive pre-training of CLIP on DataComp-1B with Recap synthetic captions and a CoCa captioning loss, yielding DeltaCLIP, and (ii) adversarial visual instruction tuning of LLaVA-1.5, yielding Delta2LLaVA. The authors evaluate clean and adversarial zero-shot classification on standard and distribution-shift benchmarks, captioning and VQA robustness, targeted attacks, visual reasoning, and hallucination. They report large robustness gains over the TeCoA and FARE baselines and argue that, unlike prior post-hoc adversarial fine-tuning, their models preserve clean performance close to non-robust baselines. The central claim is that this 'double visual defense' sets a new state of the art in VLM adversarial robustness.","tokens_in":1294,"tokens_out":2347,"duration_ms":52430,"significance":"If the headline results survive a controlled comparison, the paper is a substantive empirical contribution: it introduces adversarial visual instruction tuning as a defense stage, evaluates with AutoAttack and a broad set of benchmarks, and provides an internal comparison showing that the second training stage adds robustness, especially at larger attack radii. The paper also documents an interesting phenomenon in which typographic attacks emerge from plain L-infinity attacks on adversarially trained models. However, the claimed state-of-the-art margins are currently based on confounded cross-paper comparisons on non-shared random evaluation subsets, so the significance is conditional on the authors supplying a same-setting ablation and a reproducible evaluation protocol.","major_comments":[{"comment":"The headline ~20% ImageNet robustness improvement over TeCoA and FARE is not attributable to the proposed training method from these tables alone: DeltaCLIP-H/14-336 is trained on DataComp-1B with Recap synthetic captions and an additional CoCa captioning loss, whereas TeCoA and FARE are L/14 models fine-tuned from OpenAI CLIP on WIT-400M plus ImageNet at 224 resolution. Architecture, training data, resolution, and auxiliary losses vary simultaneously. A same-setting ablation, such as a non-adversarial CLIP trained on the same DataComp-1B recipe or an adversarially trained L/14 with the prior setup, is needed to support the attribution.","section":"Section 4.1, Tables 1-2"},{"comment":"The paper states that its random 1000-sample (CLIP) and 500-sample (LLaVA) adversarial evaluation subsets differ from those used by Schlarmann et al. and that the results are 'not directly comparable' to prior work. Because the abstract and conclusion present ~20% and ~30% improvements as state of the art, the authors should either re-evaluate prior baselines under the identical protocol, release the exact random subsets and seeds, or substantially soften the SOTA claims. Confidence intervals would also help, since some reported numbers are based on only 1000 samples.","section":"Sections 4.1 and 4.2"},{"comment":"The paper claims that adversarial visual instruction tuning on a vanilla CLIP-based LLaVA 'completely crashed,' but it provides no quantitative detail or experimental description. Since the necessity of the double defense is a load-bearing claim, please report the failure mode and the relevant numbers, or clearly mark this as a preliminary observation that was not systematically evaluated.","section":"Section 3.3"}],"minor_comments":[{"comment":"The caption reads 'TeCoA and FAR' but should read 'TeCoA and FARE'.","section":"Table 2 caption"},{"comment":"The text cites reference [5] for both JAX and PyTorch implementations, but [5] is the JAX reference; a separate PyTorch reference should be cited for the LLaVA implementation.","section":"Training Details"},{"comment":"The ScienceQA setup description contains a duplicated word: 'we use the the subset with images'.","section":"Section 4.4"},{"comment":"The Limitations section begins with 'In this paper, We focus solely...' and should be reworded; it also does not mention the lack of public code or model weights at the time of submission, which is relevant for reproducibility.","section":"Limitations"},{"comment":"The targeted-attack evaluation relies on 10 random COCO samples per target and human judgement of attack success; the paper should specify the human judgement protocol and whether multiple annotators were used, because Table 6 reports CIDEr/ASR combinations that are not self-explanatory.","section":"Section 4.3 and Table 4"}],"recommendation":"major_revision","confidential_remarks":"The main risk is overclaiming from non-comparable evaluations. If the authors can provide a same-setting ablation (e.g., a non-adversarial DataComp-1B CLIP with identical recipe, or the prior baselines re-evaluated under the same subsets) and release the exact evaluation subsets and seeds, the contribution would be solid. The statement that code and weights 'will be released' should be made concrete; without that, the reproducibility of the reported robustness numbers is limited."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The new thing here is the combination: adversarial training from scratch during web-scale CLIP pre-training and then again during LLaVA visual instruction tuning. That's a genuine first, and it produces models that keep clean accuracy close to vanilla LLaVA while being far more robust than TeCoA- or FARE-based models, which drop hard on clean benchmarks. The evaluation is broad and mostly well done: AutoAttack-based recognition, captioning and VQA under attacks, targeted attacks, reasoning, hallucination. The internal ablation of deltaLLaVA versus delta-squared-LLaVA supports the second stage, especially at larger radii. Credit where due: this is the strongest-looking recipe so far for robust VLMs that stay useful on clean inputs.\n\nThe soft spots are real, but they are concentrated in the central SOTA claim. DeltaCLIP is H/14 at 336px trained on DataComp-1B with Recap captions and an extra CoCa loss; TeCoA and FARE are L/14 at 224px fine-tuned on WIT-400M plus ImageNet. The paper itself says its random 1000-sample subsets differ from prior work and results are not directly comparable. So the ~20% ImageNet robustness margin is not attributable to double defense; architecture, data scale, resolution, and auxiliary losses all change at once. There is no same-architecture/same-data adversarial-vs-non-adversarial ablation, and no code or weights released yet. These are fixable with a controlled ablation and shared evaluation subsets.\n\nMinor note: the paper says the text encoder is kept frozen during adversarial pre-training, which is a simplification worth flagging in any revision, but it doesn't break the story.\n\nBottom line: the paper is for researchers working on robust multimodal models, and it deserves a serious referee. The recipe is interesting and the clean-utility preservation is a notable result even if the exact margin over prior SOTA is unproven. Send it to review with a request for the same-setting ablation and artifact release; without those, the headline should be softened.","headline":"A serious attempt at robust VLMs with a plausible two-stage recipe, but the headline SOTA margin is a confounded cross-paper comparison; worth peer review, not desk rejection.","tokens_in":20559,"tokens_out":1405,"would_cite":true,"duration_ms":16646,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that adversarially training vision-language models at both the contrastive pre-training stage and the visual instruction-tuning stage produces state-of-the-art robustness against visual adversarial attacks with little…","keywords":["adversarial training","vision-language models","CLIP","LLaVA","zero-shot robustness","visual instruction tuning","hallucination","web-scale pre-training"],"falsifier":"Train the identical H/14-336 architecture and DataComp-1B recipe with the same compute, but with the adversarial perturbations removed, or with only one of the two stages made adversarial, and evaluate on the exact fixed 1000-sample subsets used in [42]; if the robustness margin over the earlier models mostly disappears, the attribution of the gain to the double visual defense fails.","tokens_in":19565,"feed_emoji":"🛡️","tokens_out":11534,"duration_ms":99360,"temperature":0.7,"pith_summary":"Vision-language models inherit a serious vulnerability: small, invisible perturbations to an input image can make a CLIP or LLaVA model misclassify, hallucinate, or emit attacker-chosen text. Previous defenses tried to fix this by lightly fine-tuning an already-trained CLIP on ImageNet, which restored some robustness but degraded clean accuracy and generalization. This paper argues for a different strategy: build robustness into the model at both stages of training. It adversarially pre-trains a new CLIP, called ∆CLIP, from scratch on web-scale data, then adversarially tunes a LLaVA on top of it, producing ∆2LLaVA. Across zero-shot recognition, captioning, VQA, targeted attacks, reasoning, and hallucination benchmarks, the resulting models are claimed to set a new state of the art in adversarial defense while keeping clean-data performance close to that of the non-robust originals.","feed_headline":"Two-stage adversarial training lifts VLM attack robustness ~20-30%","feed_subtitle":"Adversarial pre-training plus instruction tuning keeps clean accuracy close to unprotected CLIP and LLaVA.","key_machinery":"The machinery is the two-stage 'double visual defense'. Stage one is adversarial contrastive pre-training: $\\min_{\\theta_I}\\max_{\\|\\delta\\|_p\\leq\\epsilon} \\mathcal{L}_{con}(x+\\delta,y)$ run at increasing resolution and attack radius (112px with PGD-2 and $\\epsilon=4/255$, then 224px with PGD-3, then 336px with PGD-4 and $\\epsilon=8/255$) on DataComp-1B with a frozen text encoder and an added CoCa captioning loss. Stage two is adversarial visual instruction tuning: $\\min_{\\phi}\\max_{\\|\\delta\\|_p\\leq\\epsilon} \\mathcal{L}_{inst}(x+\\delta,y)$ on the LLaVA-1.5 recipe with LoRA, where the visual encoder is fine-tuned at one-twentieth of the base learning rate. The argument is that the first stage prevents the catastrophic clean-performance loss seen in post-hoc ImageNet fine-tuning, and the second stage adds strength specifically under large perturbations.","core_discovery":"The paper's central claim is that adversarial training should happen at every stage of vision-language model creation, not only as a post-hoc patch. The authors replace CLIP's contrastive objective with a min-max objective in which images are perturbed to break image-text matching, and train this adversarially pre-trained model, ∆CLIP, from scratch on web-scale data. They then add a second layer of defense by adversarially tuning LLaVA with perturbed images optimized to cause next-token mispredictions, yielding ∆2LLaVA. The claimed result is that ∆CLIP and ∆2LLaVA exceed the prior robust VLM baselines by roughly 20% on ImageNet-1k zero-shot robustness, 30% on captioning robustness, and 20% on VQA robustness, while matching vanilla CLIP and LLaVA on clean inputs and hallucinating less than prior robust models.","pith_inferences":["Beyond the paper: because the new model uses a larger vision tower (H/14-336) and a far larger pre-training corpus (DataComp-1B) than the L/14 WIT-400M-plus-ImageNet models it is compared with, the share of the robustness gain due specifically to double-defense training, rather than to scale and data, is not isolated by an ablation.","Beyond the paper: the paper's own observation that typographic text attacks 'emerge' from naive $\\ell_\\infty$ attacks against its robust models suggests a testable extension: adversarially training on typographic prompts could close the remaining attack surface and may generalize to unseen text-in-image attacks.","Beyond the paper: if these gains replicate with a controlled comparison, adversarial pre-training from scratch could become the default route to robust vision-language models, and the community could compare it against lightweight fine-tuning on shared evaluation subsets."],"forward_implications":["Adversarial visual hijacking becomes far harder: at $\\epsilon=16/255$, the average targeted-attack success rate for ∆2LLaVA8 is 3.3%, versus 65.0% for the FARE4-based LLaVA and 100% for a vanilla LLaVA.","Robustness transfers across zero-shot domains: ∆CLIP improves over prior robust CLIP models on both ImageNet-variant datasets and non-ImageNet datasets such as Stanford Cars, where its adversarial accuracy is roughly seven times the second-best model.","The second stage earns its cost mainly at large attack radii: on VQAv2 at $\\epsilon=8/255$, ∆2LLaVA8 raises accuracy to 52.4% from 41.1% for ∆LLaVA.","Because clean performance and helpfulness stay close to vanilla LLaVA, the resulting models could serve as drop-in replacements for unprotected CLIP and LLaVA in applications that need both safety and utility.","The paper's own limitation statement scopes the result to visual attacks on CLIP-based architectures; text-side threats and non-CLIP VLMs are not covered by the double visual defense."],"supporting_citations":[{"why":"Defines the contrastive image-text objective and the non-robust WIT-400M CLIP baseline that all compared robust models start from.","marker":"[39]"},{"why":"Introduces TeCoA, the text-supervised adversarial fine-tuning baseline whose ImageNet-centric evaluation setup the paper follows for zero-shot robustness.","marker":"[31]"},{"why":"Introduces FARE, the prior robust CLIP state of the art, and supplies the LLaVA attack pipeline, targeted-attack protocol, and evaluation subsets the paper extends.","marker":"[42]"},{"why":"Provides the LLaVA architecture that consumes the CLIP visual encoder and is the target of the instruction-tuning stage.","marker":"[27]"},{"why":"Provides the LLaVA-1.5 training recipe used for both the baseline and the adversarial visual instruction tuning.","marker":"[26]"},{"why":"Supplies the DataComp-1B web-scale corpus used for adversarial contrastive pre-training from scratch.","marker":"[12]"},{"why":"Supplies the Recap-DataComp-1B synthetic captions that are mixed 1:1 with original captions for richer language supervision.","marker":"[21]"},{"why":"Supplies AutoAttack, the APGD and DLR attack suite used to measure zero-shot adversarial robustness.","marker":"[8]"},{"why":"Gives the projected-gradient-descent adversarial training framework that the min-max objectives are built on.","marker":"[30]"},{"why":"Gives the CoCa captioning loss that the authors add to adversarial pre-training because it improved clean and robust performance.","marker":"[53]"}],"fun_headline_variants":["Adversarial pre-training plus tuning boosts VLM robustness 20-30%","Two-stage adversarial defense lifts VLM robustness by up to 30%","Adversarial pretraining and tuning yield robust VLMs with 30% gains","New SOTA in VLM adversarial defense: 20-30% robustness gains","Two-stage adversarial training for VLMs: up to 30% better robustness"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the large robustness gains come from the two adversarial training stages rather than from the larger vision tower and much larger pre-training dataset, and that the paper's unshared 1000-sample evaluation subsets, which differ from those used for the prior results it compares against, are representative enough for direct state-of-the-art comparisons.","fun_headline_variants_meta":{"raw":{"variants":["Adversarial pre-training plus tuning boosts VLM robustness 20-30%","Two-stage adversarial defense lifts VLM robustness by up to 30%","Adversarial pretraining and tuning yield robust VLMs with 30% gains","New SOTA in VLM adversarial defense: 20-30% robustness gains","Two-stage adversarial training for VLMs: up to 30% better robustness"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001224,"raw_usage":{"total_tokens":5039,"prompt_tokens":960,"completion_tokens":4079,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":576,"completion_tokens_details":{"reasoning_tokens":3976}},"tokens_in":576,"tokens_out":4079,"duration_ms":33781,"temperature":1.0,"reasoning_tokens":3976,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:01:21.767404+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the identical H/14-336 architecture and DataComp-1B recipe with the same compute, but with the adversarial perturbations removed, or with only one of the two stages made adversarial, and evaluate on the exact fixed 1000-sample subsets used in [42]; if the robustness margin over the earlier models mostly disappears, the attribution of the gain to the double visual defense fails.","supporting_citations":[{"cited_title":"Reliable evalua- tion of adversarial robustness with an ensemble of diverse parameter-free attacks","cited_arxiv_id":null,"evidence_quote":"Supplies AutoAttack, the APGD and DLR attack suite used to measure zero-shot adversarial robustness."},{"cited_title":"Towards deep learning models resistant to adversarial attacks","cited_arxiv_id":null,"evidence_quote":"Gives the projected-gradient-descent adversarial training framework that the min-max objectives are built on."},{"cited_title":"Learn- ing transferable visual models from natural language super- vision","cited_arxiv_id":null,"evidence_quote":"Defines the contrastive image-text objective and the non-robust WIT-400M CLIP baseline that all compared robust models start from."},{"cited_title":"Understanding zero-shot adversarial robust- ness for large-scale models","cited_arxiv_id":null,"evidence_quote":"Introduces TeCoA, the text-supervised adversarial fine-tuning baseline whose ImageNet-centric evaluation setup the paper follows for zero-shot robustness."},{"cited_title":"Visual instruction tuning","cited_arxiv_id":null,"evidence_quote":"Provides the LLaVA architecture that consumes the CLIP visual encoder and is the target of the instruction-tuning stage."},{"cited_title":"Improved baselines with visual instruction tuning","cited_arxiv_id":null,"evidence_quote":"Provides the LLaVA-1.5 training recipe used for both the baseline and the adversarial visual instruction tuning."},{"cited_title":"Dat- acomp: In search of the next generation of multimodal datasets","cited_arxiv_id":null,"evidence_quote":"Supplies the DataComp-1B web-scale corpus used for adversarial contrastive pre-training from scratch."},{"cited_title":"Coca: Contrastive captioners are image-text foundation models","cited_arxiv_id":null,"evidence_quote":"Gives the CoCa captioning loss that the authors add to adversarial pre-training because it improved clean and robust performance."}],"review_version":1}