{"id":"2a1c493f-d53f-41f7-8b53-cf2f019bd7ce","arxiv_id":"2607.18958","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"DAFT fine-tunes CLIP's vision encoder with a visual-consistency loss plus a caption-triplet loss, reporting modest adversarial-robustness gains over FARE on classification, captioning, and VQA benchmarks.","lead":"The paper introduces DAFT, an adversarial fine-tuning method that trains a CLIP vision encoder with two losses: keeping adversarial image features close to clean features and aligning them with image captions via a triplet loss. It reports small robustness gains over prior defenses on zero-shot classification, captioning, and VQA when the encoder is swapped into LLaVA-1.5.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unspecified second-stage attack threshold and missing variance make the small DAFT-over-FARE margins unreliable evidence for the 'outperforms SOTA' claim.","rationale":"I read the paper as making a specific, falsifiable empirical claim: that a single adversarially fine-tuned CLIP vision encoder, when swapped into LLaVA-1.5, provides the best reported robust performance across classification, captioning, and VQA. The method itself is coherent: dual supervision via L2 visual consistency and a caption-based triplet loss is a plausible extension of prior work, and the ablations (Tables III–V) show that both terms contribute and that hardest-negative sampling is important. The paper also provides extra evidence beyond the main tables: 8/255 results in the supplementary, OpenFlamingo transfer, and black-box transfer. These are real supporting signals. The load-bearing weakness is not the method design but the strength of the evidence for the SOTA claim. The margins over FARE are consistently small, and the captioning/VQA protocol contains an unreported threshold that can materially change which samples receive the stronger second attack. Without specifying that threshold, and without any variance estimates, the reported averages are not sufficient to establish that DAFT is better than FARE rather than merely within noise or protocol sensitivity. I do not see a fatal internal inconsistency in the loss formulation or training procedure; the concern is empirical reproducibility and statistical reliability. This matches the reader's weakest-assumption analysis, so I do not propose changing the CONDITIONAL verdict: the method is plausible and likely useful, but the SOTA claim requires a more controlled, reproducible evaluation before it can be accepted at face value.","tokens_in":21744,"tokens_out":5638,"duration_ms":53821,"concrete_test":"Obtain or release DAFT and FARE checkpoints and re-run the Sec. IV-C evaluation with the stage-2 threshold fixed in advance (e.g., CIDEr > 0) and identical for all models, over at least 5 random seeds/sample draws. Compute paired 95% confidence intervals for the DAFT-minus-FARE difference on each task family and each perturbation budget. If any interval includes 0, the 'outperforms SOTA' claim is not supported. Additionally, sweep the threshold over a plausible range and verify that the ranking of DAFT versus FARE is invariant; if the ranking flips for any threshold in that range, the reported margin is an artifact of the protocol.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that DAFT outperforms the state of the art (FARE) in adversarial robustness across zero-shot classification, captioning, and VQA. The supporting evidence in Tables I and II shows average gains of only 1.2–2.3 points over FARE (e.g., +1.2% classification at eps=2/255, +1.1% VQA at eps=2/255, +2.3 CIDEr at eps=2/255). These margins are small enough that the evaluation protocol decides whether the claim holds. In Sec. IV-C, the captioning/VQA protocol applies a second 100-step FP32 APGD attack only to samples whose CIDEr exceeds 'a certain threshold' (captioning) or that were not successfully attacked (VQA). The threshold value is never reported, and the paper does not state whether the same threshold is applied to all models. If the threshold was chosen post hoc or differs across models, a model receiving fewer second-stage attacks will appear more robust without being genuinely so. Even if the threshold is fixed, no seeds, error bars, or confidence intervals are reported; on 1000-sample evaluations, a 1-point average difference can easily lie within run-to-run variance. The robustness generalization results on OpenFlamingo (Tab. VII) and black-box transfer (Tab. VIII) are supportive, but they do not remove the dependence of the headline SOTA claim on an incompletely specified, potentially unstable comparison.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DAFT, a dual-supervision adversarial fine-tuning method for the CLIP vision encoder used in LVLMs such as LLaVA-1.5. The method combines a visual supervision loss (L2 distance between adversarial features and clean features from a frozen original encoder) with a semantic supervision loss (triplet loss that aligns the adversarial image embedding with its ground-truth caption and pushes away the hardest negative caption in the batch). The vision encoder is fine-tuned on COCO captions under PGD-10 attacks and then swapped into LLaVA-1.5 or OpenFlamingo without retraining. The paper claims state-of-the-art adversarial robustness across zero-shot classification (11 datasets), image captioning (COCO, Flickr30k), and VQA (VQAv2, OKVQA, VizWiz, TextVQA), comparing against TeCoA, PMG-AFT, and FARE, with additional ablations and black-box transfer results.","tokens_in":22066,"tokens_out":5214,"duration_ms":49260,"significance":"If the empirical claims hold, DAFT would be a practical contribution: a single adversarially fine-tuned CLIP encoder that can be dropped into multiple LVLMs to improve robustness across tasks. The semantic-supervision idea, using descriptive captions instead of class labels, is well motivated and the ablations (Tables III and IV) provide useful evidence for the design choices. The cross-model evaluation on OpenFlamingo and black-box transfer results (Tables VII and VIII) are strengths and go beyond many papers in this area. However, the headline claim that DAFT outperforms the state-of-the-art method rests on small margins over FARE (0.9–2.3 points in most tables) on 1000-sample evaluations, and the evaluation protocol contains an underspecified second-stage attack threshold. These issues must be resolved before the SOTA claim is convincing.","major_comments":[{"comment":"The central SOTA claim depends on the multi-stage attack protocol, but the protocol is underspecified. The paper retains for the second FP32 APGD stage only samples whose CIDEr exceeds 'a certain threshold' for captioning, or that were 'not successfully attacked' for VQA. The threshold value and the definition of successful attack are never reported, nor is it stated that the same criterion is applied to every model. Since the reported DAFT-over-FARE margins are 0.9–2.3 points on 1000-sample evaluations, a model-dependent or post hoc threshold could determine the outcome. Please report the exact threshold/criterion, verify it is fixed across all models, and report the number of samples entering the second stage per model.","section":"§IV-C, Tables II and VI"},{"comment":"No variance estimates, seeds, or significance tests are reported. For example, under eps=2/255, DAFT exceeds FARE by 1.2% average zero-shot accuracy, 2.3 CIDEr, and 1.1% VQA accuracy; under eps=4/255 the margins are 1.4%, 2.1, and 0.9%. On a 1000-sample evaluation, such differences can easily lie within run-to-run variance. The authors should report results over multiple training seeds (or at least bootstrap confidence intervals over the evaluation samples) and a paired significance test where appropriate. Without this, the claim that DAFT is state-of-the-art is not statistically supported.","section":"Tables I, II, VII, VIII"},{"comment":"The model is trained on COCO captions and evaluated on COCO captioning, but the paper does not specify the train/evaluation split. If the COCO captioning evaluation uses images or captions from the training set, DAFT would have an unfair advantage because its semantic supervision directly uses the ground-truth captions, whereas FARE and TeCoA do not use captions. Please specify the exact split (e.g., Karpathy train/test) and confirm that the evaluation images and captions are disjoint from the fine-tuning data.","section":"§IV-A and Table II"},{"comment":"Hyperparameters gamma, a, and batch size are selected by evaluating average robust performance on the same three task families and the same epsilon=2/255 setting used in the main results. With the small margins over FARE, selection on the evaluation benchmarks can inflate the reported gains. The authors should either use a separate validation set for hyperparameter selection or provide a sensitivity analysis that shows the reported gains are not an artifact of choosing the best-performing configuration on the test benchmarks.","section":"§IV-D, Table V and Fig. 4"}],"minor_comments":[{"comment":"Typo: 'adversarial traning' should be 'adversarial training'. Please proofread the manuscript; there are also inconsistent renderings of 'LLaVA' and 'LLaV A'.","section":"Abstract / Contributions"},{"comment":"The attack protocol is said to be 'consistent with previous work [11], [19]'. If the threshold and VQA success criterion are defined in FARE, please cite the exact equations or section; otherwise, the details must be given here.","section":"§IV-C"},{"comment":"Table VI repeats Table II plus an eps=8/255 row. Consider replacing the duplicate with only the new eps=8/255 results, or clearly label it as an extension.","section":"Table VI (supplementary)"},{"comment":"The triplet loss is written with cosine similarities and a margin a. It would be helpful to state explicitly that a is in [0,2] conceptually (though the paper tunes in [0.1,0.5]) and to define the 'hardest' negative precisely when multiple captions share the maximum similarity.","section":"§III-B, Eq. (6)"},{"comment":"The statement 'We also maintain the same settings as FARE [19] for the experiments' is vague. Please list the exact training iterations, learning rate schedule, and attack hyperparameters used for each baseline so that replication is possible.","section":"§IV-A"}],"recommendation":"major_revision","confidential_remarks":"The method is plausible and the paper has a solid set of ablations, but the empirical evidence for the SOTA claim is currently too fragile: the second-stage attack threshold is underspecified and the margins over FARE are small with no variance analysis. These are fixable with additional reporting and experiments, so I recommend major revision rather than rejection. If the authors can provide code, exact thresholds, and multi-seed results, the paper could become a strong contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: DAFT is a plausible extension of the PMG-AFT and FARE line, and the empirical results are consistent enough to take seriously. The new bit is replacing class-label supervision with caption-based hardest-negative triplet loss during adversarial fine-tuning, while keeping the frozen-original-encoder L2 visual term from the authors' own PMG-AFT. That exact combination isn't in prior work, and the ablations show both branches pull their weight.\n\nThe paper does a lot right. The evaluation is broad: 11 classification datasets plus captioning and VQA on LLaVA-1.5, with extra transfer tests on OpenFlamingo and black-box transfer. DAFT lands above FARE on nearly every dataset at eps=2/255 and 4/255, with only a small clean-accuracy drop. The ablation study is honest—caption supervision alone overfits, visual supervision alone is weaker, and the pair works best. The hyperparameter analysis is also useful, even if it is tuning on the same benchmarks used for evaluation.\n\nNow the soft spots, in proportion. The average margins over FARE are 1.2 to 2.3 points, and there are no error bars, no multiple seeds, no code or checkpoints. That alone wouldn't sink the paper given the consistency across many datasets, but it does make the headline 'outperforms SOTA' fragile. More concretely, the captioning/VQA attack protocol applies a second-stage FP32 APGD only to samples whose CIDEr exceeds 'a certain threshold' (or that survived the first stage for VQA), and that threshold is never reported. If the threshold is fixed across models, the comparison is fair; if it drifts, a one- or two-point advantage could be an artifact. The paper needs to state the threshold and ideally report single-stage attack results as a check. Also, the 'first caption-supervised adversarial training' claim should be benchmarked against Waseda et al. [29], which is cited but never compared—that's a direct competitor for the caption-as-supervision idea.\n\nNet: the method is coherent, the experiments are broad, and the writing is clear. The weak spots are fixable: specify the threshold, release code or at least report seeds/variance, and add the missing comparison. This deserves peer review, not desk rejection. I'd send it to a careful referee with the expectation of moderate revisions.","headline":"A sensible incremental defense that consistently edges out FARE across tasks, but the SOTA claim rests on an under-specified attack threshold and missing variance; worth a real referee.","tokens_in":22590,"tokens_out":2556,"would_cite":true,"duration_ms":26684,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DAFT fine-tunes a CLIP vision encoder under two supervision signals—clean-image features and image captions—so that one robust encoder, swapped into LLaVA-1.5, transfers adversarial robustness to zero-shot classification, captioning, and VQ","keywords":["adversarial robustness","large vision-language models","adversarial fine-tuning","CLIP vision encoder","triplet loss","hard negative mining","image captioning","visual question answering"],"falsifier":"Re-run the paper's captioning and VQA evaluation using a single fixed, pre-registered CIDEr threshold for every model (or no second-stage filter at all), with the same 100-step FP32 APGD attack, over at least three random seeds; if DAFT's average gains over the previous unsupervised method fall within seed-to-seed variance (roughly 1 CIDEr point or 1% VQA accuracy) or reverse, the claimed superiority is not established.","tokens_in":21612,"feed_emoji":"🛡️","tokens_out":8490,"duration_ms":75221,"temperature":0.7,"pith_summary":"The paper sets out to show that the visual encoder, not the language model, is the place to defend large vision-language models against adversarial image attacks. It introduces DAFT, a dual adversarial fine-tuning scheme that trains the CLIP image encoder with two signals at once: an L2 visual-supervision term that pins adversarial features to the clean features of the frozen original encoder, and a caption triplet term that binds adversarial images to their true captions while repelling the hardest false caption in the batch. The authors claim that the resulting encoder, dropped into LLaVA-1.5 with no retraining of the language half, beats existing adversarial fine-tuning methods in robust accuracy across eleven zero-shot classification datasets and six captioning/VQA datasets while keeping clean accuracy comparable. If the claim holds, robust LVLMs become a plug-in upgrade: harden one encoder, then swap it into any compatible model.","feed_headline":"One fine-tuned CLIP encoder shields LLaVA on all three task families","feed_subtitle":"Two supervision signals let one CLIP encoder transfer robustness to LLaVA without retraining.","key_machinery":"The load-bearing mechanism is a dual-loss with two frozen reference points: a frozen original CLIP vision encoder supplying clean-image features for L2 visual supervision, and a frozen CLIP text encoder supplying caption embeddings for a triplet contrastive loss with hardest-negative sampling. The caption triplet is the novelty—descriptive captions replace category labels, and the hardest negative is the caption in the batch most similar to the adversarial features, which forces the encoder to preserve fine semantic distinctions under attack.","core_discovery":"DAFT fine-tunes only the vision encoder of CLIP ViT-L/14 on COCO captions, using PGD-10 attacks with L-infinity budget 4/255. The total loss is L_sem + γL_vis: the visual branch minimizes ||F_ori(x) − F_θ(x_a)||², keeping the adversarial embedding close to the clean embedding of the frozen original encoder; the semantic branch maximizes cosine distance to the hardest negative caption and minimizes distance to the ground-truth caption in a triplet loss with margin a. The same loss drives both attack generation and parameter updates in a min-max loop. The paper reports that DAFT achieves the best average robust accuracy at epsilon 2/255 and 4/255 (and 8/255 in supplementary material) on zero-s","pith_inferences":["The paper's own limitation section grants that DAFT is a white-box, CLIP-vision-encoder-only defense; an immediate extension question is whether the same caption-triplet supervision helps non-CLIP encoders or closed models, which is untested.","If caption diversity is the active ingredient, training DAFT on larger or more descriptive caption corpora than COCO should improve robustness further; the paper leaves this scaling test open.","Because the caption/VQA protocol filters samples by an unreported CIDEr threshold before the second attack stage, the reported 1–2 point gains over the unsupervised baseline need a re-run with one fixed, pre-registered threshold and multiple seeds before the SOTA claim can be taken at face value.","The hardest-negative result predicts that DAFT's advantage will grow on datasets with many visually similar captions or classes, since the loss explicitly sharpens separation between confusable captions under attack."],"forward_implications":["Robustness becomes a drop-in property: one DAFT-trained vision encoder can replace the original encoder in LLaVA-1.5 (and a second CLIP-based LVLM in the paper's supplementary experiments) with no task-specific retraining or architecture changes.","Caption supervision is doing the heavy lifting: in the paper's ablation, replacing captions with category labels drops average robust classification by 12.4%, caption CIDEr by 15.2, and VQA accuracy by 4.2%.","Hard-negative sampling is necessary, not just helpful: training with the easiest negative as the triplet contrast yields near-zero robust accuracy, essentially matching the undefended model.","The balance between the two supervision signals matters: for a=0.2 and γ=0.5 the method peaks; both larger and smaller values degrade robustness on all three tasks, and batch size 64 gives the best negatives.","Robustness transfers across architectures: the paper's black-box transfer tests show DAFT outperforms prior methods both when attacks are generated on one LVLM and evaluated on another."],"fun_headline_variants":["One CLIP encoder tweak, robustness across three tasks","Dual supervision fine-tune shields LLaVA from attacks","CLIP vision encoder tuning hardens LLaVA on all tasks","DAFT: fine-tune CLIP vision, secure LLaVA multitask"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The state-of-the-art claim rests on the two-stage caption/VQA evaluation in Sec. IV-C, where the unreported CIDEr threshold decides which samples face the stronger second attack—if that threshold was set per method after seeing results, the comparison is not controlled.","fun_headline_variants_meta":{"raw":{"variants":["One CLIP encoder tweak, robustness across three tasks","Dual supervision fine-tune shields LLaVA from attacks","CLIP vision encoder tuning hardens LLaVA on all tasks","DAFT: fine-tune CLIP vision, secure LLaVA multitask"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000224,"raw_usage":{"total_tokens":1312,"prompt_tokens":771,"completion_tokens":541,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":515,"completion_tokens_details":{"reasoning_tokens":466}},"tokens_in":515,"tokens_out":541,"duration_ms":5400,"temperature":1.0,"reasoning_tokens":466,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T13:47:14.610296+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the paper's captioning and VQA evaluation using a single fixed, pre-registered CIDEr threshold for every model (or no second-stage filter at all), with the same 100-step FP32 APGD attack, over at least three random seeds; if DAFT's average gains over the previous unsupervised method fall within seed-to-seed variance (roughly 1 CIDEr point or 1% VQA accuracy) or reverse, the claimed superiority is not established.","supporting_citations":[],"review_version":1}