{"id":"21191575-e08f-4991-a356-76a6af155e5e","arxiv_id":"2607.19683","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":9,"one_line_summary":"GhostPrompt is a universal adversarial text suffix that, after one optimization, steers VLMs to attacker-chosen outputs across diverse unseen images, reporting >30% ASR gains over prior prompt attacks.","lead":"The paper presents GhostPrompt, a single text suffix trained with a min-max procedure so it can misdirect vision-language models on many different images, not just one. If it works as reported, prompt-sharing sites and template-based assistants become a practical distribution channel for reusable attacks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported ASR gains hinge on undisclosed train/test split, matching rule, and image-perturbation bound; default training count chosen on the same benchmarks may inflate results.","rationale":"The reader's weakest_assumption correctly identifies the evaluation protocol as a load-bearing concern. My stress-test concurs that the train/test disjointness, ASR matching rule, and the selection of the default training-image count on the evaluation benchmarks are critical for the >30% ASR claim. I additionally flag the missing projection bound in Algorithm 1 as a related but distinct correctness risk: 'Projected Gradient Ascent' without an explicit projection makes the inner maximization ill-posed and could invalidate the mechanistic interpretation of 'worst-case image' optimization. However, this is an omission rather than an observed contradiction; the paper's core min-max formulation is plausible, and the provided code could resolve the questions. Since the reader already issued a CONDITIONAL verdict, my analysis does not change that verdict. The proposed concrete test--re-running with a disjoint split, strict matching, and a bounded perturbation--would settle whether the reported gains are real. I therefore keep the verdict as UNCHANGED while emphasizing that the burden is on the authors to disclose these details.","tokens_in":21908,"tokens_out":7687,"duration_ms":84293,"concrete_test":"Run the official code with a held-out protocol: fix 20 training images sampled from MS-COCO 'train2014' (or another split disjoint from the test subset), evaluate on the same test subset used in Table 1, and ensure no overlap. Use a strict prefix-match rule for ASR (target phrase at the start of the generated response) and constrain the image perturbation to an l_inf ball (e.g., epsilon=8/255, PGD-10) in Algorithm 1. Recompute the GhostPrompt row for target 'Yes' on MiniGPT-4 and compare against GCG-Transfer. If the ASR drops below 0.64 or the >30% relative improvement disappears, the headline claim is not robust to the missing details.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of >30% relative ASR improvement over SoTA and ~70% training-time reduction is an empirical claim. Its validity depends on three details that the paper does not disclose: (1) whether the 20 training images (Sec. 8) are disjoint from the MS-COCO/ImageNet test subsets used in Table 1; (2) the exact criterion for 'matches the attacker-specified target response' (e.g., prefix, substring, or exact match); and (3) the projection bound (if any) for the image perturbation in Algorithm 1, which is called 'Projected Gradient Ascent' but shows no projection step. The default of 20 training images is selected from ASR-vs-count curves (Fig. 7) computed on the same benchmarks as the final evaluation, so the test set is used for hyperparameter choice. If the train/test split is leaked or the matching rule is loose, the reported ASR could be inflated and the cross-image transferability claim would not be supported. Moreover, an unbounded inner maximization could cause the prompt to trivially ignore the visual stream rather than learn robust image-invariant features. These are addressable omissions, but until they are fixed the headline numbers should not be taken at face value.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"GhostPrompt proposes a text-only adversarial suffix that, once optimized, can be appended to a prompt template and steer vision-language models (MiniGPT-4, BLIP-2, InstructBLIP, LLaVA-v1.5-7b) to attacker-chosen outputs across unseen images. The method uses a min-max objective (Eq. 3) over a continuous token distribution (Gumbel-Softmax) and a projected-gradient inner loop that synthesizes 'worst-case' images, plus losses for target likelihood, text coherence, and semantic alignment. Experiments report >30% relative ASR gains over baselines on MS-COCO/ImageNet subsets, ~70% training-time reduction, black-box transfer, and some defense resistance. The manuscript also states limitations: discrete-text ASR below continuous image attacks, limited cross-model transfer, and poor performance on commercial VLMs.","tokens_in":22235,"tokens_out":5950,"duration_ms":68978,"significance":"The paper's contribution is a new attack setting (cross-image transferable adversarial prompts) and a technically plausible min-max framework; the alternating optimization with Gumbel-Softmax and semantic-alignment regularizers is a reasonable approach. Strengths: code is promised, experiments cover four VLMs and many baselines, ablations isolate loss components, and defense evaluation (including adversarial training) goes beyond ASR-only reporting. If the empirical claims hold, this is a useful and timely result for the VLM security community. However, the current manuscript does not disclose enough evaluation-protocol detail to establish the central claim; the reported gains may be inflated by test-set tuning or loose matching. I therefore treat the significance as conditional on the missing protocol being supplied and the numbers rechecked.","major_comments":[{"comment":"The central claim — a single prompt transferring to unseen images with >30% relative ASR gain — requires a clean separation between the 20 training images and the MS-COCO/ImageNet test subsets. The paper never states this split. Moreover, Fig. 7 uses ASR on these same benchmarks to choose 20 as the default, so the test benchmarks are used for hyperparameter selection. Finally, 'matches the attacker-specified target response' is undefined: Fig. 2 shows responses longer than the target 'Yes', so exact-match, prefix-match, and containment give very different ASRs. Please report the split, the matching rule, and preferably a separate validation set for choosing the number of training images.","section":"Sec. 5.1 / Table 1 / Sec. 8 / Fig. 7"},{"comment":"Algorithm 1 calls the inner update 'Projected Gradient Ascent' but line 17 has no projection or clipping; the perturbation δ_v is unbounded. With an unbounded inner maximizer, the 'worst-case image' can drift arbitrarily far from the clean image, so the trained prompt can learn to ignore the visual stream entirely rather than to transfer across realistic images. This is not the 'subtle perturbations' threat model in the abstract. Also, Algorithm 1 initializes a single x'_v and never loops over multiple images, which is inconsistent with Eq. (3)'s expectation over D and with Sec. 8's '20 training images'. Please add the projection step (budget epsilon), and show how multiple training images enter the alternating loop.","section":"Algorithm 1 / Sec. 4.4 / Eq. (3)"},{"comment":"Evaluation-critical hyperparameters are not reported: lambda_align, lambda_tc, mu, alpha1, alpha2, K, N, tau, suffix length L, and the composition of I_benign/I_harmful in Eq. (9). Without these, the experiments cannot be reproduced or compared. Since ASR is sensitive to suffix length and to the alignment-loss weight, please include a full configuration table and the exact test-subset sizes.","section":"Sec. 5.1 / Eq. (7)"}],"minor_comments":[{"comment":"The text says incorporating the semantic alignment loss improves ASR by 'an average of 7%', but Table 3 implies average relative improvements of about 11–14% (MiniGPT-4: 0.620 vs 0.558; BLIP-2: 0.597 vs 0.541; InstructBLIP: 0.512 vs 0.447). Also, the statement that removing L_tc leads to a 'slight increase' is not true for MiniGPT-4 (0.616 vs 0.620).","section":"Sec. 8 / Table 3"},{"comment":"Typo: 'avalable' should be 'available'.","section":"Abstract"},{"comment":"The sizes of the MS-COCO and ImageNet subsets are not given. Reporting only 'a subset' is insufficient to assess statistical power or to compare with prior work.","section":"Sec. 5.1"},{"comment":"The cross-model transfer ASRs are very low (roughly 0.13–0.23). The text says GhostPrompt outperforms 'on the majority of benchmarks (2 out of 3)', but the figure is hard to read; please add numeric labels or a table.","section":"Fig. 5(c)"},{"comment":"Gumbel-Softmax temperature τ appears in Eq. (4) but no annealing schedule is described. Since τ strongly affects the soft-to-hard transition, reporting the schedule would improve reproducibility.","section":"Sec. 4.3"},{"comment":"The PF column reports PPL values, but the caption only says 'ASR'. Please clarify which columns are ASR and which are PPL.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to be acceptable after revision, but the evaluation protocol must be fixed: disjoint train/test split, exact matching rule, and a bounded projection in the inner image maximization. The code link is a plus, but the manuscript itself currently lacks enough detail to verify the headline numbers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, the cross-image setting is real: prior work either fixes the image and varies the prompt (CroPA, CIA) or optimizes universal pixel perturbations (UAPs). Nobody, as far as I can tell from the cited literature, optimizes a single textual suffix to transfer across heterogeneous images in VLMs. That is a legitimate and practically relevant gap, especially for prompt-sharing platforms and template-based assistants. Second, the method itself is sound in spirit: alternating between worst-case image synthesis and Gumbel-Softmax token-distribution updates is a clean way to force the prompt to rely less on any single image. The ablations in Table 3 and Fig. 6 give credit to the losses and to the worst-case strategy, and the authors are admirably honest in Section 9 about the method's limits on black-box transfer and commercial VLMs.\n\nBut the reported numbers are not yet evidence. The stress-test note is right on all three counts. Algorithm 1 calls the image update 'Projected Gradient Ascent' but shows no projection step and no epsilon bound. An unbounded inner maximization can produce degenerate worst-case images, and the reader cannot judge whether the prompt is learning image invariance or just ignoring a corrupted visual stream. Second, the ASR criterion is never precisely defined. The example in Figure 2 shows a response that starts with 'Yes' but continues with a full sentence, so 'matches' is clearly not exact equality. Prefix? Substring? This matters a lot for short targets like 'Yes' or 'too late.' Third, the default training image count of 20 is selected from curves in Fig. 7 computed on the same MS-COCO and ImageNet subsets used for the main evaluation in Table 1. That is selection on the test set, and the paper never states that the 20 training images are disjoint from the test subsets. Any one of these three omissions would be a minor fix; together they put the >30% relative improvement and the ~70% training-time reduction on shaky ground.\n\nThe central claims are plausible and the method is well-motivated, so I would not reject this on sight. But it needs a serious referee, and it needs revisions: report the projection bound, state the matching rule exactly, and specify and ideally re-randomize the train/test split. If the authors can do that, this becomes a solid contribution. As it stands, I'd treat the headline numbers with caution.\n\nI'd bring it to the reading group — it's a good case study in how evaluation details can obscure a genuinely novel result. I'd probably cite the setting in my own work, but only after the details are resolved.","headline":"Genuinely new cross-image prompt attack with a sensible min-max formulation, but the headline ASR gains rest on undisclosed evaluation details that need to be fixed before the numbers can be trusted.","tokens_in":22733,"tokens_out":2411,"would_cite":true,"duration_ms":30381,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GhostPrompt claims that one adversarial text suffix, trained once, can steer vision-language models to attacker-chosen outputs on many unseen images.","keywords":["vision-language models","adversarial attack","adversarial prompt","cross-image transferability","min-max optimization","continuous relaxation","attack success rate","prompt injection"],"falsifier":"Re-run the attack with a fixed, pre-registered split in which the 20 training images come from a hold-out set disjoint from all test images, count success only on exact full-target-string matches, and compute ASR per image rather than per prompt-image pair; if the reported >30% relative improvement over baselines collapses or reverses, the cross-image transferability claim does not hold.","tokens_in":21782,"feed_emoji":"🤖","tokens_out":5707,"duration_ms":60781,"temperature":0.7,"pith_summary":"The paper argues that text-based adversarial prompts for vision-language models do not have to be re-optimized for each image. It introduces GhostPrompt, a single adversarial suffix that is trained once and then, when appended to a benign prompt template, steers a VLM to produce an attacker-chosen response on many unseen images. The key move is to treat the image as a variable during optimization: an inner loop synthesizes the 'worst-case' image for the current suffix, and an outer loop updates the suffix so that the target response still wins under that hard visual condition. The authors report that this joint optimization raises attack success rates by over 30% relative to prior text-based attacks on four open VLMs while cutting training time by roughly 70%. If correct, this makes prompt-reuse attacks practical in settings where users share or download prompt templates.","feed_headline":"A single prompt, trained once, hijacks VLMs across unseen images","feed_subtitle":"GhostPrompt's min-max trick lifts attack success by over 30% while cutting training time by ~70%.","key_machinery":"The load-bearing mechanism is an alternating min-max optimization over two modalities. In the inner (image maximization) stage, a small perturbation is added to a clean training image via projected gradient ascent to maximize the negative log-likelihood of the target response, producing a 'worst-case' visual condition for the current suffix. In the outer (text minimization) stage, the suffix is represented as a continuous token distribution matrix — each row a soft distribution over the vocabulary — sampled through a continuous relaxation of discrete token choices so gradients flow end-to-end. Three losses are combined: a guided adversarial loss (negative log-likelihood of the target), a sem","core_discovery":"Central claim: a suffix optimized once in continuous token space, relaxed into soft one-hot vectors by a differentiable sampling trick and trained against adversarially perturbed 'worst-case' images, encodes image-invariant adversarial features. When appended to a prompt template, this discrete suffix steers four open VLMs toward attacker-chosen outputs on unseen MS-COCO and ImageNet images. Prior text attacks hold the image fixed, entangling the suffix with one visual input; GhostPrompt solves a min-max problem where an inner step constructs hard visual conditions and an outer step updates the suffix to stay effective under them, collapsing the model's internal states into an image-agnostic","pith_inferences":["A natural extension the paper does not test: whether the same min-max recipe transfers to video or multi-image inputs, where the 'worst-case' visual condition could be a temporal perturbation rather than a single-image one.","The semantic alignment loss is described as helping the suffix evade safety alignment; a reader could test this directly by comparing trigger success on aligned versus unaligned VLM checkpoints with and without that loss.","If the t-SNE interpretation is right — the suffix collapses the model's first-token distribution into one cluster — a defense could look for abnormally low variance in internal states across images, a direction the paper does not discuss.","The paper's black-box results are limited to open models; if the optimization were run on a stronger teacher, cross-architecture transfer might extend to commercial APIs, though the paper reports low success there and lists it as a limitation."],"forward_implications":["If GhostPrompt works as reported, a single poisoned prompt template can be distributed through prompt-sharing platforms and hijack interactions on many different user images without per-image optimization.","Perplexity filtering and tokenization defenses would be largely ineffective against the optimized suffix, while adversarial training that does reduce the attack also degrades clean-task accuracy.","The attack success holds across both MS-COCO and ImageNet subsets and across four different VLM architectures, suggesting the vulnerability is not tied to one dataset or model.","The roughly 70% reduction in training time would lower the cost of generating transferable adversarial prompts, making large-scale prompt poisoning more feasible."],"fun_headline_variants":["A single adversarial prompt hijacks VLMs on unseen images","One prompt, any image: universal VLM attack lifts success 30%","GhostPrompt's min-max trick: one prompt fools all images","Image-agnostic adversarial prompt: 30% better, 70% faster"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the reported attack success rates measure true cross-image generalization: the paper does not state the exact split between the roughly 20 training images and the MS-COCO/ImageNet test subsets, nor the precise string-matching rule for success, so a favorable selection or matching could inflate the gains.","fun_headline_variants_meta":{"raw":{"variants":["A single adversarial prompt hijacks VLMs on unseen images","One prompt, any image: universal VLM attack lifts success 30%","GhostPrompt's min-max trick: one prompt fools all images","Image-agnostic adversarial prompt: 30% better, 70% faster"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000144,"raw_usage":{"total_tokens":1005,"prompt_tokens":728,"completion_tokens":277,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":472,"completion_tokens_details":{"reasoning_tokens":200}},"tokens_in":472,"tokens_out":277,"duration_ms":4621,"temperature":1.0,"reasoning_tokens":200,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T11:59:30.671887+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the attack with a fixed, pre-registered split in which the 20 training images come from a hold-out set disjoint from all test images, count success only on exact full-target-string matches, and compute ASR per image rather than per prompt-image pair; if the reported >30% relative improvement over baselines collapses or reverses, the cross-image transferability claim does not hold.","supporting_citations":[],"review_version":1}