{"id":"d680af87-05cc-4a61-bfa2-91b1a2c7ffd6","arxiv_id":"2505.07258","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A new attack scenario, Victim Data-based Attack (VDBA), generates transferable adversarial examples for text classifiers using only unlabeled victim texts, achieving over 40% attack success on several models with zero queries.","lead":"This paper introduces an adversarial attack on text classifiers that uses only the text to be attacked, with no queries to the target model and no training data. It clusters the victim texts to make a shadow dataset, trains substitute models on it, and shows the attacks transfer to BERT, RoBERTa, and LLMs like GPT-4o and Qwen2.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Assumption 4.1 is asserted without a control: the reported ASR could reflect victim-model label instability rather than cluster-boundary transfer; a random-label and random-perturbation control is needed.","rationale":"The reader's weakest assumption is Assumption 4.1, the high-level attack hypothesis, and I agree that this is the load-bearing premise. My stress-test sharpens the concern: the paper provides no control that separates genuine transfer from victim-model label instability. Without such a control, the reported ASR figures are consistent with a weaker explanation, namely that many synonym-level perturbations flip these classifiers regardless of the substitute model. The hierarchy and diversity components (Theorems 4.2 and 4.3) only guarantee improved success on the substitute models, not on the victim, so they do not compensate for the missing control. Section 5.3's finding that clustering and embedding choices have only a random impact further weakens the claim that the specific cluster structure is essential. I therefore would not raise or lower the reader's conditional verdict; the paper remains plausible but requires the control experiment before the central mechanism is established.","tokens_in":20156,"tokens_out":3512,"duration_ms":37882,"concrete_test":"Run the full VDBA pipeline on SST5 and Emotion with three pseudo-label conditions, keeping all other hyperparameters fixed: (A) K-means labels as in the paper; (B) randomly permuted K-means labels, preserving cluster sizes; (C) no substitute model at all, using random synonym substitutions matched to VDBA's average semantic similarity (Sim ~ 0.95). Compare ASR on DistilBERT and RoBERTa. If condition (B) or (C) achieves ASR within 5 percentage points of the reported values (52.08% and 45.03% on SST5), then the observed ASR does not require cluster-boundary transfer and Assumption 4.1 is not supported. If condition (A) exceeds both controls by more than 10 percentage points, the high-level attack hypothesis receives direct empirical support.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on Assumption 4.1 (High-level Attack Hypothesis, Section 4.1): adversarial examples generated on a substitute model trained with K-means pseudo-labels transfer to the victim model. This is never tested independently of the full pipeline. The paper reports that VDBA achieves 52.08% ASR on SST5 with DistilBERT and 45.03% with RoBERTa at zero query cost, but there is no baseline that measures how often arbitrary or random edits flip the victim model's prediction. If the victim models are even mildly unstable under synonym substitution, a substantial fraction of the reported ASR could occur without any alignment between the pseudo-label clusters and the victim decision boundary. The paper's own ablation results are consistent with this worry: Section 5.3 reports that clustering and embedding methods have a 'limited and somewhat random impact' on ASR, and one-hot embeddings lose only 1.5% ASR relative to pretrained embeddings. That suggests the specific cluster structure may not be doing the work attributed to it by Assumption 4.1. Theorems 4.2 and 4.3 do not rescue the argument: they only show that adding more independent candidate models or methods increases the chance that at least one candidate succeeds on the substitute model, assuming the candidate success probabilities are meaningful for the victim. They do not establish transferability. A control experiment that decouples transfer from label instability would settle whether the high-level attack hypothesis is a real mechanism or an artifact of victim-model sensitivity.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces VDBA, a textual adversarial attack that assumes access only to victim texts (no victim model queries, no gradients, no training data). The method embeds victim texts, assigns binary pseudo-labels via K-means clustering, trains substitute classifiers on the resulting shadow dataset, iteratively refines clusters to train a hierarchy of substitute models, and finally generates candidate adversarial examples with several attack methods, selecting the candidate with the best combination of predicted-probability change and semantic similarity. Experiments on SST5, Emotion, AG News, TREC6, and several closed/open LLMs report attack success rates up to 52.08% at zero query cost, substantially outperforming baselines that are allowed up to 35 accesses per text. The authors claim state-of-the-art performance in both ASR and Query metrics and conclude that even closed LLMs are vulnerable under this threat model.","tokens_in":20446,"tokens_out":3743,"duration_ms":39432,"significance":"If the central empirical claim holds, VDBA meaningfully expands the black-box threat model for text classification: it removes the need for model queries, training data, and substitute-model access, relying only on victim texts or topic-related texts. The paper's strengths include a clear problem formulation, experiments across multiple datasets and victim models, evaluation against several strong baselines, and an anonymous code release. The claim that victim texts alone can drive transferable attacks is interesting and potentially important for security auditing. However, the load-bearing mechanism, the High-level Attack Hypothesis (Assumption 4.1), is asserted rather than validated independently; without a control for victim-model label instability, the reported ASR numbers do not uniquely support the proposed clustering-based explanation. The lack of repeated runs or error bars and the use of the test sets for hyperparameter selection further weaken the quantitative claims. A revision that adds the missing controls and uncertainty quantification could make this a solid contribution.","major_comments":[{"comment":"The central mechanism of VDBA is Assumption 4.1, which states that adversarial examples generated on a substitute model trained with high-level pseudo-labels transfer to the victim model. This assumption is never tested independently of the full attack pipeline. The reported ASR could in part reflect victim-model label instability under synonym substitution rather than alignment between the pseudo-label clusters and the victim decision boundary. The paper needs a control experiment that decouples these factors, for example: (i) train substitute models with random pseudo-labels and measure the resulting ASR; and (ii) apply random synonym replacements to victim texts and measure how often the victim model changes its prediction. The paper's own ablation results in Section 5.3 and Appendix H, which find that clustering and embedding choices have a 'limited and somewhat random impact' on ASR and that one-hot embeddings lose only 1.5% ASR relative to pretrained embeddings, make this concern concrete: if the specific cluster structure were doing the work attributed to it by Assumption 4.1, one would expect clustering and embedding choices to matter more. This control is load-bearing for the paper's core claim and should be added before publication.","section":"Section 4.1, Assumption 4.1; Section 5.3"},{"comment":"Many core hyperparameters are selected through ablations on the same test sets used to report final results. Specifically, the number of clusters K, the number of hierarchical iterations, the number of attack methods, the scaling factors alpha and beta, and the thresholds epsilon' and epsilon* are all chosen after inspecting ASR and Sim on the test sets (Section 5.1, Section 5.3, Tables 8-9, Figure 8). No validation split or cross-validation is described, and no repeated runs or error bars are reported. Because the central claim is that VDBA achieves SOTA ASR, the paper should report the variance across at least several independent runs (or across seeds) and should either fix hyperparameters using a validation set or justify why the reported configuration is not over-fit to the test data.","section":"Section 5.1, Section 5.3, Eq. (9)"},{"comment":"Theorems 4.2 and 4.3 are order-statistics facts: as the number of substitute models or attack methods grows, the probability that the maximum of independent candidate success probabilities exceeds a fixed value approaches one. These theorems do not establish transferability from the substitute models to the victim model; they only say that under an independence assumption, more candidates are better. The proof of Theorem 4.2 in Appendix F explicitly assumes independence among the success probabilities, and Theorem 4.3 assumes 'approximate independence' among attack methods. These assumptions are not empirically checked, and even if they hold, the theorems cannot rescue Assumption 4.1. The paper should either present these results as heuristic motivation (with the independence assumptions clearly stated in the main text) or provide evidence that the candidate success probabilities are indeed independent and predictive of victim model behavior.","section":"Theorems 4.2 and 4.3; Appendix F"},{"comment":"The claim that 'a higher number of class labels is associated with a smaller average distance and higher ASR' is not supported by the data in Table 4. Only four training datasets are compared, and they differ in domain (sentiment vs. question classification) as well as label count. Go Emotion (28 labels, 85.68% ASR) and SST5 (5 labels, 45.03% ASR) are both sentiment, but TREC6 (6 labels, 19.37% ASR) is question classification and Financial PhraseBank (3 labels, 33.76% ASR) is sentiment. These four points cannot establish a monotonic relationship between label count and ASR; the effect is confounded by domain and dataset difficulty. This claim should be removed or supported by a controlled comparison that varies label count while holding domain and data distribution fixed.","section":"Section 5.4, Table 4"}],"minor_comments":[{"comment":"The abstract states 'achieving an ASR improvement of 52.08%', but Table 1 reports an ASR of 52.08% for SST5 with DistilBERT. The improvement over the best baseline (HQA) is about 5.97 percentage points; the wording should be corrected to 'an ASR of 52.08%' or 'an improvement of 5.97 points'.","section":"Abstract and Section 5.2"},{"comment":"There are unresolved references to 'Appendix ??' and 'Section ??' in Section 5.2 and the appendix list. These placeholders should be fixed before submission.","section":"Section 5.2 and Appendix"},{"comment":"Equation (5) introduces notation ~PM and gamma-hat but is not integrated into the surrounding text; it is unclear whether it belongs to the hierarchical substitute model design or is a leftover from a prior draft. Please remove it or explain its role.","section":"Section 4.2, Eq. (5)"},{"comment":"The text says 'As shown in Figure 2, DBA comprises three components' but the method is called VDBA. This typo should be corrected.","section":"Figure 2 and Section 4"},{"comment":"Table 2 reports a single Sim column per dataset but has four ASR columns (two models per dataset). It should either report Sim separately for each model or explain that the reported Sim is averaged across models.","section":"Table 2"},{"comment":"The LLM experiments do not report details such as the number of sampled texts, the API temperature, the number of repeated calls, or the exact version of GPT-4o and GPT-4omini. These details are needed to assess the reliability and reproducibility of the LLM attack results.","section":"Section 5.4, LLM Attack"}],"recommendation":"major_revision","confidential_remarks":"The paper's novelty claim of being the first victim-text-only attack should be checked against the broader label-free and unsupervised transfer-attack literature; the current related-work section does not discuss any prior work that uses unlabeled victim data or clustering to build substitute models. If such work exists, the authors should cite and compare with it. The main empirical claims are conditional on the unvalidated High-level Attack Hypothesis and on the absence of random-label/random-perturbation controls; I would recommend requiring those controls in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth reading, and worth a serious referee, but the load-bearing premise is under-supported. The paper introduces a threat model that I have not seen before: textual adversarial attacks with zero queries, zero model access, and zero training data, using only the victim texts. The pipeline—clustering unlabeled victim texts into binary pseudo-labels, training hierarchical substitute models, then combining multiple off-the-shelf attack methods with a selection rule—is a real combination of known parts into a new scenario. The reported ASRs, including 52.08% on SST5 with DistilBERT at zero query cost, are substantial enough that I would not dismiss the attack out of hand. Credit is also due for extending the evaluation to several datasets, two BERT-scale victim models, and actual LLM APIs.\n\nThe main soft spot, as the stress-test note says, is Assumption 4.1. The High-level Attack Hypothesis is asserted, not tested. There is no control measuring how often arbitrary or random synonym substitutions flip the victim model's prediction. If the victim models are even mildly unstable under small edits, a chunk of the reported ASR would occur without any alignment between the pseudo-label clusters and the victim decision boundary. The paper's own ablation fans this worry: clustering and embedding methods have \"limited and somewhat random impact,\" and one-hot embeddings lose only 1.5% ASR relative to pretrained embeddings. That is not what you would expect if the specific cluster structure were driving transfer. Theorems 4.2 and 4.3 are order-statistics facts—more candidates means a higher chance the best candidate succeeds—and they do not establish transferability.\n\nOther issues are real but smaller. There are no error bars or repeated runs. Several hyperparameters (α, β, thresholds, iterations, number of methods) seem to have been selected using the same test sets, which makes the reported numbers look cleaner than they probably are. The abstract says \"ASR improvement of 52.08%\" when 52.08% is the ASR itself, not an improvement. One appendix reference is missing. These are fixable in revision.\n\nWho is this for? People working on adversarial NLP threat models and on the limits of black-box attacks. It is not a conclusive paper, but it opens a new attack surface and gives enough empirical evidence to justify a full review. A random-label control, a random-perturbation control, and error bars would fix the main weakness. I would send it to peer review, and I would ask for those controls before acceptance. If the code is released with a commit hash, that also helps.","headline":"A genuinely new zero-query attack scenario that deserves referee time, but the transfer mechanism is asserted rather than tested and the headline numbers may be inflated by victim-model instability.","tokens_in":21008,"tokens_out":1844,"would_cite":true,"duration_ms":19536,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Text-only adversarial attacks need zero queries and can fool closed LLMs with up to 45.99% success.","keywords":["textual adversarial attack","victim data attack","zero-query attack","pseudo-label clustering","substitute model","transferability","LLM security","black-box attack"],"falsifier":"Run VDBA with randomly shuffled or deliberately misaligned cluster pseudo-labels on SST5 and Emotion while keeping all other components identical; if ASR stays near the reported 40–52%, the coarse semantic boundary is not what carries the attack.","tokens_in":19920,"feed_emoji":"🎯","tokens_out":8051,"duration_ms":71186,"temperature":0.7,"pith_summary":"This paper tries to establish that textual adversarial attacks can be mounted with almost no information about the target: no queries to it, no access to its weights or gradients, and no training data—only the victim texts themselves. The proposed framework, VDBA, embeds those texts, splits them into two clusters, and treats the clusters as pseudo-labels for training substitute binary classifiers. It then trains a hierarchy of finer substitutes and generates candidate adversarial examples with several existing attack methods, selecting the candidate with the best combination of predicted-probability shift and semantic similarity. On SST5 and Emotion, the paper reports attack success rates above 40% against small classifiers and up to 45.99% against closed LLMs such as Qwen2 and GPT-4o variants, all with zero queries during generation. If correct, this removes the standard prerequisites for black-box attacks and implies that exposed texts alone are a serious attack surface for modern NLP systems.","feed_headline":"Zero-query text attacks fool LLMs at 45.99 percent","feed_subtitle":"Clustering a victim's own texts into pseudo-labels trains substitutes that transfer to closed AI models.","key_machinery":"The load-bearing mechanism is the hierarchical substitute-model family trained on pseudo-labeled shadow data. VDBA starts with a binary K-means split of the victim texts into $C_{pos}$ and $C_{neg}$, trains substitute $f_s^1$, then re-clusters the texts inside each cluster to train finer substitutes $f_s^2, f_s^3$, and so on, producing $2^0 + 2^1 + \\cdots + 2^u$ models over $u$ iterations. For a victim text, every attack method $M_l$ is applied to every substitute $f_s^j$, giving $wU$ candidate perturbations; the final pick maximizes $s_i^{lj} = \\alpha\\,(p_{\\hat y}^{f_s^j}(x_i^{lj}) - p_{\\hat y}^{f_s^j}(\\tilde x_i^{lj})) + \\beta\\,\\cos(f_e(x_i), f_e(\\tilde x_i^{lj}))$, subject to thresholds $\\epsilon'$ on the probability drop and $\\epsilon^*$ on embedding similarity. The paper's theoretical support is that more substitute models make it overwhelmingly likely that at least one transfers (Theorem 4.2), and more attack methods make it overwhelmingly likely that at least one candidate satisfies the similarity requirement (Theorem 4.3). The transfer itself relies on the claim that coarse pseudo-label boundaries are aligned with the victim's fine-grained labels, so flipping 'dog' to 'non-dog' on the substitute pushes a 'Corgi' example away from 'Corgi' in the victim.","core_discovery":"On the paper's own terms, the central discovery is that a shadow dataset built from unlabeled victim texts can replace the substitute model that transfer attacks normally require. Two-cluster K-means on T5 embeddings produces high-level pseudo-labels, and a small transformer trained on those labels generates perturbations that flip the substitute's coarse decision; the claim is that the same perturbation also flips the victim's fine-grained label. The reported numbers are state-of-the-art on the tested setups, with VDBA reaching 52.08% ASR on SST5 with DistilBERT, 45.03% on SST5 with RoBERTa, and 45.99% on Qwen2, all at zero query cost. The same pipeline keeps working with five victim texts (ASR above 27%) and even with one victim text plus attribute-related online texts (up to 50.36% ASR), which is the paper's evidence that the attack is not an artifact of abundant data.","pith_inferences":["If the coarse-boundary transfer hypothesis is correct, the same pseudo-label clustering recipe should transfer to other modalities: an image or audio model whose labels refine a coarse taxonomy should be attackable with unlabeled inputs from that taxonomy alone.","A direct test of the paper's core assumption is to replace the K-means pseudo-labels with random binary labels while keeping everything else fixed; if the attack success rate barely moves, the transfer comes from training any binary substitute rather than from semantic cluster alignment.","The paper's own adversarial-training appendix (ASR dropping from roughly 49.81% to 22.27% on SST5) points to a defense that needs no access to the attacker: training on perturbed text roughly halves this attack's effectiveness, so API providers could blunt the threat without changing their public interface."],"forward_implications":["Attacks no longer need model feedback: an attacker who can read the texts being classified can generate transferable adversarial examples without spending queries, which removes the main practical barrier against API-deployed models.","Closed LLMs are exposed through the same channel: the paper reports 27.65–45.99% ASR on GPT-4o, GPT-4o mini, and Qwen2 with no API access, suggesting that current alignment and safety training do not close this transfer path.","Very small text samples still support the attack: with only five victim texts the reported ASR stays above 27%, and with one victim text plus attribute-related texts it reaches 50.36% on SST5 with DistilBERT.","Attack strength trades against similarity: raising the number of hierarchical iterations from three to four jumps ASR (to 64.49% on SST5 with DistilBERT) but lowers semantic similarity from 0.950 to 0.916, so stronger attacks are visibly less natural."],"supporting_citations":[{"why":"K-means binary clustering that assigns the pseudo-labels forming the shadow dataset.","marker":"Lloyd (1982)"},{"why":"T5 text-to-text transformer used as the embedding model to vectorize victim texts before clustering.","marker":"Raffel et al. (2020)"},{"why":"SST5 dataset supplies the victim texts and evaluation benchmark for the main experiments.","marker":"Socher et al. (2013)"},{"why":"Emotion dataset supplies victim texts and labels for the second evaluation suite.","marker":"Saravia et al. (2018)"},{"why":"BAE is one of the attack methods whose candidates VDBA generates on the substitute models.","marker":"Garg & Ramakrishnan (2020)"},{"why":"Hotflip is one of the white-box word-replacement generators used in the diverse generation step.","marker":"Ebrahimi et al. (2018)"},{"why":"PSO is another candidate generation method used in the diverse ensemble.","marker":"Zang et al. (2020)"},{"why":"TextBugger contributes character and word perturbation candidates to the selection pool.","marker":"Ren et al. (2019)"}],"fun_headline_variants":["Zero-query attacks hit 45.99% on LLMs via victim texts","No queries, just texts: attack LLMs with 45.99% success","Exploiting victim data yields zero-query LLM attacks","Clustering victim texts fools GPT and Qwen at 46%","Victim texts alone train black-box attacks on LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire attack rests on the unproven claim that an example which flips a coarse pseudo-label on the substitute (dog to non-dog) will also flip the victim's fine-grained label (Corgi to something else), even though the two models were trained on different labels.","fun_headline_variants_meta":{"raw":{"variants":["Zero-query attacks hit 45.99% on LLMs via victim texts","No queries, just texts: attack LLMs with 45.99% success","Exploiting victim data yields zero-query LLM attacks","Clustering victim texts fools GPT and Qwen at 46%","Victim texts alone train black-box attacks on LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000182,"raw_usage":{"total_tokens":1344,"prompt_tokens":1013,"completion_tokens":331,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":629,"completion_tokens_details":{"reasoning_tokens":237}},"tokens_in":629,"tokens_out":331,"duration_ms":3546,"temperature":1.0,"reasoning_tokens":237,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:21:00.995690+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run VDBA with randomly shuffled or deliberately misaligned cluster pseudo-labels on SST5 and Emotion while keeping all other components identical; if ASR stays near the reported 40–52%, the coarse semantic boundary is not what carries the attack.","supporting_citations":[{"cited_title":"Least squares quantization in pcm","cited_arxiv_id":null,"evidence_quote":"K-means binary clustering that assigns the pseudo-labels forming the shadow dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"T5 text-to-text transformer used as the embedding model to vectorize victim texts before clustering."},{"cited_title":"T., Huang, Y.-C., Wu, J., and Chen, Y.-S","cited_arxiv_id":null,"evidence_quote":"Emotion dataset supplies victim texts and labels for the second evaluation suite."},{"cited_title":"and Ramakrishnan, G","cited_arxiv_id":null,"evidence_quote":"BAE is one of the attack methods whose candidates VDBA generates on the substitute models."},{"cited_title":"Hotflip: White-box adversarial examples for text classification","cited_arxiv_id":null,"evidence_quote":"Hotflip is one of the white-box word-replacement generators used in the diverse generation step."},{"cited_title":"Word-level textual adversarial attacking as combinatorial optimization","cited_arxiv_id":null,"evidence_quote":"PSO is another candidate generation method used in the diverse ensemble."},{"cited_title":"Generating natural language adversarial examples through probability weighted word saliency","cited_arxiv_id":null,"evidence_quote":"TextBugger contributes character and word perturbation candidates to the selection pool."}],"review_version":1}