{"id":"b6bd1a16-cfd0-421f-976e-e388b6d49124","arxiv_id":"2509.06459","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":7,"one_line_summary":"Two new black-box adversarial attacks, one affine-search and one genetic, are benchmarked on four architectures and three datasets, with the genetic attack inducing higher fooling rates while augmentation benefits are inconsistent.","lead":"This paper introduces and benchmarks two black-box image attacks, one based on random geometric edits and one that adds a genetic algorithm, across four architectures and three datasets. It also claims that using these attacks as data augmentation improves classification accuracy by up to 8.82%, though that headline comparison is not a matched test.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline +8.82% accuracy improvement is an architecture gap, not an algorithm effect: it compares a no-augmentation ViT to prior CNN baselines, and ATA/AGA underperform no augmentation on that same row.","rationale":"The reader's weakest assumption is exactly the load-bearing issue: measured accuracy differences are attributed to the proposed algorithms, but the headline statistic comes from comparing different architectures and training regimes. The 8.82% figure is not an ATA/AGA augmentation gain; it is the gap between a no-augmentation ViT and Sandru et al.'s CNN baselines. If the algorithms themselves are evaluated against their own no-augmentation control, the improvement often disappears or reverses. This invalidates the paper's central 'better results than similar methods' claim. I also considered the black-box concern raised by the reader, but score-based access without gradients is a legitimate black-box setting, so this is secondary. The decisive flaw is attribution: the experiments do not isolate the effect of ATA/AGA. Therefore, the reader's REJECT verdict stands, and no verdict change is needed.","tokens_in":21063,"tokens_out":3725,"duration_ms":37194,"concrete_test":"Retrain ViT on Tiny-ImageNet-200 under identical hyperparameters and seeds in three conditions: no augmentation, ATA augmentation, and AGA augmentation. Report test accuracy and F1. If ATA/AGA do not beat no-augmentation by a statistically significant margin (or, as in Table 1, actually lower accuracy), the claim that the algorithms improve classification is refuted. Also recompute the headline by comparing the same architecture: e.g., ViT No Aug vs ViT+ATA/AGA, and DenseNet-121 No Aug vs DenseNet-121+ATA/AGA; the '+8.82%' should disappear.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (\"accuracy improvement of up to 8.82%\") is not supported by the paper's own Table 1. On Tiny-ImageNet-200, ViT with No Aug reaches 85.42% test accuracy, while Sandru et al.'s best result (DenseNet-121 FLA) is 76.60%; 85.42 - 76.60 = 8.82. This is exactly a comparison of a transformer architecture against CNN baselines from a different training recipe, not a comparison of ATA/AGA against similar methods. In the same table, on the same ViT row, ATA and AGA augmentation give 84.66% and 84.01%, both lower than the 85.42% no-augmentation baseline. Thus the headline number is an architecture/training effect, and the proposed algorithms contribute nothing to it. More generally, Table 1 shows inconsistent augmentation gains: ATA/AGA beat the no-augmentation baseline in only some configurations (e.g., Food-101 STV2) and degrade it in others (e.g., Tiny-ImageNet ViT, Caltech-256 DenseNet-121). Because the claimed \"better results than similar methods\" rests on the 8.82% statistic, and that statistic is not produced by the algorithms, the central contribution is not established. A matched same-architecture with/without augmentation evaluation is necessary.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes two black-box adversarial perturbation algorithms, Affine Transformation Attack (ATA) and Affine Genetic Attack (AGA), which combine random affine transformations and, for AGA, genetic operators and pixel noise. The authors benchmark these algorithms as data augmentation tools and as adversarial attacks on Tiny-ImageNet-200, Caltech-256, and Food-101, using ResNet-18, DenseNet-121, Swin Transformer V2, and Vision Transformer. They report attack success rates, targeted attack results, parameter sensitivity, and a qualitative comparison with Pixle and Square Attack. The abstract claims 'an accuracy improvement of up to 8.82%' over similar methods in the literature.","tokens_in":21468,"tokens_out":6857,"duration_ms":78282,"significance":"If the central claims were correct, the paper would offer simple, query-based black-box attacks and an augmentation scheme that improves classifier accuracy beyond prior affine-based methods. The experimental breadth is a genuine strength: three datasets, four architectures, repeated runs with means and standard deviations, and clearly written pseudocode for both algorithms. However, the headline accuracy improvement is not produced by the proposed algorithms: the 8.82% figure is an architecture/training-recipe difference between a no-augmentation ViT and CNN baselines from a different training setup, and the paper's own Table 1 shows that ATA and AGA augmentation frequently degrades accuracy relative to the same architecture without augmentation. The Pixle/Square comparison is anecdotal rather than a benchmark. Therefore the paper's main contribution, as stated, is not established, although the raw experimental material could be of interest if the claims were substantially reframed.","major_comments":[{"comment":"The '+8.82%' claim is an architecture gap, not an algorithm effect. On Tiny-ImageNet-200, the comparison is ViT with no augmentation (85.42%) versus Sandru et al.'s DenseNet-121 FLA (76.60%). This compares different architectures trained with different recipes, not ATA/AGA against a matched baseline. In the same ViT row, ATA (84.66%) and AGA (84.01%) are both lower than the no-augmentation result (85.42%). The central claim of accuracy improvement is therefore unsupported by the paper's own data; a matched same-architecture with/without augmentation comparison is required.","section":"Abstract; §4.1, Table 1"},{"comment":"The data augmentation benefit of ATA/AGA is inconsistent and often negative. Examples: Tiny-ImageNet-200 ViT, ATA 84.66% and AGA 84.01% versus No Aug 85.42%; Caltech-256 DenseNet-121, ATA 84.41% and AGA 83.80% versus No Aug 84.47%; Food-101 ResNet-18, ATA 72.31% versus No Aug 72.99%. Many of these differences are within one standard deviation. Thus the statement that the algorithms 'outperform a similar method' in data augmentation is not supported.","section":"Table 1; §4.1"},{"comment":"The comparison with Pixle and Square Attack is qualitative and anecdotal: 12 selected images with success counts of 6/12 for ATA, 8/12 for AGA, 5/12 for Square, and 4/12 for Pixle. There is no query budget, no perturbation norm, no aggregate test-set metric, and no measure of variance or selection protocol. This cannot support the abstract's claim of benchmarking against these methods; a quantitative attack comparison under a defined protocol is needed.","section":"§4.4, Fig. 6"},{"comment":"The 'black-box' setting is imprecise. Both algorithms compute CE(M(X), y), which requires soft scores or logits, not just hard labels. If 'black-box' is intended to mean gradient-free but score-based, the threat model should be stated explicitly and the comparison with Pixle/Square should account for the different information available to each method. As written, the algorithms do not operate in a decision-only black-box setting.","section":"§3.3, Alg. 1, Alg. 2"}],"minor_comments":[{"comment":"The dataset diversity factor df is introduced but never used in the analysis. The text says 'We apply Eq. (1) to obtain balanced difficulty insights,' but no such insights are derived. Either use df in the discussion or remove it.","section":"§3.1, Eq. (1)"},{"comment":"The limitations sentence says 'we limited the population size (ni) to 3'; this should refer to the population size np, since ni is the number of iterations in Alg. 2.","section":"§3.4"},{"comment":"Tiny-ImageNet-200 is described as having 'a total of 550 images per class.' The standard split is 500 training and 50 validation images per class; the paper later splits data into train/validation/test 0.8/0.1/0.1. Please clarify which images are used and how the 550 figure is obtained.","section":"§3.1"},{"comment":"The parameter-variation figures do not show error bars or confidence bands, even though the text quotes standard deviations up to 7–8%. Adding uncertainty information would make the sensitivity claims more interpretable.","section":"§4.3, Figs. 5 and 7"}],"recommendation":"reject","confidential_remarks":"The paper would need substantial new matched experiments and a repositioning of its claims before it could be considered. The current abstract's main quantitative claim is directly contradicted by Table 1, and the Pixle/Square comparison is not a benchmark. A future version could be viable as a more modest empirical study of affine/genetic attacks and their parameter sensitivity, with claims restricted to attack success rates and with the augmentation claims removed or re-evaluated under matched conditions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nThe paper is a solid empirical sweep of two attack algorithms that combine affine transformations and genetic search, but the headline result is not what it seems. The 8.82% accuracy improvement in the abstract comes from comparing a ViT trained without augmentation to CNN baselines from a different training recipe. In their own Table 1, on the same ViT row, ATA and AGA both underperform the no-augmentation baseline. So the central claim that their algorithms 'yield better results than similar methods' is not supported.\n\nWhat the paper does well: it runs a large number of experiments—four architectures, three datasets, parameter sweeps, defended and undefended settings, targeted attacks—with five runs and standard deviations. The parameter sensitivity analysis is useful, and the paper is honest about its memory constraints and training limitations. The observation that transformers are more vulnerable to undefended attacks but benefit more from adversarial augmentation is a genuine, if not entirely new, insight.\n\nThe soft spots are several. The most serious is the headline comparison. A matched same-architecture with/without augmentation is needed to claim any algorithm effect. The augmentation results are inconsistent, with many cells below the no-augmentation baseline, and the paper does not address this pattern. The comparison to Pixle and Square Attack is purely qualitative on 12 images, with no query budgets or distortion metrics, so it doesn't support any quantitative claim. There is no code or data release, which limits reproducibility. AGA is closely related to GenAttack, which is cited but not benchmarked; that omission weakens the novelty claim. The 'dataset diversity factor' is defined but barely used. The 'black-box' label is fair if interpreted as query-based rather than hard-label-only, since the algorithms need soft cross-entropy scores.\n\nThe math and data look sound; the experiments are internally consistent. The problem is in the framing and the claims, not the execution.\n\nWho is this for: a reader doing adversarial robustness benchmarking could use the reference numbers, but should not use the paper's own conclusions. It deserves a serious referee if the authors are willing to rewrite the claims and add matched comparisons. I'd send it to review with the expectation of major revisions rather than desk reject. The reader's rejection is right about the central flaw, but I wouldn't throw out the empirical data.\n\nRegards,","headline":"The 8.82% headline is an architecture gap, not an algorithm effect—but the empirical breadth deserves referee time if the authors fix the claims.","tokens_in":21889,"tokens_out":4701,"would_cite":false,"duration_ms":46969,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","68T45","68W50"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that two output-only adversarial algorithms, one affine-iterative and one genetic, work as both attacks and data augmentation, with accuracy gains up to 8.82% over prior baselines and attack success rates above 60% on unde","keywords":["adversarial attacks","black-box attacks","genetic algorithms","affine transformations","data augmentation","vision transformers","robustness","image classification"],"falsifier":"Train one architecture (for example, ViT on Tiny ImageNet-200) with exactly the same recipe, once with and once without ATA/AGA augmentation, and compare test accuracy; if accuracy does not rise, the augmentation claim collapses. Separately, run AGA with hard-label feedback only; if attack success drops sharply, the 'black-box' claim as framed, which uses soft cross-entropy losses, is not transferable to label-only settings.","tokens_in":21000,"feed_emoji":"🧬","tokens_out":7155,"duration_ms":74698,"temperature":0.7,"pith_summary":"The paper introduces and benchmarks two black-box iterative adversarial algorithms: ATA, which mutates images with random affine transformations and keeps the highest-scoring candidate, and AGA, which adds a genetic population, mutation, crossover, and noise to the same score-maximizing search. The authors claim both work as attacks and as data augmentation: training on adversarially augmented images raises classification accuracy, with a reported gain of up to 8.82% over similar methods in the literature, and the genetic variant is the stronger attack, reaching success rates above 60% on undefended transformers. They also report that transformer models are more vulnerable to undefended attacks but benefit disproportionately from adversarial augmentation, becoming more robust than CNNs. If true, the same cheap, model-agnostic perturbation recipe can expose model weaknesses and harden them without needing gradients or model internals.","feed_headline":"Black-box affine and genetic attacks beat prior baselines by 8.82%","feed_subtitle":"Two new algorithms, one iterative-affine and one genetic, both attack models and augment training sets.","key_machinery":"The attack score f_attack, a logistic (sigmoid) mapping of the model's cross-entropy loss, is the selection signal. ATA samples random affine transformations (rotation, translation, scaling, shearing) each iteration and keeps the candidate that maximizes f_attack, while AGA wraps the same score in a genetic loop: a population is mutated with affine transforms plus bounded uniform noise, recombined by swapping image rows, and the fittest individual is cloned to seed the next generation. This shared score function is what makes both algorithms black-box (only model outputs are needed) and what makes them usable in both attack and augmentation modes.","core_discovery":"The paper's central claim is that a single parameter-light family of black-box image perturbations, built from affine transformations and (in one variant) genetic search, can serve both as an adversarial attack and as a data-augmentation tool. ATA repeatedly applies random rotations, translations, scalings, and shears, keeping the candidate that maximizes a logistic score over the model's cross-entropy loss; AGA adds a population, mutation with bounded noise, row-swap crossover, and elitist reselection around the same score. On Caltech-256, Food-101, and Tiny ImageNet-200, across ResNet-18, DenseNet-121, Swin Transformer V2, and ViT, the authors report that training with ATA/AGA-generated ad","pith_inferences":["The headline +8.82% compares a ViT without augmentation to prior CNN baselines; a same-architecture, same-recipe ablation would likely shrink the gap. The paper's augmentation value is better supported by the paired No Aug. versus ATA/AGA columns within each architecture.","The methods are black-box in the sense of needing no gradients or parameters, but they require soft outputs (cross-entropy losses); under a true label-only API, success rates would probably fall. A label-only variant would be a direct test.","Targeted attacks with negative success rates indicate that in some class/model combinations the 'attack' improves accuracy for that class, hinting that the same machinery could be used for targeted class repair rather than only confusion.","The crossover operator, swapping whole image rows, is crude and appears insensitive; recombination at the level of patches or feature maps might make the genetic search more effective."],"forward_implications":["If ATA/AGA are as effective as reported, practitioners can use one algorithm family for both robustness auditing and training-set expansion on image classifiers.","Transformer-focused robustness work should treat undefended transformers as high-risk: AGA reaches 63.97% attack success on an undefended ViT, so deploying unhardened ViTs may be riskier than CNNs.","Adversarial augmentation with the same algorithms is a viable defense: on Tiny ImageNet-200, AGA-augmented ViT drops attack success by 49.6 percentage points, the largest defense effect observed.","Because AGA's crossover probability has little effect while iterations, mutation probability, and noise intensity drive success, tuning effort can concentrate on the latter three.","ATA's stability across iterations (0.5-4% attack-success variation) makes it a drop-in augmentation module or a component for more complex attack pipelines."],"supporting_citations":[{"why":"Provides the prior augmentation baselines (base and FLA) against which the headline accuracy comparison is measured.","marker":"[35]"},{"why":"Serves as a compared black-box attack in the benchmark and qualitative evaluation.","marker":"[3]"},{"why":"Serves as the second compared black-box attack, against which ATA/AGA success is measured.","marker":"[32]"},{"why":"Supplies the genetic black-box attack approach that AGA adapts by adding affine transformations and noise.","marker":"[2]"},{"why":"Cited as inspiration for generating adversarial examples under transformations, informing ATA's affine sampling.","marker":"[4]"},{"why":"Cited as inspiration for affine transformation-based image manipulation, informing ATA's design.","marker":"[38]"},{"why":"Defines the Vision Transformer architecture whose augmented training yields a large reported accuracy gain.","marker":"[11]"},{"why":"Defines the Swin Transformer V2 architecture that achieves the best results on two datasets.","marker":"[24]"},{"why":"Provides the analysis of vision-transformer robustness and low-frequency reliance used to interpret attack outcomes.","marker":"[37]"}],"fun_headline_variants":["Affine and genetic attacks outperform baselines by 8.82%","Iterative affine and genetic black-box attacks win by 8.82%","New black-box attacks: affine+genetic beat Square Attack by 8.82%","Affine and genetic adversarial perturbations boost accuracy by 8.82%","ATA and AGA: black-box attacks that also augment data, win by 8.82%"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The results stand on the assumption that the reported accuracy differences come from the proposed algorithms rather than from architecture or training-recipe differences; the largest claimed gain compares a ViT trained without augmentation to CNN baselines from prior work, not a matched with/without ablation.","fun_headline_variants_meta":{"raw":{"variants":["Affine and genetic attacks outperform baselines by 8.82%","Iterative affine and genetic black-box attacks win by 8.82%","New black-box attacks: affine+genetic beat Square Attack by 8.82%","Affine and genetic adversarial perturbations boost accuracy by 8.82%","ATA and AGA: black-box attacks that also augment data, win by 8.82%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000606,"raw_usage":{"total_tokens":2692,"prompt_tokens":802,"completion_tokens":1890,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":1784}},"tokens_in":546,"tokens_out":1890,"duration_ms":13577,"temperature":1.0,"reasoning_tokens":1784,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T23:35:13.718658+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train one architecture (for example, ViT on Tiny ImageNet-200) with exactly the same recipe, once with and once without ATA/AGA augmentation, and compare test accuracy; if accuracy does not rise, the augmentation claim collapses. Separately, run AGA with hard-label feedback only; if attack success drops sharply, the 'black-box' claim as framed, which uses soft cross-entropy losses, is not transferable to label-only settings.","supporting_citations":[{"cited_title":"Sandru, M.-I","cited_arxiv_id":null,"evidence_quote":"Provides the prior augmentation baselines (base and FLA) against which the headline accuracy comparison is measured."},{"cited_title":"Pomponi, S","cited_arxiv_id":null,"evidence_quote":"Serves as the second compared black-box attack, against which ATA/AGA success is measured."},{"cited_title":"Alzantot, Y","cited_arxiv_id":null,"evidence_quote":"Supplies the genetic black-box attack approach that AGA adapts by adding affine transformations and noise."},{"cited_title":"Athalye, L","cited_arxiv_id":null,"evidence_quote":"Cited as inspiration for generating adversarial examples under transformations, informing ATA's affine sampling."},{"cited_title":"Dosovitskiy, L","cited_arxiv_id":null,"evidence_quote":"Defines the Vision Transformer architecture whose augmented training yields a large reported accuracy gain."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Swin Transformer V2 architecture that achieves the best results on two datasets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the analysis of vision-transformer robustness and low-frequency reliance used to interpret attack outcomes."}],"review_version":1}