{"id":"d38b7d5b-0d3d-4813-b653-31356cc46ea7","arxiv_id":"2608.13296","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"The paper proposes black-box adversarial attacks on image classifiers as a high-dimensional benchmark for global optimization and compares seven metaheuristics under shared query budgets.","lead":"This paper proposes using black-box adversarial image attacks as a new, high-dimensional benchmark for global optimization methods, and tests seven metaheuristics against it. A generalist reader might care because it connects classical optimization benchmarking to modern machine learning safety, where query budgets and rugged landscapes are the real constraints.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Seed-dependent SGA-BSR endpoints do not establish multimodality; the central benchmark claim is unsupported without a unimodal control.","rationale":"The central claim is that BBAA is multimodal and high-dimensional, and therefore a valid benchmark for global optimization. The paper's only direct evidence for multimodality is the seed-dependence of SGA-BSR runs. The reader correctly identified that the implicit premise—that a unimodal objective would force all runs to the same point—is false. My analysis sharpens this: SGA-BSR is not even attempting to find a local optimum in the growth phase; it stops at the first misclassification, i.e. at a level-set boundary. Thus path-dependent outcomes are expected even for a strictly concave objective. This is the load-bearing weakness because without multimodality the paper's main motivation for proposing BBAA as a global-optimization benchmark collapses to a high-dimensionality argument, which is weaker than the paper claims. The secondary concerns (objective sign ambiguity in Eq. (1), lack of statistical tests for rankings) are real but do not directly undermine the central benchmark claim as much as the flawed multimodality test. The proposed control experiment is cheap and decisive: if seed-dependence on a known unimodal landscape is demonstrated, the paper must either supply a different multimodality argument or soften the claim. Since the current evidence is insufficient but the idea is plausible and fixable, the reader's CONDITIONAL verdict remains appropriate.","tokens_in":14971,"tokens_out":5947,"duration_ms":65879,"concrete_test":"Run Algorithm 1 exactly as specified on a synthetic strictly unimodal analogue: define L(δ) = -||δ - δ*||_2^2 + K over the same hypercube, choose δ* and K so that L(0) is below a threshold τ, and define 'misclassification' as L(δ) ≥ τ. Execute 1000 SGA-BSR runs with the same step size, clipping, and stopping rule. If the resulting perturbation endpoints vary across seeds—as they will for any threshold-crossing random search—then the Section 3.2 argument is invalid. To strengthen the positive case, repeat the same test on a constructed multimodal function and verify that SGA-BSR endpoints cluster around distinct known local optima.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The multimodality evidence in Section 3.2 rests on a false premise. SGA-BSR does not run until it reaches a local optimum of L; the growth phase stops at the first perturbation that flips the label, i.e. at a point on the decision boundary, which is a level-set boundary of L. Even for a strictly unimodal objective with a unique maximizer, independent random coordinate-ascent paths will cross that boundary at different points, and the refinement phase will then produce different final perturbations. The observed seed-dependent endpoints and nonzero standard deviations in Figures 2 and 3 are exactly what a threshold-crossing stochastic search would yield on a unimodal landscape. Therefore the central claim that the BBAA objective is multimodal, and hence a suitable global-optimization benchmark, is not established by the presented experiment. A control experiment on a known unimodal function is needed before accepting the claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes that black-box adversarial attack (BBAA) problems, defined by maximizing the objective in Eq. (1) over the feasible hypercube [-epsilon, epsilon]^n, constitute a realistic, high-dimensional benchmark for global optimization methods. The authors introduce a stochastic local search method (SGA-BSR) and use seed-dependent outcomes of this method to claim that the BBAA objective is multimodal. They then compare seven metaheuristics (DE, GEN, GWO, INFO, JADE, SADE, SHADE) on CIFAR-10 and ImageNet classifiers using nominal mealpy defaults, reporting success rates, query counts, perturbation strengths, and objective values. The central claims are that BBAA is a demanding global optimization task and that GEN and SHADE are the most efficient of the tested methods.","tokens_in":15145,"tokens_out":4425,"duration_ms":50702,"significance":"If the multimodality claim were established, the paper would provide a valuable new class of benchmark problems that connects global optimization to modern machine learning, with the practical advantage of being defined by real classifiers rather than synthetic functions. The paper's strengths include the release of a framework for testing new optimizers, the use of a standard attack formulation, and the absence of fitted parameters: all optimizers run with library defaults. However, the load-bearing evidence for multimodality is not sufficient, and the sign convention of the objective is inconsistent. The empirical comparison also lacks the statistical rigor needed to support the claimed ranking. These issues are fixable, but they currently prevent acceptance.","major_comments":[{"comment":"Seed-dependent SGA-BSR endpoints do not establish multimodality. The growth phase of Algorithm 1 stops as soon as the classifier's label flips; this is not convergence to a local optimum of L but rather a first crossing of the decision boundary, i.e., a level-set crossing. On a strictly unimodal objective, independent stochastic coordinate-ascent paths can cross that level set at different points, and the subsequent binary-search refinement will then produce different final perturbations. Therefore the premise stated in Section 3.2, that on a strictly unimodal problem attacks from different seeds would converge to the exact same optimal perturbation, is false in general. The observed nonzero standard deviations in Figures 2 and 3 are exactly what one would expect from a threshold-crossing stochastic search on a unimodal landscape. To support the central benchmark claim, the authors need a control experiment on a known unimodal landscape, or direct evidence of multiple local optima (for example, by a systematic restart analysis on a reduced-dimensional surrogate or by verifying that different SGA-BSR endpoints are local maxima of L in the sense of all coordinate-wise moves). Without this, the paper's main conclusion that BBAA is multimodal and therefore a suitable global-optimization benchmark is not established.","section":"Section 4.2, Tables 1 and 2"},{"comment":"The sign convention of the objective is internally inconsistent. Eq. (1) is introduced as a bound-constrained minimization problem for L(delta) = -log p_c(x+delta) - alpha ||delta||_2^2, but Algorithm 1 accepts a perturbation when L(delta_tmp) >= L(delta), i.e., it maximizes L, and Tables 1 and 2 mark the objective column with an up arrow, indicating that larger values are better. If minimization is the intended formulation, the algorithm and tables should accept and report smaller L values; if maximization is intended, Eq. (1) and the surrounding text should be rewritten accordingly. This inconsistency affects the interpretation of the reported objective values, the role of the regularization term, and the comparability of methods across tables, so it must be resolved before the experimental results can be interpreted.","section":"Section 4.2, Tables 1 and 2"},{"comment":"The ranking claim that 'GEN and SHADE dominate' is not supported by statistical analysis. The reported quantities are means and standard deviations aggregated over images and runs, but the experimental design performs 100 runs per image and therefore has a hierarchical structure: variation across images and variation across runs are conflated in the pooled standard deviations. The paper does not report paired comparisons across the same images, confidence intervals, or significance tests. For example, in Table 1 for alpha=0.1 and epsilon=0.2, GEN has success rate 97.4% and SHADE 89.6%, while the objective values are 2.63 +/- 1.60 and 1.93 +/- 1.81, respectively; without per-image paired analysis it is not clear that these differences are robust. The authors should either provide per-image results or use appropriate statistical tests before claiming that GEN and SHADE are the most efficient methods.","section":"Section 4.2, Tables 1 and 2"}],"minor_comments":[{"comment":"The number of SGA-BSR runs is inconsistent: the text states 1000 different random seeds, Figure 2's caption says 100 independent runs, and Figure 3's caption mentions 1000 runs while also saying 100 random seeds. Please clarify the actual number of runs and ensure all captions and text agree.","section":"Section 3.2, Figures 2 and 3"},{"comment":"The phrase 'proving a framework' should be 'providing a framework'; also, the sentence would benefit from a reference to the anonymous archive where the framework is made available.","section":"Section 1.3"},{"comment":"The notation p_c(x) in Eq. (1) is ambiguous: since c(x) is the ground-truth class, it would be clearer to write p_{c(x)}(x+delta) to make explicit that the probability is evaluated for the ground-truth class.","section":"Section 2, Eq. (1)"},{"comment":"The sentence 'Each run of the optimization process is terminated after exceeding the admissible limit of perturbations' is unclear: it should state which budget is exceeded (e.g., the maximum number of iterations or the query limit) and how a run is classified as unsuccessful.","section":"Section 4.1"},{"comment":"There are several typographical and stylistic issues, including 'Costa et. al' (should be 'Costa et al.'), 'inrease' (should be 'increase'), and the inconsistent use of 'eg.' and 'cf.'; a careful proofreading pass is recommended.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The central risk is the multimodality evidence in Section 3.2: as written, the seed-dependence argument is not logically valid, and the benchmark rationale depends on it. The sign-convention inconsistency in Eq. (1) versus the algorithm and tables is also load-bearing and needs resolution. The paper's positioning relative to BlackboxBench [48] deserves more explicit discussion, since that work also benchmarks black-box attacks and the paper's claim that the impact of the optimization method for score-based attacks is 'under-explored' may be too strong. The work fits the journal's scope, and the released framework plus the systematic default-parameter comparison are useful contributions if the technical concerns are addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi [Name],\n\nQuick take: this paper proposes using black-box adversarial attacks (BBAA) as a benchmark for global optimization, and runs seven optimizers on two image sets. The idea is worth discussing, but the evidence for the central claim — that the objective is multimodal — does not hold up.\n\nWhat's new: the framing of full-pixel, score-based attacks as a global optimization benchmark, and the SGA-BSR probe plus an anonymous code archive. The comparison is transparent: default mealpy parameters, shared budget, standard datasets and models. That is a step beyond synthetic 1970s functions, and the benchmark idea could be useful to both the optimization and adversarial robustness communities.\n\nThe soft spots: the multimodality claim rests on a false premise. SGA-BSR is a stochastic coordinate ascent that stops at the first misclassification (a level-set crossing) and then refines. On a strictly unimodal function with a plateau or ridge, different seeds will hit that boundary at different points, so seed-dependent endpoints and nonzero variance do not establish multiple local optima. A control on a known unimodal landscape is needed. Without that, the 'global optimization' justification is unsupported. Also, Eq. (1) is a minimization statement but the local search and tables treat L as something to maximize; the sign on the regularization term looks wrong. Minor stuff: no statistical tests on the rankings (GEN/SHADE 'dominance' is asserted from means), and the number of images per set isn't stated.\n\nIf the authors fix the multimodality evidence (e.g., add a unimodal control, or use proper local-optima counting) and correct the objective sign, the benchmark could be a real contribution. As it stands, it's a solid empirical study but the central thesis is not proven.\n\nI'd send it to peer review — it deserves serious referee attention — but with a request for major revision. Not something I'd cite yet.\n\nBest","headline":"Promising benchmark idea, but the multimodality evidence is a threshold-crossing artifact rather than proof of multiple local optima.","tokens_in":15640,"tokens_out":3635,"would_cite":false,"duration_ms":35451,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Black-box adversarial attacks can serve as realistic high-dimensional benchmarks for global optimization, supported here by evidence of multimodality and a seven-optimizer comparison.","keywords":["black-box adversarial attack","global optimization benchmark","multimodality","evolutionary algorithms","differential evolution","metaheuristics","CIFAR-10","ImageNet"],"falsifier":"Take a provably unimodal objective over the same hypercube, such as a shifted quadratic with the same clipping, and run the same SGA-BSR procedure from many seeds; if the final perturbations still differ noticeably across seeds, then seed-dependent convergence does not establish multimodality, and the benchmark claim would need a different empirical basis.","tokens_in":14782,"feed_emoji":"🎯","tokens_out":13023,"duration_ms":118340,"temperature":0.7,"pith_summary":"This paper proposes that black-box adversarial attacks—finding a barely visible perturbation to an image that flips a classifier's decision using only the classifier's output probabilities—should be treated as benchmark problems for global optimization. The authors formulate the attack as minimizing $L(\\delta) = -\\log p_{c(x)}(x+\\delta) - \\alpha\\|\\delta\\|_2^2$ over the feasible hypercube $[-\\varepsilon,\\varepsilon]^n$, an objective with thousands to hundreds of thousands of dimensions. They show that a stochastic greedy local search converges to different perturbations from different random seeds, which they read as evidence that the objective is multimodal. They then compare seven population-based optimizers on CIFAR-10 and ImageNet, reporting that GEN and SHADE find successful low-strength attacks most reliably. If the benchmark idea holds, global optimization methods can be tested on realistic, large-scale problems tied to modern machine learning rather than small analytical functions from the 1970s.","feed_headline":"1000 runs reveal a multimodal attack landscape for optimizer tests","feed_subtitle":"The paper proposes black-box adversarial attacks as a large-scale benchmark and ranks seven metaheuristics on it.","key_machinery":"The load-bearing object is the adversarial attack objective $L(\\delta)$ from equation (1), a bound-constrained minimization over the hypercube $[-\\varepsilon,\\varepsilon]^n$. The central mechanism for evaluating multimodality is the SGA-BSR local search, a stochastic pixel-increment search with binary-search refinement; its seed-dependent convergence is the paper's evidence for multiple local optima. The benchmark protocol then applies this objective, with a fixed population size and query budget, to per-image attack problems so that optimizers can be compared on success rate, queries to first success, and normalized disturbance strength.","core_discovery":"The central claim is that the black-box adversarial attack problem can be used as a large-scale global optimization benchmark. The paper defines the attack as a minimization of $L(\\delta) = -\\log p_{c(x)}(x+\\delta) - \\alpha\\|\\delta\\|_2^2$ subject to $\\delta \\in [-\\varepsilon,\\varepsilon]^n$, where $p_j$ is the classifier's output probability for class $j$ and $c(x)$ is the true label. Attack difficulty is controlled by $\\varepsilon$, the maximum per-pixel perturbation; on CIFAR-10 the search space has 3,072 dimensions and on ImageNet 150,528 dimensions. To support the benchmark claim, the authors design a local search method, SGA-BSR, that greedily grows a perturbation one pixel at a time and then shrinks each altered pixel by binary search while preserving misclassification. Running this local search from many random seeds yields different final perturbations and different objective values, which the paper takes as evidence of multiple local optima and hence multimodality. A comparison of seven metaheuristics shows that GEN (a genetic algorithm) and SHADE (an adaptive differential evolution variant) most often achieve high attack success with the smallest normalized perturbation strength, while GWO struggles without regularization and INFO behaves like a cheap local search.","pith_inferences":["An extension the authors do not pursue: the same benchmark could rank methods by query efficiency under a hard query budget, since each evaluation costs a full forward pass.","If the benchmark is adopted, the difficulty gradient observed between CIFAR-10 and ImageNet suggests that input resolution and label-set size could be used to generate problem instances of controlled difficulty.","A cautionary test: rerunning SGA-BSR on a provably unimodal surrogate with flat regions would show whether seed-dependent convergence alone is enough to infer multimodality."],"forward_implications":["If the benchmark proposal is accepted, global optimization methods can be stress-tested on problems with 3,072 to 150,528 continuous variables, far beyond the dimension range of standard analytical suites.","The same objective supplies a natural query-cost metric: every evaluation is a full forward pass of a neural network, so a method's efficiency can be measured in real classifier queries rather than abstract function calls.","The reported ranking—GEN and SHADE achieving high success rates with the smallest perturbation strength—gives future work two concrete baselines for score-based adversarial attacks.","Because attack success is strongly controlled by the perturbation bound $\\varepsilon$ and moderately helped by a moderate regularization weight, benchmark designers can tune difficulty simply by adjusting these two parameters."],"supporting_citations":[{"why":"Supplies the score-based adversarial objective that equation (1) adapts, giving the benchmark's function to optimize.","marker":"[5]"},{"why":"Introduces a genetic-algorithm attack on class-probability feedback, which the GEN optimizer comparison builds on.","marker":"[2]"},{"why":"Shows coordinate-wise zeroth-order optimization on this objective, establishing the query-based black-box attack setting.","marker":"[7]"},{"why":"Provides a random-search attack baseline that uses the same type of score-based fitness.","marker":"[3]"},{"why":"Supplies the default implementations and hyperparameter settings for all seven compared optimizers.","marker":"[46]"},{"why":"Provides the CIFAR-10 dataset used to train the attack target and run the lower-dimensional experiments.","marker":"[25]"},{"why":"Provides the ImageNet dataset used for the high-resolution, 150,528-dimensional attack experiments.","marker":"[40]"},{"why":"Provides the pretrained ResNet-18 that serves as the ImageNet black-box attack target.","marker":"[21]"},{"why":"Provides the VGG-like architecture on which the CIFAR-10 classifier used as the attack target is trained.","marker":"[41]"},{"why":"An existing benchmark of black-box attacks that the paper positions its optimizer comparison against.","marker":"[48]"}],"fun_headline_variants":["Adversarial attacks become a large-scale benchmark for optimizers","Optimizers face a 150k-dimensional attack-based test suite","Black-box attacks: a realistic stress test for global optimizers","Multimodal attack problems rank seven metaheuristics"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument that the attack objective is multimodal rests on the premise that repeated runs of a random pixel-growing search would all converge to the same perturbation if the objective had only one minimum; this premise can fail on flat or ridge-shaped unimodal functions, so the evidence is not conclusive on its own.","fun_headline_variants_meta":{"raw":{"variants":["Adversarial attacks become a large-scale benchmark for optimizers","Optimizers face a 150k-dimensional attack-based test suite","Black-box attacks: a realistic stress test for global optimizers","Multimodal attack problems rank seven metaheuristics"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000918,"raw_usage":{"total_tokens":3920,"prompt_tokens":903,"completion_tokens":3017,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":519,"completion_tokens_details":{"reasoning_tokens":2948}},"tokens_in":519,"tokens_out":3017,"duration_ms":24436,"temperature":1.0,"reasoning_tokens":2948,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:19:28.993893+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a provably unimodal objective over the same hypercube, such as a shifted quadratic with the same clipping, and run the same SGA-BSR procedure from many seeds; if the final perturbations still differ noticeably across seeds, then seed-dependent convergence does not establish multimodality, and the benchmark claim would need a different empirical basis.","supporting_citations":[{"cited_title":"2017 IEEE Symposium on Security and Privacy (SP) pp","cited_arxiv_id":null,"evidence_quote":"Supplies the score-based adversarial objective that equation (1) adapts, giving the benchmark's function to optimize."},{"cited_title":"In: Proceedings of the genetic and evolutionary computation conference","cited_arxiv_id":null,"evidence_quote":"Introduces a genetic-algorithm attack on class-probability feedback, which the GEN optimizer comparison builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows coordinate-wise zeroth-order optimization on this objective, establishing the query-based black-box attack setting."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the VGG-like architecture on which the CIFAR-10 classifier used as the attack target is trained."}],"review_version":1}