{"id":"01501853-6cce-4953-a1bd-18f8a48c7fee","arxiv_id":"1908.02658","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A hill-climbing search over randomly rotated directions generates adversarial examples with success rates competitive with gradient-based attacks, including in black-box settings.","lead":"This paper proposes RDA, an adversarial attack that searches for a perturbation direction by hill climbing over random rotations, instead of relying only on the gradient. It reports that RDA matches or beats gradient-based attacks in both white-box and black-box settings, using only model outputs in the black-box case.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Black-box comparison is asymmetric: RDA queries the target's soft labels during search while FGSM/BIM/MI-FGSM get transfer-only access; the claimed black-box advantage may be from extra query access, not the search mechanism.","rationale":"The central claim has a white-box and a black-box component. The white-box component is supported by Table IV and a plausible mechanism, although error bars and random-seed information are absent. The black-box component is where the paper makes its most distinctive claim, near-white-box performance that is 'difficult to achieve' by gradient-based attacks, and that claim is weakened by an asymmetric comparison. RDA's access to target soft-label outputs is an essential part of its algorithm, not an incidental detail; averaging over F(x + ...) is how the hill climber decides which rotated direction is better. The comparison methods are denied that access, so the experiment cannot separate the value of the proposed directional search from the value of query-based score information. The reader's weakest assumption, that a hard-label API would break the algorithm, is a related boundary condition, but the asymmetric-comparison issue is more directly load-bearing because it concerns what the experiments actually establish against the stated baselines. The data-entry anomaly in Table V, where CIFAR-10 FGSM and BIM eps=0.3 entries repeat the eps=0.03 values, gives concrete reason to doubt the integrity of the black-box comparison. These concerns do not invalidate the white-box results or the method as an algorithm; they mean the black-box contribution should be accepted only after a matched-access comparison is run. The verdict therefore remains conditional, with no change from the reader's assessment.","tokens_in":16280,"tokens_out":12087,"duration_ms":131708,"concrete_test":"Re-run the black-box comparison of Section IV-C2 with the same soft-label query access for the baselines: replace transfer-only perturbations with NES or finite-difference gradient estimates of the target model, use the same query budget as RDA, and report success rates on CIFAR-10 and ImageNet-10. If FGSM/BIM/MI-FGSM with query access reach or exceed RDA's Table V numbers, the black-box advantage is due to query access rather than to RDA's hill-climbing search. Also correct the Table V CIFAR-10 eps=0.3 FGSM/BIM entries and verify all other baseline values.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing issue is in Section IV-C2 / Table V. In the black-box experiments, RDA is given iterative query access to the target model: the substitute gradient is only the initial direction, and each hill-climbing step evaluates P = F(x + eps*sign(R_j * v_i)) on the target, using the soft-label confidence P_y to accept or reject directions (Algorithm 1, lines 9-10). The four baselines receive no such access; they are pure transfer attacks generated on the substitute and then submitted to the target. RDA therefore operates with strictly more information than the methods it is compared with. The claim in the abstract and conclusion that near-white-box black-box performance is 'difficult to achieve using existing gradient-based attack methods' is not established by these experiments: score-based gradient estimators (finite differences, NES, ZOO) can use the same soft-label queries to estimate the target gradient and may close most of the gap. An independent data-integrity check also matters: in Table V, the CIFAR-10 eps=0.3 entries for FGSM (42.15) and BIM (34.15) equal the eps=0.03 entries, which suggests copy-paste errors and undermines confidence in the baseline numbers.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RDA, a one-step adversarial attack that starts from the gradient direction of the target model (white-box) or a substitute model (black-box) and uses first-choice hill climbing over random rotations of a randomly selected subset of dimensions to find a direction that reduces the true-class confidence, finally applying a sign perturbation. The authors report attack success rates on MNIST, SVHN, CIFAR-10, and an ImageNet-10 subset, comparing with FGSM, L.L.Class, BIM, and MI-FGSM in both white-box and black-box settings, and they analyze the angle between RDA's chosen direction and the gradient direction.","tokens_in":16540,"tokens_out":2715,"duration_ms":27904,"significance":"The paper makes a useful conceptual point: the effective direction for a one-step attack need not coincide with the gradient direction, and a cheap local search over rotated gradient directions can outperform FGSM and match iterative methods in white-box settings. The empirical study spans four datasets and the authors provide source code, which supports reproducibility. However, the headline black-box claim is weakened by an asymmetric experimental setup, and some reported baseline numbers appear to contain data-entry errors, so the central claim is defensible but not yet fully established.","major_comments":[{"comment":"The black-box comparison is asymmetric. In Algorithm 1, lines 9-10, RDA evaluates candidate directions on the target model using its full probability output P = F(x + eps * sign(R_j * v_i)) and accepts a direction when the true-class confidence drops. This gives RDA iterative query access to the target's soft labels, whereas the four baselines are pure transfer attacks computed on the substitute model and then submitted once to the target. The near-white-box black-box performance of RDA may therefore be a consequence of the extra query budget rather than the proposed search mechanism. To support the claim in the abstract and Section V that such performance is 'difficult to achieve using existing gradient-based attack methods,' the authors should include score-based query baselines with a comparable query budget (e.g., ZOO, NES, or finite-difference gradient estimation), or restrict RDA to hard-label feedback and show it still performs comparably.","section":"Section IV-C2 and Table V"},{"comment":"The CIFAR-10 black-box success rates for epsilon=0.3 are identical to the epsilon=0.03 entries for three baselines: FGSM 42.15, BIM 34.15, and MI-FGSM 51.57. Since this exact duplication is implausible and contradicts the monotonic trends in the other datasets, it suggests copy-paste errors in the table. These baseline numbers need to be corrected and the experiments re-run, because the black-box comparison is the basis of the paper's central claim.","section":"Table V, CIFAR-10 block"},{"comment":"The two hyperparameters l=10 and theta=180 are selected by running experiments on the same datasets used for the headline comparisons, as shown in Figures 4 and 6 and stated in Section IV-C. This tuning on the test data may inflate RDA's reported performance relative to the baselines, whose parameters are taken from standard configurations without equivalent tuning. The authors should either fix the parameters a priori, perform a proper validation split, or report the sensitivity of the comparison to these choices.","section":"Section IV-B1, IV-B2, and IV-C"},{"comment":"The black-box protocol assumes the attacker can access the target model's complete probability vector, since the hill-climbing acceptance criterion is based on the soft-label confidence P_y. If the target returns only a hard label (as in many practical black-box settings), RDA's search objective is unavailable and the claimed near-white-box black-box performance would not transfer. This limitation should be stated explicitly in Section IV-C2, or the method should be evaluated under a hard-label setting.","section":"Section IV-C2 and Algorithm 1"}],"minor_comments":[{"comment":"The variable name 'Setp Forward' is a typo and should read 'Step Forward'.","section":"Algorithm 1, line 7"},{"comment":"The caption mentions FMNIST, but Fashion-MNIST is not among the four datasets used in the experiments; the text in Section IV-B3 refers to MNIST, SVHN, CIFAR-10, and ImageNet-10. The caption and the figure labels should be made consistent.","section":"Fig. 8 caption"},{"comment":"Only point estimates of attack success rates are reported, with no variance or confidence intervals. Given that some comparisons are close (e.g., MNIST epsilon=0.03 in Table IV), standard deviations or a statement about statistical significance would strengthen the conclusions.","section":"Tables IV and V"},{"comment":"The similarity bound derivation is informal: the step from 'similarity(A,B) = ...' to '>= 1 - 2l/m' assumes more than is stated about the magnitudes of the rotated components, and the approximation sign hides the dependence on the actual vector entries. A more careful statement of the bound, or a citation for it, would improve clarity.","section":"Section III-B3"}],"recommendation":"major_revision","confidential_remarks":"The core idea is simple and potentially useful, and the white-box results are interesting. The main risk is that the black-box claim rests on an unfair comparison and a possibly erroneous table. If the authors add query-based baselines, correct the CIFAR-10 black-box numbers, and clarify the threat model with respect to soft-label access, the paper could become a solid contribution. I would also encourage the editor to ask for the corrected data before any acceptance decision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Let me give you the short version: this paper proposes a simple one-step adversarial attack, RDA, that uses first-choice hill climbing to search over random rotations of the gradient direction, rotating a random subset of dimensions each iteration. The idea is new as far as I can tell, and the description is good enough to reimplement—code is linked. In white-box tests on MNIST, SVHN, CIFAR-10, and ImageNet-10, RDA is competitive with FGSM, BIM, and MI-FGSM, and usually better than FGSM. The angle analysis is the most interesting part: for samples where the gradient direction fails, the successful attack direction can be far from the gradient, and the deviation angles have no obvious pattern.\n\nThe main problem is the black-box comparison. In Section IV-C2, RDA searches using the target model's output probabilities directly—it evaluates P = F(x + eps*sign(...)) on the target at every hill-climbing step. The baselines (FGSM, BIM, MI-FGSM) are pure transfer attacks: generated on the substitute model and then submitted. So RDA gets strictly more information. The claim that near-white-box black-box performance is \"difficult to achieve using existing gradient-based attack methods\" is not supported by these experiments, because score-based gradient estimators like ZOO or NES can use the same soft-label queries and might close most of the gap. The comparison needs to include those methods, or at least restrict RDA to hard-label or transfer-only access for a fair test.\n\nA few other issues. There are no variance estimates for RDA, which is randomized—the reported success rates are point estimates. The hyperparameters l=10 and theta=180 are tuned on the same datasets used in the headline tables, so the numbers are mildly inflated. And Table V has a data-integrity problem: on CIFAR-10, the epsilon=0.3 entries for FGSM (42.15), BIM (34.15), and MI-FGSM (51.57) are identical to the epsilon=0.03 entries, which looks like copy-paste errors. That makes me doubt the baseline numbers in that table.\n\nThe core algorithm is plausible and easy to implement, so I don't think this is a bad paper. But the main empirical claim about black-box performance needs a fair comparison. With that fixed, this could be a useful baseline for adversarial robustness evaluation. I'd send it to a serious referee, especially at a venue that cares about reproducible attack methods. I wouldn't cite the black-box result as-is though.","headline":"A cleanly described one-step attack whose black-box claim rests on an unfair comparison: RDA gets soft-label queries on the target while the baselines are transfer-only.","tokens_in":17039,"tokens_out":3042,"would_cite":false,"duration_ms":71518,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Rotating the gradient direction improves one-step adversarial attacks.","keywords":["adversarial examples","adversarial attack","random directional attack","hill climbing search","gradient direction","black-box attack","one-step attack","deep neural networks"],"falsifier":"Run RDA against a deployed classifier that exposes only the top-1 predicted label; if the attack success rate collapses toward the transfer-only baseline while still succeeding in white-box settings, the claim that black-box performance approaches white-box performance depends on full probability outputs and fails in hard-label settings. More directly, a dataset where every successful RDA direction coincides with the gradient direction to within a few degrees would falsify the paper's central observation that effective directions deviate irregularly.","tokens_in":16077,"feed_emoji":"🎯","tokens_out":4552,"duration_ms":43829,"temperature":0.7,"pith_summary":"This paper argues that the gradient direction is a poor default for one-step adversarial attacks because deep networks are nonlinear: moving along the gradient does not reliably increase loss or force misclassification. It proposes RDA, a one-step attack that keeps the perturbation size fixed but searches for a better direction by first-choice hill climbing: repeatedly rotate the gradient direction on a small, randomly chosen set of dimensions, keep the first rotation that lowers the model's confidence in the true class, and stop when the perturbed image is misclassified. On MNIST, SVHN, CIFAR-10, and a 10-class ImageNet subset, RDA matches or exceeds FGSM, L.L.Class, BIM, and MI-FGSM at most perturbation sizes, and in black-box settings its success rate stays close to its white-box rate, unlike gradient-transfer attacks. The larger point is that effective attack directions often deviate substantially from the gradient direction, so direction search can replace gradient-following as the core of a one-step attack.","feed_headline":"Rotating the gradient direction improves one-step adversarial attacks","feed_subtitle":"A hill-climbing search for the attack angle pushes black-box success close to white-box levels.","key_machinery":"The mechanism is first-choice hill climbing over random rotations of the current direction vector. RDA generates rotation matrices that act on pairs of coordinates: for each candidate angle $\\beta$ it randomly selects an even number of dimensions and applies $2\\times 2$ rotation blocks to them; because only a few dimensions are moved, the rotated vector stays close to the original (cosine similarity no less than about $1 - 2\\ell/m$), so the search behaves as local neighbor moves. The objective minimized is the probability the model assigns to the true class, obtained from the model output; the first rotation that lowers this probability is accepted, and the process repeats until misclassification or no improving move exists. The gradient of the target model (white-box) or of a substitute model (black-box) supplies the initial direction, and the final vector's sign is the perturbation.","core_discovery":"The central claim is that a successful one-step adversarial perturbation does not need to point along the gradient; the direction itself can be treated as a search variable. RDA starts from the gradient direction and applies first-choice hill climbing: at each step it generates rotations of the current direction on a few randomly chosen coordinate pairs, evaluates each candidate by querying the classifier for the confidence in the true class, and adopts the first rotation that reduces that confidence. The final direction is applied as a single sign-based perturbation, so the attack is one-step even though finding the direction takes iterations. Experiments on four datasets show that RDA's attack success rate is competitive with or better than FGSM and its iterative variants, and in black-box settings—where only the target model's probability outputs are used—its performance in most cases remains close to the white-box result. The paper also reports that the angle between the found direction and the gradient varies widely, with no consistent law, which it reads as evidence that successful attack directions are irregularly distributed and can lie far from the gradient.","pith_inferences":["If the direction-search view is right, defenses that only obscure or clip gradients, such as gradient masking or non-differentiable preprocessing, may not stop attackers who can still query output probabilities; confidence-based direction search bypasses gradient dependence.","RDA's design suggests a natural extension: replacing the hill-climbing acceptance rule with a learned or estimated direction model could reduce the many queries per image, since RDA's iteration counts on ImageNet are in the hundreds.","The black-box result depends on full probability outputs; a testable implication is that under hard-label-only feedback RDA would need a different objective, such as decision-boundary distance, and would likely lose its near-white-box edge."],"forward_implications":["One-step attacks can be made substantially stronger than FGSM without increasing perturbation size, since RDA's advantage comes from direction choice rather than iterative pixel modification.","In black-box settings, an attacker who can read the model's output probability vector can attack almost as effectively as with full gradient access, because the search uses only confidence values.","Iterative attacks like BIM and MI-FGSM are no longer strictly necessary for high success at moderate perturbation sizes; a direction-searching one-step attack reaches comparable rates.","The measured angles between RDA and gradient directions suggest that the adversarial region around a sample is not aligned with the gradient, challenging explanations that treat local gradient ascent as the essential mechanism of adversarial examples."],"supporting_citations":[{"why":"Defines FGSM, the one-step gradient baseline that RDA starts from and is compared against.","marker":"[15]"},{"why":"Supplies BIM and L.L.Class, the iterative baselines RDA must match in success rate.","marker":"[24]"},{"why":"Supplies MI-FGSM, the momentum baseline, and the observation that one-step attacks have limited performance.","marker":"[14]"},{"why":"Source of the first-choice hill climbing strategy that RDA uses to search directions.","marker":"[48]"},{"why":"Introduces adversarial examples and the notion of imperceptible perturbations that fool DNNs.","marker":"[5]"},{"why":"Grounds the claim that strong white-box attacks transfer poorly, motivating RDA's black-box search design.","marker":"[13]"},{"why":"Provides the standardized implementations of the comparison attacks used in the experiments.","marker":"[51]"}],"fun_headline_variants":["Hill-climbing attack direction rivals gradient for one-step fooling","Random directional attack closes black-box gap to white-box in one step","Hill-climb the attack angle not the gradient for black-box attacks","One-step attacks search directions not just gradient to boost black-box","RDA: hill-climbing directions make black-box attacks rival white-box ones"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The black-box claim assumes the attacker can query the target model and read the full probability vector for each candidate direction; if the model returns only a hard class label, RDA's objective—confidence in the true class—is unavailable and the near-white-box black-box success would not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Hill-climbing attack direction rivals gradient for one-step fooling","Random directional attack closes black-box gap to white-box in one step","Hill-climb the attack angle not the gradient for black-box attacks","One-step attacks search directions not just gradient to boost black-box","RDA: hill-climbing directions make black-box attacks rival white-box ones"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000801,"raw_usage":{"total_tokens":3523,"prompt_tokens":945,"completion_tokens":2578,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":561,"completion_tokens_details":{"reasoning_tokens":2487}},"tokens_in":561,"tokens_out":2578,"duration_ms":20163,"temperature":1.0,"reasoning_tokens":2487,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:58:16.472328+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run RDA against a deployed classifier that exposes only the top-1 predicted label; if the attack success rate collapses toward the transfer-only baseline while still succeeding in white-box settings, the claim that black-box performance approaches white-box performance depends on full probability outputs and fails in hard-label settings. More directly, a dataset where every successful RDA direction coincides with the gradient direction to within a few degrees would falsify the paper's central observation that effective directions deviate irregularly.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the first-choice hill climbing strategy that RDA uses to search directions."}],"review_version":1}