{"id":"5bb6f50a-28c8-40f2-9802-acbac0d0d3a0","arxiv_id":"2501.14122","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":9,"one_line_summary":"An RL-based attack platform with custom distortion filters reports dramatically lower query counts, but its query accounting and victim-specific training undermine the comparison.","lead":"This paper presents RLAB, a reinforcement learning platform for black-box adversarial attacks that adds and removes distortions on image patches to fool classifiers. It claims large reductions in the number of queries needed to cause misclassification, but the reported query counts appear to omit thousands of sensitivity-analysis evaluations per image.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported AVG.Q excludes the 12,544 victim-model evaluations used for patch-sensitivity analysis; once counted, RLAB's query advantage over CG-Attack disappears.","rationale":"The reader's weakest assumption identifies the same load-bearing flaw: sensitivity-analysis evaluations are excluded from the reported query count. I agree. The paper explicitly calls the sweep a 'sensitivity query' and an 'inexpensive query on a GPU', but the field's AVG.Q metric counts every oracle evaluation, not GPU cycles or wall-clock time. Even the most favorable reading, where the sensitivity sweep happens once per image, adds 12,544 queries on top of the reported 178, already exceeding CG-Attack's 210 before any attack step is counted. If the sweep is repeated per step, the true count is about 12,544 times larger. This is not a disagreement with consensus; it is an internal accounting error relative to the paper's own metric and to every baseline in its tables. The dual-action RL idea is interesting, and the L2 results might survive a corrected query count, but the paper's central efficiency claim depends on this incomparable query definition. No code or data is provided, so an independent reimplementation is the only way to assess the method under a fair budget. Since this concern supports the reader's REJECT verdict rather than moving it, I keep the verdict unchanged.","tokens_in":12693,"tokens_out":5429,"duration_ms":55012,"concrete_test":"Instrument RLAB with a monotonic counter that increments on every forward pass of the victim model, including each mask-perturbed image in the sensitivity sweep, the step prediction, and the post-processing cleanup. Rerun the 1000-sample ImageNet ResNet-50 experiment and report the mean total counter per image. If the mean exceeds 12,544 + 178 ≈ 12,700 (or roughly 2.2 million if sensitivity is recomputed per step), then Table 1 understates RLAB's query cost by orders of magnitude and the comparison to CG-Attack and other baselines collapses.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is query efficiency, but RLAB's AVG.Q counts policy steps, not oracle accesses. In the section 'State Design', paragraph 'Inexpensive Query on a GPU', sensitivity analysis applies the distortion mask to every n×n patch and reads the victim model's ground-truth probability. For ImageNet 224×224 with 2×2 patches, that is 12,544 perturbed images evaluated by the victim model per sensitivity sweep. Algorithm 1 performs this sweep before the attack loop, and the workflow in Figure 5 labels it a 'Distortion Sensitivity Query'. Each of these evaluations is a black-box query under the standard definition used by every competitor in Table 1, regardless of whether the evaluations are GPU-batched and described as taking 'a few GPU cycles'. Batching changes wall-clock cost, not query count. Thus the honest cost per image is at least 12,544 initial sensitivity evaluations plus the reported number of steps; if sensitivity is recomputed at each step, the cost is roughly 178 × 12,544 ≈ 2.2 million queries, four orders of magnitude above CG-Attack's 210. Calling the sensitivity sweep a 'one-shot filter operation' redefines a query as a GPU operation, which makes the Table 1 comparison invalid. This accounting inconsistency alone removes the support for the stated efficiency advantage.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents RLAB, a reinforcement-learning platform for black-box adversarial attacks that lets users plug in custom distortion filters. The RL agent learns to add and remove distortions patch-wise, guided by a sensitivity analysis of the victim model's ground-truth probability, and the paper reports substantially lower average query counts than state-of-the-art attacks on ImageNet and CIFAR-10 (e.g., AVG.Q 178 for RLAB vs. 210 for CG-Attack on ResNet-50/ImageNet), together with 100% attack success rates and competitive L2 distortion. The paper also reports adversarial-retraining experiments showing improved robustness on corrupted datasets. The central claim is that RLAB achieves misclassification with an order-of-magnitude fewer black-box queries than existing methods.","tokens_in":12963,"tokens_out":3000,"duration_ms":30054,"significance":"If the query-efficiency claim were correct, the platform would be a practically relevant contribution to black-box robustness evaluation, especially because of its flexibility to different distortion types and its reported 100% success rates. The paper also includes adversarial-retraining results that could be of interest. However, the central efficiency claim is not supported as stated: the reported query counts exclude the many victim-model evaluations used for sensitivity analysis, and the evaluation protocol is largely in-sample. The strengths of the paper are its clear description of the platform components and the inclusion of ablations over patch sizes and distortion filters; the query-counting methodology and evaluation protocol are the main blockers.","major_comments":[{"comment":"The reported AVG.Q counts only policy steps, not oracle accesses. Sensitivity analysis applies the distortion mask to every n x n patch and reads the victim model's ground-truth probability; for a 224x224 image with 2x2 patches this is 12,544 forward passes of the victim model per sensitivity sweep. Algorithm 1 performs this sweep before the attack loop, and Figure 5 explicitly labels it a 'Distortion Sensitivity Query.' Whether these evaluations are GPU-batched changes wall-clock time, not query count, under the standard black-box definition used by every competitor in Table 1. Counting them, the honest per-image cost is at least the initial sensitivity sweep plus 12,544 times the number of steps (if recomputed each step), which is orders of magnitude above the reported 178 and far above CG-Attack's 210. The paragraph's claim that sensitivity analysis is a 'one-shot filter operation' thus redefines the attack's query cost and invalidates the Table 1 comparison, which is the central evidence for the paper's efficiency claim.","section":"State Design, 'Inexpensive Query on a GPU' and Algorithm 1, line 4"},{"comment":"The RL policy is trained on 80% of the ImageNet validation set and evaluated on the remaining 20% of the same validation set, against the same victim model whose output probabilities are used both to build the reward and to measure success. Under this protocol, the reported AVG.Q and ASR are in-sample outcomes of optimizing on the same data distribution and the same victim, not independent predictions of attack cost on new inputs. This is load-bearing because the paper claims a general advantage over competitors; to support that claim the evaluation should use a held-out victim model or a held-out distribution, and the hyperparameter selection (e.g., noise variance, Nmax, discount factor) should be justified on a separate validation split rather than selected on the test set used for the headline numbers.","section":"Experimental Details, first paragraph and 'Evaluation on ImageNet'"},{"comment":"The paper reports only point estimates for AVG.Q and ASR with no variance, standard deviation, or statistical significance testing. For instance, Table 1 reports RLAB AVG.Q 178 vs. LeBA 178.7 and CG-Attack 210; with 1000 samples and the reported method's sensitivity, these differences could easily be within noise. In addition, the text says 'failure cases were not included in any of the metrics calculated except for the success rate,' so AVG.Q is computed only on successful attacks; while this is common practice, the paper should state the number of failures and how their exclusion affects the comparison, especially because RLAB is evaluated with a 3500-query budget while competitors use up to 10000 queries.","section":"Tables 1-3 and 'Evaluation on ImageNet'"}],"minor_comments":[{"comment":"Equation (1) is typeset with ambiguous subscripts and does not clearly define n or the relationship between k_1 and k_n; it also overloads n with the patch-size notation used elsewhere.","section":"Reward, Eq. (1)"},{"comment":"Figure 5 labels the sensitivity computation as a 'Distortion Sensitivity Query,' which contradicts the text's assertion that sensitivity analysis is not a query; the terminology should be reconciled to avoid confusion about what counts as a query.","section":"Figure 5 and 'State Design'"},{"comment":"The phrase 'targetted' is a typo for 'targeted' in the contributions sentence, and the introduction uses both 'targetted' and 'targeted' inconsistently.","section":"Introduction, first paragraph of 'RLAB Platform'"},{"comment":"The paper cites 'DeepFool [4]' using numeric citation style while the rest of the references are author-year, and the relationship to the authors' prior workshop papers (Sarkar et al. 2022, 2023b, 2023c) is not clearly delineated; the text says 'This paper is inspired by earlier work' without specifying what is new in this submission.","section":"References and Related Works"}],"recommendation":"reject","confidential_remarks":"The manuscript's central claim is not supportable as written because the query-counting definition is inconsistent with the standard black-box setting, and the evaluation protocol is substantially in-sample. These are load-bearing issues that would require redoing the experiments with a correct query accounting and a truly held-out evaluation, which is beyond a normal revision. I would also note that the paper's novelty relative to the authors' prior workshop papers should be clarified for the editor."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nShort version: the dual-action add/removal idea is worth a look, but the paper's headline result—178 queries vs 210 for CG-Attack—doesn't survive contact with the query definition. The sensitivity analysis that builds LISTADD/LISTREMOVE evaluates every 2x2 patch against the victim model: 12,544 model calls per sweep for ImageNet. The paper calls this a 'one-shot filter operation' and excludes it from AVG.Q because it's GPU-batched. But every one of those is a black-box query under the standard definition every competitor in Table 1 uses. If sensitivity is recomputed per step, honest cost is on the order of 2 million queries per image. That turns the central advantage into a four-order-of-magnitude disadvantage.\n\nWhat's actually new: the dual action—adding distortion to high-sensitivity patches while removing it from low-sensitivity ones—is a reasonable alternative to a tree search, and the BYOF filter-mixing idea is genuinely useful for robustness auditing. The paper also does proper ablations on patch size and filter hyperparameters, and the retraining results show the method can produce hardened models, though the overlap between the Gaussian attack filter and CIFAR-10-C corruptions weakens that claim somewhat.\n\nThe other soft spots are more minor but real. The RL policy is trained on 80% of the same validation set and victim model it's later evaluated on, which makes the query numbers in-sample. There is no code or data. Several tables have missing or placeholder entries. None of these are fatal by themselves, but they compound the query-counting problem.\n\nWho's this for? Someone working on practical black-box attack platforms might find the mechanism interesting, but not as presented. The evidence doesn't support the performance claims. I'd send it to review, because the mechanism is plausible and the community should get a clear ruling on what counts as a query, but the authors need to report numbers with sensitivity calls included or stop comparing to methods that count every oracle access. I would not cite it in its current form.","headline":"The dual-action RL attack is a plausible incremental idea, but the query-efficiency claim disappears once you count the sensitivity-analysis model calls.","tokens_in":13533,"tokens_out":2333,"would_cite":false,"duration_ms":22002,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper presents RLAB, a reinforcement-learning platform whose dual-action agent claims to mount black-box adversarial attacks with far fewer queries than prior methods by learning which image patches to distort and which to clean at…","keywords":["adversarial attacks","black-box attacks","reinforcement learning","query efficiency","distortion filters","image classification robustness","Dueling DQN","L2 threat model"],"falsifier":"Run RLAB against the same 1000 ImageNet samples with every forward pass metered, including each per-patch sensitivity evaluation, and compare the resulting average query count with CG-Attack and SimBA-DCT under the same metering; if RLAB's average exceeds the baselines, the reported query efficiency is an artifact of the accounting rather than the algorithm.","tokens_in":12499,"feed_emoji":"🎯","tokens_out":6130,"duration_ms":51686,"temperature":0.7,"pith_summary":"RLAB is a reinforcement-learning platform for black-box adversarial attacks that lets the user choose the distortion filter. The paper's central claim is that a dual-action agent, which adds distortion to sensitive image patches and removes distortion from less effective patches at each step, can force misclassification with far fewer queries and a higher success rate than current state-of-the-art attacks. On ImageNet and CIFAR-10, with ResNet-50, Inception-V3, and VGG-16, the paper reports 100% average success rate within a 3500-query budget and average query counts below the best baselines, for example 178 versus 210 for CG-Attack on ImageNet ResNet-50. If these query counts hold under a standard black-box accounting, the result would matter because black-box attacks are usually query-hungry and therefore easy to throttle.","feed_headline":"Black-box attack agent cuts average queries to 178","feed_subtitle":"RLAB attains 100% success within a 3500-query budget, beating CG-Attack and other state-of-the-art methods.","key_machinery":"The load-bearing mechanism is the dual-action RL loop built around a per-patch sensitivity analysis. For an image divided into n-by-n patches, the platform measures how the ground-truth probability changes when each distortion filter is applied to each patch, forming ordered lists LISTADD and LISTREMOVE; the state vector also carries the per-class probability vector and recent L2 distances. The Dueling DQN agent then outputs NADD_DIST and NREM_DIST, the numbers of patches to add distortion to and remove distortion from, with NREM_DIST less than NADD_DIST to keep progress monotonic. Rewards come from a probability dilution metric, the sum over top classes of 1/log(1/p), normalized by the change in L2 distance, so the agent is pushed toward actions that shift probability mass away from the ground-truth class at the smallest distortion cost. The sensitivity analysis is described as a parallelized one-shot filter operation that is inexpensive on a GPU.","core_discovery":"The core discovery is that an RL agent can learn to attack by treating the sensitivity of the victim model's ground-truth probability as a spatial map. At each step, the agent ranks patches by how much adding distortion lowers the ground-truth probability and how much removing distortion restores it, selects a number of patches to modify, and applies the chosen distortion filter. This simultaneous addition and removal, guided by probability dilution normalized by L2 change, is what the paper credits for fast convergence. The paper reports that the resulting platform beats baselines on average queries and success rate for both untargeted and targeted attacks, reaches 100% ASR on the evaluated ImageNet models, and produces adversarial samples with a maximum L2 of 4.74, within the competitors' budget. It also reports that retraining with RLAB-generated samples improves robustness on CIFAR-10-C more than the compared augmentation methods.","pith_inferences":["If sensitivity-analysis evaluations are charged as ordinary queries, the reported average query counts would grow by orders of magnitude, since each step evaluates every patch; the paper's efficiency claim depends on not charging them.","The same per-patch sensitivity map that guides attack actions could be repurposed for visual explanation, since it already ranks image regions by their influence on the model's ground-truth probability, a direction the paper only sketches.","For fixed query meters, the platform's practical efficiency could be improved by amortizing sensitivity estimates across similar images or by recomputing them only for changed patches, but the paper does not explore this.","The dual-action principle might transfer to non-image signals such as video or ECG classification, where patch-based sensitivity could be replaced by segment-based sensitivity, though the paper leaves that as a stated future extension."],"forward_implications":["If the reported query counts hold, black-box attacks can be mounted with roughly an order of magnitude fewer queries than many prior methods, making rate-limited APIs more vulnerable to gradient-free attacks.","The dual-action add-and-remove strategy yields a direct robustness measure: the platform can report which distortion types and patch sizes a model resists, and retraining with RLAB samples lowers degradation error on CIFAR-10-C.","The Bring Your Own Filter design means the same learned policy interface can be reused for Gaussian noise, blur, brightness, dead pixels, or mixtures, without re-architecting the attack for each distortion type.","A retrained model that was hardened with RLAB-generated adversarial examples shows lower error rates when later attacked by SimBA and Square, suggesting that RLAB-style exploration produces useful training data for defense.","The patch-size ablation shows a trade-off between distortion and queries, so a user can pick a patch size that matches the acceptable perturbation for a given application."],"supporting_citations":[{"why":"Supplies the Square Attack baseline whose average query count RLAB compares against on ImageNet and CIFAR-10.","marker":"Andriushchenko et al. 2020"},{"why":"CG-Attack is the strongest query-efficient baseline that RLAB claims to beat with 178 versus 210 average queries on ImageNet ResNet-50.","marker":"Feng et al. 2022"},{"why":"LeBA and Patch Attack are the prior RL-based and transfer-prior black-box baselines that RLAB extends and compares against.","marker":"Yang et al. 2020"},{"why":"SimBA-DCT provides the orthonormal-basis attack baseline used in both untargeted and targeted comparisons.","marker":"Guo et al. 2019"},{"why":"Describes the Dueling DQN algorithm that the RLAB agent uses for its discrete action space.","marker":"Sewak 2019"},{"why":"Supplies the ImageNet ILSVRC2012 benchmark and validation split used in all ImageNet attack experiments.","marker":"Russakovsky et al. 2015"},{"why":"NES is the limited-query black-box baseline that appears in several comparison tables.","marker":"Ilyas et al. 2018"}],"fun_headline_variants":["Dual-action RL agent slashes adversarial attack queries","RLAB: sensitivity-guided RL for fast black-box attacks","Attack agent learns to add and remove noise in one pass","Fewer queries, 100% success: RL black-box attack platform","Retrain with RLAB: RL attacks also boost robustness"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The efficiency claim depends on treating the per-patch sensitivity analysis as free, even though in the standard black-box setting every model evaluation counts as a query.","fun_headline_variants_meta":{"raw":{"variants":["Dual-action RL agent slashes adversarial attack queries","RLAB: sensitivity-guided RL for fast black-box attacks","Attack agent learns to add and remove noise in one pass","Fewer queries, 100% success: RL black-box attack platform","Retrain with RLAB: RL attacks also boost robustness"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000258,"raw_usage":{"total_tokens":1552,"prompt_tokens":885,"completion_tokens":667,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":501,"completion_tokens_details":{"reasoning_tokens":584}},"tokens_in":501,"tokens_out":667,"duration_ms":6176,"temperature":1.0,"reasoning_tokens":584,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:21:17.210790+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run RLAB against the same 1000 ImageNet samples with every forward pass metered, including each per-patch sensitivity evaluation, and compare the resulting average query count with CG-Attack and SimBA-DCT under the same metering; if RLAB's average exceeds the baselines, the reported query efficiency is an artifact of the accounting rather than the algorithm.","supporting_citations":[],"review_version":1}