REVIEW 6 major objections 7 minor 42 references
Amnesia as a Catalyst for Enhancing Black Box Pixel Attacks in Image Classification and Object Detection
T0 review · 6 major / 7 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read A Remember-and-Forget reinforcement-learning loop lets a black-box pixel attack beat prior pixel attacks on ImageNet and, for the first time, attacks object detectors with scattered pixels.
desk verdict Interesting new attack idea, but the headline comparisons are not yet verifiable because baseline protocols are unspecified and query counts disagree internally. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the two-phase Remember/Forget training loop. In the Remember phase, a CNN policy trained with one-step REINFORCE outputs $N$ pixel locations and brightness actions ('Write' sets a channel to its maximum value, 'Erase' sets it to zero); the environment rewards each perturbed image by how much it lowers the victim model's confidence on the target object or class, plus a bonus for removed objects. Memory stores only the highest-reward perturbed image, and when the stored reward's relative increase stays below $\eta$ for $T$ epochs the Forget phase resets the policy and memory and restarts from that image. This reset is what prevents the agent from repeatedly exploiting the same common vulnerability, and the ablation attributes the largest gain to the memory component.
What would settle it
Re-run PRFA and GARSDC under RFPAR's exact protocol—same YOLOv8 weights, same 0.5 confidence threshold, same image subset, and the same pixel budget—and count queries needed to reach the reported 0.29 mAP reduction; if either baseline then uses no more queries than RFPAR's 1,270, the query-efficiency claim fails. Separately, enumerate the one-step pixel perturbations and check whether any intermediate image $C$ whose existence Appendix B derives from the intermediate value theorem actually yields the predicted adversarial output; if no such $C$ exists in the discrete set, the theoretical rationale is unsupported.
Extended reading notes
Core claim
RFPAR's central claim is that attacking scattered, individual pixels with a one-step reinforcement-learning policy—instead of patches—makes black-box attacks stronger on classifiers and feasible on detectors. In the Remember phase, a CNN policy chooses pixel coordinates and brightness values, and the environment rewards it for lowering the victim model's confidence; memory stores only the highest-reward perturbed image. In the Forget phase, the policy and memory are reset and the stored image becomes the new starting point. The paper reports that this loop outperforms OnePixel, ScratchThat, and PIXLE on six ImageNet-1K classifiers, and that it is the first query-based pixel attack for object detection, removing 91% of detected YOLOv8 objects on MS-COCO at $\alpha=0.05$ while matching GARSDC's 0.29 mAP reduction with 52.8% fewer queries. On Argoverse frames of 1920$\times$1200, it removes 94% of detected objects while altering about 0.1% of pixels.
Load-bearing premise
The headline query-efficiency claim assumes the comparisons to PRFA and GARSDC were made with the same victim model version, evaluation protocol, and perturbation budget; the paper does not report re-running those baselines under RFPAR's conditions.
Editorial extensions
If this is right
- On ImageNet-1K, RFPAR's reported success rates exceed those of OnePixel, ScratchThat, and PIXLE on all six tested classifiers, implying the scattered-pixel strategy works across both CNN and ViT architectures.
- The same attack transfers to object detection, so query-based pixel attacks are no longer limited to classification; RFPAR decreases YOLOv8 mAP from 0.398 to 0.111 at $\alpha=0.05$ and removes 91% of objects.
- A query cap around 1,000, which the paper proposes as a lightweight defense, would sit below RFPAR's reported object-detection query counts (1,254–1,690), so it could block the attack in that setting.
- Ablation results attribute most of the gain to memory; initialization alone without memory has negligible effect.
- On high-resolution Argoverse frames, RFPAR removes 94% of objects with 0.1% pixels changed, indicating the approach scales to 1920$\times$1200 inputs.
Reading between the lines
- If the query counts survive a re-run of PRFA and GARSDC under identical protocols, then query-rate limiting near 1,000 queries would no longer be a safe standalone defense, because RFPAR's object-detection queries are documented just above that threshold and its classification queries on CNNs are lower.
- The Forget process is effectively a restart heuristic for non-convex search; a natural extension is to test whether the same memory-and-reset schedule improves patch-based or continuous-norm attacks, where the discrete intermediate-value argument does not apply.
- Because the attack writes pixels only to 0 or 1, an input sanitizer that flags saturated and zeroed channels might neutralize many of these attacks without adversarial training; the paper does not evaluate such a defense.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes RFPAR (Remember and Forget Pixel Attack using Reinforcement Learning), a black-box query-based pixel attack for image classification and object detection. The method alternates between a Remember process, in which a one-step REINFORCE agent perturbing individual pixels is trained using a memory of the best reward, and a Forget process, which resets the agent and memory and restarts from the stored best image. The authors evaluate RFPAR on ImageNet-1K classification against OnePixel, ScratchThat, and Pixle, and on object detection with MS-COCO (YOLOv8 and DDQ) and one Argoverse sample against PRFA and GARSDC. They report higher success rates with fewer queries and lower L0 than previous pixel attacks, a comparable mAP reduction with roughly half the queries of GARSDC, and claim the first query-based pixel attack for object detection.
Significance. If the reported improvements are robust under a clearly specified and fair evaluation protocol, this is a useful contribution: it offers a derivative-free pixel attack that avoids patch constraints, releases code, and extends query-based pixel attacks to object detection. The paper includes extra experiments on transformer classifiers, adversarially trained models, and time complexity, which are valuable. However, the headline claims are currently not fully verifiable because the comparison protocol with baselines is not stated, no uncertainty estimates are provided, and the Argoverse evidence is a single video sample.
major comments (6)
- [Section 3.3, Table 3] The comparison with PRFA and GARSDC is not grounded in a stated protocol. The paper does not say whether PRFA and GARSDC were re-run under RFPAR's conditions (same YOLO version, same confidence/NMS thresholds, same input resolution, same query-count definition) or whether the numbers were taken from the original papers, which used different detector versions and non-pixel (rectangle/patch) perturbations. Without this information, the reported 52.8% query reduction and the claim of 'comparable mAP reduction' are not verifiable.
- [Section 3.1 and all result tables] Only single-run values are reported, with no error bars, confidence intervals, or number of seeds. Because the RL agent samples actions stochastically and the evaluation set is only 1000 images, the reported success rates (e.g., 64.1% vs 51.4% for ViT-B) and query counts could change with seed, so the claimed 12.1% average improvement is not established without uncertainty estimates.
- [Section 3.4, Table 4] The Argoverse experiment uses a single randomly selected video sample. A single sample cannot support the general claim that RFPAR 'effectively removed objects on a larger scale dataset'; the paper should report results over multiple videos or frames with variance.
- [Section 3.3, Table 3] The α value used for RFPAR is not specified in the comparison table. The reported RD of 0.29 and 1270 queries correspond to the α=0.05 row in Table 2, but the selection rule for choosing this variant is not stated; because α has a large effect on RD (0.18–0.29) and query count (1254–1427), this omission makes the comparison appear post-hoc.
- [Section 3.1] The definition of 'number of queries' is ambiguous. It is not stated whether the reported query counts are averaged over successful attacks only or over all 1000 images (including failures), nor whether each pixel update corresponds to one query. With success rates ranging from 8% to 95%, this choice can alter the ordering of methods and the 'fewer queries' conclusion.
- [Appendix B] The theoretical justification via the intermediate value theorem is flawed. The theorem requires continuity of f along a continuous path, while pixel perturbations form a discrete set, and the additional assumption that the intermediate image C lies among {x+a0, ..., x+a_tau*} is asserted without proof. Since the algorithm does not depend on this theorem, it should be either corrected into a heuristic motivation or removed.
minor comments (7)
- [Equation (1)] Equation (1) contains '∥x − x∥0 = ∥δ∥0', which is trivially zero; it should be a norm on the perturbed image or the difference between images.
- [Equation (2)] Equation (2) uses 'max(n(x) − n(x))', which is zero for a single x; it should compare the number of detections in the original and perturbed images, e.g., n(x̂) − n(x).
- [Table 1 caption] The caption contains a typo: 'Succes rate' should be 'Success rate'.
- [Appendix H] In Appendix H, 'PRFAR' is a typo for 'PRFA'.
- [Section 3.3] The text states RFPAR 'reduced the mAP by an average of 0.301', but averaging the YOLOv8 reduction (0.398−0.111=0.287) and the DDQ reduction (0.376−0.054=0.322) gives 0.305, not 0.301; please clarify the calculation.
- [Section 2.2] In the definition of the action set, 'a1 and a2 represent the X and Y coordinates' is ambiguous because the superscript for the pixel index is missing; please clarify whether these are a^d_1 and a^d_2.
- [Figure 3] The ablation study reports success rates graphically but does not provide numeric values or error bars; consider adding the values to the text or a table.
Circularity Check
No circularity: RFPAR's reported success rates, query counts, and mAP reductions are measured against external benchmarks and baselines, not derived from its own assumptions.
full rationale
RFPAR's central claims are empirical: attack success rate, L0 norm, query count, and mAP reduction are all measured on external datasets (ImageNet-1K, MS-COCO, Argoverse) against external victim models and compared with published baseline methods. No equation in the paper defines these reported outcomes in terms of a fitted parameter or self-citation. The reward function in Eq. (4) is aligned with the attack objective by design, but that is the algorithm's intended behavior, not a hidden equivalence: the reported numbers are measured results, not consequences of the reward definition alone. The Appendix B intermediate-value argument contains an explicit assumption ("assuming C ∈ {x + a0, x + a1, · · · , x + aτ∗}") rather than a derivation, and this assumption does not feed into the measured results; at most it weakens the theoretical motivation, which is a correctness concern, not a circular one. There are no load-bearing self-citations: the referenced adversarial-training results are external published numbers. Concerns about baseline protocol comparability and the Table 3 versus Table 2 query discrepancy are verification and fairness issues, not circularity.
Assumptions & free parameters
free parameters (5)
- alpha (pixel attack rate) =
0.01 (classification), 0.01-0.05 (detection)
- eta (convergence threshold) =
0.05
- T (convergence duration) =
3 (classification), 20 (detection)
- Maximum iterations =
100
- Confidence threshold for object detection =
0.5
assumptions (4)
- standard math Policy gradient (REINFORCE) with function approximation provides unbiased gradient estimates for the attack objective.
- ad hoc to paper The intermediate value theorem guarantees an intermediate adversarial image C that lies among the discrete pixel perturbations {x+a0, ..., x+a_tau*}.
- domain assumption The victim model's confidence scores are calibrated enough to serve as a useful reward signal for optimization.
- domain assumption One correctly classified image per ImageNet class is a representative sample for measuring attack success.
Cite this review
Pith. "Pith review of Amnesia as a Catalyst for Enhancing Black Box Pixel Attacks in Image Classification and Object Detection." pith.science (2026). https://pith.science/paper/WTBCZIZD
@misc{pith2026250207821,
author = {Pith},
title = {Pith review of: Amnesia as a Catalyst for Enhancing Black Box Pixel Attacks in Image Classification and Object Detection},
year = {2026},
howpublished = {\url{https://pith.science/paper/WTBCZIZD}},
note = {Machine review of arXiv:2502.07821}
}
read the original abstract
It is well known that query-based attacks tend to have relatively higher success rates in adversarial black-box attacks. While research on black-box attacks is actively being conducted, relatively few studies have focused on pixel attacks that target only a limited number of pixels. In image classification, query-based pixel attacks often rely on patches, which heavily depend on randomness and neglect the fact that scattered pixels are more suitable for adversarial attacks. Moreover, to the best of our knowledge, query-based pixel attacks have not been explored in the field of object detection. To address these issues, we propose a novel pixel-based black-box attack called Remember and Forget Pixel Attack using Reinforcement Learning(RFPAR), consisting of two main components: the Remember and Forget processes. RFPAR mitigates randomness and avoids patch dependency by leveraging rewards generated through a one-step RL algorithm to perturb pixels. RFPAR effectively creates perturbed images that minimize the confidence scores while adhering to limited pixel constraints. Furthermore, we advance our proposed attack beyond image classification to object detection, where RFPAR reduces the confidence scores of detected objects to avoid detection. Experiments on the ImageNet-1K dataset for classification show that RFPAR outperformed state-of-the-art query-based pixel attacks. For object detection, using the MSCOCO dataset with YOLOv8 and DDQ, RFPAR demonstrates comparable mAP reduction to state-of-the-art query-based attack while requiring fewer query. Further experiments on the Argoverse dataset using YOLOv8 confirm that RFPAR effectively removed objects on a larger scale dataset. Our code is available at https://github.com/KAU-QuantumAILab/RFPAR.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
Zaremba W. Sutskever I. Bruna J. Erhan D. Goodfellow I. J. Szegedy, C. and R. Fergus. Intrigu- ing properties of neural networks. In International Conference on Learning Representations, ICLR, 2014
work page 2014
- [2]
-
[3]
Shlens J. Goodfellow, I. J. and C. Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, ICLR, 2015
work page 2015
-
[4]
Rice L. Wong, E. and J. Z. Kolter. Fast is better than free: Revisiting adversarial training. In International Conference on Learning Representations, ICLR, 2020. 10
work page 2020
- [5]
-
[6]
Dong Y . Pang T. Su H. Cheng, S. and J. Zhu. Improving black-box adversarial attacks with a transfer-based prior. In Advances in Neural Information Processing Systems, NeurIPS, 2019
work page 2019
-
[7]
Wang S. Shi, Y . and Y . Han. Curls and whey: Boosting black-box adversarial attacks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2019
work page 2019
-
[8]
Vargas D. V . Su, J. and K. Sakurai. One pixel attack for fooling deep neural networks.IEEE Transactions on Evolutionary Computation, 23(5):828–841, 2019
work page 2019
Show all 42 references
-
[9]
Ciocarlie G
Hitaj B. Ciocarlie G. F. Jere, M. and F. Koushanfar. Scratch that! an evolution-based adversarial attack against neural networks. CoRR, abs/1912.02316, 2019. http://arxiv.org/abs/1912.02316
1912 arXiv
-
[10]
Babu A. R. Mousavi S. Ghorbanpour S. Gundecha V . Gutierrez R. L. Guillen A. Sarkar, S. and A. Naug. Reinforcement learning based black-box adversarial attack for robustness improvement. In IEEE International Conference on Automation Science and Engineering, CASE, 2023
2023
-
[11]
Nicolosi A
Dántoni D. Nicolosi A. Pomponi, J. and S. Scardapane. Rearranging pixels is a powerful black-box attack for RGB and infrared deep learning models. IEEE Access, 11:11298–11306, 2023
2023
-
[12]
Wu W. Zhang J. Deng, Y . and Z. Zheng. Blurred-dilated method for adversarial attacks. In Advances in Neural Information Processing Systems, NeurIPS, 2023
2023
-
[13]
Wu B. Fan Y . Liu L. Li Z. Feng, Y . and S.-T. Xia. Boosting black-box attack with partially transferred conditional adversarial distribution. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2022
2022
-
[14]
Chen X. Liu C. Liu, Y . and D. Song. Delving into transferable adversarial examples and black-box attacks. In International Conference on Learning Representations, ICLR, 2017
2017
-
[15]
Kortylewski A. Xie C. Cao Y . Yang, C. and A. L. Yuille. Patchattack: A black-box texture-based attack with reinforcement learning. In European Conference on Computer Vision, ECCV, 2020
2020
-
[16]
Wu B. Fan Y . Wei X. Liang, S. and X. Cao. Parallel rectangle flip attack: A query-based black-box attack against object detection. In IEEE/CVF International Conference on Computer Vision, ICCV, 2021
2021
-
[17]
Li L. Fan Y . Jia X. Li J. Wu B. Liang, S. and X. Cao. A large-scale multiple-objective method for black-box attack against object detection. In European Conference on Computer Vision, ECCV, 2022
2022
-
[18]
Demetrio L
Zolfi A. Demetrio L. Biggio B. Shapira, A. and A. Shabtai. Phantom sponges: Exploiting non-maximum suppression to attack deep object detectors. In IEEE/CVF Winter Conference on Applications of Computer Vision, WACV, 2023
2023
-
[19]
Li C. Wen S. Han Q.-L. Nepal S. Zhang X. Wang, D. and Y . Xiang. Daedalus: Breaking nonmaximum suppression in object detection via adversarial examples. IEEE Transactions on Cybernetics, 52(8):7427–7440, 2022
2022
-
[20]
He F. Huang X. Chen, S. and K. Zhang. Relevance attack on detectors. Pattern Recognition, 124:108491, 2022
2022
-
[21]
Zhang J. Li, D. and K. Huang. Universal adversarial perturbations against object detection. Pattern Recognition, 110:107584, 2021
2021
-
[22]
Córdova Esparza D. M. Terven, J. R. and J.-A. Romero-González. A comprehensive review of YOLO architectures in computer vision: From yolov1 to yolov8 and YOLO-NAS. Machine Learning and Knowledge Extraction, 5(4):1680–1716, 2023. 11
2023
-
[23]
McAllester D. A. Singh S. Sutton, R. S. and Y . Mansour. Policy gradient methods for reinforce- ment learning with function approximation. In Advances in Neural Information Processing Systems, NeurIPS, 1999
1999
-
[24]
Kolesnikov A
Beyer L. Kolesnikov A. Weissenborn D.-Zhai X. Unterthiner T. Dehghani M. Minderer M. Heigold G. Gelly S. Uszkoreit J. Dosovitskiy, A. and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representat...
2021
-
[25]
Girshick R. B. Dollár P. Tu-Z. Xie, S. and K. He. Aggregated residual transformations for deep neural networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2017
2017
-
[26]
Kosaraju R. P. Girshick R. B.-He K. Radosavovic, I. and P. Dollár. Designing network design spaces. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2020
2020
-
[27]
van der Maaten L
Liu Z. van der Maaten L. Huang, G. and K. Q. Weinberger. Densely connected convolutional networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2017
2017
-
[28]
Chen B. Pang R. Vasudevan V .-Sandler M. Howard A. Tan, M. and Q. V . Le. Mnasnet: Platform- aware neural architecture search for mobile. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2019
2019
-
[29]
Pang R. Adam H. Le Q. V . Sandler M. Chen B. Wang W. Chen L.-C. Tan M. Chu G. Vasudevan V . Howard, A. and Y . Zhu. Searching for mobilenetv3. InIEEE/CVF International Conference on Computer Vision, ICCV, 2019
2019
-
[30]
Socher R
Dong W. Socher R. Li L.-J. Li K. Deng, J. and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2009
2009
-
[31]
Belongie S
Maire M. Belongie S. J.-Hays J. Perona P. Ramanan D. Dollár P. Lin, T.-Y . and C. L. Zitnick. Microsoft COCO: common objects in context. In European Conference on Computer Vision, ECCV, 2014
2014
-
[32]
Wang Y .-X. Li, M. and D. Ramanan. Towards streaming perception. InEuropean Conference on Computer Vision, ECCV, 2020
2020
-
[33]
Pang J.-Lyu C
Wang X.-Wang J. Pang J.-Lyu C. Zhang W. Luo P. Zhang, S. and K. Chen. Dense distinct query for end-to-end object detection. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2023
2023
-
[34]
Chiurazzi M.-Roccella S
Ciuti G.-Milazzo M. Chiurazzi M.-Roccella S. Oddo C. M. Czimmermann, T. and P. Dario. Visual-based defect detection and classification approaches for industrial applications - A SURVEY. Sensors, 20(5):1459, 2020
2020
-
[35]
Covid-net: a tailored deep convolutional neural network design for detection of covid-19 cases from chest x-ray images
Linda Wang, Zhong Qiu Lin, and Alexander Wong. Covid-net: a tailored deep convolutional neural network design for detection of covid-19 cases from chest x-ray images. Scientific Reports, 10:19549, 2020
2020
-
[36]
Yu Y .-Jiao J. Xing E. P. El Ghaoui L. Zhang, H. and M. I. Jordan. Theoretically principled trade-off between robustness and accuracy. In Proceedings of the International Conference on Machine Learning, ICML, 2019
2019
-
[37]
Hu H.-Lin Y . Yao Z. Xie Z. Wei Y . Ning J. Cao Y . Zhang-Z. Dong L. Wei F. Liu, Z. and B. Guo. Swin transformer V2: scaling up capacity and resolution. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2022
2022
-
[38]
Cord M.-Douze M. Massa F. Sablayrolles A. Touvron, H. and H. Jégou. Training data-efficient image transformers & distillation through attention. In Proceedings of the International Confer- ence on Machine Learning, ICML, 2021
2021
-
[39]
Chi C.-Yao Y . Lei Z. Zhang, S. and S. Z. Li. Bridging the gap between anchor-based and anchor-free detection via adaptive training sample selection. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2020. 12
2020
-
[40]
Su W.-Lu L. Li B. Wang X. Zhu, X. and J. Dai. Deformable DETR: deformable transformers for end-to-end object detection. In International Conference on Learning Representations, ICLR, 2021
2021
-
[41]
Wu D.-Wang Y . Guo Y . Mo, Y . and Y . Wang. When adversarial training meets vision trans- formers: Recipes from training to architecture. In Advances in Neural Information Processing Systems, NeurIPS, 2022
2022
-
[42]
Original Image
Basart S.-Mu N. Kadavath S. Wang F. Dorundo E. Desai R. Zhu T. Parajuli-S. Guo M. Song D. Steinhardt J. Hendrycks, D. and J. Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. In IEEE/CVF International Conference on Computer Vision...
2021
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.