REVIEW 4 major objections 5 minor 37 references
Corrupting Attention: Evasion-Based Adversarial Attacks on Encoder Attention in Detection Transformers
T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read Corrupting the encoder attention map drops DETR mAP from 42.1 to 0.97 under a tiny bounded perturbation.
desk verdict A genuinely new attention-targeting attack with strong results, but the headline SOTA claim depends on an unverified reading of AFOG and re-running of baselines. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying object is the encoder self-attention logit row $Z^{(m)}_{t:}$ at a chosen encoder layer, restricted to token positions inside predicted boxes. A corruption operator $T$ replaces each clean logit row with a fixed target: dispersion sets all logits equal, re-ranking reverses their order, permutation shuffles key positions, and peak-suppression moves the dominant peak onto the least-attended key. The attack minimizes the MSE between perturbed and target logit rows via projected gradient descent under an $\ell_\infty$ budget, optimizing in pre-softmax logit space because post-softmax gradients are attenuated. Because the objective acts on the attention weight distribution rather than the mechanism that produces it, the same operators apply to deformable attention with $S$ replaced by the number of sampled keys $K$.
What would settle it
Re-run the baseline attack on the same DETR-R50 and DINO-Swin-L checkpoints and COCO evaluation pipeline and check whether it actually produces adversarial mAP of 4.1 and 7.3 under the stated budget and iteration count; if the reproduced baseline is stronger or weaker, the claimed 4x-to-5x improvement is not established. Also inspect the baseline's loss to confirm it does not already directly optimize an encoder-attention objective, which would invalidate the novelty claim.
Extended reading notes
Core claim
The paper claims that an imperceptible perturbation can corrupt a detector's encoder attention directly and thereby collapse detection far more effectively than optimizing the detection output. The attack computes clean attention logit rows at a selected encoder layer for tokens inside predicted boxes, replaces those rows with a corrupted target, and runs projected gradient descent to minimize the mean squared error between perturbed and target logits under an $\ell_\infty$ budget. Across four targets that destroy different structural properties of the attention row, detection falls below 3 mAP on DETR-R50 and below 10 mAP on DINO-Swin-L, while a random perturbation of the same budget leaves detection essentially unchanged. The authors conclude that the encoder attention of detection transformers, whether dense or deformable, is a directly exploitable and previously untargeted attack surface.
Load-bearing premise
The load-bearing premise is that the previously reported mAP values for the baseline attack are accurate and obtained under the same evaluation pipeline, since the authors compare against those numbers without re-running the baseline.
Editorial extensions
If this is right
- Detection transformers that rely on an encoder attention mechanism inherit this vulnerability, so defenses must protect the attention structure, not just the output loss.
- The four-target result implies the failure mode is generic: any structured corruption of the attention distribution collapses detection, so defenses cannot specialize to one corruption type.
- Cross-model transfer between DETR and DINO means an attacker can craft perturbations on one attention formulation and degrade another, weakening defenses based on architectural diversity.
- The attack's effectiveness at low budgets and iteration counts means an online attacker needs only a few gradient steps to disable a detector.
Reading between the lines
- A reader should treat the exact 4x-to-5x margins as provisional because the baseline numbers are quoted from the prior paper rather than re-run on the same checkpoints; re-benchmarking could change the margin even if the core attack remains effective.
- If the mechanism is truly attention-structure collapse, then attention-aware regularizers or Lipschitz-bounded attention layers would be a natural defense direction, though the paper does not test defenses.
- The same objective, being defined on attention weights rather than the attention mechanism, may extend to other attention-based perception tasks such as instance segmentation or video object detection; that is an extrapolation beyond the paper's experiments.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an evasion attack on detection transformers that directly optimizes an encoder-attention objective under an ℓ∞-bounded perturbation. The attacker selects object tokens from a clean forward pass, defines one of four corruption operators on their pre-softmax attention logits (dispersion, re-ranking, permutation, peak-suppression), and runs PGD to drive the logits toward the corrupted target. The paper reports that the attack reduces DETR-R50 mAP on COCO val2017 from 42.1 to 0.97 and DINO-Swin-L from 56.8 to 1.44, claiming roughly a 4× improvement over AFOG under an identical budget and iteration count. Additional experiments cover eight detection transformers, four corruption targets, layer-wise targeting, pre- versus post-softmax optimization, transferability, and budget/iteration sweeps.
Significance. If the main numerical claims hold, the paper identifies a genuinely new and practically relevant attack surface: the encoder attention of transformer detectors can be driven to a corrupted target with a small, imperceptible perturbation, and this generalizes from dense self-attention to deformable attention. The paper's design has several strengths: the corruption operators are clearly specified; the random-perturbation control in Table III shows that the effect is not due to noise alone; the pre- versus post-softmax comparison in Table VI is a useful ablation; and the layer-wise and transfer experiments help characterize the mechanism. However, the central state-of-the-art claim depends on baseline numbers quoted from a single prior paper without re-running, and one of the paper's own tables contradicts the claimed generality. These issues are load-bearing for the headline claims and need to be resolved before the results can be accepted.
major comments (4)
- [Section IV-B, Table I] The central claim of a ~4× improvement over AFOG rests entirely on baseline numbers taken from [7] without re-running AFOG under the same checkpoints, post-processing, and COCO evaluation pipeline. Because the paper states 'Baseline numbers as reported in [7]' and provides no seed-level variance or error bars, the comparison may inherit any mismatch in model versions or evaluation protocol. Please re-run AFOG with the same evaluation pipeline and report mean and standard deviation over multiple random starts for both AFOG and the proposed attack; without this, the 0.97 versus 4.1 comparison is not established.
- [Sections II-B and IV-B] The paper's novelty claim ('first attack that directly optimizes an encoder-attention objective') and the comparison against AFOG both depend on the assertion that AFOG optimizes a detection-loss objective and corrupts attention only incidentally. However, the manuscript never quotes AFOG's loss function, and AFOG's title ('Adversarial attention perturbations for large object detection transformers') makes the characterization non-obvious. Please quote AFOG's objective explicitly and demonstrate that it contains no encoder-attention term; otherwise the 'identical attack surface' premise of the comparison, and the claim of being first, are not verifiable.
- [Section IV-C, Table II] The text states that 'Our attack achieves State-of-the-art over seven models with significant margins,' but Table II shows that on ConvNeXt the proposed attack reaches 3.98 mAP while AFOG reaches 3.9 mAP, so the attack does not beat AFOG on that model. The row count is also inconsistent with the claimed 'seven models.' This contradiction directly undermines the generalization claim and must be corrected by either fixing the numbers or revising the claim to reflect the actual matrix.
- [Section IV-B, Tables I and III] The headline DETR result uses dispersion and the headline DINO result uses re-ranking as the corruption target, and the target choice appears to be made per model after evaluating all options. Since Table III shows that the ranking of targets differs between DETR and DINO, the per-model selection is a mild form of selection bias that should be disclosed in the main comparison. Please either fix a single target a priori for both models or report the headline result for all four targets side by side in Table I.
minor comments (5)
- [Section II-A] The sentence beginning 'More recently, and Nazeri et al. [6] propose...' contains a grammatical error ('More recently, and Nazeri et al.') and should be rewritten.
- [Section III-A, Eq. (3)] The text says the attack is 'blind to the detector's output during optimization,' but the target token set O is derived from a clean forward pass with a confidence threshold. This is not blindness to the detector's output; it is merely the absence of a detection loss. Please reword to avoid an inaccurate characterization.
- [Table II] The column header 'AFOGOurs' is missing a space, and the table would benefit from a note clarifying that dispersion is the fixed corruption target for all rows.
- [Abstract and Section I] The phrase 'a ~4× reduction in resulting mAP' is ambiguous; the intended meaning is that the resulting mAP is about four times lower than AFOG's, not that the drop is multiplied by four. Please rephrase for clarity.
- [Section IV-E-4, Figure 5] The figure caption states that each axis is swept with the other at its default, but the axes are not labeled with units or the default values in the caption. Please add these details.
Circularity Check
No circularity: the attack's corruption objectives and optimization are defined independently, and all headline results are benchmarked against external baselines and random-noise controls.
full rationale
This is an empirical adversarial-attack paper, not a derivation. The central quantities—adversarial mAP on COCO under a fixed l_infinity budget and iteration count—are computed by running the proposed PGD-style attack (Eq. 3, 8, 9) against frozen victim models, and are compared with numbers reported for AFOG [7] and other prior attacks, plus a random-perturbation control (Table III). The corruption targets (dispersion, re-ranking, permutation, peak-suppression) are constructed from the clean attention rows by explicit formulas (Eq. 4-7), and the attack objective is simply MSE to that fixed target; there is no parameter fitted to the evaluation metric and no 'prediction' that is the same quantity used to define the loss. The claim that detection collapses because attention is corrupted is supported empirically by the four-target ablation and the random control, not by construction. The paper does not cite its own authors for any load-bearing premise, so there is no self-citation chain. The only substantive concerns—unverified AFOG baseline numbers and the assertion that AFOG does not directly optimize an attention objective—are external-validity or correctness risks about the comparison, not circularity: the paper's own attack would stand or fall on independent re-runs. Score 0.
Assumptions & free parameters
free parameters (6)
- Perturbation budget epsilon =
8/255 (0.031)
- PGD step size alpha =
2/255
- Number of iterations =
10
- Confidence threshold tau =
0.5
- Encoder layer index =
Final encoder layer (L6 for DETR)
- Corruption target per model =
Dispersion for DETR, re-ranking for DINO
assumptions (4)
- domain assumption The victim detector is differentiable end-to-end with respect to the input, so gradients from encoder attention logits reach the input through the backbone.
- domain assumption Tokens whose spatial positions fall inside clean predicted boxes with confidence above 0.5 are the correct object tokens to corrupt.
- ad hoc to paper The four corruption operators are qualitatively distinct and their collective failure supports the conclusion that the vulnerability lies in attention structure rather than in one target.
- domain assumption Corrupted encoder attention propagates to the decoder and degrades detections sufficiently to explain the observed collapse.
Cite this review
Pith. "Pith review of Corrupting Attention: Evasion-Based Adversarial Attacks on Encoder Attention in Detection Transformers." pith.science (2026). https://pith.science/paper/DAFQJVSC
@misc{pith2026260806674,
author = {Pith},
title = {Pith review of: Corrupting Attention: Evasion-Based Adversarial Attacks on Encoder Attention in Detection Transformers},
year = {2026},
howpublished = {\url{https://pith.science/paper/DAFQJVSC}},
note = {Machine review of arXiv:2608.06674}
}
abstract
Adversarial vulnerabilities remain a major concern for the safe deployment of neural networks, particularly in object detection, a core task embedded in many safety-critical systems. Detection transformers have emerged as leading object detectors, yet their adversarial robustness remains comparatively underexplored. Most existing attacks target the detection output rather than the attention mechanism that makes these models distinctive. In this paper, we introduce the first attack that directly optimizes an encoder-attention objective under an imperceptible, bounded $\ell_\infty$ perturbation. Rather than introducing an attacker-owned sink token through a visible patch, it drives the model's own attention toward a corrupted target. We argue that encoder attention concentrates the model's spatial reasoning, so corrupting it propagates through the detection pipeline more disruptively than perturbing the detection output alone. Our attack reduces DETR-R50 mAP on COCO from 42.1 to 0.97, a $\sim 4\times$ reduction in resulting mAP over the strongest existing attack under an identical perturbation budget and iteration count. We further show that this vulnerability is not specific to a particular corruption objective: across four qualitatively distinct targets, dispersion, re-ranking, permutation, and peak-suppression, detection consistently drops below 3 mAP, suggesting that the weakness arises from disrupting the attention structure itself rather than from any single target. Finally, we demonstrate that the attack generalizes across attention formulations, reducing DINO-Swin-L from 56.8 to 1.44 mAP against 7.3 for the strongest prior attack, establishing state-of-the-art on both dense and deformable attention.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[7]
Adversarial attention perturbations for large object detection transformers,
Z. Yahn, S. F. Tekin, F. Ilhan, S. Hu, T. Huang, Y . Xu, M. Loper, and L. Liu, “Adversarial attention perturbations for large object detection transformers,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2025, pp. 3184–3193
work page 2025
-
[1]
A survey of modern deep learning based object detection models,
S. S. A. Zaidi, M. S. Ansari, A. Aslam, N. Kanwal, M. Asghar, and B. Lee, “A survey of modern deep learning based object detection models,”Digital Signal Processing, vol. 126, p. 103514, 2022
2022
-
[2]
A survey: object detection methods from cnn to transformer,
E. Arkin, N. Yadikar, X. Xu, A. Aysa, and K. Ubul, “A survey: object detection methods from cnn to transformer,”Multimedia Tools and Applications, vol. 82, no. 14, pp. 21 353–21 383, 2023
work page 2023
-
[3]
End-to-end object detection with transformers,
N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in European conference on computer vision. Springer, 2020, pp. 213– 229
2020
-
[4]
Deformable detr: Deformable transformers for end-to-end object detection,
X. Zhu, W. Su, L. Lu, B. Li, X. Wang, and J. Dai, “Deformable detr: Deformable transformers for end-to-end object detection,”arXiv preprint arXiv:2010.04159, 2020
arXiv 2010
-
[5]
Dino: Detr with improved denoising anchor boxes for end-to- end object detection,
H. Zhang, F. Li, S. Liu, L. Zhang, H. Su, J. Zhu, L. M. Ni, and H.-Y . Shum, “Dino: Detr with improved denoising anchor boxes for end-to- end object detection,”arXiv preprint arXiv:2203.03605, 2022
arXiv 2022
-
[6]
Evaluating the adversarial robustness of detection transformers,
A. Nazeri, C. Zhao, and P. Pisu, “Evaluating the adversarial robustness of detection transformers,”Applied Intelligence, vol. 56, no. 6, p. 223, 2026
work page 2026
-
[8]
Lite detr: An interleaved multi-scale encoder for efficient detr,
F. Li, A. Zeng, S. Liu, H. Zhang, H. Li, L. Zhang, and L. M. Ni, “Lite detr: An interleaved multi-scale encoder for efficient detr,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 18 558–18 567
work page 2023
Show all 37 references
-
[9]
Give me your attention: Dot-product attention considered harmful for adversarial patch robustness,
G. Lovisotto, N. Finnie, M. Munoz, C. K. Mummadi, and J. H. Metzen, “Give me your attention: Dot-product attention considered harmful for adversarial patch robustness,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 15 234–15 243
2022
-
[10]
Patch-fool: Are vision transformers always robust against adversarial perturbations?
Y . Fu, S. Zhang, S. Wu, C. Wan, and Y . C. Lin, “Patch-fool: Are vision transformers always robust against adversarial perturbations?” arXiv preprint arXiv:2203.08392, 2022
2022 arXiv
-
[11]
Adversarial examples for semantic segmentation and object detection,
C. Xie, J. Wang, Z. Zhang, Y . Zhou, L. Xie, and A. Yuille, “Adversarial examples for semantic segmentation and object detection,” inProceed- ings of the IEEE international conference on computer vision, 2017, pp. 1369–1378
2017
-
[12]
Transferable adversarial attacks for image and video object detection,
X. Wei, S. Liang, N. Chen, and X. Cao, “Transferable adversarial attacks for image and video object detection,”arXiv preprint arXiv:1811.12641, 2018
2018 arXiv
-
[13]
Robust ad- versarial perturbation on deep proposal-based models,
Y . Li, D. Tian, M.-C. Chang, X. Bian, and S. Lyu, “Robust ad- versarial perturbation on deep proposal-based models,”arXiv preprint arXiv:1809.05962, 2018
2018 arXiv
-
[14]
Adversarial objectness gradient attacks in real-time object detection systems,
K.-H. Chow, L. Liu, M. Loper, J. Bae, M. E. Gursoy, S. Truex, W. Wei, and Y . Wu, “Adversarial objectness gradient attacks in real-time object detection systems,” in2020 Second IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications...
2020
-
[15]
Explaining and harnessing adversarial examples,
I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,”arXiv preprint arXiv:1412.6572, 2014
2014 arXiv
-
[16]
Towards deep learning models resistant to adversarial attacks,
A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” inInternational Conference on Learning Representations, 2018. [Online]. Available: https://openreview.net/forum?id=rJzIBfZAb
2018
-
[17]
Towards evaluating the robustness of neural networks,
N. Carlini and D. Wagner, “Towards evaluating the robustness of neural networks,” in2017 ieee symposium on security and privacy (sp). Ieee, 2017, pp. 39–57
2017
-
[18]
An image is worth 16x16 words: Transformers for image recognition at scale,
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gellyet al., “An image is worth 16x16 words: Transformers for image recognition at scale,”arXiv preprint arXiv:2010.11929, 2020
2010 arXiv
-
[19]
Training data-efficient image transformers & distillation through attention,
H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. J ´egou, “Training data-efficient image transformers & distillation through attention,” inInternational conference on machine learning. PMLR, 2021, pp. 10 347–10 357
2021
-
[20]
Localized query attack toward transformer-based visible object detectors,
Y . Wang, A. Li, Z. Yang, and X. Liu, “Localized query attack toward transformer-based visible object detectors,”Sensors, vol. 26, no. 6, p. 1987, 2026
1987
-
[21]
Ad- versarial attention deficit: Fooling deformable vision transformers with collaborative adversarial patches,
Q. M. Alam, B. Tarchoun, I. Alouani, and N. Abu-Ghazaleh, “Ad- versarial attention deficit: Fooling deformable vision transformers with collaborative adversarial patches,” in2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). IEEE, 2025, pp. 7123– 7132
2025
-
[22]
The lipschitz constant of self- attention,
H. Kim, G. Papamakarios, and A. Mnih, “The lipschitz constant of self- attention,” inInternational Conference on Machine Learning. PMLR, 2021, pp. 5562–5571
2021
-
[23]
Attention is all you need,
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,”Advances in neural information processing systems, vol. 30, 2017
2017
-
[24]
Microsoft coco: Common objects in context,
T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll ´ar, and C. L. Zitnick, “Microsoft coco: Common objects in context,” inEuropean conference on computer vision. Springer, 2014, pp. 740–755
2014
-
[25]
Pytorch: An imperative style, high-performance deep learning library,
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antigaet al., “Pytorch: An imperative style, high-performance deep learning library,”Advances in neural information processing systems, vol. 32, 2019
2019
-
[26]
A large-scale multiple-objective method for black-box attack against object detection,
S. Liang, L. Li, Y . Fan, X. Jia, J. Li, B. Wu, and X. Cao, “A large-scale multiple-objective method for black-box attack against object detection,” inEuropean Conference on Computer Vision. Springer, 2022, pp. 619– 636
2022
-
[27]
Improving transferable adversarial attack for vision transformers via global attention and local drop,
T. Li and Y . Han, “Improving transferable adversarial attack for vision transformers via global attention and local drop,”Multimedia Systems, vol. 29, no. 6, pp. 3467–3480, 2023
2023
-
[28]
Relevance attack on detectors,
S. Chen, F. He, X. Huang, and K. Zhang, “Relevance attack on detectors,”arXiv preprint arXiv:2008.06822, 2020
2008 arXiv
-
[29]
Gradient- guided hierarchical feature attack for object detector,
Y . Wang, Y . Zheng, L. Chen, Z. Yang, J. Wu, and T. Cao, “Gradient- guided hierarchical feature attack for object detector,”Journal of King Saud University-Computer and Information Sciences, vol. 36, no. 1, p. 101901, 2024
2024
-
[30]
Ensemble-based blackbox attacks on dense prediction,
Z. Cai, Y . Tan, and M. S. Asif, “Ensemble-based blackbox attacks on dense prediction,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 4045–4055
2023
-
[31]
Object-aware transfer-based black-box adversarial attack on object detector,
Z. Leng, Z. Cheng, P. Wei, and J. Chen, “Object-aware transfer-based black-box adversarial attack on object detector,” inChinese Conference on Pattern Recognition and Computer Vision (PRCV). Springer, 2023, pp. 278–289
2023
-
[32]
Deep residual learning for image recognition,
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778
2016
-
[33]
A convnet for the 2020s,
Z. Liu, H. Mao, C.-Y . Wu, C. Feichtenhofer, T. Darrell, and S. Xie, “A convnet for the 2020s,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 11 976–11 986
2022
-
[34]
Swin transformer: Hierarchical vision transformer using shifted windows,
Z. Liu, Y . Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” inProceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 10 012–10 022
2021
-
[35]
Internimage: Exploring large-scale vision foundation models with deformable convolutions,
W. Wang, J. Dai, Z. Chen, Z. Huang, Z. Li, X. Zhu, X. Hu, T. Lu, L. Lu, H. Liet al., “Internimage: Exploring large-scale vision foundation models with deformable convolutions,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 14 408–14 419
2023
-
[36]
Focal modulation networks,
J. Yang, C. Li, X. Dai, and J. Gao, “Focal modulation networks,” Advances in Neural Information Processing Systems, vol. 35, pp. 4203– 4217, 2022
2022
-
[37]
Eva: Exploring the limits of masked visual representation learning at scale,
Y . Fang, W. Wang, B. Xie, Q. Sun, L. Wu, X. Wang, T. Huang, X. Wang, and Y . Cao, “Eva: Exploring the limits of masked visual representation learning at scale,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 19 358–19 369
2023
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.