REVIEW 5 major objections 7 minor 40 references
Harmonizing Intra-coherence and Inter-divergence in Ensemble Attacks for Adversarial Transferability
T0 review · 5 major / 7 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read HEAT synthesizes shared gradient directions with SVD and dynamically weights surrogate models to boost adversarial transferability.
desk verdict Plausible ensemble attack with impressive tables, but the central 'consensus gradient' mechanism is undercut by an untested SVD scale issue and a contradictory formula; deserves a careful referee rather than rejection. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the consensus gradient direction $V_k$, produced by C-GRADS via SVD: gradients from $M$ surrogate models are stacked row-wise into $G\in\mathbb{R}^{M\times D}$, and the right-singular vectors weighted by their singular values give a shared perturbation direction. The second mechanism is D-HARMO's dual weight: the intra-domain weight $w^{\mathrm{intra}}_m$ compares how much model $m$'s own adversarial step raises other models' losses, and the inter-domain weight $w^{\mathrm{inter}}_m$ combines a normalized loss-contribution factor with gradient alignment (cosine similarity) into an information-entropy score; the final gradient is $\sum_m w^{\mathrm{intra}}_m w^{\mathrm{inter}}_m \nabla_x L_m(x^{V_k}_{\mathrm{adv}})$. These two machinery pieces are what carry the claim that shared directions survive projection while per-model noise is weighted down.
What would settle it
Run HEAT while sweeping the cumulative variance ratio $p$ from small values to $p=1$ (keeping all singular directions) on the same surrogate and target models. If attack success rate does not drop sharply when $p$ is reduced well below 0.7, or if $p=1$ matches or beats $p=0.7$, the claim that the top-$k$ consensus subspace carries the transferable signal is falsified; a second check is to repeat with ensembles whose gradients are intentionally made diverse (e.g., models trained on disjoint data), where the shared-subspace assumption should break down.
Extended reading notes
Core claim
HEAT's central claim is that the transferable component of an ensemble of surrogate gradients lies in a low-dimensional consensus subspace. Concretely, the paper constructs the gradient matrix $G$ by stacking per-model gradients and decomposes it as $G = U\Sigma V^\top$; it then keeps the $k$ singular directions that account for fraction $p=0.7$ of total variance and forms a shared direction $V_k$ as a singular-value-weighted sum of those top-$k$ vectors. Around this shared direction, D-HARMO computes per-model weights: intra-domain coherence rewards models whose gradient direction, when applied alone, also raises the loss of other models, while inter-domain divergence rewards models with large loss contribution and gradient alignment with the ensemble. The final perturbation updates along the sign of the weighted sum of model gradients at the C-GRADS example. The paper asserts that this combination yields substantially higher attack success rates against eight unseen CNN and ViT models across three datasets, and that each component contributes in ablation.
Load-bearing premise
The method's gains depend on the assumption that the transferable gradient signal of the surrogate ensemble is concentrated in the top few singular directions (captured at $p=0.7$), so that throwing away the remaining directions removes noise rather than useful signal.
Editorial extensions
If this is right
- If the consensus subspace really captures transferable signal, then HEAT should raise black-box attack success rate on held-out CNN and ViT targets, which the paper reports across CIFAR-10, CIFAR-100, and ImageNet.
- Because D-HARMO adapts weights per input and per iteration, HEAT should remain effective when the ensemble is heterogeneous (CNNs plus ViTs), not only homogeneous.
- The SVD projection plus entropy weighting should compose with base attack methods such as I-FGSM, MI-FGSM, and DI2-FGSM, which the paper confirms.
- If both modules are load-bearing, removing either should degrade attack success rate; the ablation shows C-GRADS contributes the largest gain, especially against ViT targets.
Reading between the lines
- The paper fixes the SVD retention ratio at $p=0.7$; an immediate extension would sweep $p$ per ensemble and per dataset, since the optimal subspace rank should depend on how diverse the surrogate models are.
- Because the method's success depends on shared singular directions, a defense could try to detect or destroy adversarial perturbations by projecting them onto the top-$k$ subspace of the input's gradient ensemble, if gradient information is available.
- The intra-domain weighting resembles a leave-one-model-out generalization estimator; a natural test is whether the same weights predict which single model transfers best, which would turn D-HARMO into a model-selection tool.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript proposes HEAT, an ensemble adversarial attack that aims to improve black-box transferability by combining two modules: C-GRADS, which performs SVD on a matrix of white-box surrogate gradients and uses a weighted sum of leading right-singular vectors as a 'shared' attack direction, and D-HARMO, which reweights each surrogate gradient using an intra-domain coherence term and an inter-domain divergence/alignment term. The reported experiments on CIFAR-10, CIFAR-100, and ImageNet against eight black-box architectures show average attack success rate gains over the Ens baseline, with particularly large improvements on vision transformers, and an ablation study attributes the gains to the proposed components.
Significance. If the results hold, HEAT provides a simple and effective recipe for ensemble attacks: SVD-based direction synthesis plus adaptive gradient weighting yields state-of-the-art transferability on CNNs and ViTs, and it integrates naturally with I-FGSM, MI-FGSM, and DI2-FGSM. The paper should be credited for the breadth of its evaluation (three datasets, eight black-box models, multiple base attacks) and for including an ablation study. However, the claimed 'first formal connection' to domain generalization is not formalized, and the lack of code, error bars, and sensitivity analyses means the empirical claims are not yet fully supported.
major comments (5)
- [Section 3.2.2, Eq. (16)] The prose states that 'Models with higher gradient similarity (i.e., greater alignment) are assigned larger alignment contribution factors,' but Eq. (16) defines A_m as the reciprocal of the mean cosine similarity, so higher P_mj produces a smaller A_m. Because A_m enters the entropy H_m (Eq. 19) and hence the inter-domain weights, this contradiction changes which models are emphasized. Please correct the equation or the prose and rerun the affected experiments and ablations.
- [Section 3.2.1, Eqs. (4)-(7)] G is built from unnormalized gradients, so the leading singular vectors are scale-dependent: a surrogate with a large gradient norm can dominate the top singular subspace, and V_k may approximate that model's gradient rather than a shared direction. The central claim that C-GRADS synthesizes consensus directions is therefore not established as written. Please add an experiment with row-normalized gradients (or another scale-invariant variant) and report whether the gains persist; otherwise the improvement over Ens may be an implicit magnitude reweighting rather than consensus extraction.
- [Section 3.2.1, Eq. (6)] The number of retained singular vectors k is determined by the hand-set p=0.7, but no ablation over p is reported, so the sensitivity of the headline results to this key hyperparameter is unknown. In addition, the text calls the ratio a 'contribution to total variance' while using raw singular values; with SVD the variance interpretation applies to squared singular values. Please either report an ablation over p and justify the variance terminology, or use squared singular values.
- [Section 3, Algorithm 1 and Eqs. (8)/(24)] The text describes an iterative process and the base attacks I-FGSM, MI-FGSM, and DI2-FGSM are iterative, but Algorithm 1 shows a single forward pass ending in one update x*_adv = Clip(x + alpha sign(g)), with no loop over iterations and no indication of how C-GRADS and D-HARMO are reused in each iteration. This makes the method ambiguous and hard to reproduce. Please present the full iterative procedure (e.g., with x^{i-1}_{adv} on the right-hand side of Eqs. 8 and 24) and specify which quantities are recomputed per iteration.
- [Section 4.2] The quantitative claims in the text are not consistent with the tables. The sentence 'HEAT achieves ASR improvements of 5.11%, 22.76%, and 27.22%' for I-FGSM, MI-FGSM, and DI2-FGSM does not match Table 2 (for I-FGSM, HEAT is 45.42 vs Ens 16.87, an absolute difference of 28.55 points), and Table 1 contains several cells where HEAT is below AdaEA (e.g., CIFAR-100 BiT-101: 46.90 vs 55.38), contradicting 'consistently outperforms.' Please restate these claims precisely, define the metric used for 'improvement,' and add error bars or significance tests over multiple runs.
minor comments (7)
- [Table 3] The two rows marked '✓ ✓ ✓' are not distinguished in the header; please label the component sets explicitly (e.g., A+B+C, A+B+D, A+B+C+D).
- [Section 4.1] The temperature parameter τ appears in Algorithm 1 and Eqs. (17)-(18) but its value is not reported; please provide it.
- [Section 3.2.1] The SVD sign ambiguity is not addressed; since v_i are defined up to sign, the sum V_k in Eq. (7) may flip direction. Please specify a sign convention (e.g., align each v_i with the mean gradient).
- [Section 3.2.2] The intra-domain weight in Eq. (11) can be negative when the adversarial example of model m does not transfer to model j; after normalization this can produce negative weights, which is not discussed.
- [Section 1] There are several typos: 'Introdution' in Section 1, 'HEA T' in the introduction, 'D-HarMony' in Section 3.2, and 'Comparision' in Section 4.1. Figure 2 also spells the module as 'C-GARDS' in one place.
- [Sections 1 and 5] The claim of a 'first formal connection between domain generalization and adversarial attacks' is not backed by any formal statement in the paper; please soften it or add a formal treatment.
- [Reproducibility] No code is provided; given the number of hyperparameters (p, τ, ε) and the ambiguity in Algorithm 1, releasing code would materially improve reproducibility.
Circularity Check
No significant circularity: HEAT is evaluated as an empirical ensemble attack against held-out black-box models, and the core modules are constructed from white-box gradients without fitting to the target metric.
full rationale
The paper's central claim is that HEAT improves black-box transferability over existing ensemble attacks. The attack pipeline (Eqs. 4-24) uses only white-box surrogate gradients: C-GRADS builds a gradient matrix from the white-box models and applies SVD, D-HARMO computes weights from white-box losses and gradient alignments, and the final perturbation is evaluated on unseen black-box models in Tables 1-4. No parameter of the method is fitted to the black-box success rate; the cumulative contribution ratio p=0.7 is a fixed hyperparameter, and the temperature τ is also a preset constant. The comparisons with Ens, SVRE, and AdaEA are external baselines, and the ablation study attributes gains to the proposed components rather than to any hidden reuse of the target labels. The self-citations (e.g., Ma et al. 2023 and Gao et al. 2024) appear only as motivational examples in the introduction and are not load-bearing. The paper's 'domain generalization' framing is rhetorical and not backed by a formal transfer bound, and the SVD-based consensus direction could in principle be dominated by a single large-norm gradient, but that is a correctness or robustness concern about the heuristic, not circularity. The derivation chain does not reduce any predicted quantity to its own input; the transferability claim is an empirical finding on held-out models, so no circular step is present.
Assumptions & free parameters
free parameters (3)
- p (cumulative contribution ratio) =
0.7
- tau (temperature parameter) =
not reported
- epsilon (stability constant) =
not reported
assumptions (3)
- domain assumption The top-k singular subspace of G captures the gradient directions most likely to transfer to black-box models.
- ad hoc to paper The alignment factor in Eq. 16 implements the 'higher similarity yields larger weight' behavior described in the prose.
- domain assumption Treating surrogate models as domains makes domain generalization principles directly applicable to attack design.
Cite this review
Pith. "Pith review of Harmonizing Intra-coherence and Inter-divergence in Ensemble Attacks for Adversarial Transferability." pith.science (2026). https://pith.science/paper/EYEIBFAQ
@misc{pith2026250501168,
author = {Pith},
title = {Pith review of: Harmonizing Intra-coherence and Inter-divergence in Ensemble Attacks for Adversarial Transferability},
year = {2026},
howpublished = {\url{https://pith.science/paper/EYEIBFAQ}},
note = {Machine review of arXiv:2505.01168}
}
read the original abstract
The development of model ensemble attacks has significantly improved the transferability of adversarial examples, but this progress also poses severe threats to the security of deep neural networks. Existing methods, however, face two critical challenges: insufficient capture of shared gradient directions across models and a lack of adaptive weight allocation mechanisms. To address these issues, we propose a novel method Harmonized Ensemble for Adversarial Transferability (HEAT), which introduces domain generalization into adversarial example generation for the first time. HEAT consists of two key modules: Consensus Gradient Direction Synthesizer, which uses Singular Value Decomposition to synthesize shared gradient directions; and Dual-Harmony Weight Orchestrator which dynamically balances intra-domain coherence, stabilizing gradients within individual models, and inter-domain diversity, enhancing transferability across models. Experimental results demonstrate that HEAT significantly outperforms existing methods across various datasets and settings, offering a new perspective and direction for adversarial attack research.
Figures
Reference graph
Works this paper leans on
-
[1]
and Wagner, D
Carlini, N. and Wagner, D. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp), pp.\ 39--57. Ieee, 2017
2017
-
[2]
An adaptive model ensemble adversarial attack for boosting adversarial transferability
Chen, B., Yin, J., Chen, S., Chen, B., and Liu, X. An adaptive model ensemble adversarial attack for boosting adversarial transferability. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 4489--4498, 2023 a
work page 2023
-
[3]
Rethinking model ensemble in transfer-based adversarial attacks
Chen, H., Zhang, Y., Dong, Y., Yang, X., Su, H., and Zhu, J. Rethinking model ensemble in transfer-based adversarial attacks. In The Twelfth International Conference on Learning Representations, 2023 b
work page 2023
-
[4]
Imagenet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.\ 248--255. Ieee, 2009
2009
-
[5]
Discovering adversarial examples with momentum
Dong, Y., Liao, F., Pang, T., Hu, X., and Zhu, J. Discovering adversarial examples with momentum. arXiv preprint arXiv:1710.06081, 5, 2017
arXiv 2017
-
[6]
Boosting adversarial attacks with momentum
Dong, Y., Liao, F., Pang, T., Su, H., Zhu, J., Hu, X., and Li, J. Boosting adversarial attacks with momentum. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 9185--9193, 2018
work page 2018
-
[7]
Benchmarking adversarial robustness on image classification
Dong, Y., Fu, Q.-A., Yang, X., Pang, T., Su, H., Xiao, Z., and Zhu, J. Benchmarking adversarial robustness on image classification. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 321--331, 2020
work page 2020
-
[8]
An image is worth 16x16 words: Transformers for image recognition at scale
Dosovitskiy, A. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020
arXiv 2010
Show all 40 references
-
[9]
and Young, G
Eckart, C. and Young, G. The approximation of one matrix by another of lower rank. Psychometrika, 1 0 (3): 0 211--218, 1936
1936
-
[10]
Ethical alignment decision-making for connected autonomous vehicle in traffic dilemmas via reinforcement learning from human feedback
Gao, X., Luan, T., Li, X., Liu, Q., Ma, Z., Meng, X., and Li, Z. Ethical alignment decision-making for connected autonomous vehicle in traffic dilemmas via reinforcement learning from human feedback. IEEE Internet of Things Journal, 2024
2024
-
[11]
J., Shlens, J., and Szegedy, C
Goodfellow, I. J., Shlens, J., and Szegedy, C. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014
2014 arXiv
-
[12]
Deep residual learning for image recognition
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 770--778, 2016
2016
-
[13]
Big transfer (bit): General visual representation learning
Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., and Houlsby, N. Big transfer (bit): General visual representation learning. In European Conference on Computer Vision, pp.\ 491--507, 2020
2020
-
[14]
Learning multiple layers of features from tiny images
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009
2009
-
[15]
J., and Bengio, S
Kurakin, A., Goodfellow, I. J., and Bengio, S. Adversarial examples in the physical world. In Artificial intelligence safety and security, pp.\ 99--112. Chapman and Hall/CRC, 2018
2018
-
[16]
Adaptive momentum variance for attention-guided sparse adversarial attacks
Li, C., Yao, W., Wang, H., and Jiang, T. Adaptive momentum variance for attention-guided sparse adversarial attacks. Pattern Recognition, 133: 0 108979, 2023 a
2023
-
[17]
Visualizing the loss landscape of neural nets
Li, H., Xu, Z., Taylor, G., Studer, C., and Goldstein, T. Visualizing the loss landscape of neural nets. Advances in neural information processing systems, 31, 2018
2018
-
[18]
Making substitute models more bayesian can enhance transferability of adversarial examples
Li, Q., Guo, Y., Zuo, W., and Chen, H. Making substitute models more bayesian can enhance transferability of adversarial examples. In The Eleventh International Conference on Learning Representations, 2023 b
2023
-
[19]
Learning transferable adversarial examples via ghost networks
Li, Y., Bai, S., Zhou, Y., Xie, C., Zhang, Z., and Yuille, A. Learning transferable adversarial examples via ghost networks. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp.\ 11458--11465, 2020
2020
-
[20]
Lin, J., Song, C., He, K., Wang, L., and Hopcroft, J. E. Nesterov accelerated gradient and scale invariance for adversarial attacks. In International Conference on Learning Representations
-
[21]
Delving into transferable adversarial examples and black-box attacks
Liu, Y., Chen, X., Liu, C., and Song, D. Delving into transferable adversarial examples and black-box attacks. In International Conference on Learning Representations, 2022
2022
-
[22]
Swin transformer: Hierarchical vision transformer using shifted windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp.\ 10012--10022, 2021
2021
-
[23]
A homologous and heterogeneous multi-view inter-patient adaptive network for arrhythmia detection
Ma, Z., Wang, J., Yue, J., and Lin, Y. A homologous and heterogeneous multi-view inter-patient adaptive network for arrhythmia detection. Computer Methods and Programs in Biomedicine, 241: 0 107740, 2023
2023
-
[24]
Spatial-frequency gradient fusion based model augmentation for high transferability adversarial attack
Pang, J., Yuan, C., Xia, Z., Li, X., and Fu, Z. Spatial-frequency gradient fusion based model augmentation for high transferability adversarial attack. Knowledge-Based Systems, 301: 0 112241, 2024
2024
-
[25]
D., Neff, C., Noghre, G
Pazho, A. D., Neff, C., Noghre, G. A., Ardabili, B. R., Yao, S., Baharani, M., and Tabkhi, H. Ancilia: Scalable intelligent video surveillance for the artificial intelligence of things. IEEE Internet of Things Journal, 10 0 (17): 0 14940--14951, 2023
2023
-
[26]
Improving the adversarial transferability with relational graphs ensemble adversarial attack
Pi, J., Luo, C., Xia, F., Jiang, N., Wu, H., and Wu, Z. Improving the adversarial transferability with relational graphs ensemble adversarial attack. Frontiers in Neuroscience, 16: 0 1094795, 2023
2023
-
[27]
Intriguing properties of neural networks
Szegedy, C. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013
2013 arXiv
-
[28]
Rethinking the inception architecture for computer vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 2818--2826, 2016
2016
-
[29]
Ags: Transferable adversarial attack for person re-identification by adaptive gradient similarity attack
Tao, Z., Lu, Z., Peng, J., and Wang, H. Ags: Transferable adversarial attack for person re-identification by adaptive gradient similarity attack. Knowledge-Based Systems, 304: 0 112506, 2024
2024
-
[30]
Training data-efficient image transformers & distillation through attention
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and J \'e gou, H. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pp.\ 10347--10357. PMLR, 2021
2021
-
[31]
Ensemble adversarial training: Attacks and defenses
Tram \`e r, F., Kurakin, A., Papernot, N., Goodfellow, I., Boneh, D., and McDaniel, P. Ensemble adversarial training: Attacks and defenses. In International Conference on Learning Representations, 2018
2018
-
[32]
Boosting adversarial transferability by block shuffle and rotation
Wang, K., He, X., Wang, W., and Wang, X. Boosting adversarial transferability by block shuffle and rotation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 24336--24346, 2024
2024
-
[33]
Structure invariant transformation for better adversarial transferability
Wang, X., Zhang, Z., and Zhang, J. Structure invariant transformation for better adversarial transferability. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 4607--4619, 2023
2023
-
[34]
Improving transferable targeted adversarial attacks with model self-enhancement
Wu, H., Ou, G., Wu, W., and Zheng, Z. Improving transferable targeted adversarial attacks with model self-enhancement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 24615--24624, 2024
2024
-
[35]
Xie, C., Zhang, Z., Zhou, Y., Bai, S., Wang, J., Ren, Z., and Yuille, A. L. Improving transferability of adversarial examples with input diversity. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2019
2019
-
[36]
E., and He, K
Xiong, Y., Lin, J., Zhang, M., Hopcroft, J. E., and He, K. Stochastic variance reduced ensemble adversarial attack for boosting the adversarial transferability. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 14983--14992, 2022
2022
-
[37]
Wide residual networks
Zagoruyko, S. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016
2016 arXiv
-
[38]
Boosting adversarial transferability via gradient relevance attack
Zhu, H., Ren, Y., Sui, X., Yang, L., and Jiang, W. Boosting adversarial transferability via gradient relevance attack. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 4741--4750, 2023
2023
-
[39]
Learning to transform dynamically for better adversarial transferability
Zhu, R., Zhang, Z., Liang, S., Liu, Z., and Xu, C. Learning to transform dynamically for better adversarial transferability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 24273--24283, 2024
2024
-
[40]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.