REVIEW 4 major objections 5 minor 38 references
From Sharpness to Better Generalization for Speech Deepfake Detection
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Sharpness is proposed as a theoretical predictor of how well speech deepfake detectors generalize to unseen data.
desk verdict Useful empirical study with a real diagnostic corner, but the abstract oversells it and the sharpness-EER correlation has scale and grouping confounds that need addressing. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
$$s(w,S) = \max_{\|\epsilon\|_2 \le \rho} \frac{1}{|S|} \sum_{i:(x_i,y_i)\in S} \bigl(\ell_i(w+\epsilon) - \ell_i(w)\bigr),$$ computed per batch and averaged. The paper pairs this measure with Sharpness-Aware Minimization, whose objective replaces the training loss by its maximum over a $\rho$-bounded perturbation neighborhood, approximated by a first-order gradient step. The fixed radius $\rho=0.05$ and batch size $m=32$ are kept constant when sharpness is compared across 24 systems; this measure is what carries the correlation analysis and the claim that SAM flattens the landscape.
What would settle it
A direct check would be to recompute the paper's 24-system sharpness–EER correlations with $\rho$ scaled per model, for example by parameter norm or layer-wise variance; if the reported correlations such as SRCC 0.87 on In-The-Wild disappear or reverse, the fixed-radius choice carried the result. A second check is to train a deliberately low-error model whose loss landscape is sharp; if its EER is low despite high sharpness, the claimed indicator fails in that setting.
Extended reading notes
Core claim
The paper's central claim is that sharpness is a usable theoretical indicator of generalization for speech deepfake detection, not just a heuristic. Concretely, the authors define m-sharpness on a test set as the maximum average loss increase under weight perturbations bounded by $\rho$, fix $\rho=0.05$ and batch size $m=32$, and show that sharper systems tend to have higher EER on unseen datasets, with statistically significant correlations in five of six out-of-domain settings. They also argue that minimizing sharpness through Sharpness-Aware Minimization, which optimizes the worst-case loss over a perturbation neighborhood, flattens the loss landscape and yields consistently better and more stable EER, with the largest gains on the most mismatched sets such as ASVspoof 2021 LA and ADD 2022.
Load-bearing premise
The central claim depends on the assumption that one fixed perturbation radius, $\rho=0.05$, gives fair and comparable sharpness values for models as different as AASIST and W2V-XLSR, and that the observed sharpness–EER correlation is not an artifact of that radius or of the Adam-versus-SAM grouping.
Editorial extensions
If this is right
- A practitioner can measure sharpness on held-out data to predict which detectors will transfer, instead of relying on error-rate ranking alone.
- SAM-trained models are a direct robustness recipe for deployment under language, channel, or attack mismatch, since gains are largest in those conditions.
- Combining SSL pretraining with SAM is supported: even the flattest pretrained models usually improve further on out-of-domain test sets.
- Sharpness can act as a diagnostic for which mismatch factors hurt a detector, since language, attack, and channel shifts raise it while speaker variability does not.
- Training with SAM also reduces run-to-run variance, so the improvement is not just in the mean error rate.
Reading between the lines
- Editorial inference: if the sharpness–error relationship is causal rather than merely predictive, tracking sharpness on a small validation set during training could serve for model selection without needing a labeled target-domain test set.
- Editorial inference: the fixed-radius comparison may bundle flatness together with model scale; recomputing the correlations with $\rho$ scaled per parameter norm would test whether the ranking survives, an experiment the paper does not run.
- Editorial inference: SAM's objective includes an explicit L2 term and per-model radius selection, so the improvement over Adam could partly come from those choices; an ablation holding weight decay fixed would isolate the perturbation term.
- Editorial inference: the same sharpness measure could rank feature-level and data-level defenses as well as optimizers, giving a single scale on which to compare robustness interventions.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes to use sharpness as a predictor of generalization for speech deepfake detection. It defines m-sharpness via Eq. (1), observes that sharpness increases under domain shifts for two models, and then applies Sharpness-Aware Minimization (SAM) to train AASIST and several wav2vec/XLS-R based systems on ASVspoof 2019 LA, evaluating on seven unseen benchmarks. The authors report that SAM lowers EER for many configurations (e.g., AASIST on 21LA: 8.10 to 4.62) and that sharpness correlates with EER across 24 systems, with the strongest correlations on ITW, 21LA, and ADD. The paper concludes that sharpness can serve as a theoretical indicator of generalization and that SAM improves robustness.
Significance. If the correlation claim were established with appropriate controls, the paper would be valuable: sharpness could serve as a cheap diagnostic for model selection and failure analysis, and SAM as a drop-in training method. The study is commendably broad, covering seven test sets, six model configurations, three seeds, and released code, and the concrete improvements on high-mismatch sets are practically interesting. However, the current analysis does not rule out that the sharpness-EER correlations are driven by architecture scale or by the Adam/SAM grouping, and the attribution of SAM gains to flatness is confounded with L2 regularization and per-model radius selection. These issues are fixable with additional analyses, so the work is a promising empirical contribution but not yet an established theoretical indicator.
major comments (4)
- [Section 2, Eq. (1); Section 4.1] The sharpness diagnostic in Eq. (1) uses a fixed perturbation radius ρ=0.05 for every model, whereas Section 4.1 selects SAM's radius per model from {0.05, 0.01, 0.005, 0.001}, with smaller models assigned larger values. Because Eq. (1) is not normalized by parameter norm, the same L2 radius produces very different relative perturbations for AASIST versus SSL encoders with much larger weight norms, so the lower sharpness of SSL models and the correlations in Table 2 may reflect architecture scale rather than a general sharpness-generalization relationship. Please report within-family correlations, within-optimizer correlations, or partial correlations controlling for parameter count, and include a sensitivity analysis over ρ.
- [Table 2 and Figure 3] The correlation coefficients in Table 2 are computed over all 24 systems pooled across Adam and SAM. Since SAM simultaneously reduces both EER and sharpness for most model-dataset pairs (e.g., AASIST on 21LA in Table 1), the between-optimizer separation can inflate Spearman and Pearson coefficients even if no within-optimizer trend exists. Figure 3 appears to show separate regression lines for the two groups, but the corresponding within-group coefficients are not reported. Please report correlations for Adam-only and SAM-only subsets, or include the optimizer as a covariate.
- [Section 4.2, Table 1] The text says SAM 'consistently improves generalization across nearly all models and datasets,' but Table 1 contains several clear regressions: W2V-Base+Linear on WF (30.34 to 36.95), W2V-Large+Linear on FOR (13.24 to 14.65) and ADD (30.97 to 37.31), W2V-XLSR+Linear on 21LA (1.32 to 1.87), and W2V-XLSR+AASIST on SC (27.50 to 29.93). Please quantify the fraction of pairwise improvements and either soften the abstract/conclusion claims or analyze what distinguishes the configurations where SAM hurts.
- [Eq. (2) and Section 4.1] The paper attributes SAM's performance gains to flatness, but the SAM objective in Eq. (2) also includes an explicit L2 term λ||w||^2, and the SAM radius is tuned per model in Section 4.1. Without ablating against standard Adam with equivalent L2 regularization, or evaluating sharpness at each model's own selected ρ, the causal link from reduced sharpness to improved EER is not isolated. Please add such an ablation or restrict the claim to a correlational one.
minor comments (5)
- [Section 4.1] There is a duplicated phrase 'in in 4.1' in the sentence describing the dataset and training configurations.
- [Table 1] The header 'W2V-Base+AASISTAdam' is missing a space between 'AASIST' and 'Adam'.
- [Figure 2] The loss-landscape visualization covers only one model-dataset pair, so the claim that SAM 'results in a noticeably flatter loss landscape' would be stronger if additional configurations were shown; as presented, the visualization is illustrative only.
- [Table 2] The p-values for the correlation coefficients are not adjusted for multiple comparisons across seven datasets and three metrics; please state whether the reported statistical significance survives a multiple testing correction.
- [Abstract and Conclusion] The phrase 'theoretical indicator' overstates what is currently an empirical correlation; consider using 'empirical indicator' or 'diagnostic' unless the requested confound controls change the strength of the evidence.
Circularity Check
No circularity: the sharpness-EER relationship is empirical, and SAM's objective does not by construction force EER improvement.
full rationale
The paper's load-bearing claim is empirical rather than derivational: sharpness is measured on test sets using Eq. (1) (adopted from Foret et al. [21] and Andriushchenko & Flammarion [23]), EER is measured on unseen evaluation sets, and the association is assessed via PCC/SRCC/KTAU in Table 2. No parameter is fitted to the target EER, and the sharpness values are not defined in terms of EER. Although SAM's objective in Eq. (2) contains the same sharpness term as Eq. (1), minimizing sharpness does not logically guarantee lower EER; indeed Table 1 contains counterexamples (e.g., W2V-Base+Linear on WF: SAM 36.95 vs Adam 30.34; W2V-XLSR+AASIST on SC: SAM 29.93 vs Adam 27.50), so the correlation is not forced by construction. Citations to the authors' own prior work are used as datasets/benchmarks or standard external references, not as the justification for the sharpness-generalization relationship. Potential confounds (fixed rho=0.05 across differently scaled models, pooling Adam and SAM groups) are validity and generalizability concerns, not circularity. Therefore no specific circular step can be exhibited.
Assumptions & free parameters
free parameters (4)
- SAM perturbation bound rho (per model) =
chosen from {0.05, 0.01, 0.005, 0.001}
- Sharpness evaluation rho =
0.05
- Batch size m for sharpness =
32
- Class weights for weighted cross-entropy =
0.9 bona fide / 0.1 spoof
assumptions (4)
- domain assumption The sharpness definition of Eq. (1) is a meaningful proxy for generalization in SDD.
- standard math The first-order perturbation approximation of SAM (Eqs. 4-5) is accurate enough to reduce sharpness as defined in Eq. (1).
- domain assumption Sharpness values computed with a fixed rho are comparable across architectures with very different parameter scales (AASIST vs W2V-Base/XLSR).
- domain assumption EER is the appropriate ground-truth generalization measure for SDD.
Cite this review
Pith. "Pith review of From Sharpness to Better Generalization for Speech Deepfake Detection." pith.science (2026). https://pith.science/paper/6CNFGZ4Z
@misc{pith2026250611532,
author = {Pith},
title = {Pith review of: From Sharpness to Better Generalization for Speech Deepfake Detection},
year = {2026},
howpublished = {\url{https://pith.science/paper/6CNFGZ4Z}},
note = {Machine review of arXiv:2506.11532}
}
read the original abstract
Generalization remains a critical challenge in speech deepfake detection (SDD). While various approaches aim to improve robustness, generalization is typically assessed through performance metrics like equal error rate without a theoretical framework to explain model performance. This work investigates sharpness as a theoretical proxy for generalization in SDD. We analyze how sharpness responds to domain shifts and find it increases in unseen conditions, indicating higher model sensitivity. Based on this, we apply Sharpness-Aware Minimization (SAM) to reduce sharpness explicitly, leading to better and more stable performance across diverse unseen test sets. Furthermore, correlation analysis confirms a statistically significant relationship between sharpness and generalization in most test settings. These findings suggest that sharpness can serve as a theoretical indicator for generalization in SDD and that sharpness-aware training offers a promising strategy for improving robustness.
Figures
Reference graph
Works this paper leans on
-
[1]
Introduction Speech Deepfake Detection (SDD) aims to differentiate gen- uine from synthetic speech, which plays a vital role in com- bating audio-based misinformation, fraud, and security threats. However, generalization remains a major challenge, as mod- els trained on specific datasets often struggle when tested on unseen data, particularly under mismat...
arXiv 2025
-
[2]
Sharpness and Domain Mismatch Sharpness measures the sensitivity of the loss function to per- turbations in the model’s parameters, which is often associated with the model’s robustness [21]. Given this characteristic, we aim to first explore whether sharpness can serve as a diagnos- tic tool to evaluate how domain mismatch impacts the model’s sensitivity...
work page 2019
-
[3]
Sharpness-Aware Minimization Sharpness-Aware Minimization is proposed in [21] to improve model generalization by simultaneously minimizing the loss value and its sharpness. Similar to Section 2, let S = (xi, yi)n i=1 be a set of la- beled data, and let ℓi(w) denote the loss evaluated at the point (xi, yi). For simplicity, we use LS(w) = 1/|S| Pn i=1 ℓi(w)...
-
[4]
Experimental Settings Datasets
Experiments and Analysis 4.1. Experimental Settings Datasets. Systems were trained on the ASVspoof 2019 LA training set [24], which consists of 25k clean English utter- ances, including spoofed samples generated using 6 different voice conversion and text-to-speech attacks. The evaluation was conducted on eight unseen datasets representing diverse domain ...
work page 2019
-
[5]
Conclusion In this work, we examined sharpness as a theoretical proxy for generalization in speech deepfake detection. Our findings show that sharpness increases under domain shifts, indicating model sensitivity to unseen conditions. Our correlation anal- ysis confirmed its positive association with generalization, es- pecially in high-mismatch scenarios....
-
[6]
This study is partially supported by JST AIP Acceleration Research (JPMJCR24U3)
Acknowledgements This work was conducted during the first author’s intern- ship at NII, Japan. This study is partially supported by JST AIP Acceleration Research (JPMJCR24U3). This work was also supported in part by China NSFC projects under Grants 62122050 and 62071288, in part by Shanghai Munici- pal Science and Technology Commission Project under Grant...
-
[7]
ASVspoof 2021: Accelerating progress in spoofed and deep- fake speech detection,
J. Yamagishi, X. Wang, M. Todisco, M. Sahidullah, J. Patino, A. Nautsch, X. Liu, K. A. Lee, T. Kinnunen, N. Evans et al. , “ASVspoof 2021: Accelerating progress in spoofed and deep- fake speech detection,” in ASVspoof 2021 Workshop-Automatic Speaker Verification and Spoofing Coutermeasures Challenge , 2021
work page 2021
-
[8]
Does audio deepfake detection generalize?
N. M. M ¨uller, P. Czempin, F. Dieckmann, A. Froghyar, and K. B¨ottinger, “Does audio deepfake detection generalize?” Proc. ISCA Interspeech, 2022
work page 2022
Show all 38 references
-
[9]
Raw- Boost: A raw data boosting and augmentation method applied to automatic speaker verification anti-spoofing,
H. Tak, M. Kamble, J. Patino, M. Todisco, and N. Evans, “Raw- Boost: A raw data boosting and augmentation method applied to automatic speaker verification anti-spoofing,” in Proc. ICASSP. IEEE, 2022, pp. 6382–6386
2022
-
[10]
SpecAugment: A simple data augmen- tation method for automatic speech recognition,
D. S. Park, W. Chan, Y . Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V . Le, “SpecAugment: A simple data augmen- tation method for automatic speech recognition,” arXiv preprint arXiv:1904.08779, 2019
1904 arXiv
-
[11]
Spoofed training data for speech spoofing countermeasure can be efficiently created using neural vocoders,
X. Wang and J. Yamagishi, “Spoofed training data for speech spoofing countermeasure can be efficiently created using neural vocoders,” in Proc. ICASSP. IEEE, 2023, pp. 1–5
2023
-
[12]
Can large-scale vocoded spoofed data improve speech spoofing countermeasure with a self-supervised front end?
——, “Can large-scale vocoded spoofed data improve speech spoofing countermeasure with a self-supervised front end?” in Proc. ICASSP. IEEE, 2024, pp. 10 311–10 315
2024
-
[13]
CPAUG: Refining copy-paste augmentation for speech anti-spoofing,
L. Zhang, K. A. Lee, L. Zhang, L. Wang, and B. Niu, “CPAUG: Refining copy-paste augmentation for speech anti-spoofing,” in ICASSP 2024-2024 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP) . IEEE, 2024, pp. 10 996–11 000
2024
-
[14]
AASIST: Audio anti-spoofing using integrated spectro-temporal graph attention networks,
J.-w. Jung, H.-S. Heo, H. Tak, H.-j. Shim, J. S. Chung, B.-J. Lee, H.-J. Yu, and N. Evans, “AASIST: Audio anti-spoofing using integrated spectro-temporal graph attention networks,” in Proc. ICASSP. IEEE, 2022, pp. 6367–6371
2022
-
[15]
Spoofing speech detection by modeling local spectro-temporal and long-term dependency,
H. Wu, W. Guo, Z. Zhang, W. Zhao, S. Peng, and J. Zhang, “Spoofing speech detection by modeling local spectro-temporal and long-term dependency,” inProc. ISCA Interspeech, 2024, pp. 507–511
2024
-
[16]
Temporal-channel modeling in multi-head self- attention for synthetic speech detection,
D.-T. Truong, R. Tao, T. Nguyen, H.-T. Luong, K. A. Lee, and E. S. Chng, “Temporal-channel modeling in multi-head self- attention for synthetic speech detection,” in Proc. ISCA Inter- speech, 2024, pp. 537–541
2024
-
[17]
wav2vec 2.0: A framework for self-supervised learning of speech repre- sentations,
A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech repre- sentations,” Advances in neural information processing systems , vol. 33, pp. 12 449–12 460, 2020
2020
-
[18]
XLS-R: Self-supervised cross-lingual speech rep- resentation learning at scale,
A. Babu, C. Wang, A. Tjandra, K. Lakhotia, Q. Xu, N. Goyal, K. Singh, P. von Platen, Y . Saraf, J. Pino, A. Baevski, A. Conneau, and M. Auli, “XLS-R: Self-supervised cross-lingual speech rep- resentation learning at scale,” arXiv, vol. abs/2111.09296, 2021
2021 arXiv
-
[19]
WavLM: Large-scale self- supervised pre-training for full stack speech processing,
S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiaoet al., “WavLM: Large-scale self- supervised pre-training for full stack speech processing,” IEEE Journal of Selected Topics in Signal Processing , vol. 16, no. 6, pp. 1505–1518, 2022
2022
-
[20]
One-class learning towards syn- thetic voice spoofing detection,
Y . Zhang, F. Jiang, and Z. Duan, “One-class learning towards syn- thetic voice spoofing detection,” IEEE Signal Processing Letters, vol. 28, pp. 937–941, 2021
2021
-
[21]
One-class learning with adap- tive centroid shift for audio deepfake detection,
H. M. Kim, K. Jang, and H. Kim, “One-class learning with adap- tive centroid shift for audio deepfake detection,” in Proc. ISCA Interspeech, 2024, pp. 4853–4857
2024
-
[22]
Generalizable audio deepfake detection via latent space refinement and augmen- tation,
W. Huang, Y . Gu, Z. Wang, H. Zhu, and Y . Qian, “Generalizable audio deepfake detection via latent space refinement and augmen- tation,” in Proc. ICASSP, 2025, pp. 1–5
2025
-
[23]
Tandem assessment of spoofing countermeasures and au- tomatic speaker verification: Fundamentals,
T. Kinnunen, H. Delgado, N. Evans, K. A. Lee, V . Vestman, A. Nautsch, M. Todisco, X. Wang, M. Sahidullah, J. Yamagishi et al., “Tandem assessment of spoofing countermeasures and au- tomatic speaker verification: Fundamentals,” IEEE Trans. Audio, Speech, Language Process., vol...
2020
-
[24]
t- EER: Parameter-free tandem evaluation of countermeasures and biometric comparators,
T. H. Kinnunen, K. A. Lee, H. Tak, N. Evans, and A. Nautsch, “t- EER: Parameter-free tandem evaluation of countermeasures and biometric comparators,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023
2023
-
[25]
On large-batch training for deep learning: Gen- eralization gap and sharp minima,
N. S. Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy, and P. T. P. Tang, “On large-batch training for deep learning: Gen- eralization gap and sharp minima,” 2017
2017
-
[26]
Fantastic generalization measures and where to find them,
Y . Jiang, B. Neyshabur, H. Mobahi, D. Krishnan, and S. Bengio, “Fantastic generalization measures and where to find them,” 2020
2020
-
[27]
Sharpness- aware minimization for efficiently improving generalization,
P. Foret, A. Kleiner, H. Mobahi, and B. Neyshabur, “Sharpness- aware minimization for efficiently improving generalization,” in International Conference on Learning Representations, 2021
2021
-
[28]
Multi-dataset co- training with sharpness-aware optimization for audio anti- spoofing,
H.-j. Shim, J.-w. Jung, and T. Kinnunen, “Multi-dataset co- training with sharpness-aware optimization for audio anti- spoofing,” in Proc. ISCA Interspeech, 2023, pp. 3804–3808
2023
-
[29]
Towards understanding sharpness-aware minimization,
M. Andriushchenko and N. Flammarion, “Towards understanding sharpness-aware minimization,” in International Conference on Machine Learning. PMLR, 2022, pp. 639–668
2022
-
[30]
ASVspoof 2019: A large-scale public database of synthe- sized, converted and replayed speech,
X. Wang, J. Yamagishi, M. Todisco, H. Delgado, A. Nautsch, N. Evans, M. Sahidullah, V . Vestman, T. Kinnunen, K. A. Lee et al., “ASVspoof 2019: A large-scale public database of synthe- sized, converted and replayed speech,” Computer Speech & Lan- guage, vol. 64, p. 101114, 2020
2019
-
[31]
FoR: A dataset for synthetic speech detection,
R. Reimao and V . Tzerpos, “FoR: A dataset for synthetic speech detection,” in 2019 International Conference on Speech Technol- ogy and Human-Computer Dialogue (SpeD) . IEEE, 2019, pp. 1–10
2019
-
[32]
WaveFake: A data set to facilitate audio deepfake detection,
J. Frank and L. Sch ¨onherr, “WaveFake: A data set to facilitate audio deepfake detection,” in Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021
2021
-
[33]
The LJ Speech dataset,
K. Ito and L. Johnson, “The LJ Speech dataset,” https://keithito. com/LJ-Speech-Dataset/, 2017
2017
-
[34]
JSUT corpus: Free large-scale japanese speech corpus for end-to-end speech synthe- sis,
R. Sonobe, S. Takamichi, and H. Saruwatari, “JSUT corpus: Free large-scale japanese speech corpus for end-to-end speech synthe- sis,” arXiv preprint arXiv:1711.00354, 2017
2017 arXiv
-
[35]
ADD 2022: the first audio deep synthesis detection challenge,
J. Yi, R. Fu, J. Tao, S. Nie, H. Ma, C. Wang, T. Wang, Z. Tian, Y . Bai, C. Fanet al., “ADD 2022: the first audio deep synthesis detection challenge,” in Proc. ICASSP. IEEE, 2022, pp. 9216– 9220
2022
-
[36]
SpoofCeleb: Speech deepfake detection and SASV in the wild,
J.-w. Jung, Y . Wu, X. Wang, J.-H. Kim, S. Maiti, Y . Matsunaga, H.-j. Shim, J. Tian, N. Evans, J. S. Chung et al., “SpoofCeleb: Speech deepfake detection and SASV in the wild,” IEEE Open Journal of Signal Processing, 2025
2025
-
[37]
Automatic speaker verification spoofing and deep- fake detection using wav2vec 2.0 and data augmentation,
H. Tak, M. Todisco, X. Wang, J.-w. Jung, J. Yamagishi, and N. Evans, “Automatic speaker verification spoofing and deep- fake detection using wav2vec 2.0 and data augmentation,” in The Speaker and Language Recognition Workshop, 2022
2022
-
[38]
Visualizing the loss landscape of neural nets,
H. Li, Z. Xu, G. Taylor, C. Studer, and T. Goldstein, “Visualizing the loss landscape of neural nets,”Advances in neural information processing systems, vol. 31, 2018
2018
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.