REVIEW 5 major objections 5 minor 29 references
End-to-end Audio Deepfake Detection from RAW Waveforms: a RawNet-Based Approach with Cross-Dataset Evaluation
T0 review · 5 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read A lightweight raw-waveform network, RawNetLite, reaches over 99.7% F1 on in-domain audio deepfakes and up to 83.4% F1 on a mixed cross-dataset benchmark, with the gains attributed to domain-mixed training, Focal Loss, and waveform…
desk verdict RawNetLite is a useful lightweight detector for matched-domain audio deepfake detection, but the paper's central generalization claim is overstated because the headline 'out-of-distribution' test set is actually matched-domain by the authors' own split design. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the RawNetLite architecture: a 1D convolutional layer followed by three residual blocks, adaptive average pooling, a bidirectional GRU with 128 units per direction, and a two-layer fully connected head with sigmoid output. It is trained with Focal Loss ($\gamma=2$, $\alpha=0.25$), which down-weights well-classified samples and steers gradient effort toward hard and ambiguous examples. The other two levers are domain-mix training, which oversamples multiple spoofing corpora, and on-the-fly waveform augmentation (pitch shift within $\pm 2$ semitones, time stretch in $[0.9, 1.1]$, Gaussian noise in $[0.001, 0.015]$). These components together are what the paper credits for the reported robustness.
What would settle it
The decisive calculation is already in Table XI: when CodecFake is left out of training, the model scores only 15.1% fake F1 and 50.7% EER on CodecFake, so the 83.4% headline rests on that dataset's presence in training; a fresh test on a TTS or VC generator absent from all training splits would confirm whether any training recipe yields true open-world robustness.
Extended reading notes
Core claim
On its own terms, the discovery is that a small raw-waveform model can match or beat much larger pretrained systems on in-domain spoofed audio while remaining cheap to deploy, and that its cross-domain behavior is governed more by the composition of training data and the loss function than by architecture. The paper reports that adding 20% AVSpoof2021 data to FakeOrReal training lifts fake recall on AVSpoof2021 from 22.1% to 77.6%, that Focal Loss alone raises cross-domain fake F1 from 55.8% to 79.5%, and that including CodecFake in the training mix brings its fake F1 from 15.1% to 78.5%. The headline 83.4% F1 on the combined AVSpoof2021+CodecFake test is presented as evidence that this strategy tames the open-world gap.
Load-bearing premise
The load-bearing assumption is that the AVSpoof2021+CodecFake test is genuinely new to the model, even though 12,800 of its clips (6,400 per class) already appeared in training; remove that overlap and the headline 83.4% F1 drops to 15.1% F1 on the only fully unseen dataset.
Editorial extensions
If this is right
- A lightweight raw-waveform detector can reach over 99% F1 on generators seen in training, making it a candidate for low-resource deployment where computational budget is tight.
- Adding a modest slice of target-domain data (about 20% of a dataset) to the training mix can shift fake recall by over 50 percentage points, implying that data selection, not backbone size, is the dominant lever in this task.
- Focal Loss appears to act as a surrogate for harder example mining in cross-domain settings, improving fake F1 by roughly 24 points on AVSpoof2021 without degrading in-domain performance.
- Waveform-level augmentation alone can lift fake F1 on a codec-degraded dataset from 15.1% to 74.7%, suggesting that simulating acoustic variability is a cheap route to robustness.
- Combining all three strategies yields the best reported numbers (83.4% F1, 16.4% EER) on the joint AVSpoof2021+CodecFake test, indicating the authors view the combination as the practical recipe for open-world detection.
Reading between the lines
- The headline cross-dataset number is better read as a matched-domain measurement than an open-world one, because training includes 6,400 real and 6,400 fake samples from each of AVSpoof2021 and CodecFake; the genuinely unseen result in Table XI (CodecFake: 15.1% fake F1, 50.7% EER) is the more honest estimate of open-world transfer.
- A direct consequence is that an auditor holding out an entirely new TTS or VC generator, with different codecs and recording conditions, should expect performance closer to the 15% F1 regime than the 83% regime unless the generator's artifacts overlap with the training corpora.
- The augmentation results suggest a testable extension: if codec-specific distortions (e.g., MP3 or Opus artifacts) are simulated instead of generic noise and time stretch, the remaining CodecFake gap might close further; this could be verified without acquiring new datasets.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes RawNetLite, a lightweight end-to-end convolutional-recurrent network that classifies raw waveforms as real or fake speech, and evaluates it on FakeOrReal, ASVspoof2021, and CodecFake. The training pipeline combines domain mixing, Focal Loss, and waveform augmentations. The authors report near-perfect in-domain performance on FakeOrReal (up to 99.75% F1, 0.25% EER in the Focal-loss cross-domain configuration) and claim up to 83.4% F1 with 16.4% EER on a so-called out-of-distribution AVSpoof2021+CodecFake test set. The paper emphasizes that the lightweight architecture competes with larger pretrained systems while remaining deployable, and makes code and pretrained models available.
Significance. If the headline generalization claim were valid, the paper would make a useful empirical contribution: a small raw-waveform model that is near-perfect in-domain and substantially robust across codec-degraded speech without large self-supervised backbones. The in-domain results are competitive with much larger systems, and the systematic comparison of BCE versus Focal Loss, domain mixing, and augmentation is a useful design study. The explicit reporting of failure cases on truly unseen datasets (e.g., the poor CodecFake results when CodecFake is not in training) is also a strength. However, the central cross-domain claim is not supported by the experimental design as written: the datasets used to form the headline ``out-of-distribution'' test set are also partially included in training, so the 83.4% F1 / 16.4% EER result is a matched-domain measurement, not an open-world one. The genuine cross-dataset numbers in the paper tell a much weaker and partly contradictory story.
major comments (5)
- [Section III.F, Tables II--III, Section V.H] The headline result is not out-of-distribution. Tables II and III state that 6,400 real and 6,400 fake samples from each of ASVspoof2021 and CodecFake are included in training in the triple-domain configuration, while Section III.F claims that cross-dataset evaluations are conducted on datasets not used in training. These statements are mutually inconsistent. The held-out test samples share spoofing generators, codecs, and recording conditions with the training samples from the same datasets; sample-level disjointness does not make the test set out-of-distribution. Therefore the abstract's claim of ``challenging out-of-distribution test set (AVSpoof2021 + CodecFake)'' and Section V.H's claim of 83.4% F1 on that set describe matched-domain evaluation. The truly cross-dataset results are the CodecFake rows in Tables V, VI, and XI, which show 15--18% F1 and about 50% EER and support the opposite conclusion.
- [Tables XI and XIV, Section V.G] There is an unexplained contradiction between the two CodecFake results for training on FakeOrReal+ASVspoof2021. Table XI reports 15.11% fake F1 and 50.68% EER on CodecFake under cross-domain training with Focal Loss, while Table XIV reports 74.66% fake F1 on CodecFake under training on FakeOrReal+ASVspoof2021 with augmentation. If the only difference is the on-the-fly augmentation, the magnitude of the improvement is surprising and needs a detailed explanation, including whether the same test subset, preprocessing, and number of epochs were used. As presented, the discrepancy undermines the paper's claim that augmentation alone confers open-world robustness, because the two tables appear to describe the same base training set with and without augmentation.
- [Table X, Section V.E] The text states that replacing BCE with Focal Loss produced a ``substantial performance gain'' on ASVspoof2021, but Table X shows that EER worsens from 16.60% to 17.34%. EER is a threshold-independent metric and is the standard summary for spoofing detection; a result that improves recall and F1 while increasing EER is a trade-off, not an unqualified gain. The discussion should acknowledge this directly and explain which operating point the F1 numbers correspond to, otherwise the reader cannot tell whether Focal Loss actually improves the detection trade-off.
- [Table IV, Table I] The support counts in Table IV are inconsistent with the dataset description. Table I specifies a test set with 3,200 real and 3,200 fake samples, but Table IV reports support of 32,496 real and 32,428 fake samples, summing to 64,924 total, which exceeds the entire FakeOrReal dataset size of 64,000. These numbers are impossible under the stated 80/10/10 split and need to be corrected or explained.
- [Abstract, Table IV, Table XI] The abstract's in-domain numbers do not match the baseline table. The abstract states ``over 99.7% F1 and 0.25% EER'' on FakeOrReal, while Table IV (baseline) reports 99.27% F1 and 0.29% EER, and Table XI (Focal-loss cross-domain training) reports 99.75% F1 and 0.25% EER. The abstract appears to mix the baseline F1 with the Focal-loss EER. The text should consistently report which configuration produced each number.
minor comments (5)
- [Section III.F and Section V.A] The claim in Section V.A that test splits are ``strictly disjoint, ensuring no overlap in generation methods or identities'' is contradicted by Tables II--III, which include the same datasets in training; the wording should be corrected to ``no overlap at the sample level.''
- [Abstract and throughout] The dataset name is typeset with an internal space as ``AVSpoof2021'' in several places; this should be fixed to ``ASVspoof2021'' for consistency with the official challenge name and references.
- [Section V.G, Table XIII] The text says Table XIII shows a fake F1-score of 78.05% on the test set, but the table shows fake F1 of 80.84% and accuracy of 80.96%; similarly, the text mentions a balanced accuracy of 77.74% that does not appear in the table. These numbers need to be reconciled.
- [Section II, Reference [19]] Reference [19] is listed with an incomplete arXiv identifier ``arXiv:2503.xxxxx''; the full identifier or a published venue should be provided.
- [Table XV] In Table XV, the RawNet2 baseline is cited as [12] (the ASVspoof2021 evaluation plan), but the RawNet2 architecture is described in [13]; the citation should be corrected.
Circularity Check
Headline OOD result is matched-domain: AVSpoof2021 and CodecFake are in the triple-domain training set, so 83.4% F1/16.4% EER is not out-of-distribution.
-
fitted input called prediction
[Abstract; Section III.B Table II; Section III.C Table III; Section V.B 'Triple-domain' definition; Section V.H]
"The proposed model achieves over 99.7% F1 and 0.25% EER on in-domain data (FakeOrReal), and up to 83.4% F1 with 16.4% EER on a challenging out-of-distribution test set (AVSpoof2021 + CodecFake). ... We included a balanced subset of 6400 real and 6400 fake samples during training. ... CodecFake was initially used exclusively for cross-dataset testing. In later configurations (triple-dataset and augmentation), a subset was included in training ... Triple-domain: training and testing involve all three datasets."
The abstract calls AVSpoof2021+CodecFake an 'out-of-distribution test set,' but the triple-domain configuration that produces the 83.4% F1 / 16.4% EER number trains on 6,400 real and 6,400 fake samples from each of AVSpoof2021 and CodecFake (Tables II and III), and Section V.B defines triple-domain as 'training and testing involve all three datasets.' Held-out splits from the same datasets share spoofing generators, codecs, and recording conditions with the training data. The headline 'OOD' claim therefore reduces, by the paper's own experimental design, to a matched-domain evaluation. The genuinely external benchmark in Table XI (CodecFake with no training exposure) gives 15.1% F1 and 50.7% EER, which supports the opposite conclusion.
full rationale
The paper's derivation chain is empirical rather than formal, and most architectural and loss choices are tested independently rather than assumed circularly. The load-bearing circular step is the labeling of the combined AVSpoof2021+CodecFake test set as out-of-distribution in the triple-domain setting. Tables II and III explicitly place training subsets from both datasets in that configuration, and Section V.B defines triple-domain as training and testing involving all three datasets; Section III.F's claim that cross-dataset evaluations were conducted exclusively on data from datasets not used in training is contradicted by those tables. Consequently, the 83.4% F1 / 16.4% EER headline is a held-out-same-distribution measurement, not an open-world result. The paper does include one genuinely external evaluation (CodecFake without any training exposure), where it reports 15.1% F1 and 50.7% EER, so the circularity is localized to the headline generalization claim rather than the whole study. No load-bearing self-citation or ansatz-smuggling is present. Score 6: a central claimed prediction reduces by construction to a matched-domain fit, while a genuinely cross-domain result is reported and is poor.
Assumptions & free parameters
free parameters (5)
- Focal Loss focusing parameter gamma =
2
- Focal Loss class-balance alpha =
0.25
- Augmentation ranges =
pitch [-2,2] semitones, time stretch [0.9,1.1], noise [0.001,0.015], p=0.5
- Domain-mix training subset size =
6,400 real + 6,400 fake per dataset
- Architecture capacity =
64 Conv1D filters, 3 residual blocks, 128 GRU units
assumptions (4)
- ad hoc to paper Held-out splits of AVSpoof2021 and CodecFake are treated as out-of-distribution despite training on subsets of those datasets (Section V.A, Tables II-III).
- domain assumption The chosen public datasets (FakeOrReal, AVSpoof2021, CodecFake) collectively represent open-world audio conditions (Section III).
- domain assumption Waveform-level augmentations with the given ranges simulate realistic acoustic conditions (Section III.E).
- domain assumption The 3-second fixed-length truncation preserves enough information to detect spoofing (Section III.D).
Cite this review
Pith. "Pith review of End-to-end Audio Deepfake Detection from RAW Waveforms: a RawNet-Based Approach with Cross-Dataset Evaluation." pith.science (2026). https://pith.science/paper/XF4GNGON
@misc{pith2026250420923,
author = {Pith},
title = {Pith review of: End-to-end Audio Deepfake Detection from RAW Waveforms: a RawNet-Based Approach with Cross-Dataset Evaluation},
year = {2026},
howpublished = {\url{https://pith.science/paper/XF4GNGON}},
note = {Machine review of arXiv:2504.20923}
}
read the original abstract
Audio deepfakes represent a growing threat to digital security and trust, leveraging advanced generative models to produce synthetic speech that closely mimics real human voices. Detecting such manipulations is especially challenging under open-world conditions, where spoofing methods encountered during testing may differ from those seen during training. In this work, we propose an end-to-end deep learning framework for audio deepfake detection that operates directly on raw waveforms. Our model, RawNetLite, is a lightweight convolutional-recurrent architecture designed to capture both spectral and temporal features without handcrafted preprocessing. To enhance robustness, we introduce a training strategy that combines data from multiple domains and adopts Focal Loss to emphasize difficult or ambiguous samples. We further demonstrate that incorporating codec-based manipulations and applying waveform-level audio augmentations (e.g., pitch shifting, noise, and time stretching) leads to significant generalization improvements under realistic acoustic conditions. The proposed model achieves over 99.7% F1 and 0.25% EER on in-domain data (FakeOrReal), and up to 83.4% F1 with 16.4% EER on a challenging out-of-distribution test set (AVSpoof2021 + CodecFake). These findings highlight the importance of diverse training data, tailored objective functions and audio augmentations in building resilient and generalizable audio forgery detectors. Code and pretrained models are available at https://iplab.dmi.unict.it/mfs/Deepfakes/PaperRawNet2025/.
Figures
Reference graph
Works this paper leans on
-
[1]
Deepfake media forensics: Status and future challenges,
I. Amerini, M. Barni, S. Battiato, P. Bestagini, G. Boato, V . Bruni, R. Caldelli, F. De Natale, R. De Nicola, L. Guarnera et al. , “Deepfake media forensics: Status and future challenges,” Journal of Imaging , vol. 11, no. 3, p. 73, 2025
work page 2025
-
[2]
Deepfakes and beyond: A survey of face manipulation and fake detection,
R. Tolosana, R. Vera-Rodriguez, J. Fierrez, A. Morales, and J. Ortega- Garcia, “Deepfakes and beyond: A survey of face manipulation and fake detection,” Information Fusion , vol. 64, pp. 131–148, 2020
work page 2020
-
[3]
The creation and detection of deepfakes: A survey,
Y . Mirsky and W. Lee, “The creation and detection of deepfakes: A survey,” ACM Computing Surveys (CSUR) , vol. 54, no. 1, pp. 1–41, 2021
work page 2021
-
[4]
Audio deepfake detection: A survey,
J. Yi, C. Wang, J. Tao, X. Zhang, C. Y . Zhang, and Y . Zhao, “Audio deepfake detection: A survey,” arXiv preprint arXiv:2308.14970 , 2023
arXiv 2023
-
[5]
M. Casu, L. Guarnera, P. Caponnetto, and S. Battiato, “GenAI mirage: The impostor bias and the deepfake detection challenge in the era of artificial illusions,” F orensic Science International: Digital Investigation, vol. 50, p. 301795, 2024
work page 2024
-
[6]
Breaking the silence: Detecting ai-converted voices in the quietest moments,
S. Borz `ı, L. Mongelli, F. Stanco, S. Battiato, and D. Allegra, “Breaking the silence: Detecting ai-converted voices in the quietest moments,” in Pattern Recognition. ICPR 2024 International Workshops and Chal- lenges, S. Palaiahnakote, S. Schuckers, J.-M. Ogier, P. Bhattacharya, U. Pal, and S. Bhattacharya, Eds. Cham: Springer Nature Switzerland, 2025, ...
work page 2024
-
[7]
Deepfake style transfer mixture: A first forensic ballistics study on synthetic images,
L. Guarnera, O. Giudice, and S. Battiato, “Deepfake style transfer mixture: A first forensic ballistics study on synthetic images,” in Image Analysis and Processing – ICIAP 2022 , vol. 13232. Cham: Springer International Publishing, 2022, pp. 151–163
work page 2022
-
[8]
AI Chatbots for Mental Health: A Scoping Review of Effectiveness, Feasibility, and Applications,
M. Casu, S. Triscari, S. Battiato, L. Guarnera, and P. Caponnetto, “AI Chatbots for Mental Health: A Scoping Review of Effectiveness, Feasibility, and Applications,” Applied Sciences , vol. 14, p. 5889, 07 2024
work page 2024
Show all 29 references
-
[9]
Mastering deepfake detection: A cutting-edge approach to distinguish GAN and diffusion-model im- ages,
L. Guarnera, O. Giudice, and S. Battiato, “Mastering deepfake detection: A cutting-edge approach to distinguish GAN and diffusion-model im- ages,” ACM Transactions on Multimedia Computing, Communications and Applications , vol. 20, no. 11, pp. 1–24, 2024
2024
-
[10]
Is synthetic voice detection research going into the right direction?
S. Borz `ı, O. Giudice, F. Stanco, and D. Allegra, “Is synthetic voice detection research going into the right direction?” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2022, pp. 71–80
2022
-
[11]
Detection of synthetic speech for spoofing attack using frame-level deep features,
P. Nagarsheth, X. Song, J. Hines, and J. H. Hansen, “Detection of synthetic speech for spoofing attack using frame-level deep features,” in Proc. INTERSPEECH, 2017, pp. 706–710
2017
-
[12]
End-to-end anti-spoofing with rawnet2,
H. Tak, J. Patino, M. Todisco, A. Nautsch, N. Evans, and A. Larcher, “End-to-end anti-spoofing with rawnet2,” 2021
2021
-
[13]
To- rawnet: Improving rawnet with tcn and orthogonal regularization for fake audio detection,
C. Wang, J. Yi, J. Tao, C. Zhang, S. Zhang, R. Fu, and X. Chen, “To- rawnet: Improving rawnet with tcn and orthogonal regularization for fake audio detection,” arXiv preprint arXiv:2305.13701 , 2023
2023 arXiv
-
[14]
Attentive filtering network for audio deepfake detection,
Y . Lai, Y . Liu, X. Liang, J. Zhu, T. Zhao, S. Yu, and C. Wang, “Attentive filtering network for audio deepfake detection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 3849–3858
2023
-
[15]
Specrnet: A lightweight end-to-end model for audio deepfake detection,
A. Janicki and R. Heusdens, “Specrnet: A lightweight end-to-end model for audio deepfake detection,” in 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2022, pp. 3064–3068
2022
-
[16]
Asvspoof 2021: Automatic speaker verification spoofing and deepfake detection challenge evaluation plan,
H. Tak, M. Todisco, X. Wang, N. Evans, J. Yamagishi, and F. L. Alegre, “Asvspoof 2021: Automatic speaker verification spoofing and deepfake detection challenge evaluation plan,” in Proc. ASVspoof Workshop 2023, 2023, challenge paper presenting baseline systems using Wav2Vec 2....
2021
-
[17]
Audio deepfake detection with self- supervised wavlm and multi-fusion attentive classifier,
Y . Guo, H. Huang, X. Chen et al., “Audio deepfake detection with self- supervised wavlm and multi-fusion attentive classifier,” inProc. ICASSP, 2024
2024
-
[18]
Audio deepfake detection with self- supervised xls-r and sls classifier,
Q. Zhang, S. Wen, and T. Hu, “Audio deepfake detection with self- supervised xls-r and sls classifier,” in Proc. ACM MM , 2024
2024
-
[19]
Anomaly detection and localiza- tion for speech deepfakes via feature pyramid matching,
E. Coletta, D. Salvi, V . Negroni et al., “Anomaly detection and localiza- tion for speech deepfakes via feature pyramid matching,” arXiv preprint arXiv:2503.xxxxx, 2025
2025
-
[20]
One-class learning with adaptive centroid shift for audio deepfake detection,
H. M. Kim, K. Jang, and H. Kim, “One-class learning with adaptive centroid shift for audio deepfake detection,” in Proc. Interspeech, 2024
2024
-
[21]
End-to-end spoofing detection with rawnet2 on logical access and physical access,
S. Chettri and H. A. Patil, “End-to-end spoofing detection with rawnet2 on logical access and physical access,” in Proc. Odyssey 2020 The Speaker and Language Recognition Workshop , 2021, pp. 132–138
2020
-
[22]
For: A dataset for synthetic speech detection,
R. Reimao and V . Tzerpos, “For: A dataset for synthetic speech detection,” in 2019 International Conference on Speech Technology and Human-Computer Dialogue (SpeD) , 2019, pp. 1–10
2019
-
[23]
Asvspoof 2019: Future horizons in spoofed and fake audio detection,
M. Todisco, X. Wang, V . Vestman, M. Sahidullah, H. Delgado, A. Nautsch, J. Yamagishi, N. Evans, T. Kinnunen, and K. A. Lee, “Asvspoof 2019: Future horizons in spoofed and fake audio detection,” arXiv preprint arXiv:1904.05441 , 2019
2019 arXiv
-
[24]
The codecfake dataset and countermeasures for the universally detection of deepfake audio,
Y . Xie, Y . Lu, R. Fu, Z. Wen, Z. Wang, J. Tao, X. Qi, X. Wang, Y . Liu, H. Cheng, L. Ye, and Y . Sun, “The codecfake dataset and countermeasures for the universally detection of deepfake audio,” 2025
2025
-
[25]
A survey on deepfake detection: fundamentals, current challenges, and future trends,
Y . Zhang, H. Xie, J. Xing, X. Zhen, Z. Wang, and F. Liu, “A survey on deepfake detection: fundamentals, current challenges, and future trends,” Computers & Security , vol. 112, p. 102494, 2021
2021
-
[26]
Focal loss for dense object detection,
T.-Y . Lin, P. Goyal, R. Girshick, K. He, and P. Doll ´ar, “Focal loss for dense object detection,” in Proceedings of the IEEE International Conference on Computer Vision (ICCV) , 2017, pp. 2980–2988
2017
-
[27]
A systematic analysis of performance measures for classification tasks,
M. Sokolova and G. Lapalme, “A systematic analysis of performance measures for classification tasks,” Information Processing & Manage- ment, vol. 45, no. 4, pp. 427–437, 2009
2009
-
[28]
The det curve in assessment of detection task performance,
A. Martin, G. Doddington, T. Kamm, M. Ordowski, and M. Przybocki, “The det curve in assessment of detection task performance,” in Proc. Eurospeech, 1997
1997
-
[29]
Leveraging mixture of experts for improved speech deepfake detection,
V . Negroni, D. Salvi, A. Ilic Mezza et al., “Leveraging mixture of experts for improved speech deepfake detection,” in Proc. ICASSP, 2025
2025
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.