REVIEW 2 major objections 5 minor 1 cited by
Replay Attacks Against Audio Deepfake Detection
T0 review · 2 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Replaying deepfake audio through speakers and microphones masks synthetic traces, raising the best detector's equal error rate from 4.7% to 18.2%.
desk verdict First real-world replay dataset for audio deepfake detection; the vulnerability is real, but the headline EER is driven by poor-quality setups and needs an intelligibility filter. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is ReplayDF, a dataset of 52,320 recordings built by playing bona fide and spoofed samples through 109 distinct loudspeaker-microphone setups and re-recording them, with full metadata including room impulse responses, setup photos, language, TTS model, and quality scores. This air-gap pipeline supplies the physical distortion—playback, re-recording, room acoustics, and hardware coloration—that the paper argues removes the artifacts detectors rely on. The RIR augmentation experiments use those recorded room impulse responses to retrain a detector, providing the paper's test of whether the vulnerability can be adapted away.
What would settle it
Collect replay recordings made with commodity consumer devices (laptop speakers, phone microphones, smart speakers) in ordinary rooms and measure whether a detector such as W2V2-AASIST shows the same EER jump; if the effect largely disappears or requires very high-quality playback, then the vulnerability is specific to the lab setups rather than the air-gap itself.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that replay attacks function as a universal disguising mechanism for deepfake audio: when spoofed clips are played and recorded through real hardware, detectors increasingly label them bona fide, while genuine clips are still labeled correctly. The effect appears across all six tested detector architectures and all five training datasets, and is not reproduced by simply adding noise at comparable levels, suggesting that the re-recording removes model-relevant synthetic artifacts rather than merely corrupting the signal. Adaptive defense via RIR augmentation helps but does not close the gap, leaving replayed deepfakes substantially less detectable than original ones.
Load-bearing premise
The load-bearing premise is that ReplayDF's 109 loudspeaker-microphone setups and lab rooms represent realistic replay attacks; if actual attackers use different hardware, room acoustics, or playback levels, the measured error rates may not transfer to deployed systems.
Editorial extensions
If this is right
- Deployed deepfake detection systems that assume digital input can be evaded by a cheap physical replay step; detectors should be evaluated on re-recorded audio before deployment.
- The vulnerability is broad: all six open-source detectors and all five training datasets showed degradation, so the issue is not specific to one architecture.
- RIR augmentation during training improves robustness (EER from 18.2% to 11.0%) but does not restore baseline performance, so current augmentation strategies are insufficient.
- Because bona fide detection stays high, replay attacks specifically target the spoof class, suggesting that detectors could be tuned with a replay-aware decision threshold, at the cost of usability.
- The dataset release enables standardized benchmarking of replay robustness for future detection models.
Reading between the lines
- If this vulnerability generalizes beyond the lab setups, forensic detection pipelines that rely on artifact analysis should treat physical replay as a first-class threat, similar to replay attacks in speaker verification.
- The noise experiment suggests that detectors are sensitive to convolutional (filter-like) distortions rather than additive noise, pointing toward training on RIR-like transformations at scale or on invariant representations as a research direction the paper does not fully explore.
- A testable extension would be to replay audio through consumer devices (laptops, phones, smart speakers) in varied rooms and measure EER; if the effect shrinks or grows with device quality, the representativeness of the 109 setups can be calibrated.
- The threshold between 'deepfake' and 'bona fide' may become a physical-channel question: if a detector cannot distinguish an authentic recording from a replayed one, then for spoofed samples the replay attack essentially converts the detection problem into a source-authentication problem.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces ReplayDF, a large dataset of re-recorded (air-gapped) bona fide and TTS-generated speech from 109 loudspeaker-microphone setups across six languages, and uses it to evaluate the vulnerability of several open-source audio deepfake detectors. The central empirical claim is that replay attacks make synthetic speech appear authentic: for the best model, W2V2-AASIST trained on ODSS, the equal error rate rises from 4.7% on the original files to 18.2% on the re-recorded ReplayDF files. The paper also reports that the degradation is concentrated in spoofed rather than bona fide utterances, that additive noise alone does not reproduce the effect, and that room-impulse-response augmentation during training only partially mitigates the vulnerability. The dataset and evaluation code are released for non-commercial use.
Significance. If the central claim is correct, the paper addresses an important and understudied gap: most deepfake detection research evaluates synthetic audio directly, whereas a realistic attacker can play the audio through a loudspeaker and re-record it. The paper's controlled air-gap design, with baseline and replayed samples originating from the same files, cleanly isolates the effect of the physical propagation path. The release of ReplayDF, including metadata, setup photographs, and room impulse responses, is a valuable community resource. The multi-model and multi-training-dataset evaluation strengthens the generality of the observed degradation. However, as detailed in the major comments, the scope of the headline claim needs to be tempered by an analysis of how the vulnerability depends on recording quality and intelligibility, and the adaptive-defense result is weakened by test-condition leakage.
major comments (2)
- [§4.6, Table 3, Fig. 2] The headline result (EER rising from 4.7% to 18.2%) is pooled over all 109 setups, but Figure 2 shows that detection accuracy correlates with MOS (r=0.423) and PESQ (r=0.509). The paper never reports performance for the subset of high-quality, intelligible recordings. Without this analysis, the abstract's claim that replay attacks make spoofed audio 'appear authentic' is only established for the average over setups that include many very low-quality recordings. A real vishing or voice-biometric attack requires intelligible speech, so the practical threat may be substantially smaller than the pooled EER suggests. Please report EER and accuracy for high-quality subsets (e.g., MOS ≥ 3 or PESQ ≥ 3) and, ideally, an intelligibility metric such as WER for ReplayDF. This is a load-bearing point for the paper's main contribution.
- [§4.5, Table 4] The RIR augmentation defense is trained using room impulse responses from ReplayDF itself, which are then used to evaluate on the same setups. This is test-condition leakage: the defender in the experiment has access to the attacker's exact channel characteristics. The paper discloses this usage, but the conclusion that 'adaptive retraining using RIRs from ReplayDF can help to lessen the effects of replay attacks' is weaker than a defender would need. Please add a held-out evaluation (e.g., train on RIRs from a subset of setups and test on the rest) or explicitly frame the current experiment as an upper bound on the benefit of RIR augmentation.
minor comments (5)
- [Introduction, §2] The term 'air-gap' is used without definition; please define it at first occurrence as the process of playing audio through a loudspeaker and re-recording it with a microphone.
- [§2] There are several typos: 'suing' should be 'using' in the back-end sentence; 'V oice' in the Introduction should be 'Voice'; 'performace' in Section 4.2 should be 'performance'.
- [§3] The text states 'The data generation pipeline, outlined in Section 3' but Section 3 is the current section; this should refer to the algorithm description within the same section or to a different section.
- [Figure 2] The x-axis is labeled only as 'Recording UID'; please clarify the ordering (e.g., sorted by quality score) and label the axes with the respective score ranges so the scatter plot is self-contained.
- [§4.7] The noise experiment varies SNR between 15 and 40 dB but the paper does not report the distribution of resulting SNRs or the EER for each noise type; adding this would strengthen the claim that additive noise is not the cause of the degradation.
Circularity Check
No derivation-level circularity; disclosed test-condition RIR leakage in the defense experiment is the only caveat.
full rationale
The paper is an empirical measurement study; it contains no derivation that maps inputs to outputs by construction. ReplayDF is built by physically playing and re-recording samples from M-AILABS and MLAAD across 109 loudspeaker/microphone setups (Algorithm 1), and the detection models are either publicly released checkpoints (Table 2) or models retrained on standard public datasets and evaluated without adaptation (Table 3). No parameter is fitted to ReplayDF and then renamed as a prediction; the headline EER increase (4.7% to 18.2%) is a held-out evaluation on data the models never saw in training. The author-affiliated citations ([19] In-the-Wild, [21] shortcut learning) are used as a training/evaluation dataset and an explanatory hypothesis, not as a uniqueness theorem or as justification for the central vulnerability claim. The only design element that could be mistaken for circularity is the Section 4.5 adaptive defender, which uses ReplayDF RIRs during training and evaluates on ReplayDF ('During training, we convolve the training data with RIRs from ReplayDF, then evaluate performance on both the Baseline and ReplayDF datasets as before'). This is disclosed test-condition leakage and it does not support the main claim, which is established by the non-adaptive results in Tables 2 and 3. Concerns about pooled low-quality setups driving the EER (Figure 2 correlations of 0.423/0.509) are generalizability/correctness issues, not circularity. No step reduces to its own input by definition or by self-citation.
Assumptions & free parameters
assumptions (4)
- domain assumption The 109 speaker-microphone setups in ReplayDF are representative of realistic replay attack conditions.
- domain assumption The four TTS models used (Bark, VITS, XTTS v1.1, XTTS v2.0) are representative of current deepfake generators.
- domain assumption The six open-source detection models and their checkpoints are representative of current audio deepfake detection.
- domain assumption Subjective quality metrics (MOS, PESQ) correlate with the features that detection models rely on.
Cite this review
Pith. "Pith review of Replay Attacks Against Audio Deepfake Detection." pith.science (2026). https://pith.science/paper/DWVQ4YSM
@misc{pith2026250514862,
author = {Pith},
title = {Pith review of: Replay Attacks Against Audio Deepfake Detection},
year = {2026},
howpublished = {\url{https://pith.science/paper/DWVQ4YSM}},
note = {Machine review of arXiv:2505.14862}
}
read the original abstract
We show how replay attacks undermine audio deepfake detection: By playing and re-recording deepfake audio through various speakers and microphones, we make spoofed samples appear authentic to the detection model. To study this phenomenon in more detail, we introduce ReplayDF, a dataset of recordings derived from M-AILABS and MLAAD, featuring 109 speaker-microphone combinations across six languages and four TTS models. It includes diverse acoustic conditions, some highly challenging for detection. Our analysis of six open-source detection models across five datasets reveals significant vulnerability, with the top-performing W2V2-AASIST model's Equal Error Rate (EER) surging from 4.7% to 18.2%. Even with adaptive Room Impulse Response (RIR) retraining, performance remains compromised with an 11.0% EER. We release ReplayDF for non-commercial research use.
Figures
Forward citations
Cited by 1 Pith paper
-
Robust Localization of Partially Fake Speech: Metrics and Out-of-Domain Evaluation
Segment-level EER overstates deployment readiness for partial fake speech localizers, which drop from 7.6% to above 40% EER on out-of-domain test sets.
Reference graph
Works this paper leans on
-
[1]
Introduction Text-to-speech (TTS) and V oice Conversion (VC) technolo- gies have facilitated numerous advancements across various do- mains. In the entertainment industry, they enable independent studios in television and film to generate a diverse array of char- acter voices, including the recreation of legacy character voices. In the healthcare sector, ...
arXiv 2025
-
[2]
Related Work Audio deepfake detection has been largely driven by ASVspoof [7–9], a challenge that initially focused on voice biometrics but recently expanded to include deepfake detec- tion. The urgency of the problem is underscored by the rise of commercial text-to-speech providers like Resemble AI, Re- speecher, and ElevenLabs [10–12], some of which hav...
work page 2021
-
[3]
ReplayDF Database To systematically assess the impact of replay attacks on deep- fake detection, we introduceReplayDF, a dataset of audio recordings generated by playing and re-recording both bona fide and spoofed samples using a diverse set of loudspeakers and microphones. The dataset spans six languages and incorporates attacks from four TTS models, ens...
-
[4]
Experiments 4.1. Evaluation Approach We evaluateReplayDFacross multiple scenarios to assess the impact of replay attacks on audio deepfake detection models. We define two key data partitions: first,ReplayDF(setR): All audio files generated as in Section 3, containing equal amounts of bona fide and spoofed instances. Second, theBaselinedataset (setO): the ...
-
[5]
Conclusion In this work, we investigate the impact of replay attacks on audio deepfake detection systems by introducingReplayDF, a comprehensive dataset of132.5hours of re-recorded spoof and bona fide audio. Our results demonstrate that replay attacks sig- nificantly degrade detection performance, effectively disguising deepfake audio as authentic, while ...
work page 2024
-
[6]
F. Biadsy, R. J. Weiss, P. J. Moreno, D. Kanvesky, and Y . Jia, “Par- rotron: An End-to-End Speech-to-Speech Conversion Model and its Applications to Hearing-Impaired Speech and Speech Separa- tion,” inProc. Interspeech 2019, 2019, pp. 4115–4119
work page 2019
-
[7]
How deepfake videos are used to spread disinformation - the new york times,
“How deepfake videos are used to spread disinformation - the new york times,” https://www.nytimes.com/2023/02/ 07/technology/artificial-intelligence-training-deepfake.html, (Accessed: 16.10.2024)
work page 2023
-
[8]
Taylor swift nude deepfake goes viral on x, despite platform rules,
K. Tenbarge, “Taylor swift nude deepfake goes viral on x, despite platform rules,” 1 2024, [Online; accessed 03.02.2025]. [On- line]. Available: https://www.nbcnews.com/tech/misinformation/ taylor-swift-nude-deepfake-goes-viral-x-platform-rules-rcna135669
work page 2024
Show all 39 references
-
[9]
A voice deepfake was used to scam a ceo out of $243,000,
“A voice deepfake was used to scam a ceo out of $243,000,” https://www.forbes.com/sites/jessedamiani/2019/09/03/ a-voice-deepfake-was-used-to-scam-a-ceo-out-of-243000/, (Accessed: 16.10.2024)
2019
-
[10]
NSE CEO deepfake: NSE urges caution after fake videos of CEO Ashish Chauhan recommend- ing stocks go viral - The Economic Times,
“NSE CEO deepfake: NSE urges caution after fake videos of CEO Ashish Chauhan recommend- ing stocks go viral - The Economic Times,” https: //economictimes.indiatimes.com/markets/stocks/news/ beware-of-deepfake-of-ceo-recommending-stocks-says-nse/ articleshow/109189329.cms, (Acc...
-
[11]
A deepfake video showing volodymyr ze- lenskyy surrendering worries experts : Npr,
“A deepfake video showing volodymyr ze- lenskyy surrendering worries experts : Npr,” https://www.npr.org/2022/03/16/1087062648/ deepfake-video-zelenskyy-experts-war-manipulation-ukraine-russia, (Accessed: 16.10.2024)
2022
-
[12]
ASVspoof 2021: accelerating progress in spoofed and deepfake speech detection,
J. Yamagishi, X. Wang, M. Todisco, M. Sahidullah, J. Patino, A. Nautsch, X. Liu, K. A. Lee, T. Kinnunen, N. Evans, and H. Delgado, “ASVspoof 2021: accelerating progress in spoofed and deepfake speech detection,” inProc. of Automatic Speaker Verification and Spoofing Countermea...
2021
-
[13]
ASVspoof 2019: Future Horizons in Spoofed and Fake Au- dio Detection,
M. Todisco, X. Wang, V . Vestman, M. Sahidullah, H. Delgado, A. Nautsch, J. Yamagishi, N. Evans, T. H. Kinnunen, and K. A. Lee, “ASVspoof 2019: Future Horizons in Spoofed and Fake Au- dio Detection,” inProc. Interspeech 2019, 2019, pp. 1008–1012
2019
-
[14]
ASVspoof 5: Crowdsourced speech data, deepfakes, and adversarial attacks at scale,
X. Wang, H. Delgado, H. Tak, J.-w. Jung, H.-j. Shim, M. Todisco, I. Kukanov, X. Liu, M. Sahidullah, T. Kinnunen, N. Evans, K. A. Lee, and J. Yamagishi, “ASVspoof 5: Crowdsourced speech data, deepfakes, and adversarial attacks at scale,” inASVspoof Work- shop 2024, 2024
2024
-
[15]
AI V oice Cloning: Clone your V oice in Sec- onds,
Resemble AI, “AI V oice Cloning: Clone your V oice in Sec- onds,” https://www.resemble.ai/voice-cloning/, 2024, Accessed: 17.10.2024
2024
-
[16]
AI V oice Cloning,
Respeecher, “AI V oice Cloning,” https://www.respeecher.com/ ai-voice-cloning, 2024, Accessed: 17.10.2024
2024
-
[17]
Create a replica of your voice that sounds just like you,
Eleven Labs, “Create a replica of your voice that sounds just like you,” https://elevenlabs.io/voice-cloning, 2024, Accessed: 17.10.2024
2024
-
[18]
Researchers say the deepfake biden robocall was likely made with tools from ai startup elevenlabs — wired,
K. Knibbs, “Researchers say the deepfake biden robocall was likely made with tools from ai startup elevenlabs — wired,” 1 2024, [Online; accessed 2025-01-14]. [Online]. Available: https: //www.wired.com/story/biden-robocall-deepfake-elevenlabs/
2024
-
[19]
A comparative study on recent neu- ral spoofing countermeasures for synthetic speech detection,
X. Wang and J. Yamagishi, “A comparative study on recent neu- ral spoofing countermeasures for synthetic speech detection,” in Interspeech 2021, 2021, pp. 4259–4263
2021
-
[20]
End-to-End anti-spoofing with RawNet2,
H. Tak, J. Patino, M. Todisco, A. Nautsch, N. Evans, and A. Larcher, “End-to-End anti-spoofing with RawNet2,” inIEEE ICASSP 2021, 2021, pp. 6369–6373
2021
-
[21]
wav2vec 2.0: A framework for self-supervised learning of speech repre- sentations,
A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech repre- sentations,”Advances in neural information processing systems, vol. 33, pp. 12 449–12 460, 2020
2020
-
[22]
Aasist: Audio anti-spoofing using integrated spectro- temporal graph attention networks,
J. Jung, H. Heo, H. Tak, H. Shim, J. Chung, B. Lee, H. Yu, and N. Evans, “Aasist: Audio anti-spoofing using integrated spectro- temporal graph attention networks,” inProc. of ICASSP, 2022
2022
-
[23]
Temporal-channel modeling in multi-head self-attention for synthetic speech detection,
D.-T. Truong, R. Tao, T. Nguyen, H.-T. Luong, K. A. Lee, and E. S. Chng, “Temporal-channel modeling in multi-head self-attention for synthetic speech detection,”arXiv preprint arXiv:2406.17376, 2024
2024 arXiv
-
[24]
Does Audio Deepfake Detection Generalize?
N. M. M ¨uller, P. Czempin, F. Dieckmann, A. Froghyar, and K. B ¨ottinger, “Does Audio Deepfake Detection Generalize?” in Interspeech, 2022
2022
-
[25]
Towards generalisable and calibrated audio deepfake detection with self- supervised representations,
O. Pascu, A. Stan, D. Oneata, E. Oneata, and H. Cucu, “Towards generalisable and calibrated audio deepfake detection with self- supervised representations,” inProc. of Interspeech, 2024
2024
-
[26]
Speech is Silver, Silence is Golden: What do ASVspoof-trained Models Really Learn?
N. M ¨uller, F. Dieckmann, P. Czempin, R. Canals, K. B ¨ottinger, and J. Williams, “Speech is Silver, Silence is Golden: What do ASVspoof-trained Models Really Learn?” inAutomatic Speaker Verification and Spoofing Countermeasures Challenge, 2021
2021
-
[27]
Toward robust real-world audio deepfake detection: Closing the explain- ability gap,
G. Channing, J. Sock, R. Clark, P. Torr, and C. S. de Witt, “Toward robust real-world audio deepfake detection: Closing the explain- ability gap,”arXiv preprint arXiv:2410.07436, 2024
2024 arXiv
-
[28]
Adversarial attack and defense: A survey,
H. Liang, E. He, Y . Zhao, Z. Jia, and H. Li, “Adversarial attack and defense: A survey,”Electronics, vol. 11, no. 8, p. 1283, 2022
2022
-
[29]
Im- perceptible, robust, and targeted adversarial examples for auto- matic speech recognition,
Y . Qin, N. Carlini, G. Cottrell, I. Goodfellow, and C. Raffel, “Im- perceptible, robust, and targeted adversarial examples for auto- matic speech recognition,” inProc of ICML, 2019
2019
-
[30]
Synthesizing robust adversarial examples,
A. Athalye, L. Engstrom, A. Ilyas, and K. Kwok, “Synthesizing robust adversarial examples,” inProc. of ICML, 2018
2018
-
[31]
Room impulse responses help attackers to evade deep fake detection,
H.-T. Luong, D.-T. Truong, K. A. Lee, and E. S. Chng, “Room impulse responses help attackers to evade deep fake detection,” in 2024 IEEE SLT Workshop. IEEE, 2024, pp. 623–629
2024
-
[32]
Im- proved DeepFake Detection Using Whisper Features,
P. Kawa, M. Plata, M. Czuba, P. Szyma ´nski, and P. Syga, “Im- proved DeepFake Detection Using Whisper Features,” inProc. INTERSPEECH 2023, 2023, pp. 4009–4013
2023
-
[33]
Raw Differentiable Architecture Search for Speech Deepfake and Spoofing Detec- tion,
W. Ge, J. Patino, M. Todisco, and N. Evans, “Raw Differentiable Architecture Search for Speech Deepfake and Spoofing Detec- tion,” inProc. of the Automatic Speaker Verification and Spoofing Countermeasures Challenge, 2021, pp. 22–28
2021
-
[34]
End-to-end spectro-temporal graph attention networks for speaker verification anti-spoofing and speech deepfake detec- tion,
H. Tak, J. weon Jung, J. Patino, M. Kamble, M. Todisco, and N. Evans, “End-to-end spectro-temporal graph attention networks for speaker verification anti-spoofing and speech deepfake detec- tion,” inProc. of the Automatic Speaker Verification and Spoofing Countermeasures Chall...
2021
-
[35]
Automatic speaker verification spoofing and deepfake detection using wav2vec 2.0 and data augmentation,
H. Tak, M. Todisco, X. Wang, J.-w. Jung, J. Yamagishi, and N. Evans, “Automatic speaker verification spoofing and deepfake detection using wav2vec 2.0 and data augmentation,” inProc. of Speaker and Language Recognition Workshop, 2022
2022
-
[36]
Datasets – APTLY and LaSSoftE,
“Datasets – APTLY and LaSSoftE,” [Accessed 24.01.2025]. [Online]. Available: https://bil.eecs.yorku.ca/datasets/
2025
-
[37]
An Open Dataset of Synthetic Speech,
A. Yaroshchuk, C. Papastergiopoulos, L. Cuccovillo, P. Aichroth, K. V otis, and D. Tzovaras, “An Open Dataset of Synthetic Speech,” inIEEE International Workshop on Information Foren- sics and Security (WIFS). IEEE, 2023, pp. 1–6
2023
-
[38]
Shortcut learning in deep neural networks,
R. Geirhos, J.-H. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann, “Shortcut learning in deep neural networks,”Nature Machine Intelligence, vol. 2, no. 11, 2020
2020
-
[39]
Per- ceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs,
A. W. Rix, J. G. Beerends, M. P. Hollier, and A. P. Hekstra, “Per- ceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs,” in Proc. of ICASSP, 2001
2001
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.