Pith. sign in

REVIEW 3 major objections 5 minor 26 references

Robust Localization of Partially Fake Speech: Metrics and Out-of-Domain Evaluation

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read EER-based evaluation makes partial-fake speech localization look far more deployable than it is, and the paper argues for threshold-dependent metrics plus out-of-domain testing.

desk verdict A mostly solid evaluation-analysis paper: the out-of-domain degradation numbers are real and worth publishing, but the overfitting narrative leans on an uncontrolled reproduction comparison. read the letter →

arxiv 2507.03468 v3 pith:5QPF2IQE submitted 2025-07-04 cs.SD eess.AS

Hieu-Thi Luong , Inbal Rimon , Haim Permuter , Kong Aik Lee , Eng Siong Chng This is my paper
classification cs.SDeess.AS
keywords partialfakespeechaudiodeepfakelocalizationout-of-domaingeneralizationequalerrorrateanomalydetectionself-supervisedlearningwaveformaugmentationthreshold-dependentmetrics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that current evaluations of partial-fake speech localization overstate real-world reliability because they rely on Equal Error Rate (EER), a threshold-independent metric. The authors show that an open-source state-of-the-art model, CFPRF, reports a 7.61% EER on the in-domain PartialSpoof test set but degrades to 43.25% EER on the out-of-domain LlamaPartialSpoof set and 27.59% on the Half-Truth set, and that the score threshold where EER is obtained shifts from 0.088 in-domain to 0.883 out-of-domain. They propose reframing localization as sequential anomaly detection and reporting threshold-dependent metrics such as accuracy, precision, recall, and F1. They also find that adding bona fide or fully synthetic audio to training data often hurts performance, while adding partially fake utterances helps, and that waveform augmentation gives no guaranteed out-of-domain gain.

What carries the argument

The central mechanism is the Equal Error Rate (EER) itself: a threshold-independent summary that hides the actual operating point. Each reported EER is achieved at a different decision threshold, and the paper's evidence is the gap between the in-domain EER threshold (0.088) and the out-of-domain EER threshold (0.883) for the released CFPRF; that gap is what turns a nominally good EER into poor single-threshold deployment. The paper also introduces the reframing of the localization output as a sequential anomaly score, which justifies using accuracy, precision, recall, and F1 at a fixed threshold, and uses segment-resolution sweeps and score-distribution plots to expose where models fail.

What would settle it

Retrain CFPRF using the released checkpoint's exact training schedule, hyperparameters, and data pipeline, then measure its EER threshold on PartialSpoof and LlamaPartialSpoof; if the clone shows a small threshold gap (within about 0.1) and much better out-of-domain EER, then the released model's large 0.088-to-0.883 threshold shift is not an inherent property of the architecture and training setup but something specific to that checkpoint.

Watch

Extended reading notes

Core claim

The paper's central empirical claim is that self-supervised-learning-based partial-fake localization models do not generalize across domains, and that this failure is masked by EER-based reporting. On the in-domain PartialSpoof evaluation set, the released CFPRF achieves a 20-ms EER of 7.61%, but the same model reaches only 43.25% EER on LlamaPartialSpoof and 27.59% on Half-Truth; the authors' reproduced version scores worse in-domain (9.84%) yet better out-of-domain (41.72% and 14.98%), which they read as evidence that the released model was over-optimized for in-domain EER. The decisive observation is threshold drift: each EER corresponds to a different decision threshold, and CFPRF's threshold moves from 0.088 on in-domain data to 0.883 on LlamaPartialSpoof, so a single threshold deployed in practice cannot reproduce benchmark performance. The paper further reports that these models misclassify both unseen synthetic audio and unfamiliar bona fide audio, and that adding partial-fake utterances to training data improves out-of-domain localization while adding bona fide or fully fake utterances often degrades it.

Load-bearing premise

The overfitting explanation relies on the authors' re-trained CFPRF being a faithful reproduction of the released checkpoint; if the training recipes really differed, the gap could be a training artifact rather than proof of over-optimization for in-domain EER.

Editorial extensions

If this is right

  • A benchmark EER below 8% on PartialSpoof should not be read as deployment readiness, because the same system can exceed 40% EER on related out-of-domain data.
  • Comparing models by EER alone can misorder them: the released CFPRF beats the reproduction in-domain but loses to it on both out-of-domain sets.
  • Deploying a single fixed threshold requires reporting performance at that threshold; otherwise the operating point behind the headline number is unknown.
  • Training-data composition matters more than volume: adding partial-fake utterances improves out-of-domain localization, while adding bona fide or fully fake utterances often hurts.
  • Waveform augmentations that help in-domain detection do not reliably transfer to out-of-domain localization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test calibration or threshold adaptation, but the 0.088-to-0.883 threshold shift suggests a concrete follow-up: apply a post-hoc calibration procedure on a small out-of-domain probe set and measure whether deployment accuracy recovers.
  • If the anomaly-detection framing is adopted, an unsupervised per-utterance score-variance or threshold-drift monitor could flag when a deployed system is operating out of domain.
  • The finding that partial-fake samples are the useful training signal implies that data collection should prioritize constructing locally manipulated utterances rather than simply accumulating large volumes of standard spoofed audio.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper empirically examines out-of-domain generalization of two SSL-based partial fake speech localization models, MRM and CFPRF, using the in-domain PartialSpoof set and the out-of-domain LlamaPartialSpoof and Half-Truth sets. It argues that EER, being threshold-independent, obscures deployment behavior and proposes evaluating localization as sequential anomaly detection with accuracy, precision, recall, and F1. The central experimental findings are large EER degradations on out-of-domain data (e.g., CFPRF at 20 ms: 7.61% EER in-domain vs. 43.25% on LlamaPartialSpoof and 27.59% on Half-Truth), a reproduced reCFPRF with higher in-domain EER but lower out-of-domain EER, inconsistent effects of waveform augmentation, and a fine-tuning experiment showing that adding bona fide or fully fake utterances often harms performance while adding partially fake utterances helps.

Significance. If the results hold, the paper makes a useful contribution by quantifying cross-domain degradation of partial fake speech localization systems and by providing an out-of-domain benchmark, LlamaPartialSpoof, together with an open-sourced metrics library. The independent Half-Truth set corroborates the generalization failure, and the numerical claims in the abstract are consistent with the reported tables. The main weakness is that the causal 'over-optimizing for in-domain EER' narrative rests on an uncontrolled comparison against a single reproduced checkpoint, so the headline interpretation is stronger than the evidence currently supports.

major comments (3)
  1. [Section III.A and Abstract] The abstract and Section III.A attribute the CFPRF/reCFPRF trade-off to the released CFPRF being overfitted to its training data ("the original model may have been overfitted to its training data"). The only support is a single reCFPRF run, and the manuscript does not report the training recipe: number of epochs, optimizer and learning-rate schedule, batch size, random seed, augmentation, or model-selection criterion. Without evidence of training comparability, the in-domain/OOD gap between CFPRF and reCFPRF could reflect training configuration rather than overfitting. This matters because the abstract's "risks of over-optimizing for in-domain EER" is one of the paper's headline claims. Please provide the full training recipe and ideally multiple seeds, or explicitly soften the claim to "models trained in this study".
  2. [Sections III.B and III.C] The central methodological claim that EER obscures generalization and that models can show strong EER scores while behaving poorly under a single fixed threshold is asserted rather than directly demonstrated. Table V reports accuracy, precision, recall, and F1 at each dataset's own EER threshold, and Table VI uses a 0.5 threshold only in the fine-tuning experiment. To make the deployment argument quantitative, report the performance produced by applying the in-domain EER threshold (e.g., CFPRF's 0.088 on PartialSpoof) to LlamaPartialSpoof and Half-Truth, alongside the dataset-specific EER thresholds.
  3. [Section III.D] The fine-tuning experiment's conclusion that adding bona fide or fully synthetic utterances "often degrades" performance is presented without fine-tuning details such as learning rate, number of epochs, batch size, data balancing, and early stopping. Since the authors attribute the benefit specifically to partially fake utterances, the missing hyperparameters make it difficult for readers to separate data-composition effects from optimization effects. Please report these details or provide a simple sensitivity check.
minor comments (5)
  1. [Table IV] The caption says "Table III reports the Precision, Recall, and F1" but the table is actually Table IV.
  2. [Section II.C] The phrase "default setting of their respectable setups" should read "respective setups."
  3. [Section III.B] The statement that all models output scores "approximately between 0 and 1 (uncapped, as logits)" is internally inconsistent; Table V lists EER thresholds above 1 (e.g., 1.0420). Please reconcile the wording with the fact that logits are not bounded to [0,1].
  4. [Table VI] The legend "Gray cells indicate results that is better than the original checkpoints" is not visible in the text; please use an explicit marker, such as bold face or a footnote symbol, that survives the formatting.
  5. [Figure 1] In the first panel the y-axis label "0.2" appears detached from the axis; please check the axis labeling and tick placement.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's generalization findings rest on fixed model checkpoints and are independently corroborated by an external dataset.

full rationale

This is an empirical evaluation paper rather than a derivation, and I found no step in which a prediction reduces to its input by construction. The core out-of-domain results (CFPRF EER rising from 7.61% on PartialSpoof to 43.25% on LlamaPartialSpoof and 27.59% on Half-Truth) are computed from released model checkpoints with no fitted parameter renamed as a prediction. The threshold-shift evidence (EER threshold 0.088 in-domain vs. 0.883 on LlamaPartialSpoof) is an observed property of the released model, not an artifact of the paper's own fitting procedure. The reCFPRF comparison is a reproducibility check whose uncontrolled training recipe is a validity threat to the overfitting narrative, but it is not circular because the two runs are independent empirical objects and the raw OOD degradation stands without it. The self-cited LlamaPartialSpoof dataset is a publicly released resource rather than an unverified theorem, and its use as an OOD benchmark is corroborated by Half-Truth, which was created by a different research group; therefore the self-citation is not load-bearing in a circular way. The metric re-framing (EER to threshold-dependent accuracy/precision/recall/F1) is a proposed evaluation convention, not a derivation that assumes its conclusion.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claims rest on standard evaluation assumptions: the ground-truth segment labels in the three public datasets are correct and comparably defined, and the 20-speaker LlamaPartialSpoof subset is a representative emerging-data scenario. One ad hoc assumption enables the overfitting narrative: the authors' reCFPRF is treated as a comparably trained re-implementation of the released CFPRF. There are no fitted free parameters; the operating points used (threshold 0.5, EER threshold, recall targets 90% and 95%) are reporting choices, not parameters that the claims depend on. No new entities are introduced.

assumptions (3)
  • domain assumption Frame-level ground-truth labels in PartialSpoof, LlamaPartialSpoof, and Half-Truth are reliable and comparably defined across datasets.
    All EER, accuracy, precision, recall, and F1 computations in Tables III-VI inherit the correctness and comparability of the segment annotations in the three public datasets. Section II.A defines the labeling rule (any fake content makes a segment fake) and assumes it applies across datasets.
  • ad hoc to paper The authors' retrained reCFPRF is a faithful, comparably trained re-implementation of CFPRF, so performance differences isolate overfitting.
    Section III.A uses the CFPRF-versus-reCFPRF gap to suggest the released model 'may have been overfitted to its training data.' The training recipe for reCFPRF is only described as the upstream 'default settings,' so comparability is assumed rather than demonstrated.
  • domain assumption The approximately 20-speaker LlamaPartialSpoof train subset is a representative emerging-data scenario for the tuning study.
    Section III.D and Table I: the tuning study's conclusions about which data types help or hurt depend on this subset being representative of how new data would arrive in deployment.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Robust Localization of Partially Fake Speech: Metrics and Out-of-Domain Evaluation." pith.science (2026). https://pith.science/paper/5QPF2IQE

@misc{pith2026250703468,
  author       = {Pith},
  title        = {Pith review of: Robust Localization of Partially Fake Speech: Metrics and Out-of-Domain Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5QPF2IQE}},
  note         = {Machine review of arXiv:2507.03468}
}
read the original abstract

Partial audio deepfake localization poses unique challenges and remain underexplored compared to full-utterance spoofing detection. While recent methods report strong in-domain performance, their real-world utility remains unclear. In this analysis, we critically examine the limitations of current evaluation practices, particularly the widespread use of Equal Error Rate (EER), which often obscures generalization and deployment readiness. We propose reframing the localization task as a sequential anomaly detection problem and advocate for the use of threshold-dependent metrics such as accuracy, precision, recall, and F1-score, which better reflect real-world behavior. Specifically, we analyze the performance of the open-source Coarse-to-Fine Proposal Refinement Framework (CFPRF), which achieves a 20-ms EER of 7.61% on the in-domain PartialSpoof evaluation set, but 43.25% and 27.59% on the LlamaPartialSpoof and Half-Truth out-of-domain test sets. Interestingly, our reproduced version of the same model performs worse on in-domain data (9.84%) but better on the out-of-domain sets (41.72% and 14.98%, respectively). This highlights the risks of over-optimizing for in-domain EER, which can lead to models that perform poorly in real-world scenarios. It also suggests that while deep learning models can be effective on in-domain data, they generalize poorly to out-of-domain scenarios, failing to detect novel synthetic samples and misclassifying unfamiliar bona fide audio. Finally, we observe that adding more bona fide or fully synthetic utterances to the training data often degrades performance, whereas adding partially fake utterances improves it.

Figures

Figures reproduced from arXiv: 2507.03468 by the authors.

Figure 1
Figure 1. Distributions of 20-ms segment score produced by the detections models. Blues are bona fide segments while oranges are [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

26 extracted references · 16 canonical work pages

  1. [1]

    Asvspoof: the automatic speaker verification spoofing and countermeasures challenge,

    Z. Wu, J. Yamagishi, T. Kinnunen, C. Hanilc ¸i, M. Sahidullah, A. Sizov, N. Evans, M. Todisco, and H. Delgado, “Asvspoof: the automatic speaker verification spoofing and countermeasures challenge,” IEEE Journal of Selected Topics in Signal Processing, vol. 11, no. 4, pp. 588–604, 2017

    work page 2017

  2. [2]

    Mlaad: The multi-language audio anti-spoofing dataset,

    N. M. M ¨uller, P. Kawa, W. H. Choong, E. Casanova, E. G¨olge, T. M¨uller, P. Syga, P. Sperl, and K. B ¨ottinger, “Mlaad: The multi-language audio anti-spoofing dataset,” in Proc. IJCNN. IEEE, 2024, pp. 1–7

    work page 2024

  3. [3]

    Asvspoof 5: Design, collection and validation of resources for spoofing, deepfake, and adversarial attack detection using crowdsourced speech,

    X. Wang, H. Delgado, H. Tak, J.-w. Jung, H.-j. Shim, M. Todisco, I. Kukanov, X. Liu, M. Sahidullah, T. Kinnunen et al. , “Asvspoof 5: Design, collection and validation of resources for spoofing, deepfake, and adversarial attack detection using crowdsourced speech,” Computer Speech & Language , p. 101825, 2025

    work page 2025

  4. [4]

    The partialspoof database and countermeasures for the detection of short fake speech segments embedded in an utterance,

    L. Zhang, X. Wang, E. Cooper, N. Evans, and J. Yamagishi, “The partialspoof database and countermeasures for the detection of short fake speech segments embedded in an utterance,” IEEE/ACM Trans. Audio, Speech, Language Process., vol. 31, pp. 813–825, 2023

    work page 2023

  5. [5]

    Add 2023: the second audio deepfake detection challenge,

    J. Yi, J. Tao, R. Fu, X. Yan, C. Wang, T. Wang, C. Y . Zhang, X. Zhang, Y . Zhao, Y . Renet al., “Add 2023: the second audio deepfake detection challenge,” arXiv preprint arXiv:2305.13774 , 2023

    arXiv 2023

  6. [6]

    Llamapartial- spoof: An llm-driven fake speech dataset simulating disinformation generation,

    H.-T. Luong, H. Li, L. Zhang, K. A. Lee, and E. S. Chng, “Llamapartial- spoof: An llm-driven fake speech dataset simulating disinformation generation,” in Proc. ICASSP. IEEE, 2025, pp. 1–5

    work page 2025

  7. [7]

    Av-deepfake1m: A large-scale llm-driven audio-visual deepfake dataset,

    Z. Cai, S. Ghosh, A. P. Adatia, M. Hayat, A. Dhall, T. Gedeon, and K. Stefanov, “Av-deepfake1m: A large-scale llm-driven audio-visual deepfake dataset,” in Proceedings of the 32nd ACM International Con- ference on Multimedia , 2024, pp. 7414–7423

    work page 2024

  8. [8]

    Coarse-to-fine proposal refinement framework for audio temporal forgery detection and localization,

    J. Wu, W. Lu, X. Luo, R. Yang, Q. Wang, and X. Cao, “Coarse-to-fine proposal refinement framework for audio temporal forgery detection and localization,” in Proceedings of the 32nd ACM International Conference on Multimedia, 2024, pp. 7395–7403

    2024

Show all 26 references
  1. [9]

    Integrating frame-level boundary detection and deepfake detection for locating manipulated regions in partially spoofed audio forgery attacks,

    Z. Cai and M. Li, “Integrating frame-level boundary detection and deepfake detection for locating manipulated regions in partially spoofed audio forgery attacks,” Computer Speech & Language , vol. 85, p. 101597, 2024

    2024

  2. [10]

    Does audio deepfake detection generalize?

    N. M ¨uller, P. Czempin, F. Diekmann, A. Froghyar, and K. B ¨ottinger, “Does audio deepfake detection generalize?” in Proc. Interspeech, 2022, pp. 2783–2787

    2022

  3. [11]

    A robust audio deepfake detection system via multi-view feature,

    Y . Yang, H. Qin, H. Zhou, C. Wang, T. Guo, K. Han, and Y . Wang, “A robust audio deepfake detection system via multi-view feature,” in Proc. ICASSP. IEEE, 2024, pp. 13 131–13 135

    2024

  4. [12]

    Slim: Style-linguistics mismatch model for generalized audio deepfake detection,

    Y . Zhu, S. Koppisetti, T. Tran, and G. Bharaj, “Slim: Style-linguistics mismatch model for generalized audio deepfake detection,” Advances in Neural Information Processing Systems , vol. 37, pp. 67 901–67 928, 2024

    2024

  5. [13]

    Trident of poseidon: A generalized approach for detecting deepfake voices,

    T.-P. Doan, H. Dinh-Xuan, T. Ryu, I. Kim, W. Lee, K. Hong, and S. Jung, “Trident of poseidon: A generalized approach for detecting deepfake voices,” in Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security , 2024, pp. 2222–2235

    2024

  6. [14]

    Easy, interpretable, effective: opensmile for voice deepfake detection,

    O. Pascu, D. Oneata, H. Cucu, and N. M. M ¨uller, “Easy, interpretable, effective: opensmile for voice deepfake detection,” arXiv preprint arXiv:2408.15775, 2024

    2024 arXiv

  7. [15]

    Anomaly detection and localization for speech deepfakes via feature pyramid matching,

    E. Coletta, D. Salvi, V . Negroni, D. U. Leonzio, and P. Bestagini, “Anomaly detection and localization for speech deepfakes via feature pyramid matching,” arXiv preprint arXiv:2503.18032 , 2025

    2025 arXiv

  8. [16]

    Half- Truth: A Partially Fake Audio Detection Dataset,

    J. Yi, Y . Bai, J. Tao, H. Ma, Z. Tian, C. Wang, T. Wang, and R. Fu, “Half- Truth: A Partially Fake Audio Detection Dataset,” in Proc. Interspeech, 2021, pp. 1654–1658

    2021

  9. [17]

    Tandem assessment of spoofing countermeasures and automatic speaker veri- fication: Fundamentals,

    T. Kinnunen, H. Delgado, N. Evans, K. A. Lee, V . Vestman, A. Nautsch, M. Todisco, X. Wang, M. Sahidullah, J. Yamagishi et al. , “Tandem assessment of spoofing countermeasures and automatic speaker veri- fication: Fundamentals,” IEEE/ACM Trans. Audio, Speech, Language Process....

    2020

  10. [18]

    Range- based equal error rate for spoof localization,

    L. Zhang, X. Wang, E. Cooper, N. Evans, and J. Yamagishi, “Range- based equal error rate for spoof localization,” in Proc. Interspeech, 2023, pp. 3212–3216

    2023

  11. [19]

    Asvspoof 5: Crowdsourced speech data, deepfakes, and adversarial attacks at scale,

    X. Wang, H. Delgado, H. Tak, J.-w. Jung, H.-j. Shim, M. Todisco, I. Kukanov, X. Liu, M. Sahidullah, T. Kinnunen et al. , “Asvspoof 5: Crowdsourced speech data, deepfakes, and adversarial attacks at scale,” arXiv preprint arXiv:2408.08739 , 2024

    2024 arXiv

  12. [20]

    Rawboost: A raw data boosting and augmentation method applied to automatic speaker verification anti-spoofing,

    H. Tak, M. Kamble, J. Patino, M. Todisco, and N. Evans, “Rawboost: A raw data boosting and augmentation method applied to automatic speaker verification anti-spoofing,” in Proc. ICASSP. IEEE, 2022, pp. 6382–6386

    2022

  13. [21]

    Unmasking deepfakes: Leveraging augmentations and features variability for deepfake speech detection,

    I. Rimon, O. Gal, and H. Permuter, “Unmasking deepfakes: Leveraging augmentations and features variability for deepfake speech detection,” arXiv preprint arXiv:2501.05545 , 2025

    2025

  14. [22]

    Cross- domain audio deepfake detection: Dataset and analysis,

    Y . Li, M. Zhang, M. Ren, X. Qiao, M. Ma, D. Wei, and H. Yang, “Cross- domain audio deepfake detection: Dataset and analysis,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024, pp. 4977–4983

    2024

  15. [23]

    Con- tinual unsupervised domain adaptation for audio deepfake detection,

    X. Chen, W. Lu, R. Zhang, J. Xu, X. Lu, L. Zhang, and J. Wei, “Con- tinual unsupervised domain adaptation for audio deepfake detection,” in Proc. ICASSP. IEEE, 2025, pp. 1–5

    2025

  16. [24]

    Leveraging distance information for generalized spoofing speech detection,

    J. Lu, Y . Zhang, Z. Li, Z. Shang, W. Wang, and P. Zhang, “Leveraging distance information for generalized spoofing speech detection,” Com- puter Speech & Language , p. 101804, 2025

    2025

  17. [25]

    Audiobox: Unified audio gener- ation with natural language prompts,

    A. Vyas, B. Shi, M. Le, A. Tjandra, Y .-C. Wu, B. Guo, J. Zhang, X. Zhang, R. Adkins, W. Ngan et al., “Audiobox: Unified audio gener- ation with natural language prompts,” arXiv preprint arXiv:2312.15821, 2023

    2023 arXiv

  18. [26]

    Replay attacks against audio deepfake detection,

    N. M ¨uller, P. Kawa, W.-H. Choong, A. Stan, A. T. Bukkapatnam, K. Pizzi, A. Wagner, and P. Sperl, “Replay attacks against audio deepfake detection,” arXiv preprint arXiv:2505.14862 , 2025

    2025 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.