REVIEW 5 major objections 6 minor 36 references
Reject Threshold Adaptation for Open-Set Model Attribution of Deepfake Audio
T0 review · 5 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read ReTA replaces hand-tuned rejection thresholds with per-class adaptive thresholds derived from reconstruction error distributions, and reports the best open-set attribution accuracy on the SFR deepfake audio benchmark.
desk verdict A competent transplant of an existing SAR OSR method to deepfake audio, with a plausible but unvalidated proxy assumption and a missing threshold objective; worth a serious referee but needs revision. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the pair of reconstruction error distributions built by the reconstruction error learning module. For each training sample the decoder receives the fingerprint representation multiplied either by the conditional latent feature of its true class (matching) or by a randomly drawn latent feature from another known class (non-matching), producing two error sets per class; the adaptive threshold module then fits a Gaussian kernel density estimate to each set and chooses the class-specific reject threshold by probability minimization. This mechanism converts the open-set question into a per-class anomaly detection problem in reconstruction-error space.
What would settle it
Take a held-out deepfake generator that was never used in training, feed its audio through ReTA, and measure the reconstruction error for its predicted label; if that error distribution overlaps substantially with the matching-error distributions of the known classes, the per-class thresholds will be miscalibrated and the open-set F1 will drop close to the fixed-threshold baselines. Concretely, training only on five of the seven vendors and testing on the other two as unknowns, then varying which vendors are held out, would reveal whether the simulated non-matching errors are a faithful proxy.
Extended reading notes
Core claim
The central claim is that a closed-set deepfake attribution classifier can be turned into an open-set one by calibrating a separate rejection threshold for each known class, where those thresholds are computed from the distributions of matching and non-matching reconstruction errors rather than chosen by hand. The paper shows that when a sample's fingerprint representation is decoded together with the correct class label the reconstruction error is small, while decoding with a randomly chosen other class label produces a larger error, and it fits a Gaussian kernel density estimate to each set of errors to locate the threshold that minimizes the probability of misclassification. Experiments on the SFR dataset, with Tencent and iFLYTEK as unknown classes, show average F1 of 77.35% and average total accuracy of 70.09%, beating the OpenMax and CROSR baselines while largely retaining closed-set ID accuracy.
Load-bearing premise
The method assumes that the reconstruction errors obtained by pairing a known class fingerprint with a different known class label behave like the reconstruction errors of genuinely unknown classes, so that calibrating thresholds on those simulated mismatches transfers to real unknown classes at test time.
Editorial extensions
If this is right
- ReTA achieves the best average F1 and total accuracy on SFR among the compared methods, including on the compressed set where unknown detection is harder.
- The per-class thresholds are derived without manual validation-set tuning, so deployment to a new open-set scenario does not require re-setting a global threshold.
- The framework turns an existing closed-set fingerprint classifier into an open-set one by adding the reconstruction decoder and threshold module, so it can be attached to other ResNet-based extractors.
- The design predicts that matching reconstruction errors cluster tightly while non-matching errors spread out, which is what the threshold calculation exploits.
Reading between the lines
- If the proxy assumption holds across domains, ReTA could serve as a general open-set recognition wrapper for any fingerprint-based attribution system, replacing hand-tuned confidence thresholds in production.
- A natural testable extension is to model the unknown-class error distribution with a mixture or heavy-tailed density instead of a Gaussian, since real unknown generators may produce multi-modal errors; the probability minimization criterion could be sensitive to tail overlap.
- The same reconstruction-error geometry could be used to detect out-of-distribution audio beyond attribution, for example by treating real human speech as an additional known class and rejecting everything else.
- The reported gap between clean and compressed performance suggests that compression changes the error distributions substantially; an extension could condition thresholds on audio quality or estimate distribution shift during inference.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes ReTA, a framework for open-set model attribution of deepfake audio. It consists of a system fingerprint recognition module, a reconstruction error learning module that produces matching and non-matching reconstruction samples, and an adaptive reject threshold module that uses kernel density estimation and a "probability minimization criterion" to set per-class rejection thresholds. The method is evaluated on the SFR dataset, where ReTA is compared with Softmax, OpenMax, and CROSR, reporting the highest average F1 score and total accuracy.
Significance. If the method is fully specified and validated, it addresses a genuine practical problem: manual rejection thresholds in open-set audio attribution are brittle and do not transfer across data distributions. The paper has a fair experimental setup in that thresholds are fit to training data and unknown vendors (Tencent, iFLYTEK) are held out, so the central result is not circular. The reported gains, however, are modest (average F1 77.35 vs 75.66 for CROSR), and the current manuscript does not specify the probability minimization criterion, lacks error bars or significance tests, and does not validate the key proxy assumption that known-class mismatched reconstruction errors model unknown-class errors. The contribution is promising but not yet established.
major comments (5)
- [Section 2.1.3] The "probability minimization criterion" is the core of the adaptive threshold mechanism but is never defined mathematically or algorithmically. The text only states that "optimal reject thresholds are determined using the probability minimization criterion" and provides a flowchart in Figure 4. Please specify the objective function in a formula, including how the two KDEs are combined, what is minimized (e.g., a weighted sum of error probabilities), and any hyperparameters such as false-positive/false-negative costs or grid resolution.
- [Section 2.2 vs Section 3.1] The training procedure is internally inconsistent: Section 2.2 states that the modules "use stochastic gradient descent (SGD) to iteratively update parameters," while Section 3.1 states "The training was conducted with the Adam optimizer." These are different optimization algorithms. Please clarify which optimizer is actually used and provide the exact training configuration.
- [Table 2 and Section 3.3] The performance comparison reports only point estimates. The average F1 improvement over CROSR is 1.69 points (77.35 vs 75.66), and the clean-set F1 improvement is only 0.41 points (89.35 vs 88.94). Without repeated runs, standard deviations, or a paired significance test, the claim that ReTA "overall achieves the best" is not statistically supported. Please report mean±std over at least five random seeds and apply an appropriate significance test (e.g., paired bootstrap).
- [Section 2.1.2 and Section 2.1.3] The central assumption of the method is that non-matching reconstruction errors, obtained by pairing a known-class fingerprint with a randomly selected other known-class label, are a faithful proxy for the reconstruction errors of genuinely unknown classes such as Tencent and iFLYTEK. This assumption is never validated. Please add a leave-one-known-out experiment: hold out one of the five training vendors as an unknown class, calibrate thresholds using the remaining four, and compare the simulated non-matching error distribution with the held-out vendor's actual error distribution. Report a quantitative separation measure (e.g., AUROC, KL divergence, or overlap coefficient) to demonstrate that the proxy holds.
- [Section 3.3] The baseline threshold selection is unspecified. The paper states that Softmax, OpenMax, and CROSR use manually set thresholds and that all methods use the same LFCC features and ResNet backbone, but it does not describe how each baseline's threshold was chosen (e.g., grid search on the development set, a fixed percentile, or a single scalar). Without this information, the comparison may be biased in favor of ReTA if the baselines' thresholds are poorly tuned. Please document the threshold selection protocol for each baseline and provide threshold sensitivity curves analogous to Figure 2.
minor comments (6)
- [Section 3 heading] The section heading "Experiences" should be "Experiments."
- [Conclusion] The word "atribution" is a typo and should be "attribution."
- [Figure 3] The label "X mm" in Figure 3 appears to be a typo and should likely be "X^nm" (non-matching), and the caption uses "Mismatching Features" while the text uses "non-matching"; please make the terminology consistent.
- [Equation (1)] The notation in Eq. (1) is ambiguous: X'_i is described as "a randomly sampled sample from other target classes," but it is not clear whether the non-matching reconstruction target is the original input X_i reconstructed under a wrong class condition, or a different sample X'_i reconstructed under its own or another wrong class condition. Please clarify the exact pairing and loss computation.
- [Section 2.1.1] The phrase "constraining the activation of the penultimate layer [26]" is vague; please specify the constraint (e.g., ReAct or a bounded activation) and how it is applied to the ResNet-18 encoder.
- [Introduction and References] Reference [16] describes a similar "reject threshold adaptation" method for SAR target recognition, but the paper does not discuss the differences between ReTA and [16] or what is newly contributed beyond adapting that approach to audio model attribution. Please add a brief comparison.
Circularity Check
No significant circularity: ReTA's thresholds are fit to training reconstruction errors and evaluated on held-out unknown vendors, with self-citations used only as independent benchmark and motivation.
full rationale
ReTA's rejection thresholds are computed entirely from training-time reconstruction error distributions: matching errors (same-class reconstruction) and non-matching errors (fingerprint paired with a randomly chosen other known class label), via KDE and a probability-minimization criterion (Sections 2.1.2 and 2.1.3). The test protocol holds out Tencent and iFLYTEK as unknown classes (Table 1) and never uses their labels or errors to set thresholds. Thus the headline F1 gains (77.35 average vs. CROSR's 75.66 in Table 2) are not forced by construction: thresholds are fit to known-class data and then evaluated on genuinely unseen vendors. The self-citations are to the SFR dataset [2] and to a vocoder-fingerprint study [23] used as motivation (Figure 2); these are independent artifacts or benchmarks rather than fitted parameters or unverified uniqueness theorems. The weakest point is the empirical proxy assumption that non-matching known-class reconstruction errors approximate unknown-class errors; if false, thresholds would be miscalibrated. But that is an empirical validity concern, not a circular derivation, because the unknown-class labels and errors are not inputs to threshold fitting. No circular step requiring Eq. X = Eq. Y by construction, or a renamed fitted parameter presented as a prediction, was found in the paper.
Assumptions & free parameters
free parameters (3)
- alpha (loss weighting) =
not reported
- KDE bandwidth =
not reported
- probability minimization criterion parameters =
not reported
assumptions (4)
- domain assumption Non-matching reconstruction errors from randomly selected known classes approximate unknown-class reconstruction errors
- domain assumption Gaussian kernel density estimation faithfully models reconstruction error distributions with the chosen bandwidth
- domain assumption LFCC features plus ResNet-18 capture distinguishing system fingerprints
- domain assumption Per-class thresholds computed on training data transfer to unseen data distributions
Cite this review
Pith. "Pith review of Reject Threshold Adaptation for Open-Set Model Attribution of Deepfake Audio." pith.science (2026). https://pith.science/paper/7O6OZBSL
@misc{pith2026241201425,
author = {Pith},
title = {Pith review of: Reject Threshold Adaptation for Open-Set Model Attribution of Deepfake Audio},
year = {2026},
howpublished = {\url{https://pith.science/paper/7O6OZBSL}},
note = {Machine review of arXiv:2412.01425}
}
read the original abstract
Open environment oriented open set model attribution of deepfake audio is an emerging research topic, aiming to identify the generation models of deepfake audio. Most previous work requires manually setting a rejection threshold for unknown classes to compare with predicted probabilities. However, models often overfit training instances and generate overly confident predictions. Moreover, thresholds that effectively distinguish unknown categories in the current dataset may not be suitable for identifying known and unknown categories in another data distribution. To address the issues, we propose a novel framework for open set model attribution of deepfake audio with rejection threshold adaptation (ReTA). Specifically, the reconstruction error learning module trains by combining the representation of system fingerprints with labels corresponding to either the target class or a randomly chosen other class label. This process generates matching and non-matching reconstructed samples, establishing the reconstruction error distributions for each class and laying the foundation for the reject threshold calculation module. The reject threshold calculation module utilizes gaussian probability estimation to fit the distributions of matching and non-matching reconstruction errors. It then computes adaptive reject thresholds for all classes through probability minimization criteria. The experimental results demonstrate the effectiveness of ReTA in improving the open set model attributes of deepfake audio.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
Introduction The Text-to-Speech (TTS) generation Application Program- ming Interface (API) 1 2 3 provides a method for converting text into natural speech, producing audio that is almost indis- tinguishable from real human speech [1], [2], [3]. However, the misuse of such generation tools or models can lead to the spread of misinformation and violations o...
work page 2023
-
[2]
Proposed Method The overall network architecture of ReTA, as shown in Figure 3, consists of three modules: A) System Fingerprint Recogni- tion module; B) Reconstruction Error Learning module; and C) Adaptive Reject Threshold module. The System Finger- print Recognition module includes 1) fingerprint extraction sub- network; and 2) classification sub-netwo...
-
[3]
Experimental Settings ReTA was validated on the SFR dataset
Experiences 3.1. Experimental Settings ReTA was validated on the SFR dataset. [2] constructed the first audio dataset for system fingerprint recognition. The au- dio in the SFR dataset was generated by models from main- stream TTS systems of Chinese vendors, including both clear and compressed sets. The generated speech comes from seven mainstream open so...
-
[4]
To address this challenge, we propose a novel adaptive threshold framework named ReTA
Conclusions Manually preset rejection thresholds for model attribution of deepfake audio pose challenges to both the performance of open-set recognition and data adaptability. To address this challenge, we propose a novel adaptive threshold framework named ReTA. ReTA employs a reconstruction error strategy to simulate the unique characteristics of unknown...
-
[5]
Acknowledgments This work is supported by the National Natural Science Foun- dation of China (NSFC) (No. 62322120, No.U21B2010, No. 62306316, No. 62206278)
-
[6]
Recent advances in open set recognition: A survey,
C. Geng, S.-j. Huang, and S. Chen, “Recent advances in open set recognition: A survey,”IEEE transactions on pattern analysis and machine intelligence, vol. 43, no. 10, pp. 3614–3631, 2020
work page 2020
-
[7]
Opengan: Open-set recognition via open data generation,
S. Kong and D. Ramanan, “Opengan: Open-set recognition via open data generation,” in Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, 2021, pp. 813–822
work page 2021
-
[8]
For: A dataset for synthetic speech detection,
R. Reimao and V . Tzerpos, “For: A dataset for synthetic speech detection,” in 2019 International Conference on Speech Technol- ogy and Human-Computer Dialogue (SpeD) . IEEE, 2019, pp. 1–10
work page 2019
Show all 36 references
-
[9]
System fingerprint recognition for deepfake audio: An initial dataset and investigation,
X. Yan, J. Yi, J. Tao, C. Wang, C. Zhang, and R. Fu, “System fingerprint recognition for deepfake audio: An initial dataset and investigation,”arXiv preprint arXiv:2208.10489, 2022
2022 arXiv
-
[10]
Leveraging speech ptm, text llm, and emotional tts for speech emotion recognition,
Z. Ma, W. Wu, Z. Zheng, Y . Guo, Q. Chen, S. Zhang, and X. Chen, “Leveraging speech ptm, text llm, and emotional tts for speech emotion recognition,” in ICASSP 2024-2024 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 11 146–11 150
2024
-
[11]
Attributing fake images to gans: Learning and analyzing gan fingerprints,
N. Yu, L. S. Davis, and M. Fritz, “Attributing fake images to gans: Learning and analyzing gan fingerprints,” in Proceedings of the IEEE/CVF international conference on computer vision , 2019, pp. 7556–7566
2019
-
[12]
Are you stealing my model? sam- ple correlation for fingerprinting deep neural networks,
J. Guan, J. Liang, and R. He, “Are you stealing my model? sam- ple correlation for fingerprinting deep neural networks,”Advances in Neural Information Processing Systems , vol. 35, pp. 36 571– 36 584, 2022
2022
-
[13]
Pro- gressive open space expansion for open-set model attribution,
T. Yang, D. Wang, F. Tang, X. Zhao, J. Cao, and S. Tang, “Pro- gressive open space expansion for open-set model attribution,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 15 856–15 865
2023
-
[14]
Open-set recog- nition: A good closed-set classifier is all you need?
S. Vaze, K. Han, A. Vedaldi, and A. Zisserman, “Open-set recog- nition: A good closed-set classifier is all you need?”OpenReview, 2021
2021
-
[15]
C2ae: Class conditioned auto-encoder for open-set recognition,
P. Oza and V . M. Patel, “C2ae: Class conditioned auto-encoder for open-set recognition,” in Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition , 2019, pp. 2307–2316
2019
-
[16]
Divergent angular repre- sentation for open set image recognition,
J. Park, C. Y . Low, and A. B. J. Teoh, “Divergent angular repre- sentation for open set image recognition,” IEEE Transactions on Image Processing, vol. 31, pp. 176–189, 2021
2021
-
[17]
Orientational dis- tribution learning with hierarchical spatial attention for open set recognition,
Z.-g. Liu, Y .-m. Fu, Q. Pan, and Z.-w. Zhang, “Orientational dis- tribution learning with hierarchical spatial attention for open set recognition,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 7, pp. 8757–8772, 2022
2022
-
[18]
[19] utilized a la- tent representation of the network for reconstruction, leading to robust unknown detection
proposed an alternative to using one-versus-rest units (one- versus-rest units) instead of softmax layers. [19] utilized a la- tent representation of the network for reconstruction, leading to robust unknown detection. [8] proposes the C2AE method, arXiv:2412.01425v1 [cs.SD] 2...
2024 arXiv
-
[19]
Energy-based open-world uncertainty modeling for confidence calibration,
Y . Wang, B. Li, T. Che, K. Zhou, Z. Liu, and D. Li, “Energy-based open-world uncertainty modeling for confidence calibration,” in Proceedings of the IEEE/CVF International Conference on Com- puter Vision, 2021, pp. 9302–9311
2021
-
[20]
Generative-discriminative feature representations for open-set recognition,
P. Perera, V . I. Morariu, R. Jain, V . Manjunatha, C. Wigington, V . Ordonez, and V . M. Patel, “Generative-discriminative feature representations for open-set recognition,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion, 2020, pp. 11 8...
2020
-
[21]
A tutorial on hidden Markov models and selected applications in speech recognition,
L. R. Rabiner, “A tutorial on hidden Markov models and selected applications in speech recognition,” Proceedings of the IEEE , vol. 77, no. 2, pp. 257–286, Feb. 1989
1989
-
[22]
A baseline for detecting misclas- sified and out-of-distribution examples in neural networks,
D. Hendrycks and K. Gimpel, “A baseline for detecting misclas- sified and out-of-distribution examples in neural networks,” inIn- ternational Conference on Learning Representations, 2022
2022
-
[23]
An open set recognition for sar targets based on encoding-conditional de- coding network with reject threshold adaptation,
Y . Li, L. Du, J. Chen, J. Song, Z. Wang, and Y . Guo, “An open set recognition for sar targets based on encoding-conditional de- coding network with reject threshold adaptation,”IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sens- ing, 2024
2024
-
[24]
Towards open set deep networks,
A. Bendale and T. E. Boult, “Towards open set deep networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 1563–1572
2016
-
[25]
Doc: Deep open classification of text documents,
L. Shu, H. Xu, and B. Liu, “Doc: Deep open classification of text documents,” in Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2017, pp. 2911–2916
2017
-
[26]
Classification-reconstruction learning for open-set recognition,
R. Yoshihashi, W. Shao, R. Kawakami, S. You, M. Iida, and T. Naemura, “Classification-reconstruction learning for open-set recognition,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 4016–4025
2019
-
[27]
is computed between the predicted labels and the ground truth labels. 2.1.2. Reconstruction Error Learning Module The reconstruction decoding sub-network consists of four de- convolution layers, and its input consists of two parts: the fingerprint representation generated by t...
-
[28]
Detect- ing unknown speech spoofing algorithms with nearest neighbors
J. Lu, Y . Zhang, Z. Li, Z. Shang, W. Wang, and P. Zhang, “Detect- ing unknown speech spoofing algorithms with nearest neighbors.” in DADA@ IJCAI, 2023, pp. 89–94
2023
-
[29]
Why relu net- works yield high-confidence predictions far away from the train- ing data and how to mitigate the problem,
M. Hein, M. Andriushchenko, and J. Bitterwolf, “Why relu net- works yield high-confidence predictions far away from the train- ing data and how to mitigate the problem,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion, 2019, pp. 41–50
2019
-
[30]
An initial investigation for detecting vocoder fingerprints of fake audio,
X. Yan, J. Yi, J. Tao, C. Wang, H. Ma, T. Wang, S. Wang, and R. Fu, “An initial investigation for detecting vocoder fingerprints of fake audio,” in Proceedings of the 1st International Workshop on Deepfake Detection for Audio Multimedia, 2022, pp. 61–68
2022
-
[31]
Deep residual learning for image recognition,
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778
2016
-
[32]
A comparison of features for synthetic speech detection,
M. Sahidullah, T. Kinnunen, and C. Hanilçi, “A comparison of features for synthetic speech detection,” ISCA (the International Speech Communication Association), 2015
2015
-
[33]
React: Out-of-distribution detection with rectified activations,
Y . Sun, C. Guo, and Y . Li, “React: Out-of-distribution detection with rectified activations,” Advances in Neural Information Pro- cessing Systems, vol. 34, pp. 144–157, 2021
2021
-
[34]
Cross-entropy loss functions: Theoretical analysis and applications,
A. Mao, M. Mohri, and Y . Zhong, “Cross-entropy loss functions: Theoretical analysis and applications,” in International confer- ence on Machine learning. PMLR, 2023, pp. 23 803–23 828
2023
-
[35]
Kernel density estimation and its application,
S. W˛ eglarczyk, “Kernel density estimation and its application,” in ITM Web of Conferences, vol. 23. EDP Sciences, 2018, p. 00037
2018
-
[36]
Multi-class metrics made simple, part ii: the f1- score,
B. Shmueli, “Multi-class metrics made simple, part ii: the f1- score,” 2023
2023
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.