{"id":"21761f7a-b360-4add-82ab-cd1983bd520b","arxiv_id":"2412.01425","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"ReTA improves open-set deepfake audio source attribution by learning reconstruction error distributions and computing per-class reject thresholds automatically.","lead":"This paper presents ReTA, a way for deepfake audio attribution systems to set their own rejection cut-offs per class instead of relying on a manually tuned threshold. On a Chinese commercial TTS dataset it reports better open-set accuracy and F1 than three conventional baselines.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The core assumption that known-class mismatched reconstruction errors approximate unknown-class errors is unvalidated; if it fails, ReTA's adaptive thresholds are miscalibrated and the claimed gains may not hold.","rationale":"The reader's weakest_assumption identifies precisely the same load-bearing concern: ReTA's thresholds are calibrated using non-matching errors from other known classes, and this is used as a stand-in for errors from genuinely unknown classes. I agree that this is the most load-bearing point. The paper's experimental section does not validate this proxy, and the missing probability minimization derivation and lack of error bars make the reported gains hard to interpret. My proposed leave-one-known-out experiment directly tests the proxy by treating a known class as unknown during calibration, which is a stronger check than only using the two real unknown vendors because it can be repeated over all six known classes and isolates generalization to a class never seen during threshold fitting. Since the reader already rendered a CONDITIONAL verdict based on this assumption, my read does not change that verdict; it reinforces the condition. I do not see an additional independent fatal flaw: the method is internally coherent, the comparison with prior methods uses the same backbone, and the reported improvements, while modest, are directionally positive. The concern is empirical validity of the proxy, not a logical contradiction. Therefore 'UNCHANGED' is appropriate.","tokens_in":6978,"tokens_out":4027,"duration_ms":36146,"concrete_test":"Hold out one of the six known SFR classes (e.g., Baidu Ai Cloud) as a pseudo-unknown: train ReTA on the remaining five, compute matching/non-matching errors and thresholds from those five, then evaluate open-set F1 and total accuracy on the held-out class plus the real unknowns. Repeat for each known class. If the average pseudo-unknown F1 is substantially below the Table 2 value (or below the oracle threshold set using the held-out class's true errors), the known-class proxy does not generalize and ReTA's adaptive thresholds are miscalibrated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.1.2 calibrates each class's reject threshold using two error populations: matching errors (same-class reconstruction) and non-matching errors (known-class fingerprint paired with a randomly chosen other known class label). Section 2.1.3 then fits KDEs and sets tau_k by a probability minimization criterion. The central claim—that ReTA outperforms Softmax/OpenMax/CROSR on SFR—rests on the premise that the non-matching error distribution is a faithful proxy for the reconstruction error distribution of genuinely unseen vendors (Tencent, iFLYTEK). This premise is never tested in the paper. Unknown vendors may produce errors that are systematically larger (novel acoustic characteristics), systematically smaller (near-duplicate of a known vendor), or concentrated in a different region of reconstruction space. If so, the threshold that minimizes the known matching/non-matching error overlap will misclassify unknown samples, and the reported F1 gains (77.35 vs 75.66) could shrink or reverse. The paper also omits the exact probability minimization objective and reports no error bars, so we cannot tell whether the improvement is within noise. A leave-one-known-out validation would settle whether the proxy holds.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ReTA, a framework for open-set model attribution of deepfake audio. It consists of a system fingerprint recognition module, a reconstruction error learning module that produces matching and non-matching reconstruction samples, and an adaptive reject threshold module that uses kernel density estimation and a \"probability minimization criterion\" to set per-class rejection thresholds. The method is evaluated on the SFR dataset, where ReTA is compared with Softmax, OpenMax, and CROSR, reporting the highest average F1 score and total accuracy.","tokens_in":7245,"tokens_out":4545,"duration_ms":39527,"significance":"If the method is fully specified and validated, it addresses a genuine practical problem: manual rejection thresholds in open-set audio attribution are brittle and do not transfer across data distributions. The paper has a fair experimental setup in that thresholds are fit to training data and unknown vendors (Tencent, iFLYTEK) are held out, so the central result is not circular. The reported gains, however, are modest (average F1 77.35 vs 75.66 for CROSR), and the current manuscript does not specify the probability minimization criterion, lacks error bars or significance tests, and does not validate the key proxy assumption that known-class mismatched reconstruction errors model unknown-class errors. The contribution is promising but not yet established.","major_comments":[{"comment":"The \"probability minimization criterion\" is the core of the adaptive threshold mechanism but is never defined mathematically or algorithmically. The text only states that \"optimal reject thresholds are determined using the probability minimization criterion\" and provides a flowchart in Figure 4. Please specify the objective function in a formula, including how the two KDEs are combined, what is minimized (e.g., a weighted sum of error probabilities), and any hyperparameters such as false-positive/false-negative costs or grid resolution.","section":"Section 2.1.3"},{"comment":"The training procedure is internally inconsistent: Section 2.2 states that the modules \"use stochastic gradient descent (SGD) to iteratively update parameters,\" while Section 3.1 states \"The training was conducted with the Adam optimizer.\" These are different optimization algorithms. Please clarify which optimizer is actually used and provide the exact training configuration.","section":"Section 2.2 vs Section 3.1"},{"comment":"The performance comparison reports only point estimates. The average F1 improvement over CROSR is 1.69 points (77.35 vs 75.66), and the clean-set F1 improvement is only 0.41 points (89.35 vs 88.94). Without repeated runs, standard deviations, or a paired significance test, the claim that ReTA \"overall achieves the best\" is not statistically supported. Please report mean±std over at least five random seeds and apply an appropriate significance test (e.g., paired bootstrap).","section":"Table 2 and Section 3.3"},{"comment":"The central assumption of the method is that non-matching reconstruction errors, obtained by pairing a known-class fingerprint with a randomly selected other known-class label, are a faithful proxy for the reconstruction errors of genuinely unknown classes such as Tencent and iFLYTEK. This assumption is never validated. Please add a leave-one-known-out experiment: hold out one of the five training vendors as an unknown class, calibrate thresholds using the remaining four, and compare the simulated non-matching error distribution with the held-out vendor's actual error distribution. Report a quantitative separation measure (e.g., AUROC, KL divergence, or overlap coefficient) to demonstrate that the proxy holds.","section":"Section 2.1.2 and Section 2.1.3"},{"comment":"The baseline threshold selection is unspecified. The paper states that Softmax, OpenMax, and CROSR use manually set thresholds and that all methods use the same LFCC features and ResNet backbone, but it does not describe how each baseline's threshold was chosen (e.g., grid search on the development set, a fixed percentile, or a single scalar). Without this information, the comparison may be biased in favor of ReTA if the baselines' thresholds are poorly tuned. Please document the threshold selection protocol for each baseline and provide threshold sensitivity curves analogous to Figure 2.","section":"Section 3.3"}],"minor_comments":[{"comment":"The section heading \"Experiences\" should be \"Experiments.\"","section":"Section 3 heading"},{"comment":"The word \"atribution\" is a typo and should be \"attribution.\"","section":"Conclusion"},{"comment":"The label \"X mm\" in Figure 3 appears to be a typo and should likely be \"X^nm\" (non-matching), and the caption uses \"Mismatching Features\" while the text uses \"non-matching\"; please make the terminology consistent.","section":"Figure 3"},{"comment":"The notation in Eq. (1) is ambiguous: X'_i is described as \"a randomly sampled sample from other target classes,\" but it is not clear whether the non-matching reconstruction target is the original input X_i reconstructed under a wrong class condition, or a different sample X'_i reconstructed under its own or another wrong class condition. Please clarify the exact pairing and loss computation.","section":"Equation (1)"},{"comment":"The phrase \"constraining the activation of the penultimate layer [26]\" is vague; please specify the constraint (e.g., ReAct or a bounded activation) and how it is applied to the ResNet-18 encoder.","section":"Section 2.1.1"},{"comment":"Reference [16] describes a similar \"reject threshold adaptation\" method for SAR target recognition, but the paper does not discuss the differences between ReTA and [16] or what is newly contributed beyond adapting that approach to audio model attribution. Please add a brief comparison.","section":"Introduction and References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is written at short-paper length and does not provide enough implementation detail for reproduction. Given that the central threshold mechanism is defined only in prose, I suggest requesting a significantly expanded version with an algorithm box, a hyperparameter table, and a reproducibility statement, as well as statistical evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about this paper. First, it is a straightforward transplant of Li et al.'s reject-threshold adaptation for SAR open-set recognition onto deepfake audio attribution, and the authors cite that paper. Second, the empirical evaluation is clean enough that the headline result—ReTA beats Softmax, OpenMax, and CROSR on SFR—deserves a serious look, but the paper omits the exact threshold objective and never tests the central proxy assumption, so the gains may not replicate.\n\nWhat is genuinely new: per-class adaptive rejection thresholds for open-set audio attribution, trained on matching and non-matching reconstruction errors, evaluated on the SFR dataset with Tencent and iFLYTEK held out as unknown vendors. That is not circular: thresholds are fit to training errors and tested on truly unseen vendors. The comparison is fair in the sense that all methods share LFCC features and a ResNet backbone. The improvements are consistent across clean and compressed splits (average F1 77.35 vs 75.66 for CROSR), which is modest but real-looking.\n\nThe soft spots are real but not fatal. The biggest is the one the stress-test flagged: Section 2.1.2 calibrates each class's threshold using mismatch errors from other known classes, then assumes these approximate the reconstruction errors of genuinely unknown vendors. That assumption is never tested. Unknown vendors could produce systematically larger errors (new acoustic characteristics) or smaller ones (near-duplicates of known vendors), and if so the adaptive thresholds will be miscalibrated. A leave-one-known-out experiment would settle this and should be required.\n\nSecond, the probability minimization criterion is described only in prose; there is no formula. That is a genuine replication barrier. Neither the KDE bandwidth nor alpha are reported, and there are no error bars or significance tests. The gains over CROSR are about 1.7 F1 points on average; without variance estimates we cannot tell if that is within noise. Third, baseline thresholds are not specified; if Softmax/OpenMax thresholds were tuned on the test set, the comparison would be biased in favor of ReTA, which computes thresholds from training data only.\n\nOverall: the paper is a plausible, honest adaptation of an existing method to a new domain, with a clean dataset split and a testable empirical claim. It is not a new capability and not a field-changer, but it is worth a serious referee. I would send it to peer review with a strong request for the missing details and the leave-one-known-out validation.","headline":"A competent transplant of an existing SAR OSR method to deepfake audio, with a plausible but unvalidated proxy assumption and a missing threshold objective; worth a serious referee but needs revision.","tokens_in":7757,"tokens_out":2206,"would_cite":false,"duration_ms":18548,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ReTA replaces hand-tuned rejection thresholds with per-class adaptive thresholds derived from reconstruction error distributions, and reports the best open-set attribution accuracy on the SFR deepfake audio benchmark.","keywords":["open set recognition","deepfake audio","model attribution","system fingerprint","reject threshold adaptation","reconstruction error","kernel density estimation","adaptive threshold"],"falsifier":"Take a held-out deepfake generator that was never used in training, feed its audio through ReTA, and measure the reconstruction error for its predicted label; if that error distribution overlaps substantially with the matching-error distributions of the known classes, the per-class thresholds will be miscalibrated and the open-set F1 will drop close to the fixed-threshold baselines. Concretely, training only on five of the seven vendors and testing on the other two as unknowns, then varying which vendors are held out, would reveal whether the simulated non-matching errors are a faithful proxy.","tokens_in":6834,"feed_emoji":"🎧","tokens_out":4337,"duration_ms":34922,"temperature":0.7,"pith_summary":"This paper argues that the manual rejection threshold used in open-set recognition of deepfake audio is the main bottleneck, and proposes ReTA, a framework that learns each known generator class's reconstruction error distribution and then computes a per-class reject threshold automatically. On the SFR benchmark, ReTA reports the best open-set attribution results, raising average F1 to 77.35% from 75.66% for CROSR and 73.44% for OpenMax, and average total accuracy to 70.09% from 67.91%. The pitch is that thresholds should be derived from the data rather than tuned by hand, so the same method carries over to new data distributions without re-tuning.","feed_headline":"Adaptive reject thresholds outscore fixed cutoffs on deepfake attribution","feed_subtitle":"ReTA learns each class's reconstruction error distribution and sets its own threshold, improving average F1 to 77.35% on SFR.","key_machinery":"The load-bearing mechanism is the pair of reconstruction error distributions built by the reconstruction error learning module. For each training sample the decoder receives the fingerprint representation multiplied either by the conditional latent feature of its true class (matching) or by a randomly drawn latent feature from another known class (non-matching), producing two error sets per class; the adaptive threshold module then fits a Gaussian kernel density estimate to each set and chooses the class-specific reject threshold by probability minimization. This mechanism converts the open-set question into a per-class anomaly detection problem in reconstruction-error space.","core_discovery":"The central claim is that a closed-set deepfake attribution classifier can be turned into an open-set one by calibrating a separate rejection threshold for each known class, where those thresholds are computed from the distributions of matching and non-matching reconstruction errors rather than chosen by hand. The paper shows that when a sample's fingerprint representation is decoded together with the correct class label the reconstruction error is small, while decoding with a randomly chosen other class label produces a larger error, and it fits a Gaussian kernel density estimate to each set of errors to locate the threshold that minimizes the probability of misclassification. Experiments on the SFR dataset, with Tencent and iFLYTEK as unknown classes, show average F1 of 77.35% and average total accuracy of 70.09%, beating the OpenMax and CROSR baselines while largely retaining closed-set ID accuracy.","pith_inferences":["If the proxy assumption holds across domains, ReTA could serve as a general open-set recognition wrapper for any fingerprint-based attribution system, replacing hand-tuned confidence thresholds in production.","A natural testable extension is to model the unknown-class error distribution with a mixture or heavy-tailed density instead of a Gaussian, since real unknown generators may produce multi-modal errors; the probability minimization criterion could be sensitive to tail overlap.","The same reconstruction-error geometry could be used to detect out-of-distribution audio beyond attribution, for example by treating real human speech as an additional known class and rejecting everything else.","The reported gap between clean and compressed performance suggests that compression changes the error distributions substantially; an extension could condition thresholds on audio quality or estimate distribution shift during inference."],"forward_implications":["ReTA achieves the best average F1 and total accuracy on SFR among the compared methods, including on the compressed set where unknown detection is harder.","The per-class thresholds are derived without manual validation-set tuning, so deployment to a new open-set scenario does not require re-setting a global threshold.","The framework turns an existing closed-set fingerprint classifier into an open-set one by adding the reconstruction decoder and threshold module, so it can be attached to other ResNet-based extractors.","The design predicts that matching reconstruction errors cluster tightly while non-matching errors spread out, which is what the threshold calculation exploits."],"supporting_citations":[{"why":"Provides the SFR dataset of deepfake audio from seven vendors and the Softmax with threshold baseline that ReTA improves upon.","marker":"[2]"},{"why":"Introduces the encoding-conditional decoding network with reject threshold adaptation for SAR targets, which ReTA adapts to audio attribution.","marker":"[16]"},{"why":"Defines the OpenMax baseline that calibrates open-set probabilities with the Weber distribution; ReTA is compared against it.","marker":"[17]"},{"why":"Presents the CROSR baseline that combines classification and reconstruction for open-set recognition; ReTA is compared against it.","marker":"[19]"},{"why":"Demonstrates that the optimal rejection threshold varies across attribution tasks, motivating the need for adaptive thresholds.","marker":"[23]"},{"why":"Supplies the kernel density estimation technique used to fit matching and non-matching reconstruction error distributions.","marker":"[28]"},{"why":"Explains why neural networks produce overconfident predictions far from training data, motivating the failure of fixed thresholds.","marker":"[22]"}],"fun_headline_variants":["ReTA tunes per-class reject thresholds for open-set deepfake attribution","Per-class reject thresholds adapt to each deepfake model's fingerprint","Open-set deepfake attribution boosts F1 by per-class adaptive thresholds","Adaptive reject thresholds bring open-set capability to deepfake attribution","ReTA learns per-class reject thresholds, improving open-set deepfake attribution"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the reconstruction errors obtained by pairing a known class fingerprint with a different known class label behave like the reconstruction errors of genuinely unknown classes, so that calibrating thresholds on those simulated mismatches transfers to real unknown classes at test time.","fun_headline_variants_meta":{"raw":{"variants":["ReTA tunes per-class reject thresholds for open-set deepfake attribution","Per-class reject thresholds adapt to each deepfake model's fingerprint","Open-set deepfake attribution boosts F1 by per-class adaptive thresholds","Adaptive reject thresholds bring open-set capability to deepfake attribution","ReTA learns per-class reject thresholds, improving open-set deepfake attribution"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001067,"raw_usage":{"total_tokens":4464,"prompt_tokens":928,"completion_tokens":3536,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":3460}},"tokens_in":544,"tokens_out":3536,"duration_ms":21829,"temperature":1.0,"reasoning_tokens":3460,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:23:13.548740+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out deepfake generator that was never used in training, feed its audio through ReTA, and measure the reconstruction error for its predicted label; if that error distribution overlaps substantially with the matching-error distributions of the known classes, the per-class thresholds will be miscalibrated and the open-set F1 will drop close to the fixed-threshold baselines. Concretely, training only on five of the seven vendors and testing on the other two as unknowns, then varying which vendors are held out, would reveal whether the simulated non-matching errors are a faithful proxy.","supporting_citations":[{"cited_title":"The System Finger- print Recognition module includes 1) fingerprint extraction sub- network; and 2) classification sub-network","cited_arxiv_id":null,"evidence_quote":"Provides the SFR dataset of deepfake audio from seven vendors and the Softmax with threshold baseline that ReTA improves upon."},{"cited_title":"Divergent angular repre- sentation for open set image recognition,","cited_arxiv_id":null,"evidence_quote":"Introduces the encoding-conditional decoding network with reject threshold adaptation for SAR targets, which ReTA adapts to audio attribution."},{"cited_title":"Orientational dis- tribution learning with hierarchical spatial attention for open set recognition,","cited_arxiv_id":null,"evidence_quote":"Defines the OpenMax baseline that calibrates open-set probabilities with the Weber distribution; ReTA is compared against it."},{"cited_title":"Energy-based open-world uncertainty modeling for confidence calibration,","cited_arxiv_id":null,"evidence_quote":"Presents the CROSR baseline that combines classification and reconstruction for open-set recognition; ReTA is compared against it."},{"cited_title":"An open set recognition for sar targets based on encoding-conditional de- coding network with reject threshold adaptation,","cited_arxiv_id":null,"evidence_quote":"Demonstrates that the optimal rejection threshold varies across attribution tasks, motivating the need for adaptive thresholds."},{"cited_title":"Detect- ing unknown speech spoofing algorithms with nearest neighbors","cited_arxiv_id":null,"evidence_quote":"Supplies the kernel density estimation technique used to fit matching and non-matching reconstruction error distributions."},{"cited_title":"A baseline for detecting misclas- sified and out-of-distribution examples in neural networks,","cited_arxiv_id":null,"evidence_quote":"Explains why neural networks produce overconfident predictions far from training data, motivating the failure of fixed thresholds."}],"review_version":1}