{"id":"7e5408e3-9a8e-4751-b694-f291044acd87","arxiv_id":"2504.20923","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"RawNetLite, a lightweight raw-waveform model, reaches near-perfect F1 on FakeOrReal, but the headline out-of-distribution result uses test splits of datasets that were partly included in training.","lead":"This paper presents RawNetLite, a small neural network that flags synthetic speech directly from raw audio waveforms. It matters because cheap, fast synthetic-voice detectors could protect voice authentication and social media, though the reported out-of-distribution gains are overstated.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline OOD result is not out-of-distribution: AVSpoof2021 and CodecFake train/test splits share generators and codecs, so the 83.4% F1/16.4% EER is a matched-domain measurement, not open-world evidence.","rationale":"The reader's weakest_assumption identifies the correct load-bearing issue: the headline OOD result is not actually out-of-distribution because training includes thousands of samples from the same datasets used for testing. I agree with this assessment and with the CONDITIONAL verdict. My read adds one nuance: the paper does contain a genuinely cross-domain CodecFake evaluation in Table XIV, where training uses only FakeOrReal + AVSpoof2021 and reports 74.7% F1 on CodecFake. So the evidence is not limited entirely to the weak Table XI CodecFake row. However, the contradiction between Table XI and Table XIV for the same training domains (with vs. without augmentation) is itself a serious reproducibility concern; a jump from 15.1% to 74.7% F1 from three simple augmentations is surprising and needs independent verification. The paper also contains numeric inconsistencies (e.g., FakeOrReal F1 99.27% in Table IV vs. 99.75% in Table XI; EER values 0.29%, 0.25%, and 2.66% across sections) that reduce confidence in the exact numbers. The most actionable fix is to re-label the evaluation, release exact splits, and provide a true leave-one-domain-out protocol. My concern reinforces the reader's conditional verdict rather than changing it.","tokens_in":12162,"tokens_out":4572,"duration_ms":41944,"concrete_test":"Release exact train/test split identifiers (e.g., file lists or hashes) and rerun the triple-domain protocol with an evaluation set restricted to spoofing methods, codecs, and speakers completely absent from the training subsets. For example, use only the ASVspoof2021 DF evaluation conditions not represented in the 6,400-sample training subset, or CodecFake codec settings excluded from training. Compare F1/EER against Table XII; if the truly OOD numbers fall to the level of Table XI (about 15% F1 on CodecFake), the headline generalization claim is unsupported. Also rerun Table XIV with a fixed seed and report EER to resolve the conflict with Table XI.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central generalization claim rests on the 83.4% F1/16.4% EER reported for 'AVSpoof2021+CodecFake' (abstract, Section V.H, Table XII, 'Triple-domain'). This test set is not out-of-distribution. Sections III.B and III.C and Tables II-III state that 6,400 real and 6,400 fake samples from each of AVSpoof2021 and CodecFake are included in training in the triple-domain configuration. Section III.F claims strict disjointness, but disjointness is only at the sample level; the held-out test samples share the same spoofing generators, codecs, and recording conditions as the training samples from the same datasets. Thus the model has been exposed to the target distributions, and the 83.4% F1 is a matched-domain result. The paper's genuinely cross-domain numbers are contradictory: Table XI (FOR+ASV training, Focal Loss) reports 15.1% F1/50.7% EER on CodecFake, while Table XIV (FOR+ASV training with augmentation) reports 74.7% F1 on CodecFake. The discrepancy is unexplained and undermines the claim that augmentation alone yields open-world robustness. The load-bearing assumption, that the headline evaluation is OOD, is therefore false as stated, and the generalization evidence is at best mixed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RawNetLite, a lightweight end-to-end convolutional-recurrent network that classifies raw waveforms as real or fake speech, and evaluates it on FakeOrReal, ASVspoof2021, and CodecFake. The training pipeline combines domain mixing, Focal Loss, and waveform augmentations. The authors report near-perfect in-domain performance on FakeOrReal (up to 99.75% F1, 0.25% EER in the Focal-loss cross-domain configuration) and claim up to 83.4% F1 with 16.4% EER on a so-called out-of-distribution AVSpoof2021+CodecFake test set. The paper emphasizes that the lightweight architecture competes with larger pretrained systems while remaining deployable, and makes code and pretrained models available.","tokens_in":12515,"tokens_out":4145,"duration_ms":42245,"significance":"If the headline generalization claim were valid, the paper would make a useful empirical contribution: a small raw-waveform model that is near-perfect in-domain and substantially robust across codec-degraded speech without large self-supervised backbones. The in-domain results are competitive with much larger systems, and the systematic comparison of BCE versus Focal Loss, domain mixing, and augmentation is a useful design study. The explicit reporting of failure cases on truly unseen datasets (e.g., the poor CodecFake results when CodecFake is not in training) is also a strength. However, the central cross-domain claim is not supported by the experimental design as written: the datasets used to form the headline ``out-of-distribution'' test set are also partially included in training, so the 83.4% F1 / 16.4% EER result is a matched-domain measurement, not an open-world one. The genuine cross-dataset numbers in the paper tell a much weaker and partly contradictory story.","major_comments":[{"comment":"The headline result is not out-of-distribution. Tables II and III state that 6,400 real and 6,400 fake samples from each of ASVspoof2021 and CodecFake are included in training in the triple-domain configuration, while Section III.F claims that cross-dataset evaluations are conducted on datasets not used in training. These statements are mutually inconsistent. The held-out test samples share spoofing generators, codecs, and recording conditions with the training samples from the same datasets; sample-level disjointness does not make the test set out-of-distribution. Therefore the abstract's claim of ``challenging out-of-distribution test set (AVSpoof2021 + CodecFake)'' and Section V.H's claim of 83.4% F1 on that set describe matched-domain evaluation. The truly cross-dataset results are the CodecFake rows in Tables V, VI, and XI, which show 15--18% F1 and about 50% EER and support the opposite conclusion.","section":"Section III.F, Tables II--III, Section V.H"},{"comment":"There is an unexplained contradiction between the two CodecFake results for training on FakeOrReal+ASVspoof2021. Table XI reports 15.11% fake F1 and 50.68% EER on CodecFake under cross-domain training with Focal Loss, while Table XIV reports 74.66% fake F1 on CodecFake under training on FakeOrReal+ASVspoof2021 with augmentation. If the only difference is the on-the-fly augmentation, the magnitude of the improvement is surprising and needs a detailed explanation, including whether the same test subset, preprocessing, and number of epochs were used. As presented, the discrepancy undermines the paper's claim that augmentation alone confers open-world robustness, because the two tables appear to describe the same base training set with and without augmentation.","section":"Tables XI and XIV, Section V.G"},{"comment":"The text states that replacing BCE with Focal Loss produced a ``substantial performance gain'' on ASVspoof2021, but Table X shows that EER worsens from 16.60% to 17.34%. EER is a threshold-independent metric and is the standard summary for spoofing detection; a result that improves recall and F1 while increasing EER is a trade-off, not an unqualified gain. The discussion should acknowledge this directly and explain which operating point the F1 numbers correspond to, otherwise the reader cannot tell whether Focal Loss actually improves the detection trade-off.","section":"Table X, Section V.E"},{"comment":"The support counts in Table IV are inconsistent with the dataset description. Table I specifies a test set with 3,200 real and 3,200 fake samples, but Table IV reports support of 32,496 real and 32,428 fake samples, summing to 64,924 total, which exceeds the entire FakeOrReal dataset size of 64,000. These numbers are impossible under the stated 80/10/10 split and need to be corrected or explained.","section":"Table IV, Table I"},{"comment":"The abstract's in-domain numbers do not match the baseline table. The abstract states ``over 99.7% F1 and 0.25% EER'' on FakeOrReal, while Table IV (baseline) reports 99.27% F1 and 0.29% EER, and Table XI (Focal-loss cross-domain training) reports 99.75% F1 and 0.25% EER. The abstract appears to mix the baseline F1 with the Focal-loss EER. The text should consistently report which configuration produced each number.","section":"Abstract, Table IV, Table XI"}],"minor_comments":[{"comment":"The claim in Section V.A that test splits are ``strictly disjoint, ensuring no overlap in generation methods or identities'' is contradicted by Tables II--III, which include the same datasets in training; the wording should be corrected to ``no overlap at the sample level.''","section":"Section III.F and Section V.A"},{"comment":"The dataset name is typeset with an internal space as ``AVSpoof2021'' in several places; this should be fixed to ``ASVspoof2021'' for consistency with the official challenge name and references.","section":"Abstract and throughout"},{"comment":"The text says Table XIII shows a fake F1-score of 78.05% on the test set, but the table shows fake F1 of 80.84% and accuracy of 80.96%; similarly, the text mentions a balanced accuracy of 77.74% that does not appear in the table. These numbers need to be reconciled.","section":"Section V.G, Table XIII"},{"comment":"Reference [19] is listed with an incomplete arXiv identifier ``arXiv:2503.xxxxx''; the full identifier or a published venue should be provided.","section":"Section II, Reference [19]"},{"comment":"In Table XV, the RawNet2 baseline is cited as [12] (the ASVspoof2021 evaluation plan), but the RawNet2 architecture is described in [13]; the citation should be corrected.","section":"Table XV"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful lightweight architecture and an honest set of baseline failures, but the framing systematically overstates generalization. The central ``out-of-distribution'' evaluation is not OOD because training includes subsets of both test datasets, and the internal numerical inconsistencies (Table IV supports, Table X EER, Tables XI vs XIV) need resolution before the claims can be trusted. I believe these issues are fixable within the manuscript's scope by reframing the claims as matched-domain robustness and thoroughly auditing the experimental tables, hence major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper delivers a compact, efficient raw-waveform detector (RawNetLite) that hits near-perfect in-domain numbers on FakeOrReal and shows how domain-mix training plus waveform augmentation can improve robustness on codec-corrupted and multi-domain data. The integration of known techniques is honest and clearly framed—the authors explicitly say they are not proposing entirely new components—and the cross-dataset benchmarking effort (five test scenarios, three public datasets) is more systematic than what most short papers in this space do. Credit where due: the architecture is small enough to deploy, the ablations are extensive, and the promise of code and pretrained models, if fulfilled, would make this a useful baseline for training-recipe comparisons.\n\nThe soft spots are real and not minor. The headline generalization result—83.4% F1 and 16.4% EER on 'AVSpoof2021 + CodecFake'—is described throughout as out-of-distribution, but Tables II and III show that 6,400 real and 6,400 fake samples from each of those datasets were included in training under the triple-domain configuration. Section III.F claims strict disjointness, but that disjointness is only at the sample level: the held-out test samples share the same spoofing generators, codecs, and recording conditions as the training samples. So the 83.4% F1 is a matched-domain measurement, not open-world evidence. The paper's genuinely cross-domain numbers tell a different story: Table XI reports 15.1% F1 on CodecFake when trained on FakeOrReal+AVSpoof, while Table XIV reports 74.7% F1 on the same test set with augmentation instead of CodecFake training—an unexplained discrepancy that undermines the claim that augmentation alone yields open-world robustness. There are also internal numeric inconsistencies: the abstract says 99.7% F1 and 0.25% EER while Table IV gives 99.27% F1 and 0.29% EER, and Table X shows Focal Loss raising EER from 16.6% to 17.3% while the text calls it a substantial gain.\n\nThese are fixable issues, but they are load-bearing. The paper deserves a serious referee, not because the headline result is trustworthy as written, but because the underlying model and ablations are worth having in the literature once the evaluation is re-labeled, the numbers reconciled, and exact splits/code released with a commit hash. If those corrections come through, this becomes a solid empirical contribution and a fair baseline for future work. As is, treat the generalization claims with caution.\n\nFor peer review: accept under heavy revision, with an explicit request to fix the OOD framing and the table-text discrepancies.","headline":"RawNetLite is a useful lightweight detector for matched-domain audio deepfake detection, but the paper's central generalization claim is overstated because the headline 'out-of-distribution' test set is actually matched-domain by the authors' own split design.","tokens_in":13057,"tokens_out":1734,"would_cite":false,"duration_ms":19259,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A lightweight raw-waveform network, RawNetLite, reaches over 99.7% F1 on in-domain audio deepfakes and up to 83.4% F1 on a mixed cross-dataset benchmark, with the gains attributed to domain-mixed training, Focal Loss, and waveform…","keywords":["audio deepfake detection","raw waveform","RawNetLite","Focal Loss","domain-mix training","waveform augmentation","cross-dataset evaluation","synthetic speech detection"],"falsifier":"The decisive calculation is already in Table XI: when CodecFake is left out of training, the model scores only 15.1% fake F1 and 50.7% EER on CodecFake, so the 83.4% headline rests on that dataset's presence in training; a fresh test on a TTS or VC generator absent from all training splits would confirm whether any training recipe yields true open-world robustness.","tokens_in":11975,"feed_emoji":"🎧","tokens_out":9705,"duration_ms":86289,"temperature":0.7,"pith_summary":"The paper introduces RawNetLite, a compact convolutional-recurrent network that classifies audio as real or synthetic directly from raw waveforms, avoiding spectrograms, handcrafted features, and large pretrained models. The authors claim over 99.7% F1 and 0.25% EER on the FakeOrReal in-domain test, and up to 83.4% F1 with 16.4% EER on a combined AVSpoof2021 and CodecFake test set. They argue that these robustness gains come from three training choices: mixing data from multiple domains, using Focal Loss to focus on hard samples, and applying waveform-level augmentations such as pitch shift, time stretch, and noise. A sympathetic reader would take the contribution as a reproducible demonstration that lightweight end-to-end detectors can be competitive when training data and loss design are aligned.","feed_headline":"RawNetLite catches known audio deepfakes at 99.7% F1","feed_subtitle":"A lightweight raw-waveform detector shows near-perfect in-domain accuracy, and which training choices actually transfer.","key_machinery":"The central object is the RawNetLite architecture: a 1D convolutional layer followed by three residual blocks, adaptive average pooling, a bidirectional GRU with 128 units per direction, and a two-layer fully connected head with sigmoid output. It is trained with Focal Loss ($\\gamma=2$, $\\alpha=0.25$), which down-weights well-classified samples and steers gradient effort toward hard and ambiguous examples. The other two levers are domain-mix training, which oversamples multiple spoofing corpora, and on-the-fly waveform augmentation (pitch shift within $\\pm 2$ semitones, time stretch in $[0.9, 1.1]$, Gaussian noise in $[0.001, 0.015]$). These components together are what the paper credits for the reported robustness.","core_discovery":"On its own terms, the discovery is that a small raw-waveform model can match or beat much larger pretrained systems on in-domain spoofed audio while remaining cheap to deploy, and that its cross-domain behavior is governed more by the composition of training data and the loss function than by architecture. The paper reports that adding 20% AVSpoof2021 data to FakeOrReal training lifts fake recall on AVSpoof2021 from 22.1% to 77.6%, that Focal Loss alone raises cross-domain fake F1 from 55.8% to 79.5%, and that including CodecFake in the training mix brings its fake F1 from 15.1% to 78.5%. The headline 83.4% F1 on the combined AVSpoof2021+CodecFake test is presented as evidence that this strategy tames the open-world gap.","pith_inferences":["The headline cross-dataset number is better read as a matched-domain measurement than an open-world one, because training includes 6,400 real and 6,400 fake samples from each of AVSpoof2021 and CodecFake; the genuinely unseen result in Table XI (CodecFake: 15.1% fake F1, 50.7% EER) is the more honest estimate of open-world transfer.","A direct consequence is that an auditor holding out an entirely new TTS or VC generator, with different codecs and recording conditions, should expect performance closer to the 15% F1 regime than the 83% regime unless the generator's artifacts overlap with the training corpora.","The augmentation results suggest a testable extension: if codec-specific distortions (e.g., MP3 or Opus artifacts) are simulated instead of generic noise and time stretch, the remaining CodecFake gap might close further; this could be verified without acquiring new datasets."],"forward_implications":["A lightweight raw-waveform detector can reach over 99% F1 on generators seen in training, making it a candidate for low-resource deployment where computational budget is tight.","Adding a modest slice of target-domain data (about 20% of a dataset) to the training mix can shift fake recall by over 50 percentage points, implying that data selection, not backbone size, is the dominant lever in this task.","Focal Loss appears to act as a surrogate for harder example mining in cross-domain settings, improving fake F1 by roughly 24 points on AVSpoof2021 without degrading in-domain performance.","Waveform-level augmentation alone can lift fake F1 on a codec-degraded dataset from 15.1% to 74.7%, suggesting that simulating acoustic variability is a cheap route to robustness.","Combining all three strategies yields the best reported numbers (83.4% F1, 16.4% EER) on the joint AVSpoof2021+CodecFake test, indicating the authors view the combination as the practical recipe for open-world detection."],"supporting_citations":[{"why":"RawNet2, the base architecture that RawNetLite simplifies and adapts for raw-waveform anti-spoofing.","marker":"[12]"},{"why":"To-Rawnet, a later RawNet variant with orthogonal regularization, used as a robustness-improvement reference.","marker":"[13]"},{"why":"FakeOrReal dataset, the in-domain training and evaluation corpus for the near-perfect detection results.","marker":"[22]"},{"why":"ASVspoof benchmark, used as the cross-domain dataset for training and testing with varied spoofing techniques.","marker":"[23]"},{"why":"CodecFake dataset, providing codec-compressed audio that drives the generalization experiments and the augmentation motivation.","marker":"[24]"},{"why":"Focal Loss, the loss function whose reweighting of easy vs. hard samples is central to the cross-domain improvements.","marker":"[26]"}],"fun_headline_variants":["RawNetLite audio deepfake detector: 99.7% F1 in-domain, 83.4% OOD","Training data mix and Focal Loss boost cross-domain deepfake detection","From 22% to 78% recall: how training mix fixes audio deepfake generalization","Cheap raw-waveform model matches big pretrained systems on audio deepfakes","RawNetLite: lightweight detector with 99.7% F1 and cross-domain robustness"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the AVSpoof2021+CodecFake test is genuinely new to the model, even though 12,800 of its clips (6,400 per class) already appeared in training; remove that overlap and the headline 83.4% F1 drops to 15.1% F1 on the only fully unseen dataset.","fun_headline_variants_meta":{"raw":{"variants":["RawNetLite audio deepfake detector: 99.7% F1 in-domain, 83.4% OOD","Training data mix and Focal Loss boost cross-domain deepfake detection","From 22% to 78% recall: how training mix fixes audio deepfake generalization","Cheap raw-waveform model matches big pretrained systems on audio deepfakes","RawNetLite: lightweight detector with 99.7% F1 and cross-domain robustness"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00039,"raw_usage":{"total_tokens":2090,"prompt_tokens":1021,"completion_tokens":1069,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":637,"completion_tokens_details":{"reasoning_tokens":951}},"tokens_in":637,"tokens_out":1069,"duration_ms":10267,"temperature":1.0,"reasoning_tokens":951,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:16:15.937379+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"The decisive calculation is already in Table XI: when CodecFake is left out of training, the model scores only 15.1% fake F1 and 50.7% EER on CodecFake, so the 83.4% headline rests on that dataset's presence in training; a fresh test on a TTS or VC generator absent from all training splits would confirm whether any training recipe yields true open-world robustness.","supporting_citations":[{"cited_title":"End-to-end anti-spoofing with rawnet2,","cited_arxiv_id":null,"evidence_quote":"RawNet2, the base architecture that RawNetLite simplifies and adapts for raw-waveform anti-spoofing."},{"cited_title":"For: A dataset for synthetic speech detection,","cited_arxiv_id":null,"evidence_quote":"FakeOrReal dataset, the in-domain training and evaluation corpus for the near-perfect detection results."},{"cited_title":"The codecfake dataset and countermeasures for the universally detection of deepfake audio,","cited_arxiv_id":null,"evidence_quote":"CodecFake dataset, providing codec-compressed audio that drives the generalization experiments and the augmentation motivation."}],"review_version":1}