{"id":"65ac0022-1e6b-4e16-9d7f-cdd1c7dfdf62","arxiv_id":"2505.09633","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"On the FakeMusicCaps benchmark, an ImageNet-pretrained ResNet18 trained on mel spectrograms distinguishes human from AI-generated music with about 88% F1, and stays above 80% F1 on clips with pitch and tempo changes, though the robustness test is weaker than claimed.","lead":"This preprint tests a standard convolutional network, ResNet18, at telling human music from AI-generated music using the public FakeMusicCaps dataset, with pitch and tempo changes added to mimic evasions. It reports F1 scores near 88% on clean clips and above 80% on modified clips, but the adversarial setup is weaker than claimed and no code or error bars are provided.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Main robustness claim is untested: manipulated audio appears in training, so Table I measures in-distribution classification, not adversarial evasion.","rationale":"The reader's weakest assumption is exactly the load-bearing concern. The paper's own Section IV.D claim depends on the manipulations in Section III.C being a meaningful adversarial simulation; because the model is trained and tested on the same manipulation distribution, the results cannot distinguish robustness from memorization of augmentation. The proposed test would settle this. I do not see an internal inconsistency that would warrant rejection: the baseline result may still be a valid demonstration on FakeMusicCaps, and the paper explicitly labels the manipulations as simulations rather than real attacks. However, the headline conclusion is conditional on a held-out evaluation that is absent. Secondary issues such as missing error bars, seeds, and code reinforce the need for revision but are not the primary reason the conclusion is unsupported. Therefore the appropriate verdict remains CONDITIONAL, unchanged from the reader's assessment.","tokens_in":8682,"tokens_out":4670,"duration_ms":50942,"concrete_test":"Retrain the same ResNet18 exactly as in Section IV.B, but only on the unmodified FakeMusicCaps training split. Evaluate it on the held-out test clips after applying pitch shifts sampled from [-4,-2) ∪ (2,4] semitones and tempo stretches sampled from [0.6,0.8) ∪ (1.2,1.4], i.e., ranges never seen in training. Also evaluate on in-range shifts for reference. If the clean-trained model's F1 on out-of-range manipulations is materially below the Table I values (e.g., <0.75), the claimed robustness fails. Repeat with 5 seeds and report mean ± std.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section IV.D concludes that adversarial evasion is 'not an insurmountable' problem based on Table I. The experiments, however, never present the detector with a manipulation family it has not already been trained on. Section III.C defines the attack distribution (pitch shift uniform in [-2,+2] semitones, tempo stretch uniform in [0.8,1.2]); Section IV.A creates modified datasets from the same distribution; Section IV.B trains on those exact datasets; the test splits are drawn from the same modified pools. The 'Continuous Learning' variant additionally trains on all manipulated versions before evaluation. Thus Table I shows that ResNet18 can learn to classify audio perturbed in ways represented in its training set. That is data augmentation, not adversarial robustness. A real adversary selects transformations at inference time, after the model is fixed, and may use magnitudes or types outside the training ranges. The stated conclusion would require at minimum a clean-trained model evaluated on modified test clips, and ideally held-out manipulation ranges/types. Without that, the 'not insurmountable' claim is unsupported. Secondary reporting gaps (no seeds, error bars, or code) would matter even after this design issue is fixed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript reports experiments on detecting AI-generated music (deepfakes) using the FakeMusicCaps dataset. The author converts audio clips to mel spectrograms, applies pitch-shift and tempo-stretch modifications to simulate adversarial evasion, and trains a transfer-learned ResNet18 to classify clips as human or deepfake. Table I reports F1, accuracy, recall, false positive rate, false negative rate, precision, and specificity for five conditions: baseline, tempo stretch, pitch shift, pitch shift plus tempo stretch, and a continuous-learning condition. The paper concludes in Section IV.D that 'the problem of adversarial techniques to evade deepfake detection is not an insurmountable one.' A large portion of the paper is devoted to ethical, legal, and societal implications of musical deepfakes, including impacts on musicians, copyright and right-of-publicity issues, and positive uses of the technology.","tokens_in":8907,"tokens_out":4713,"duration_ms":42895,"significance":"If the robustness conclusion were supported, the paper would provide a useful baseline showing that a standard transfer-learned CNN with simple augmentation can resist common audio manipulations. The study has genuine strengths: the labels are external ground truth from FakeMusicCaps rather than the model's own predictions, so there is no definitional circularity; it uses a public dataset and standard tools (librosa, torchvision); and Table I reports a complete set of metrics that others can build on. However, the experimental design does not actually measure adversarial evasion: manipulated audio is present in the training distribution, so the results demonstrate in-distribution classification under augmentation, not robustness against an adversary who chooses transformations after the model is fixed. The significance of the paper is therefore limited unless the design is corrected; the ethical discussion, while broad, is not empirically load-bearing.","major_comments":[{"comment":"The conclusion in Section IV.D that 'the problem of adversarial techniques to evade deepfake detection is not an insurmountable one' is not supported by the experiments. The pitch-shifted and tempo-stretched datasets are created in Section III.C and used as training data in Section IV.B, and the test splits are drawn from the same modified pools; the 'Continuous Learning' condition additionally trains on every manipulated version before evaluation. An adversary, by contrast, applies transformations at inference time to a fixed model, possibly using magnitudes or types outside the training range. As designed, Table I measures the model's ability to classify audio perturbed by the same augmentation family seen in training; that is data augmentation, not adversarial robustness. The claim would require at least a model trained only on clean clips and evaluated on modified test clips, and preferably held-out manipulation ranges or types.","section":"Section IV.B / IV.C / IV.D, Table I"},{"comment":"The paper reports a single run with no random seeds, no standard deviations, and no confidence intervals. With 1,074 test tensors, the 2-4 percentage point differences between conditions (for example, pitch shift versus tempo stretch) may well be within run-to-run noise, so the qualitative claims in Section IV.C, such as 'it appears easier to detect rudimentary pitch shifts compared to temporal stretches,' are not statistically supported. Reporting multiple seeds with mean and confidence intervals is necessary before comparing conditions.","section":"Section IV.B / Table I"},{"comment":"The choice of manipulation parameters (-2 to +2 semitones, 0.8 to 1.2 tempo stretch) is not justified as a proxy for a 'moderately skilled malicious actor,' and no evidence is provided that these ranges are representative of real-world evasion attempts. Even after adding a clean-trained evaluation, the model should be tested on out-of-range magnitudes (e.g., ±3 semitones, 0.7 and 1.3 stretch factors) and on manipulation types not seen in training (e.g., reverb, EQ, dynamic range compression) to support the general claim that evasion is 'not insurmountable.'","section":"Section III.C"}],"minor_comments":[{"comment":"The sentence '5,373 tracks were selected from the 5 different deepfake platform subsets of the FakeMusicCaps dataset' is ambiguous: does it mean 5,373 deepfake tracks in total (about 1,075 per platform) or 5,373 per platform? The reported total of 10,746 tracks suggests the former, but the wording should be explicit.","section":"Section III.A"},{"comment":"The manuscript contains two 'Fig. 1' labels; the second figure (Recall by Data Source Type) should be numbered Fig. 2, and the first figure should be referenced in the text.","section":"Figure numbering"},{"comment":"The URL 'http://www.ctan.org/tex-archive/macros/latex/contrib/IEEEtran/' appears to be incorrectly embedded in reference [8]; it should be removed or placed in an appropriate entry.","section":"References"},{"comment":"The preprocessing details for the mel spectrograms are missing: sample rate, number of mel bins, FFT and hop length, handling of clips of variable duration after stretching, and the input image size fed to ResNet18. These details are essential for reproducibility.","section":"Section IV.A"},{"comment":"The wording 'To simulate a real-world adversarial entity' is awkward; consider 'To simulate real-world adversarial conditions.'","section":"Abstract and Introduction"},{"comment":"The sentence 'The other models that were trained on one type of dataset were each within 3-4 percentage points of the Baseline experiment' is vague; specify which metric and which models are meant.","section":"Section IV.C"}],"recommendation":"major_revision","confidential_remarks":"The paper is a short empirical report combined with a broad ethics discussion. The central technical claim is undermined by train/test contamination on the manipulation distribution, but the issue is fixable within the manuscript's scope by re-running the evaluation with a clean-trained model and held-out manipulations. The statistical and reproducibility gaps (no seeds, no code, no preprocessing details) also need to be addressed. The references for the ethical sections rely heavily on non-peer-reviewed sources, but that is secondary to the technical concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about this paper. First, it's a straightforward ResNet18-on-mel-spectrograms baseline for FakeMusicCaps, trained with basic audio perturbations; the numbers are plausible but nothing more. Second, the headline conclusion that adversarial evasion of deepfake detection is 'not insurmountable' is not supported by the experiments, because the model trains and tests on the same manipulation family.\n\nWhat the paper does well: it's clearly written, uses a public dataset, and reports a compact set of metrics. The ethics section is broad and cites a lot of relevant work, which is useful for context. The author explicitly acknowledges future directions and doesn't oversell the method itself. The continuous learning setup is an interesting idea, though it's under-specified.\n\nThe main problem is that the 'malicious manipulation' is applied the same way in training and evaluation. The model sees pitch shifts and tempo stretches from the same ranges at train time, so Table I measures in-distribution classification under augmentation, not robustness to an adversary. A real adversary picks transformations after the model is fixed, often outside the training range. To support the claim, you'd need to train on clean audio and evaluate on manipulated audio, or hold out manipulation types/ranges. The paper's own numbers show moderate drops (F1 from 0.878 to 0.837) when both manipulations are applied, but again that's trained on those manipulations. There are also reporting gaps: no code, no seeds, no error bars, fewer than the full set of deepfake tracks, and no comparison to the original FakeMusicCaps results or Deezer's model. That makes it hard to judge whether the 84-88% F1 is actually competitive. The novelty is thin: the related work itself notes ResNet18 on spectrograms was already used with this dataset, and pitch/tempo manipulation was already explored by Deezer. So the only really new element is the continuous learning variant, which is described in two sentences.\n\nWho is this for? Someone wanting a quick, checkable baseline on FakeMusicCaps might find it a starting point. Someone looking for evidence about adversarial robustness in music deepfake detection should look elsewhere. The paper is honest and clearly written, but the central claim overreaches the design.\n\nI would not send this to peer review as is. It would deserve a serious referee only after the robustness evaluation is redesigned and the reporting is tightened. As a workshop paper or technical report, it's fine.","headline":"A clean but incremental ResNet18 baseline on FakeMusicCaps; the claim about adversarial robustness is not supported because training and testing share the same manipulation distribution.","tokens_in":9431,"tokens_out":2956,"would_cite":false,"duration_ms":29633,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A standard image-classification network can identify AI-generated music even after pitch shifting and tempo stretching.","keywords":["musical deepfakes","text-to-music detection","mel spectrograms","deep learning classifier","pitch shifting","tempo stretching","adversarial audio manipulation"],"falsifier":"Train the same model on only unmodified clips, then test on clips manipulated at inference time with pitch shifts outside -2 to +2 semitones or tempo factors outside 0.8 to 1.2, such as a three-semitone shift or a 1.4x stretch; if accuracy drops toward chance, the not-insurmountable conclusion would fail for unmatched adversarial edits.","tokens_in":8441,"feed_emoji":"🎵","tokens_out":8107,"duration_ms":78675,"temperature":0.7,"pith_summary":"The paper sets out to establish that a relatively simple computer-vision pipeline can tell human-recorded music from music generated by text-to-music models, even after the audio has been tampered with in ways a malicious actor would use. It converts ten-second clips to mel spectrograms and trains a standard pretrained convolutional network as a human-versus-deepfake binary classifier. On clean clips the classifier reports F1 0.878 and accuracy 0.885; with random pitch shifts, tempo stretches, or both, F1 stays above 0.83. The author reads this as evidence that adversarial evasion of musical deepfake detectors is not an insurmountable problem. If the result holds, an ordinary image-classifier baseline could serve as a practical screening tool, and detection systems could be developed to protect artists rather than simply to restrict generative music.","feed_headline":"Detector flags AI music even after pitch and tempo edits","feed_subtitle":"A standard image classifier on mel spectrograms holds F1 above 0.83 on modified clips.","key_machinery":"The load-bearing machinery is the mel-spectrogram image representation plus a pretrained ResNet18 classifier. Mel spectrograms display time, frequency, and amplitude in one image, so audio classification becomes image classification; ResNet18, an 18-layer convolutional network pretrained on image data, is adapted by replacing its final layer with a two-class output. The adversarial-simulation machinery is the random manipulation: each clip gets its own pitch shift drawn from -2 to +2 semitones, tempo stretch from 0.8 to 1.2, or both, before spectrogram generation.","core_discovery":"On the paper's own terms, the central discovery is that a transfer-learned convolutional network trained on spectrogram images provides a workable detector for synthetic music under light adversarial modification. Using a paired corpus of 5,373 human and 5,373 machine-generated ten-second clips, the author converts audio to mel spectrograms and trains ResNet18 with two output classes. The classifier reaches F1 0.878 and accuracy 0.885 on clean clips; with per-clip random pitch shifts between -2 and +2 semitones, tempo stretches between 0.8 and 1.2, or both, F1 remains above 0.83. The paper interprets these results as showing that adversarial evasion of musical deepfake detection is not insurmountable, and that detection systems can therefore be part of a protective response to generative music platforms.","pith_inferences":["A stronger adversarial test would train only on clean audio and then apply previously unseen manipulations at test time; the reported numbers leave that version of robustness open.","False positives may matter more than false negatives in real deployment: human musicians who use MIDI, effects, or AI-assisted production could be flagged by the same detector.","The same pipeline could be extended to other cheap manipulations such as reverb, echo, distortion, volume changes, and codec compression, and to deeper networks, as the paper's own future-work section suggests."],"forward_implications":["A detector trained on clean clips generalizes to mild pitch and tempo changes, with F1 falling from 0.878 on clean clips to no lower than 0.837 when both manipulations are applied.","Sequential training across manipulated datasets can raise recall for human clips to 0.889, but it doubles the false-positive rate relative to the single-dataset baseline.","With only 20 training epochs, every configuration stays above 0.80 in F1, accuracy, recall, and precision, so the approach is inexpensive to reproduce.","The best single configuration reaches 0.885 accuracy and 0.911 precision on unmodified clips, giving a plausible screening baseline for streaming platforms."],"supporting_citations":[{"why":"Supplies the paired human and synthetic ten-second clips that form the training and test data.","marker":"[1]"},{"why":"Provides the caption-based music collection from which the synthetic clips were generated.","marker":"[2]"},{"why":"Demonstrates that pitch shifting and temporal stretching make deepfake detection harder, motivating the manipulation simulation.","marker":"[4]"},{"why":"Supports processing audio as spectrogram images, the classification pipeline adopted here.","marker":"[3]"},{"why":"Supports the choice of a pretrained ResNet18 classifier for mel-spectrogram inputs.","marker":"[6]"},{"why":"Describes the pitch-shifting procedure used to create the modified audio sets.","marker":"[5]"},{"why":"Provides the audio-processing library used for pitch shifting, tempo stretching, and spectrogram extraction.","marker":"[43]"}],"fun_headline_variants":["AI music detector stays strong after audio editing","CNN spots deepfake songs despite pitch and tempo shifts","Robust AI-music detection survives audio alterations","Deepfake music detection holds under tempo and pitch tweaks","Detecting synthetic music even after audio modifications"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that random pitch and tempo changes applied to the training and test sets the same way behave like a real adversary trying to evade detection.","fun_headline_variants_meta":{"raw":{"variants":["AI music detector stays strong after audio editing","CNN spots deepfake songs despite pitch and tempo shifts","Robust AI-music detection survives audio alterations","Deepfake music detection holds under tempo and pitch tweaks","Detecting synthetic music even after audio modifications"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000214,"raw_usage":{"total_tokens":1366,"prompt_tokens":829,"completion_tokens":537,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":445,"completion_tokens_details":{"reasoning_tokens":465}},"tokens_in":445,"tokens_out":537,"duration_ms":6284,"temperature":1.0,"reasoning_tokens":465,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:06:09.321732+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same model on only unmodified clips, then test on clips manipulated at inference time with pitch shifts outside -2 to +2 semitones or tempo factors outside 0.8 to 1.2, such as a three-semitone shift or a 1.4x stretch; if accuracy drops toward chance, the not-insurmountable conclusion would fail for unmatched adversarial edits.","supporting_citations":[{"cited_title":"GTZAN – Mel Spectrogram + Resnet18","cited_arxiv_id":null,"evidence_quote":"Supports the choice of a pretrained ResNet18 classifier for mel-spectrogram inputs."},{"cited_title":"Using Librosa to Change the Pitch of an Audio File","cited_arxiv_id":null,"evidence_quote":"Describes the pitch-shifting procedure used to create the modified audio sets."},{"cited_title":"Available at https://librosa.org/doc/latest/index.html","cited_arxiv_id":null,"evidence_quote":"Provides the audio-processing library used for pitch shifting, tempo stretching, and spectrogram extraction."}],"review_version":1}