{"id":"f612c383-cda9-4851-afa1-3a9f74debcc8","arxiv_id":"2501.01333","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Newly annotated YouTube cover dataset shows state-of-the-art identification models rank correctly far less often than on community datasets, with instrumental, drum-only, and vocal-only versions being especially hard.","lead":"The authors built a new benchmark dataset of cover songs from YouTube, with human annotations for relevance and difficulty. State-of-the-art cover identification models perform much worse on this dataset than on standard community datasets, showing these models are less robust to real-world YouTube content than previously assumed.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline MAP drop is confounded by uncertainty-sampling selection and a hard-negative test distribution; the benchmark does not yet establish general robustness on YouTube.","rationale":"The reader's weakest assumption identifies the core threat: the uncertainty-sampled candidate set may not represent the general distribution of YouTube covers. My analysis sharpens this into a concrete benchmark-composition problem. Section 3.2 selects candidates by disagreement and mutual uncertainty between an audio proxy (Re-MOVE) and a text proxy (Ditto), and Section 5 evaluates models on the resulting set, including 29% irrelevant videos in SHS-YT+2Q. The negative set in SHS-YT contains many near-miss, same-artist, and non-music videos that are not present in SHS100K-Test, so the MAP gap is not an apples-to-apples comparison. The fact that CoverHunter's MAP is 0.83 on SHS-YT+AllQ, close to its SHS100K-Test value, supports the concern that the headline 0.52 depends on the choice of benchmark construction. This does not invalidate the dataset, the taxonomy, or the distributional analyses, which are useful contributions. The paper's own limitation statement mentions sampling bias only for the proxy models, not for the general robustness claim, making this an addressable but load-bearing gap. A random-sample control or a negative-replacement experiment would settle the issue; the published repository makes such a check feasible. Therefore the conditional verdict is appropriate, and I do not recommend changing it.","tokens_in":14682,"tokens_out":5605,"duration_ms":61012,"concrete_test":"Build a control set by randomly sampling candidates from YT-CRAWL (Section 3.1) for the same 100 works, annotate them with the same protocol, and recompute CoverHunter MAP with the same two SHS query versions per work. If the random-sample MAP approaches 0.86 while SHS-YT+2Q stays at 0.52, the uncertainty-sampling selection drives the reported gap; if the random-sample MAP is also low, the robustness concern is real. A cheaper secondary check is to keep the SHS-YT+2Q positives but replace NonVersion/NoMusic negatives with randomly sampled negatives from SHS100K-Test and see whether MAP recovers.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim rests on comparing CoverHunter's MAP of 0.52 on SHS-YT+2Q with 0.86 on SHS100K-Test (Table 5), but the two test collections are not comparable in composition. Section 3.2 deliberately retains only the three candidates per work with the largest disagreement between Re-MOVE and Ditto or the highest mutual uncertainty, with the explicit goal of focusing on out-of-distribution data. Consequently, SHS-YT is enriched for cases that state-of-the-art audio VI models find hard, and SHS-YT+2Q inherits this selection. The negative distribution also differs: Table 4 shows 622 of 900 annotated candidates are NonVersion and 77 are NoMusic, many drawn from the same YouTube search result sets, so negatives are frequently same-artist, same-genre, or musically similar songs rather than the random unrelated works used as negatives in SHS100K-Test. A lower MAP under this design can reflect the difficulty of the negative set and the preselection of positive candidates, not a general inability of CoverHunter to rank YouTube covers. The paper acknowledges this sampling bias only for Re-MOVE and Ditto (Section 5.1), but the threat extends to all evaluated models because the dataset itself is uncertainty-filtered. The sensitivity of the headline gap to composition is visible in Table 5: on SHS-YT+AllQ, where most candidates come from SHS-SEED, CoverHunter's MAP is 0.83, close to the 0.86 baseline. The central claim may still be true, but the reported evidence does not separate selection bias from genuine distribution shift.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper asks whether version identification (VI) models validated on SecondHandSongs-based benchmarks generalize to cover versions as they actually appear on YouTube. The authors build a new dataset, SHS-YT: from the first 100 works of SHS100K-Test (SHS-SEED), they retrieve 94,358 YouTube candidates by text queries, then select nine candidates per work via multi-modal uncertainty sampling using Re-MOVE (audio) and Ditto (text), forming three groups (DisagrAudio, DisagrText, MutualUnc). The resulting 900 candidates are labeled by MTurk workers on an ordinal scale (NoMusic, NonVersion, Version, Match) and curated by two experts who add 19 uncertainty classes and a taxonomy of alterations. The authors benchmark CoverHunter, CQTNet, Re-MOVE, Ditto, and a fuzzy baseline on SHS-YT+2Q and SHS-YT+AllQ, finding a strong MAP drop on SHS-YT+2Q (CoverHunter 0.52 vs 0.86 on SHS100K-Test) but a much smaller gap at SHS-YT+AllQ (0.83 vs 0.86), where CQTNet and Re-MOVE even improve on their SHS100K-Test scores. Cosine-similarity analyses (Tables 6-7) show that YouTube positives are less similar and YouTube negatives more similar than their SHS counterparts, with drum-only, instrumental, medley, and vocal-only versions being the hardest classes.","tokens_in":14978,"tokens_out":19842,"duration_ms":178450,"significance":"The paper has genuine value even though the headline drop is not a clean measurement of general robustness on YouTube. The dataset's ordinal labels (Match vs Version vs NonVersion vs NoMusic), the expert uncertainty classes, and the alteration taxonomy are reusable resources; the repository ships URLs, queries, and CQT/CREMA features, making most quantitative claims checkable; and the falsifiable finding that drum-only, instrumental, medley, and vocal-only versions are hardest for all three audio models (Table 7) is directly actionable for training-augmentation research. The paper is also partially transparent about its own confound: it reports SHS-YT+AllQ where the drop shrinks to 0.83 vs 0.86, and it flags the Re-MOVE/Ditto selection bias in a closing remark. The significance is conditional: the abstract-level claim of significantly lower ranking performance is not yet separated from dataset-construction effects, so the current evidence supports a narrower claim about uncertainty-selected hard cases rather than a general robustness deficit of VI models on YouTube.","major_comments":[{"comment":"The headline comparison, CoverHunter MAP 0.86 on SHS100K-Test vs 0.52 on SHS-YT+2Q, is not like-for-like, and the paper draws a broader conclusion from it than the design supports. SHS-YT+2Q is composed by construction of the nine most uncertain candidates per work (Section 3.2), and its negatives come largely from the same YouTube search result sets: Table 6 shows that both construction effects move the metrics, with YT-Positives less similar to SHS-SEED queries than SHS-Positives (0.80 vs 0.88 for CoverHunter) and YT-Negatives more similar than SHS-Negatives (0.72 vs 0.68). The AllQ column in the same table is decisive: once SHS-SEED versions are added back, CoverHunter's MAP is 0.83 (near the 0.86 baseline) and CQTNet and Re-MOVE score higher than on SHS100K-Test (0.72 vs 0.66 and 0.56 vs 0.53). The claims 'existing models achieve significantly lower ranking performance on our dataset' (Abstract) and 'we in fact confirmed a significant difference' (§6) should therefore be recast as statements about the uncertainty-sampled regime. I recommend adding a control evaluation of the same models on a random, non-uncertainty-filtered subsample of YT-CRAWL, and reporting results per sampling group; the per-group positive rates already differ sharply (93/293, 82/300, 26/307 for DisagrAudio, DisagrText, MutualUnc in Table 4), so aggregate MAP hides which part of the dataset drives the drop.","section":"Abstract; §5.1 (Table 5); §6"},{"comment":"The sampling-bias caveat is confined to Re-MOVE and Ditto, but the bias is a property of the dataset rather than of the proxy models: because every candidate in SHS-YT was selected for proxy disagreement or mutual uncertainty, CoverHunter and CQTNet are evaluated on the same preselected hard set, and the aggregate numbers cannot separate the selection effect from model-level robustness. The issue is compounded for Re-MOVE itself: since DisagrText selects candidates with low Re-MOVE audio similarity, Re-MOVE's YT-Positive row in Table 7 (0.45 vs 0.62 for SHS-Positive) is partly by construction. Reporting the similarity statistics separately for each sampling group, or on a random control subset, would let the reader quantify the by-construction component; at minimum, the closing remark in §5.1 should be extended to state that the selection affects the evaluation of all benchmarked models, not only the modality proxies.","section":"§5.1 closing remark; §5.2 (Tables 6-7)"},{"comment":"No uncertainty is attached to the MAP/MR1 point estimates, yet the Abstract's 'significantly lower' is a statistical claim. With only 100 works, bootstrap confidence intervals over query works (or per-work AP distributions) are needed to judge whether the 0.52 vs 0.86 gap, and especially the 0.83 vs 0.86 AllQ gap, is distinguishable from noise. Relatedly, the two-sample t-tests in Tables 6-7 use supports up to 5.6 million pairs, so p<0.01 is reported for mean differences as small as 0.04 (YT-Negative 0.72 vs SHS-Negative 0.68 for CoverHunter); effect sizes or Cohen's d should be reported alongside, and the YT-Match row of Table 6 rests on only 44 pairs.","section":"§5.1; Abstract"}],"minor_comments":[{"comment":"Typo: 'experiements' should be 'experiments'.","section":"§5.1"},{"comment":"Figure 3 caption: 'the SSHS-YT dataset' and 'bandwith' should read 'SHS-YT' and 'bandwidth'.","section":"§4.1"},{"comment":"In the description of SHS-YT+2Q, 'one additional work from SHS-SEED' should read 'one additional version from SHS-SEED', since each work already contributes two query versions.","section":"§5"},{"comment":"Table 7 caption: the baseline is given as '(SHS-Positive or SHS-Positive respectively)'; it should be 'SHS-Positive or SHS-Negative respectively', since the lower block is contrasted with SHS-Negative.","section":"§5.2"},{"comment":"Twenty SHS-YT candidates also appear in SHS100K (13 from training, 2 from validation, 5 from test), all but one labeled NonVersion; since Section 5 excludes only SHS-SEED versions from the benchmark, please clarify whether the single non-NonVersion candidate drawn from the training or validation subset is retained in SHS-YT+2Q and SHS-YT+AllQ, which would constitute a mild training contamination for CoverHunter and CQTNet.","section":"§4.1"},{"comment":"The sentence 'We publish one human intelligence tasks (HITs) on MTurk per work' should read 'one Human Intelligence Task (HIT)'.","section":"§3.3"},{"comment":"Airoldi, Beraldo, and Gandini (2016) is listed twice as references [3] and [4] with identical bibliographic data; one entry should be removed.","section":"References"},{"comment":"The paper does not explain how 'the first 100 works' from SHS100K-Test are ordered; if the repository listing order is arbitrary the choice is acceptable, but the ordering criterion should be stated so that the seed selection cannot introduce an unexamined bias.","section":"§3.1"},{"comment":"Figure 5 appears to show the same similarity matrix twice and lacks a legend and explicit axis labels for the uncertainty classes; please clarify the layout so that the upper and lower halves are interpretable.","section":"Figure 5"}],"recommendation":"major_revision","confidential_remarks":"I concur with the conditional reader verdict: the load-bearing risk is that the abstract's robustness claim will be cited independently of the uncertainty-sampling proviso. A random-subsample control from the already-collected YT-CRAWL (94,358 videos) is the cleanest fix and is feasible; if the authors cannot run it, they should restrict the abstract and conclusion claims to the uncertainty-sampled setting. The paper is a good fit for the venue: it ships reproducible artifacts, and the taxonomy with per-class similarity statistics is a concrete contribution. One editorial note: MR1 comparisons across collections of very different sizes (1,092 vs 10,547 items) would be more interpretable if normalized, e.g., as the percentile rank of the first relevant item."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead the cover version ID robustness paper (arXiv:2501.01333). The real contribution is SHS-YT, a 900-video annotated dataset from YouTube with ordinal relevance labels and 19 uncertainty classes, plus a taxonomy of alterations. That is a genuinely useful resource for MIR. The annotation pipeline is careful: MTurk with quality checks, expert curation, and public features (CQT/CREMA) even though raw audio can't be shared. The cosine similarity breakdown by uncertainty class is the strongest part: drum-only, instrumental, and vocal-only versions are clearly harder for all models, and same-artist/similar negatives are measurably closer to positives. That evidence supports the paper's RQ2.\n\nThe soft spot is the headline claim. Comparing CoverHunter MAP 0.86 on SHS100K-Test vs 0.52 on SHS-YT+2Q is not an apples-to-apples robustness measure. The dataset was built by uncertainty sampling: candidates are the three per work with the highest disagreement or mutual uncertainty between the audio and text proxies. So SHS-YT is deliberately enriched for hard cases. The negative set also differs—many negatives are same-artist, same-genre, or musically similar songs rather than random works. A lower MAP under that design can reflect the harder test distribution, not a general failure on YouTube. The paper does acknowledge the bias for Re-MOVE and Ditto, but the same selection applies to CoverHunter and CQTNet because the dataset itself is filtered. The sensitivity is visible in the paper's own Table 5: on SHS-YT+AllQ, where most candidates come from SHS-SEED, CoverHunter recovers to MAP 0.83, close to the 0.86 baseline. So the dramatic drop is not robust to composition.\n\nThe fix is to reframe the claim. The paper should present the benchmark as an evaluation on a deliberately hard, uncertainty-selected subset, not as evidence about YouTube in general. They could also report confidence intervals and consider a random sample for comparison. The similarity-based analysis is less confounded and should carry more weight.\n\nOverall: a solid, honest empirical study with a useful dataset and taxonomy. The central robustness claim needs softening, and the evaluation framing needs revision. I'd send it to review.\n\nRecommendation: accept for peer review, expect major revision.","headline":"Good dataset and taxonomy, but the headline robustness claim is confounded by the uncertainty-sampling design; the paper deserves review with revisions.","tokens_in":15473,"tokens_out":1950,"would_cite":true,"duration_ms":19171,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that state-of-the-art cover song identification models, which score well on community-built SecondHandSongs benchmarks, lose substantial ranking accuracy on YouTube-style cover versions, especially when the version…","keywords":["cover song identification","version identification","YouTube","benchmark dataset","uncertainty sampling","robustness","music information retrieval","taxonomy of alterations"],"falsifier":"Take the same 100 works from SHS-SEED, retrieve a random sample of YouTube videos without uncertainty sampling, annotate them with the same ordinal labels, and measure CoverHunter's MAP on the resulting benchmark; if MAP approaches the 0.86 seen on SHS100K-Test, the drop reported here is an artifact of hard-case sampling rather than a property of YouTube covers.","tokens_in":14491,"feed_emoji":"🎵","tokens_out":5951,"duration_ms":50781,"temperature":0.7,"pith_summary":"This paper asks whether cover song identification models, which are trained and evaluated on collections derived from the community database SecondHandSongs, transfer to cover versions as they actually appear on YouTube. To answer it, the authors build SHS-YT, a 900-video benchmark selected by a multi-modal uncertainty sampling procedure and labelled by crowd workers and music experts, then evaluate several state-of-the-art models on it. They find a large performance drop: CoverHunter's mean average precision falls from 0.86 on SHS100K-Test to 0.52 on SHS-YT+2Q. The paper also identifies which alteration types are hardest—instrumental, drum-only, and vocal-only versions, plus medleys and videos mixing music and speech—and offers a taxonomy of alterations in online-video covers. If correct, the result means current benchmarks overstate how robust cover song identification is in the wild.","feed_headline":"Cover-song AI ranking drops from 0.86 to 0.52 on YouTube covers","feed_subtitle":"A new YouTube benchmark shows state-of-the-art models struggle with instrumental, drum-only, and vocal-only versions.","key_machinery":"The load-bearing mechanism is the multi-modal uncertainty sampling pipeline used to build the dataset. For each candidate YouTube video, an audio proxy (pre-trained Re-MOVE, which embeds CREMA features into 256-dimensional vectors) computes mean cosine similarity against all seed versions of a work, while a text proxy (Ditto, a BERT-based entity matcher) computes a maximum matching confidence from titles and channel names. Candidates are ranked by disagreement between the two proxies (audio-high/text-low and text-high/audio-low) and by mutual uncertainty, defined as negative Euclidean distance to the per-work center of uncertainty in the two-dimensional similarity–confidence space. This procedure is designed to over-sample out-of-distribution candidates, i.e., videos whose relatedness the audio model and the text model cannot agree on. The benchmark then combines the 900 annotated candidates with query versions from SHS-SEED to form SHS-YT+2Q and SHS-YT+AllQ, and evaluation is done with MIREX metrics MAP and MR1.","core_discovery":"The central discovery is that cover version identification models are substantially less robust on YouTube-style covers than standard benchmarks suggest, and that the gap is driven by identifiable alteration types. Using 100 western-popular-music works from SHS100K-Test as seed, the authors retrieve 94,358 candidate YouTube videos, select 900 candidates with disagreement and mutual-uncertainty sampling between an audio similarity proxy (Re-MOVE embeddings) and a text matching proxy (Ditto), and annotate them on an ordinal relevance scale (NoMusic, NonVersion, Version, Match). On the resulting benchmark SHS-YT+2Q, all tested models rank worse than on SHS100K-Test; CoverHunter drops from 0.86 to 0.52 MAP, CQTNet from 0.66 to 0.50, Re-MOVE from 0.53 to 0.40. Similarity analysis shows that positive pairs from YouTube are significantly less similar than SHS positive pairs, and that non-versions from YouTube are more similar than random negative pairs, which explains the ranking degradation. The paper's taxonomy classifies alterations into Song-level attributes (melody, harmony, timbre, key, timing, tempo) and Video-level context (multiple songs, non-music noise, chunking, fidelity, stem isolation, in-background), with stem isolation and video context as the hardest cases.","pith_inferences":["The uncertainty sampling deliberately enriches for hard cases, so the 0.52 MAP on SHS-YT+2Q is likely an upper bound on the difficulty of typical YouTube covers; a random sample of YouTube covers for the same works would probably show a smaller but still positive gap.","If stem-isolated versions (drum-only, vocal-only) are the hardest class, then integrating sound-source separation into training augmentation, or evaluating query-by-humming systems on the same SHS-YT candidates, could be a cheap way to improve robustness without new annotation.","The taxonomy suggests a testable prediction: models trained with synthetic concatenations of songs plus speech noise, or with sliding-window matching, should reduce the performance gap specifically for the Video: Multiple Songs and Video: With Non-Music classes.","Because YouTube is dynamic, the benchmark's URLs may decay; the released CQT and CREMA features are the durable artifact, so future comparisons should rely on those features rather than re-downloading audio."],"forward_implications":["If the gap is real, published MAP numbers on SHS100K-Test and Da-Tacos overstate real-world robustness on YouTube-like content, especially for copyright-detection and recommendation applications.","Instrumental, drum-only, vocal-only, and single-instrument versions are consistently hard for current models, pointing to the absence of the main melody and harmony as a key failure cause.","Medleys, videos with multiple versions, and videos with non-musical speech break the alignment assumptions baked into current models, so chunk-based matching and noise-mixing augmentation are natural next steps.","The ordinal labels (NoMusic, NonVersion, Version, Match) and 19 uncertainty classes give a finer-grained evaluation target than binary same-work labels."],"supporting_citations":[{"why":"Provides CoverHunter, the state-of-the-art model whose MAP drops from 0.86 to 0.52 on the new benchmark.","marker":"[25]"},{"why":"Provides CQTNet, the second evaluated model whose MAP drops from 0.66 to 0.50.","marker":"[39]"},{"why":"Provides Re-MOVE, the audio proxy used in uncertainty sampling and the third evaluated model.","marker":"[37]"},{"why":"Provides SHS100K-Test, the seed dataset and the community benchmark baseline for comparison.","marker":"[34]"},{"why":"Provides Da-Tacos, the second community benchmark baseline used to contextualize the performance drop.","marker":"[38]"},{"why":"Provides Ditto, the text-based entity matching proxy used in the sampling pipeline.","marker":"[23]"},{"why":"Provides the existing taxonomy of version identification elements that the paper extends to online-video covers.","marker":"[35]"},{"why":"Provides the query formulation method used to retrieve candidate YouTube videos for each work.","marker":"[18]"}],"fun_headline_variants":["Cover-song ID models fail on YouTube's instrumental and vocal-only versions","Instrumental and drum-only YouTube covers trip up song ID models","Cover-song AI struggles with YouTube stems and video context","Why cover-song AI fails on YouTube: stems and video context","YouTube covers expose weakness in cover-song identification"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole comparison rests on the assumption that the uncertainty-sampled candidates represent the distribution of cover versions on YouTube; if the sampling over-selects hard or unusual cases, the measured performance drop may not generalize to YouTube covers at large.","fun_headline_variants_meta":{"raw":{"variants":["Cover-song ID models fail on YouTube's instrumental and vocal-only versions","Instrumental and drum-only YouTube covers trip up song ID models","Cover-song AI struggles with YouTube stems and video context","Why cover-song AI fails on YouTube: stems and video context","YouTube covers expose weakness in cover-song identification"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000723,"raw_usage":{"total_tokens":3251,"prompt_tokens":961,"completion_tokens":2290,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":2209}},"tokens_in":577,"tokens_out":2290,"duration_ms":16724,"temperature":1.0,"reasoning_tokens":2209,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:29:20.583330+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same 100 works from SHS-SEED, retrieve a random sample of YouTube videos without uncertainty sampling, annotate them with the same ordinal labels, and measure CoverHunter's MAP on the resulting benchmark; if MAP approaches the 0.86 seen on SHS100K-Test, the drop reported here is an artifact of hard-case sampling rather than a property of YouTube covers.","supporting_citations":[{"cited_title":"In: 2023 IEEE International Conference on Mul- timedia and Expo (ICME)","cited_arxiv_id":null,"evidence_quote":"Provides CoverHunter, the state-of-the-art model whose MAP drops from 0.86 to 0.52 on the new benchmark."},{"cited_title":"In: ICASSP 2020-2020 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP)","cited_arxiv_id":null,"evidence_quote":"Provides CQTNet, the second evaluated model whose MAP drops from 0.66 to 0.50."},{"cited_title":"In: Proc","cited_arxiv_id":null,"evidence_quote":"Provides Re-MOVE, the audio proxy used in uncertainty sampling and the third evaluated model."},{"cited_title":"In: 2018 IEEE International Conference on Multi- media and Expo (ICME)","cited_arxiv_id":null,"evidence_quote":"Provides SHS100K-Test, the seed dataset and the community benchmark baseline for comparison."},{"cited_title":"In: Proc","cited_arxiv_id":null,"evidence_quote":"Provides Da-Tacos, the second community benchmark baseline used to contextualize the performance drop."},{"cited_title":"(eds.) Proceedings of the LWDA 2021 Work- shops: FGWM, FGKD, and FGDB, Hildesheim (Germany), Oktober 5-7th, 2022","cited_arxiv_id":null,"evidence_quote":"Provides the query formulation method used to retrieve candidate YouTube videos for each work."}],"review_version":1}