{"id":"c5384b5f-f07c-4827-998a-15193fc84055","arxiv_id":"2506.00733","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A perceptual listening audit of 2,280 utterance pairs sets a cosine-similarity threshold of 0.354 for removing likely different-speaker utterances from Common Voice client IDs.","lead":"The authors used voice-embedding similarity scores to detect multiple speakers hiding under one anonymous ID in the Mozilla Common Voice corpus, and set a similarity threshold of 0.354 from a human listening audit. The threshold rejects about 3.5% of utterances per language on average, giving corpus phoneticians a way to clean speaker labels before analysis.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Data-loss estimates depend on which utterance is arbitrarily chosen as enrollment; no robustness check is reported, so the 'minimizing data loss' claim is not yet established.","rationale":"The reader's weakest assumption concerned the perceptual audit: five non-native annotators with moderate agreement provide a crosslinguistically valid threshold. That is a real concern and the paper itself acknowledges it. However, the most load-bearing issue for the central claim may be more basic: even if the 0.354 threshold were perfectly valid, the paper applies it to scores computed against a single enrollment utterance per client ID, with no justification for why the final recording is the right reference. Because client IDs can contain multiple speakers, the enrollment utterance's identity determines which other utterances look 'different.' The reported data loss and the 'minimizing data loss' conclusion are therefore conditional on an arbitrary choice. This concern is concrete, internal to the method, and directly testable by recomputation. The reader did not flag this, so my agreement is partial. The finding does not refute the paper's central contribution; the released similarity scores and audit framework remain useful. But the headline claim about effectiveness and minimal data loss should be accompanied by an enrollment-robustness analysis, which strengthens the case for a conditional verdict rather than a clean acceptance. Since the reader already reached CONDITIONAL, no verdict change is needed.","tokens_in":8855,"tokens_out":5437,"duration_ms":59412,"concrete_test":"Recompute similarity scores and data loss for a stratified sample of client IDs from several languages spanning low and high data loss (e.g., et, ca, nan-tw) under three enrollment rules: (a) first recording, (b) a randomly selected recording, repeated over multiple draws, and (c) the mean/centroid embedding of all utterances in the client ID. Compare per-language percentages of utterances below 0.354, the proportion of client IDs with >10% data loss, and the overlap of retained utterance sets (e.g., IoU). If the relative change in mean data loss exceeds about 20%, or the IoU between retained sets falls below 0.9, the reported data-loss figures and the 'minimizing data loss' claim are not robust to the enrollment rule.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 2.2, every test utterance is scored against a single enrollment utterance per client ID, defined as 'the final recording associated with each client ID.' Because Common Voice client IDs may contain multiple speakers, this enrollment utterance can belong to any one of those speakers. All similarity scores, and therefore all utterances falling below the 0.354 threshold, are relative to that one arbitrary reference. If the final recording happens to be from a minority speaker in a heterogeneous client ID, utterances from the majority speaker will receive low scores and be discarded, inflating data loss; if the final recording is from the majority speaker, the minority speaker's utterances are discarded instead. The reported mean 3.5% utterance data loss, the per-language figures, and the 'minimizing data loss' conclusion are thus not stable properties of the corpus but artifacts of a single, unjustified enrollment choice. This issue is independent of annotator quality: even a perfect threshold cannot fix the fact that the reference identity is unknown and arbitrary. The paper does not report any sensitivity analysis with respect to enrollment selection, such as using the first recording, a random recording, or a centroid embedding as the enrollment. This is an internal robustness gap, not a disagreement with external consensus.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses the known problem that Mozilla Common Voice client IDs can contain multiple speakers, which is problematic for phonetic analysis. The authors use a ResNet-293 speaker verification model pretrained on VoxBlink2 to compute cosine similarity between an enrollment utterance (the final recording of each client ID) and all other utterances under that ID. To set a threshold for flagging potentially different speakers, five authors perceptually audited 2,280 utterance pairs across 76 languages, and a GLMM estimated the crossover point at similarity 0.354. Applying this threshold yields a mean utterance loss of 3.5% across languages (median 1.6%, range <0.0001% to 21.8%). The authors release the similarity scores and argue that 0.354 effectively reduces speaker heterogeneity while minimizing data loss, and that it is reliable because it is close to the VoxCeleb1-H EER threshold of 0.405.","tokens_in":9081,"tokens_out":3278,"duration_ms":34496,"significance":"If the central claims hold, this work would be a useful practical resource for corpus phonetics: it quantifies speaker heterogeneity in a large multilingual corpus, provides a simple utterance-level filtering criterion, and releases the underlying similarity scores so that other researchers can choose their own thresholds. The use of a multilingual speaker verification model and a perceptual audit that is independent of the embedding model are strengths. However, the current evidence does not yet support the paper's headline claim that the 0.354 threshold 'effectively reduces speaker heterogeneity while minimizing data loss': the data-loss figures are conditional on an arbitrary enrollment choice, and the threshold is a single point estimate with no reported uncertainty and only moderate inter-annotator agreement.","major_comments":[{"comment":"The data-loss estimates and the 'minimizing data loss' conclusion depend on the arbitrary choice of the final recording as the enrollment utterance for each client ID. Because Common Voice client IDs may contain multiple speakers, all cosine similarities are relative to that single reference utterance. If the final recording belongs to a minority speaker within a heterogeneous client ID, utterances from the majority speaker will receive low similarity scores and be discarded, inflating data loss; if it belongs to the majority speaker, the minority speaker's utterances are discarded instead. The reported mean of 3.5% utterance loss, the per-language figures, and the client-ID-level loss proportions are therefore not stable properties of the corpus but artifacts of one enrollment convention. The manuscript provides no sensitivity analysis (e.g., first recording, random recording, or centroid embedding as enrollment). This is a load-bearing gap for the central claim and should be addressed with robustness checks.","section":"Section 2.2, Section 3.2, Section 5"},{"comment":"The threshold of 0.354 is estimated from a GLMM on 2,280 trials with only moderate inter-annotator agreement (Fleiss kappa = 0.45), and no confidence interval or measure of uncertainty is reported for the crossover point. Given that the paper recommends this as a crosslinguistically reasonable universal threshold, the authors should report the sampling uncertainty of 0.354 (e.g., bootstrap or profile-likelihood CI) and examine heterogeneity of the crossover across languages. The Discussion itself acknowledges a language-familiarity effect and suggests that by-language thresholds may be more suitable, which is in tension with the conclusion's universal framing. At minimum, the paper should quantify how much the estimated threshold varies across languages or explicitly temper the crosslinguistic claim.","section":"Section 2.3, Section 3.2, Section 4"},{"comment":"The closeness of the audited threshold (0.354) to the VoxCeleb1-H EER threshold (0.405) is presented as evidence that the threshold is 'highly reasonable and reliable.' This comparison is not quantitatively grounded: the two thresholds come from different tasks (perceptual same/different judgments versus speaker verification equal-error-rate on a specific English benchmark), and the 0.051 difference has no stated significance. Without a distribution or standard error for the audited threshold, this concordance is anecdotal. I recommend removing this claim or replacing it with a principled comparison that accounts for uncertainty.","section":"Section 4"}],"minor_comments":[{"comment":"The GLMM description ('uncorrelated random intercepts and slopes by score for annotator and language') does not make clear whether the reported 0.354 is the fixed-effect crossover or some averaged quantity; please clarify the exact model and the formula used to derive the reported point estimate.","section":"Section 2.3"},{"comment":"The x-axis labels in Figures 4 and 5 contain many overlapping language codes and are unreadable; consider using a table, rotating labels, or showing only selected languages.","section":"Figures 4 and 5"},{"comment":"The title and abstract contain 'Common V oice' with an unintended space; this should be corrected to 'Common Voice'.","section":"Title and Abstract"},{"comment":"The statement that the approach is 'superior to using a threshold trained on a single language or arbitrarily selecting a threshold' is not directly supported by the results, since no comparison to those baselines is presented; please either add such a comparison or soften the claim.","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a real and widely recognized problem, and the release of similarity scores is a valuable contribution. The major revision should focus on the enrollment-choice robustness and the uncertainty of the threshold; these are fixable within the manuscript's scope. The work fits the venue (Interspeech-style speech corpus processing), but the current version overstates the stability and precision of the data-loss and threshold claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper ships something worth having: utterance-level similarity scores for 9.2M Common Voice utterance pairs and a 76-language perceptual audit that yields a recommended threshold of 0.354. That is a step beyond Hintz and Siegert's client-ID-level cleaning, and the released scores let other researchers apply their own thresholds. The method is documented clearly, the annotation was blind to the scores, and the GLMM crossover is a sensible way to set a threshold without ground truth.\n\nThe soft spots are real but not fatal. The stress-test concern about enrollment is valid. Section 2.2 says the enrollment is 'the final recording associated with each client ID.' In a client ID that actually contains multiple speakers, that reference is arbitrary. All similarity scores and the resulting data-loss percentages are relative to that one recording. If the final recording belongs to a minority speaker, majority-speaker utterances will look dissimilar and get discarded, inflating the reported 3.5% mean loss. If it belongs to the majority speaker, minority utterances get discarded instead. The paper reports no sensitivity analysis—no first-recording, random, or centroid enrollment—so the 'minimizing data loss' claim in the conclusion is not yet established as a stable property of the corpus. This is an internal robustness gap, not a dispute with the field.\n\nThe second weakness is the threshold itself. The five authors, mostly non-native in the languages they judged, produced moderate agreement (kappa 0.45). The paper acknowledges this and even suggests by-language thresholds may be better. That is honest, but it means 0.354 should be read as a rough crosslinguistic recommendation, not a precise estimate. The lack of a confidence interval on the crossover point adds to that. The sanity check against the English VoxCeleb1-H threshold (0.405) is weaker than it looks, since the embedding model was fine-tuned on VoxCeleb2; still, the perceptual audit is independent of the model, so the threshold is not circular.\n\nWhat the paper does well: it compares against prior work fairly, it releases the scores so others can tailor cleaning, and it frames the threshold as a recommendation rather than a gold standard. The citation pattern looks appropriate.\n\nWho is this for? Empirical researchers using Common Voice or VoxCommunis for crosslinguistic phonetics, and anyone building speaker-based technology on messy crowdsourced data. A serious referee should engage with it, but the review should ask for enrollment robustness checks and a more diverse annotator pool (or by-language thresholds). The core resource is useful; the headline numbers should be treated as provisional.\n\nRecommendation: send to peer review. The paper is a legitimate empirical contribution with fixable weaknesses.","headline":"A useful resource with a real caveat: the utterance-level similarity scores and the 76-language audit are worth having, but the data-loss numbers are tied to an arbitrary enrollment choice and the threshold rests on moderate non-native agreement.","tokens_in":9596,"tokens_out":2404,"would_cite":true,"duration_ms":24148,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Using voice embeddings and listener judgments, the paper sets a cosine threshold of 0.354 to flag multiple speakers inside one Common Voice client ID, at a mean cost of 3.5% of utterances per language.","keywords":["corpus phonetics","speaker verification","voice embeddings","Common Voice","crosslinguistic phonetics","data cleaning","perceptual audit","speaker heterogeneity"],"falsifier":"A reader could test the claim by sampling utterance pairs from the 0.25-0.45 similarity range across several of the 76 languages and having native or fluent speakers label them same/different; if per-language crossover points vary widely or the pooled value shifts away from 0.354, the single threshold fails to generalize.","tokens_in":8674,"feed_emoji":"🎙️","tokens_out":9266,"duration_ms":82026,"temperature":0.7,"pith_summary":"The Mozilla Common Voice Corpus assigns recordings a client ID that is widely used as a speaker ID, but several speakers can contribute under one ID. This paper claims that cosine similarity between ResNet-293 voice embeddings flags those mixed-speaker pairs, and that a perceptual audit by five listeners locates a usable cutoff at similarity 0.354. Applied across 76 languages, that cutoff removes the low-similarity, likely other-speaker utterances while discarding a mean 3.5% and median 1.6% of utterances per language. The authors release the per-utterance similarity scores so downstream phonetic analyses can choose their own cleaning threshold.","feed_headline":"Voice similarity 0.354 separates same from different speakers","feed_subtitle":"Cleans cluttered Common Voice client IDs while dropping a median of 1.6% of utterances per language.","key_machinery":"The load-bearing object is a pair of voice embeddings computed by ResNet-293, a 293-layer residual neural network for speaker verification pretrained on VoxBlink2 and fine-tuned on VoxCeleb2, joined by cosine similarity. A perceptual audit maps those continuous scores onto same-speaker and different-speaker judgments, and a generalized linear mixed model with random intercepts and slopes by annotator and language fits the crossover of the logistic curve, computed as -β0/β1 = 0.354. This machinery translates an embedding-based similarity score into a concrete cleaning rule for dropping utterances from mixed-speaker client IDs.","core_discovery":"On the paper's own terms, the discovery is that speaker heterogeneity inside a Common Voice client ID can be quantified by embedding each utterance with a multilingual ResNet-293 speaker-verification model and taking the cosine similarity against the client ID's enrollment recording. Because no ground-truth speaker labels exist for Common Voice, the authors treat five listeners' same/different judgments as ground truth. The logistic crossover of those judgments, where a listener is equally likely to call a pair same or different, falls at similarity 0.354. At that threshold, across 123,737 client IDs and 9,204,867 utterance pairs, the mean utterance loss is 3.5% per language, the median is 1.6%, and 91.9% of client IDs lose no more than 10% of their utterances. The closeness of the audited threshold to the English-only VoxCeleb1-H equal-error-rate threshold of 0.405 is taken as evidence that the system's notion of voice similarity is reasonably language-independent.","pith_inferences":["Extension: because the similarity-score interquartile range varies across languages, with Q1 ranging from 0.37 to 0.75, a single global threshold may over-clean high-variance languages and under-clean low-variance ones; a native-speaker audit could yield per-language cutoffs and would directly test this.","Extension: the released per-pair similarity scores can double as a diagnostic for utterance quality, not just speaker identity; trials labeled 'audio quality issue' and 'missing speech' formed 12.1% of the audit sample yet were excluded from the threshold fit, so score-quality links remain unexplored.","Extension: if the 0.354 threshold is approximately language-independent, as the paper's closeness to the English-only 0.405 suggests, the same embedding-based approach could estimate effective speaker counts in other crowdsourced speech corpora that also lack ground-truth speaker IDs.","Extension: the paper focuses on multiple speakers sharing one client ID, but a complementary problem is one speaker spread across many client IDs, which would bias variance estimates in the opposite direction."],"forward_implications":["Applying the 0.354 cutoff to VoxCommunis-derived data gives phonetic analyses a speaker-heterogeneity-reduced subset while retaining the vast majority of utterances: 70 of 76 languages lose less than 10% of their data.","Researchers no longer need to discard entire client IDs, as in earlier ID-level cleaning; utterance-level scores preserve usable recordings from mixed client IDs.","The audited threshold being close to the English VoxCeleb1-H equal-error-rate threshold of 0.405 suggests that English-trained verification thresholds can serve as a rough default on multilingual data.","Because the authors release similarity scores for all utterance pairs, each downstream study can select its own trade-off between speaker purity and data retention."],"supporting_citations":[{"why":"Defines the Common Voice corpus and states that the client ID is an anonymized speaker approximation whose heterogeneity this paper targets.","marker":"[1]"},{"why":"Defines VoxCommunis, the forced-aligned derivative of Common Voice whose 76 languages set the scope of the study.","marker":"[2]"},{"why":"Provides the prior client-ID cleaning method that this paper contrasts with; its utterance-level alternative lowers data loss.","marker":"[11]"},{"why":"Supplies the ResNet-293 speaker verification model and VoxBlink2 pretraining data from which embeddings and similarity scores are computed.","marker":"[15]"},{"why":"Points to the released VoxCommunis dataset, which determines the language selection and downstream phonetic use.","marker":"[16]"},{"why":"VoxCeleb2 is the fine-tuning set for the model, and VoxCeleb1-H supplies the English equal-error-rate threshold of 0.405 compared with the audited 0.354.","marker":"[17]"},{"why":"Provides evidence that language shifts speaker embeddings, motivating multilingual pretraining rather than a single-language model.","marker":"[18]"},{"why":"Documents language dependence in multilingual speaker verification, supporting the need for a crosslinguistic audit.","marker":"[25]"},{"why":"Documents the language familiarity effect used to explain annotator disagreement in the second audit round.","marker":"[26]"}],"fun_headline_variants":["0.354: the voice-similarity line between same and different speakers","Voice threshold 0.354 cleans up Common Voice speaker IDs","Language-independent voice similarity at 0.354","Speaker-ID cleanup: threshold 0.354, median 1.6% utterance loss"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole threshold rests on five listeners' same/different judgments in languages most of them did not speak, with only moderate inter-annotator agreement; if those judgments mislocate the crossover, then 0.354 is an artifact of the audit rather than a property of voice similarity across the corpus.","fun_headline_variants_meta":{"raw":{"variants":["0.354: the voice-similarity line between same and different speakers","Voice threshold 0.354 cleans up Common Voice speaker IDs","Language-independent voice similarity at 0.354","Speaker-ID cleanup: threshold 0.354, median 1.6% utterance loss"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000814,"raw_usage":{"total_tokens":3542,"prompt_tokens":895,"completion_tokens":2647,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":511,"completion_tokens_details":{"reasoning_tokens":2570}},"tokens_in":511,"tokens_out":2647,"duration_ms":17275,"temperature":1.0,"reasoning_tokens":2570,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:58:37.387722+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could test the claim by sampling utterance pairs from the 0.25-0.45 similarity range across several of the 76 languages and having native or fluent speakers label them same/different; if per-language crossover points vary widely or the pooled value shifts away from 0.354, the single threshold fails to generalize.","supporting_citations":[{"cited_title":"Quantifying and Reducing Speaker Heterogeneity within the Common Voice Corpus for Phonetic Analysis","cited_arxiv_id":"2506.00733","evidence_quote":"Defines the Common Voice corpus and states that the client ID is an anonymized speaker approximation whose heterogeneity this paper targets."},{"cited_title":"same speaker","cited_arxiv_id":null,"evidence_quote":"Defines VoxCommunis, the forced-aligned derivative of Common Voice whose 76 languages set the scope of the study."},{"cited_title":"V oxlingua107: A dataset for spoken lan- guage recognition,","cited_arxiv_id":null,"evidence_quote":"Provides the prior client-ID cleaning method that this paper contrasts with; its utterance-level alternative lowers data loss."},{"cited_title":"CMU Wilderness Multilingual Speech Dataset,","cited_arxiv_id":null,"evidence_quote":"Supplies the ResNet-293 speaker verification model and VoxBlink2 pretraining data from which embeddings and similarity scores are computed."},{"cited_title":"FLEURS: Few-shot learning evaluation of universal representations of speech,","cited_arxiv_id":null,"evidence_quote":"Points to the released VoxCommunis dataset, which determines the language selection and downstream phonetic use."},{"cited_title":"Phonetic segmentation of the UCLA Phonetics Lab Archive,","cited_arxiv_id":null,"evidence_quote":"VoxCeleb2 is the fine-tuning set for the model, and VoxCeleb1-H supplies the English equal-error-rate threshold of 0.405 compared with the audited 0.354."},{"cited_title":"CommonBench: A larger scale speaker verification benchmark,","cited_arxiv_id":null,"evidence_quote":"Provides evidence that language shifts speaker embeddings, motivating multilingual pretraining rather than a single-language model."},{"cited_title":"Analysis of deep generative model impact on feature extraction and dimension reduction for short ut- terance text-independent speaker verification,","cited_arxiv_id":null,"evidence_quote":"Documents language dependence in multilingual speaker verification, supporting the need for a crosslinguistic audit."},{"cited_title":"PyCantonese: Cantonese linguistics and NLP in python,","cited_arxiv_id":null,"evidence_quote":"Documents the language familiarity effect used to explain annotator disagreement in the second audit round."}],"review_version":1}