{"id":"616b0ba0-6b2a-4a86-97ed-fd77d999437e","arxiv_id":"2508.16858","paper_version":1,"verdict":"UNVERDICTED","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":2,"one_line_summary":"WildSpoof is a two-track competition plan: generate spoofed speech with cloned voices (TTS) and detect spoofed speech in speaker verification (SASV), both on in-the-wild audio.","lead":"This paper lays out the rules, data, and scoring for the WildSpoof Challenge, where one track builds text-to-speech systems that read text in a target speaker's voice and a parallel track builds systems that detect such spoofed audio. A generalist might read it to see how the speech community plans to benchmark voice cloning and deepfake detection on real-world recordings, not just clean lab data.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing scoring tooling: Versa is undefined, and the a-DCF reference implementation points to an evaluation package that Section 4 never names.","rationale":"The reader's weakest_assumption already identified the scoring tooling as load-bearing: Versa is undefined and the a-DCF reference implementation is not actually located. My stress-test confirms this is the most concrete, internally checkable flaw in the protocol. Unlike the evaluation-data release (which the document itself flags as 'to be released' and is a scheduling matter), the missing tooling references create an internal inconsistency: Section 3.4 promises an evaluation package that Section 4 does not mention, and Section 2.4 references an undefined tool. This means the protocol's metric specifications are not currently actionable, and the central claim about the challenge running with specified metrics cannot be verified. I therefore recommend CONDITIONAL: accept the evaluation plan only if the organizers publicly release or explicitly identify the evaluation package and define Versa before the challenge opens. This does not reject the plan, but it marks the missing pieces as necessary conditions for the plan to be executable.","tokens_in":3891,"tokens_out":3328,"duration_ms":38224,"concrete_test":"Visit the WildSpoof GitHub organization and the two repositories linked in Section 4 (github.com/wildspoof/TTS_baselines and github.com/wildspoof/SASV_baselines). If neither contains an 'evaluation' package with the a-DCF reference implementation and a Versa-based metric script, then Sections 2.4 and 3.4's tooling claims are not publicly available as described.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The protocol's central claim—that the challenge will be run with the specified metrics—is undercut by two missing tooling references. Section 2.4 states that MCD, UTMOS, DNSMOS, WER, and SPK-sim 'will be calculated by Versa' but does not cite, describe, or link Versa; the exact metric implementations (e.g., which UTMOS checkpoint, how SPK-sim is computed) are therefore not reproducible. Section 3.4 says the reference implementation of a-DCF 'is included in the evaluation package in GitHub repository mentioned in Section 4,' but Section 4 mentions only TTS_baselines and SASV_baselines repositories—no evaluation package is named. A reader cannot locate the a-DCF implementation from the information in the paper, and the TTS metric code is likewise unspecified. Because these tools are the sole specified scoring mechanism for both tracks, the protocol as written does not actually specify how challenge results will be computed, making the claim that the challenge will execute 'with the specified metrics' unverifiable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript is an evaluation plan for the WildSpoof Challenge, a community challenge with two parallel tracks: a TTS synthesis track and a spoofing-robust SASV track. The TTS track uses the TITW-Easy/Hard training sets and the TITW-KSKT/KSUT evaluation protocols; participants submit 9,113 and 8,000 waveform files, and the organizers compute MCD, UTMOS, DNSMOS, WER, and SPK-sim, with no system ranking. The SASV track uses SpoofCeleb training data and a forthcoming evaluation package; participants submit trial scores, and systems are ranked by a-DCF with priors pi_tar=0.9405, pi_non=0.0095, pi_spf=0.05 and costs C_miss=1, C_fa=10, C_fa_spoof=10. The plan also specifies baselines, submission formats, rules, ethics, registration, and a schedule. The central content is the protocol itself; there are no experimental results or derivations.","tokens_in":4121,"tokens_out":2095,"duration_ms":28087,"significance":"If the challenge executes as specified, it would provide a useful community resource: it explicitly targets in-the-wild speech data, connects TTS generation with spoofing detection, and provides open baselines with public repositories. The protocol is largely concrete: submission file counts, formats, sampling rates, score file schemas, and the a-DCF priors/costs are all explicitly stated, and the priors sum to one. The two baseline repositories are named with URLs. The plan is less complete on evaluation tooling: the TTS metrics are delegated to an undefined tool called Versa, and the a-DCF reference implementation is said to live in an evaluation package that is never identified. Since these tools are the sole mechanism for computing both tracks' metrics, the protocol as written is not fully reproducible. These gaps are fixable and do not undermine the challenge's motivation, but they need to be resolved before the plan can serve as the definitive specification.","major_comments":[{"comment":"The statement 'All these metrics will be calculated by Versa' is load-bearing for the TTS track, yet Versa is not cited, described, or linked. The reader cannot tell which UTMOS checkpoint is used, how SPK-sim is computed (e.g., which speaker embedding model), how MCD is aligned, or how WER is decoded. Without this information, the metric definitions are not reproducible and participants cannot verify the organizers' scoring. Please name the Versa repository or API, its version, and the exact configuration of each metric.","section":"Section 2.4"},{"comment":"The a-DCF reference implementation is described as 'included in the evaluation package in GitHub repository mentioned in Section 4,' but Section 4 mentions only the TTS_baselines and SASV_baselines repositories, neither of which is identified as containing an evaluation package. The path to the a-DCF implementation is therefore missing. Since a-DCF is the sole ranking metric for the SASV track, the plan must give a direct URL or repository name and, ideally, a version/commit for the reference implementation.","section":"Section 3.4"}],"minor_comments":[{"comment":"'Both evaluation protocols cover 40.' appears to be missing a noun (presumably '40 speakers' or '40 utterances'). Please clarify.","section":"Section 2.2"},{"comment":"The Evaluation row for the TTS track says 'to be released,' while Section 2.2 refers to the TITW-KSKT/KSUT evaluation protocols. Clarify whether the evaluation protocols already exist (as part of TITW) and only the file lists are being released, or whether the protocols themselves are new.","section":"Table 1"},{"comment":"In the last bullet, 'If a participants is uncertain' contains a subject-verb agreement typo.","section":"Section 3.5"},{"comment":"The challenge objectives and ethics are described twice in similar terms (Introduction and Ethics); consolidating would reduce redundancy, though this is not essential.","section":"Section 1 and 5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an evaluation plan, not a research contribution with technical claims. If the venue expects archival research papers, the fit should be considered separately. The missing tooling references (Versa and the a-DCF evaluation package) are the main technical weaknesses; they are easily fixed by adding URLs and implementation details. I would be comfortable with acceptance after those additions, assuming the challenge proceeds on schedule."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a challenge evaluation plan, not a research paper. It contains no new data, methods, or results; its contribution is organizational, pairing a TTS generation track and an SASV detection track on in-the-wild speech. And that's fine—the community needs shared benchmarks that go beyond clean, controlled datasets, and WildSpoof looks like a sensible step in that direction.\n\nWhat's good: the protocol is mostly concrete. Data partitions (TITW, SpoofCeleb) are specified, submission formats are clear, the a-DCF priors sum correctly (0.9405+0.0095+0.05=1.0) and costs are stated explicitly. The rules are sensible, including the ban on cross-participation between tracks and the prohibition on using evaluation data for normalization. The choice not to rank TTS systems is honest, given how noisy metrics like UTMOS and DNSMOS are. Baselines (Grad-TTS + DiffWave; integrated SASV system) are drawn from prior work and links are given. The schedule is realistic.\n\nThe soft spots are real but minor in proportion. The main one, which the stress-test correctly flags, is that the scoring tooling is underspecified. Section 2.4 says all TTS metrics 'will be calculated by Versa' but gives no citation or description—so a reader cannot know exactly which implementations (e.g., which UTMOS checkpoint, how SPK-sim is computed) will be used. Section 3.4 says the a-DCF reference implementation is in 'the evaluation package in GitHub repository mentioned in Section 4,' but Section 4 only names the TTS_baselines and SASV_baselines repositories, not an evaluation package. As written, you cannot locate the scoring code for either track. That's a genuine gap in the protocol, though it's likely to be fixed when the evaluation packages are actually released. There are also small editorial slips: the sentence 'Both evaluation protocols cover 40.' is truncated, and the version date (Sept 12, 2025) postdates the arXiv submission (Aug 23, 2025).\n\nA note on citation pattern: the plan leans heavily on the organizers' own prior work (TITW, SpoofCeleb, a-DCF, SASV baseline). That's not a flaw—these are the natural resources for this task—but it does mean the novelty is purely organizational.\n\nWho is this for? Participants in WildSpoof, and anyone designing or studying evaluation methodology for speech deepfakes. It is not a research contribution and should not be evaluated as one.\n\nRecommendation: if you're an editor at a venue that accepts challenge plans, send it to a reviewer with a quick sanity check. A reviewer could catch the missing references and tighten the protocol. It's a worthwhile document, just don't expect any science in it.","headline":"A concrete, mostly well-specified challenge plan for in-the-wild TTS and spoofing detection; the science is absent by design, and the scoring tooling is underspecified.","tokens_in":4697,"tokens_out":3716,"would_cite":false,"duration_ms":38397,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"WildSpoof pairs in-the-wild TTS with spoof-aware speaker verification","keywords":["in-the-wild speech","text-to-speech","spoofing-robust speaker verification","deepfake detection","a-DCF","challenge evaluation plan","TITW","SpoofCeleb"],"falsifier":"If, by the announced data-release date, the TITW-KSKT and TITW-KSUT trial lists or the SASV evaluation waveform package are not distributed, or the reference a-DCF implementation is absent from the evaluation package, the plan cannot be executed as written. A simpler check: run the provided baseline systems and confirm the metric code produces finite scores on the released trial lists.","tokens_in":3738,"feed_emoji":"🎙️","tokens_out":4540,"duration_ms":46077,"temperature":0.7,"pith_summary":"The WildSpoof Challenge Evaluation Plan lays out a two-track competition that ties text-to-speech generation to spoof-aware speaker verification. The TTS track asks participants to build systems that read arbitrary text in a target speaker's voice using the TITW dataset, while the SASV track asks participants to score enrollment-test trials from the SpoofCeleb dataset to reject both non-target and spoofed speech. The plan fixes the evaluation protocols: file formats, submission formats, metrics (MCD, UTMOS, DNSMOS, WER, and speaker similarity for TTS; a-DCF with set priors and costs for SASV), and baseline systems. It also states that no ranking will be given for TTS systems, only a summary. By pairing generation and detection on in-the-wild data, the challenge aims to push both communities beyond clean and controlled benchmark conditions.","feed_headline":"In-the-wild challenge pairs TTS with deepfake detection","feed_subtitle":"A single protocol ties TTS synthesis to spoof-aware verification on TITW and SpoofCeleb, with a-DCF scoring.","key_machinery":"The load-bearing object is the challenge protocol itself: the pairing of two independently treated tracks, each with its own data, submission format, and metric. The SASV side is anchored by the a-DCF metric, which assigns a single cost to a system by weighting one miss rate and two false-alarm rates; the specified priors and costs make the formula numerically concrete. The TTS side is anchored by the five fixed metrics computed by the designated scoring tool. Together these define the only basis on which systems will be compared, and the plan states explicitly that TTS systems will not be ranked.","core_discovery":"This document is an evaluation plan rather than a research result. Its central claim is that a coordinated pair of tasks—generating spoofed speech and detecting it—can be run on in-the-wild datasets under a single protocol. The TTS track fixes the training data (TITW), the evaluation lists (TITW-KSKT and TITW-KSUT), and five metrics computed by a single scoring tool, with no participant ranking. The SASV track fixes the SpoofCeleb training data, a three-class trial list (target, non-target, spoof), and the a-DCF metric with priors and costs specified exactly: π_tar = 0.9405, π_non = 0.0095, π_spf = 0.05, and C_miss = 1, C_fa = 10, C_fa,spoof = 10. The plan's feasibility depends on evaluation","pith_inferences":["The paired-track design implicitly assumes that better TTS systems produce harder spoofs; if that coupling holds, improvements in generation quality should translate into more challenging detection trials in future iterations.","Because the same participants cannot enter both tracks, the challenge does not directly measure how well a single team's TTS fools a single team's detector; the organizers' summary may need to cross-evaluate submissions to test that coupling.","The plan's TTS metrics cover intelligibility, naturalness, and speaker similarity but not human listening; adding a small human evaluation could ground the neural metrics in perception.","The dependency on materials 'to be released' means the protocol could execute even without a TTS ranking, but a missing a-DCF reference implementation in the evaluation package would block the SASV track entirely."],"forward_implications":["If the plan executes, TTS participants produce 9,113 and 8,000 WAV files for the two evaluation conditions, forming a new in-the-wild TTS evaluation corpus.","The SASV track will produce a public score set over target, non-target, and spoof trials, creating a common benchmark for spoofing-robust speaker verification on wild audio.","Because participants must choose one track, the challenge generates independent solutions from each community rather than a single joint system.","The absence of a TTS ranking means the public outcome will be a summary rather than a leaderboard, reducing competitive pressure to game the metrics.","The fixed priors and costs of a-DCF make the SASV results directly comparable to future challenges that adopt the same metric."],"supporting_citations":[{"why":"Supplies the TITW training and evaluation data used by the TTS track.","marker":"[1]"},{"why":"Supplies the SpoofCeleb training data and the in-the-wild evaluation design for the SASV track.","marker":"[2]"},{"why":"Defines the a-DCF metric with the exact priors and costs that the SASV track adopts.","marker":"[3]"},{"why":"Provides the no-public-comparison rule that the SASV track incorporates.","marker":"[4]"},{"why":"The acoustic model component of the TTS track baseline system.","marker":"[5]"},{"why":"The vocoder component of the TTS track baseline system.","marker":"[6]"},{"why":"The end-to-end integrated system used as the SASV track baseline.","marker":"[7]"}],"fun_headline_variants":["WildSpoof challenge pairs TTS with spoof detection","Generate and detect spoofed speech on in-the-wild data","Two tracks, one protocol: TTS and anti-spoofing in the wild","In-the-wild challenge for TTS and spoof-aware verification","WildSpoof: TTS synthesis meets spoof detection in real world"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The challenge can only run if the evaluation trial lists, waveform packages, and scoring scripts described as 'to be released' are actually distributed before the submission deadline, and if the designated scoring tool computes the TTS metrics as described.","fun_headline_variants_meta":{"raw":{"variants":["WildSpoof challenge pairs TTS with spoof detection","Generate and detect spoofed speech on in-the-wild data","Two tracks, one protocol: TTS and anti-spoofing in the wild","In-the-wild challenge for TTS and spoof-aware verification","WildSpoof: TTS synthesis meets spoof detection in real world"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0002,"raw_usage":{"total_tokens":1207,"prompt_tokens":734,"completion_tokens":473,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":478,"completion_tokens_details":{"reasoning_tokens":378}},"tokens_in":478,"tokens_out":473,"duration_ms":5262,"temperature":1.0,"reasoning_tokens":378,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T17:07:59.603539+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"If, by the announced data-release date, the TITW-KSKT and TITW-KSUT trial lists or the SASV evaluation waveform package are not distributed, or the reference a-DCF implementation is absent from the evaluation package, the plan cannot be executed as written. A simpler check: run the provided baseline systems and confirm the metric code produces finite scores on the released trial lists.","supporting_citations":[{"cited_title":"Text-to-speech synthesis in the wild,","cited_arxiv_id":null,"evidence_quote":"Supplies the TITW training and evaluation data used by the TTS track."},{"cited_title":"Spoofceleb: Speech deepfake detection and sasv in the wild,","cited_arxiv_id":null,"evidence_quote":"Supplies the SpoofCeleb training data and the in-the-wild evaluation design for the SASV track."},{"cited_title":"Asvspoof 5 evaluation plan,","cited_arxiv_id":null,"evidence_quote":"Provides the no-public-comparison rule that the SASV track incorporates."},{"cited_title":"Grad-tts: A diffusion probabilistic model for text-to-speech,","cited_arxiv_id":null,"evidence_quote":"The acoustic model component of the TTS track baseline system."},{"cited_title":"Diffwave: A versatile diffusion model for audio synthesis,","cited_arxiv_id":null,"evidence_quote":"The vocoder component of the TTS track baseline system."},{"cited_title":"Towards single integrated spoofing-aware speaker veri- fication embeddings,","cited_arxiv_id":null,"evidence_quote":"The end-to-end integrated system used as the SASV track baseline."}],"review_version":1}