{"id":"64414434-8991-40f5-b8b8-7572ead77d0f","arxiv_id":"2505.24291","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A zero-shot voice conversion model that uses discrete prosody tokens and a mask transformer to control speaking style while converting voice.","lead":"Discl-VC is a voice conversion system that separates speech into content, prosody, and timbre, then re-synthesizes a target speaker's voice while controlling the speaking style. It shows a modest edge over two baselines in quality and prosody similarity, though it trails one baseline on word error rate.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim depends on VQ prosody tokens being content-independent, but no objective enforces this and the only evidence is qualitative; prosody transfer may corrupt content.","rationale":"The reader's conditional verdict is appropriate. I identify a slightly more specific failure mode than the reader's stated weakest assumption: not just speaker leakage into prosody tokens, but content leakage caused by the absence of any content-independence objective in the VQ prosody encoder. The reader cited the w/o ContentVec ablation as evidence of residual speaker information; I argue the same architectural incentive could also encode content, since ContentVec contains content and prosody jointly and reconstruction pressure favors preserving content cues. This concern is load-bearing because the prosody mask transformer's entire controllable-conversion mechanism assumes that source content can be combined with reference prosody tokens cleanly. If those tokens carry source content, the generated speech may retain wrong content or blend content from the reference. The proposed probe directly measures whether prosody tokens predict content; it would settle the concern without relying on subjective or downstream metrics. If the probe finds leakage, the paper's interpretation of the Table 2 gains would need major revision, but the current evidence does not yet justify rejecting the work. The conditional verdict stands, pending this specific verification.","tokens_in":8649,"tokens_out":5393,"duration_ms":75681,"concrete_test":"Take 100 held-out VCTK utterances not used in the main tables; extract prosody tokens p and HuBERT content tokens c. Train a lightweight probe (e.g., a 1-layer BiLSTM) to predict the content token sequence c from p only, with a shuffled utterance-level p-to-c mapping as chance. If probe accuracy is significantly above chance, the prosody tokens carry content information, and the central disentanglement assumption is violated. A task-level complement: in the prosody-converted setup, hold source content fixed and vary the reference audio's content while keeping its prosody; if output WER changes systematically with reference content, content has leaked through the prosody path.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim of controllable zero-shot voice conversion rests on the assumption that the discrete prosody tokens from the VQ Prosody Encoder carry prosody only, with no content or speaker information. The stage-1 objective (Eq. 3) is reconstruction-based: L_stage1 = L_Dur + L_SimVQ + L_FMT + L_F0. Nothing in this objective penalizes mutual information between prosody tokens and content tokens. ContentVec is described as having 'already disentangled speaker information,' but ContentVec is designed primarily to remove speaker identity, not to separate content from prosody; its representation contains 'almost all of the content and prosody information.' The only bottleneck separating content from prosody is the VQ codebook (size 2048), optimized for reconstruction. To help the flow matching transformer reconstruct the mel spectrogram from HuBERT content tokens plus prosody tokens, the VQ encoder is incentivized to preserve residual phonetic content inside the prosody tokens, especially because HuBERT kmeans tokens are lossy and duration-deduplicated. The paper's evidence of disentanglement is Figure 3, a qualitative mel-spectrogram with either prosody or content tokens removed, plus one ablation replacing ContentVec with first-20 mel dims. That ablation does not establish content-independence: the drop when using mel dims could simply reflect poorer prosody capture. The paper even notes that in the mel-based ablation 'the extracted prosody tokens inevitably still contain some speaker information,' but it never runs the corresponding content-leakage probe for the ContentVec pipeline. If prosody tokens encode content, then the prosody mask transformer's cross-utterance inference (source content tokens + reference prosody prompt) will not transfer style cleanly; the elevated WER in the prosody-converted condition (4.139 in Table 2) could reflect content corruption rather than expressive prosody transfer.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Discl-VC, a zero-shot voice conversion system that factorizes speech into discrete content tokens (HuBERT k-means), discrete prosody tokens (a trained VQ encoder on ContentVec features), and timbre via a flow-matching transformer with in-context learning. A non-autoregressive mask generative transformer predicts prosody tokens from a reference prompt to enable prosody conversion. The system is trained in two stages and evaluated on VCTK and ESD against FAcodec and Vevo. The paper claims superior zero-shot voice conversion performance and accurate prosody control.","tokens_in":9117,"tokens_out":4608,"duration_ms":52537,"significance":"The proposed architecture is well motivated, and the two-stage token-prediction scheme is a sensible way to add prosody controllability to a zero-shot voice conversion pipeline. If the empirical results were fully substantiated, a 131M-parameter model with state-of-the-art naturalness (N-MOS 4.31, UTMOS 4.079) and competitive speaker similarity (SECS 0.929) would be a useful contribution. The public demo and comparisons to two strong baselines are valuable. However, the current evidence is mixed: WER is worse than FAcodec, SECS is lower than Vevo in the prosody-converted setting, and no significance tests are given for objective metrics.","major_comments":[{"comment":"The central claim of 'superior performance' in zero-shot voice conversion is not supported by the reported WER: Discl-VC achieves 1.946, clearly worse than FAcodec's 1.341. Since WER is the primary content-preservation metric, the model is not uniformly superior; the statement in §3.2 that 'the WER performance falls between the two baselines, demonstrating the superiority' does not follow. Either temper the claim to a trade-off or show that the WER difference is within noise after statistical testing.","section":"Abstract, §3.2 (Table 1)"},{"comment":"None of the objective metrics (UTMOS, WER, F0Corr, SECS) is reported with error bars or significance tests. Several decisive comparisons rest on small differences (e.g., F0Corr 0.973 vs 0.968 in Table 1; SECS 0.847 vs 0.892 in the prosody-converted row of Table 2; UTMOS 4.119 vs 4.057). Since these are single-number estimates, the reported advantage of Discl-VC over Vevo and FAcodec on several metrics could be within run-to-run variability. The authors should provide variance estimates (e.g., bootstrap CIs or multiple evaluation seeds) and, if feasible, significance tests.","section":"§3.2, Tables 1 and 2"},{"comment":"The disentanglement claim—that the VQ prosody tokens carry prosody but not content or speaker information—is load-bearing for the prosody-control application, yet it is never directly measured. The stage-1 objective in Eq. (3) is reconstruction-based (duration, SimVQ, flow matching, F0); nothing penalizes mutual information between prosody tokens and content tokens, and the codebook is optimized for reconstruction, which can incentivize residual phonetic content in prosody tokens. The evidence provided (Figure 3 spectrogram and the w/o ContentVec ablation) is qualitative or indirect. I recommend adding a quantitative probe: train a linear classifier on the prosody tokens to predict either the content token cluster or the speaker identity, and report accuracy versus chance; and/or measure WER when prosody tokens are exchanged between utterances with different content but the content tokens are held fixed.","section":"§2.3 (Eq. 3) and §3.3"}],"minor_comments":[{"comment":"The phrase 'ground true prosody tokens' should be 'ground truth prosody tokens'.","section":"§2.3"},{"comment":"The sentence 'We use the pre-trained RMVPE to extract F0' would benefit from a citation to the RMVPE paper.","section":"§3.1.2"},{"comment":"The 'w/o F0 loss' row shows a lower WER (1.279) than the full model (1.946); this is worth discussing, since it suggests a possible trade-off between content preservation and prosody modeling.","section":"Table 3"},{"comment":"In the SimVQ description, the codebook vectors are randomly initialized and never updated, with a learned linear layer W producing the quantized result; this choice should be explained more clearly, in particular how the codebook is selected and why this avoids collapse.","section":"§2.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a typical speech-conference submission. The main risk is overclaiming: the abstract and conclusion state superiority despite mixed metrics. If the authors add statistical tests and a direct disentanglement probe, the contribution is publishable. No concerns about novelty overlap beyond the usual; the system is a combination of known components (SimVQ, flow matching, mask token prediction) but the application to prosody-controlled zero-shot voice conversion is novel enough."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: Discl-VC is a genuinely assembled system, not a toy. It reuses HuBERT/kmeans content tokens, ContentVec, a VQ prosody encoder, flow matching and a mask transformer for prosody control. The new bit is the non-autoregressive prosody mask transformer that predicts VQ prosody tokens from a reference prompt, which is a legitimate application of mask-and-predict to controllable VC. The paper is honest about its ablations and compares against two recent baselines (FAcodec, Vevo) on standard datasets.\n\nWhat it does well: the two-stage training is clean, the ablations show each component matters. Subjective MOS is competitive and the demo makes the prosody transfer concrete. I particularly like the explicit F0 supervision with Z-score normalization to strip speaker information. That shows the authors thought about the disentanglement problem.\n\nSoft spots, in proportion. The reader's report and the stress-test note are right that the core disentanglement claim—that the VQ prosody tokens carry no content—is never directly measured. Equation (3)'s reconstruction objective has no mutual-information penalty between prosody and content tokens, and HuBERT kmeans tokens are lossy, so the VQ encoder has an incentive to stash phonetic residue in prosody tokens. The only evidence is Figure 3's qualitative spectrogram and the mel-dims ablation, which doesn't actually probe content leakage. That said, this is not a fatal flaw: the system still works better than Vevo on several metrics, and the elevated WER in prosody-converted condition could be style transfer rather than content corruption. But it does mean the headline claim of disentanglement is stronger than the evidence supports.\n\nOther issues: no significance tests on objective metrics, only N-MOS has confidence intervals. No code release (demo only). Two baselines is thin but acceptable for a conference paper. Minor inconsistency: the parameter count for Discl-VC changes from 131M in Table 1 to 269M in Table 2's prosody-converted block; that needs a footnote.\n\nWho is this for: speech/audio researchers working on zero-shot VC and prosody control. It's a useful system paper, maybe not a landmark. Deserves a serious referee, but the referee should ask for content-leakage probes (e.g., speaker/content probe on prosody tokens, or measuring WER under content-only reconstruction) and significance tests.\n\nRecommendation: send to peer review. Conditional acceptance with revisions, not desk reject. I wouldn't cite it without code, but I wouldn't mind seeing it in a reading group.","headline":"A credible controllable VC system whose prosody-disentanglement claim needs direct verification; worth peer review.","tokens_in":9566,"tokens_out":3004,"would_cite":false,"duration_ms":34945,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Discl-VC splits speech into discrete content tokens, prosody tokens, and an in-context timbre prompt, making zero-shot voice conversion controllable in both who speaks and how.","keywords":["zero-shot voice conversion","in-context learning","speech disentanglement","prosody control","flow matching","masked generative transformer","vector quantization","discrete speech tokens"],"falsifier":"A direct test would be to train a speaker classifier on prosody tokens alone, extracted from many speakers' audio without any content or timbre input: if that classifier performs far above chance, the tokens are not speaker-independent and the disentanglement claim weakens. A more behavioral falsifier: hold content fixed, generate the same sentence with prosody tokens taken from several different reference speakers, and measure the speaker-embedding similarity of the outputs to the target voice; if the similarity shifts with the prosody source, prosody is leaking timbre.","tokens_in":8499,"feed_emoji":"🎙️","tokens_out":10560,"duration_ms":119629,"temperature":0.7,"pith_summary":"Discl-VC is a voice-conversion system built on the idea that speech can be separated into three re-combinable streams: content tokens, prosody tokens, and an in-context timbre prompt. The paper argues that once content and prosody are discretized separately, the target speaker's timbre can be rendered by a flow-matching transformer that learns from surrounding context, while a non-autoregressive mask transformer can regenerate the prosody tokens to imitate a reference clip's speaking style. A single frozen pipeline can then convert any utterance to an unseen speaker's voice and independently control how it is spoken, without per-speaker fine-tuning. In the reported comparisons, the system attains the best naturalness score (N-MOS 4.31) and UTMOS (4.079) among the tested systems, together with the smallest parameter count.","feed_headline":"Two token streams split speech into words, style, and voice","feed_subtitle":"New system converts any utterance to an unseen voice while copying a reference clip's prosody.","key_machinery":"The machinery has two token streams and two non-autoregressive transformers. Content tokens are obtained by clustering the hidden units of a self-supervised speech model into 1024 classes and removing adjacent duplicates, which strips duration-related prosody. Prosody tokens are obtained by passing a speaker-disentangled continuous representation through an inverted length regulator and a vector-quantization bottleneck trained with a SimVQ-style loss and an F0 regression loss; this bottleneck is meant to filter out content and timbre while retaining expressive variation. The flow-matching transformer predicts masked mel-spectrogram regions from the content tokens, the prosody tokens, and the surrounding acoustic context, thereby learning timbre as an in-context task. The prosody mask transformer follows the mask-and-predict paradigm: given content tokens and a reference prompt's prosody tokens, it iteratively unmasks all prosody tokens in parallel, so that a reference speaker's style can be transplanted onto the source content.","core_discovery":"The paper's central claim is that disentangled discrete tokens give voice conversion a clean control surface. Content is encoded as deduplicated cluster units of a self-supervised speech model, prosody is encoded as vector-quantized codes taken from a speaker-disentangled continuous representation under an F0-supervision constraint, and timbre is never extracted at all: it is left for the flow-matching transformer to reconstruct by in-context learning from a short acoustic prompt. The prosody mask transformer then predicts what the prosody tokens of the source content should be when told to follow a reference speaker's style, and the flow-matching transformer renders those tokens in the target voice. The authors report that this two-stage, fully non-autoregressive design beats the comparison systems on naturalness, UTMOS, and F0 correlation in zero-shot voice conversion, and enables both prosody-preserving and prosody-converted generation.","pith_inferences":["The paper leaves the speaker-independence of prosody tokens unmeasured; an explicit probe of that property would either confirm the central control claim or reveal a ceiling on style transfer accuracy.","The same content-token/prosody-token split could be lifted into text-to-speech, expressive dubbing, and emotion conversion, where content and style are likewise separable and a single reference clip should suffice as a style prompt.","If the prosody tokens are truly content-free, they become an editable layer: one could modify a single pitch or rhythm token and re-render the utterance without re-synthesizing from text, enabling fine-grained prosody editing.","The classifier-free guidance drop probability is fixed at 0.2 in both transformers; scanning this value would likely reveal a naturalness-versus-style-fidelity trade-off that the paper does not explore."],"forward_implications":["A user can take one recorded sentence and re-speak it in an unseen target voice while instructing the model to copy the happy, sad, or neutral delivery of a different reference clip, all from the same trained checkpoint.","Because both transformers generate in parallel rather than token-by-token, controllable voice conversion can run without the latency of autoregressive synthesis.","The deduplicated content tokens make duration a predicted quantity, which opens the door to explicit control of speaking rate and rhythm.","The reported system uses 131M parameters in zero-shot voice conversion, compared with 922M for the larger baseline, so the control gains are not purchased with a larger model."],"supporting_citations":[{"why":"Supplies the speaker-disentangled continuous representation that the VQ prosody encoder discretizes into prosody tokens.","marker":"[7]"},{"why":"Supplies the self-supervised speech model whose hidden units are clustered into content tokens.","marker":"[29]"},{"why":"Provides the optimal-transport flow-matching objective that trains the acoustic generation transformer.","marker":"[19]"},{"why":"Establishes the mask-and-predict in-context-learning recipe for generating masked acoustic features with a flow-matching transformer.","marker":"[20]"},{"why":"Introduces the SimVQ objective that prevents codebook collapse in the prosody tokenizer.","marker":"[30]"},{"why":"Supplies the F0-supervised prosody modeling and factorized-codec design that the prosody encoder builds on.","marker":"[27]"},{"why":"Provides the masked-token-modeling training and iterative unmasking procedure used by the prosody mask transformer.","marker":"[28]"},{"why":"The main comparison system for controllable zero-shot voice conversion, serving as the baseline to beat.","marker":"[18]"}],"fun_headline_variants":["Disentangled tokens let voice conversion switch prosody at will","Zero-shot voice cloning with controllable prosody via discrete tokens","Two token streams split speech into words, style, and voice","In-context learning lets unseen speakers copy a reference prosody","Discl-VC: disentangle content and prosody for controllable conversion"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the prosody tokens encode speaking style without also hiding speaker identity or content, so that replacing them with tokens predicted from a reference clip changes only the way the words are spoken and not what is said or whose voice it is.","fun_headline_variants_meta":{"raw":{"variants":["Disentangled tokens let voice conversion switch prosody at will","Zero-shot voice cloning with controllable prosody via discrete tokens","Two token streams split speech into words, style, and voice","In-context learning lets unseen speakers copy a reference prosody","Discl-VC: disentangle content and prosody for controllable conversion"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000304,"raw_usage":{"total_tokens":1710,"prompt_tokens":872,"completion_tokens":838,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":488,"completion_tokens_details":{"reasoning_tokens":733}},"tokens_in":488,"tokens_out":838,"duration_ms":9395,"temperature":1.0,"reasoning_tokens":733,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:25:10.434618+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test would be to train a speaker classifier on prosody tokens alone, extracted from many speakers' audio without any content or timbre input: if that classifier performs far above chance, the tokens are not speaker-independent and the disentanglement claim weakens. A more behavioral falsifier: hold content fixed, generate the same sentence with prosody tokens taken from several different reference speakers, and measure the speaker-embedding similarity of the outputs to the target voice; if the similarity shifts with the prosody source, prosody is leaking timbre.","supporting_citations":[{"cited_title":"Sef-vc: Speaker embedding free zero-shot voice conversion with cross attention,","cited_arxiv_id":null,"evidence_quote":"Supplies the speaker-disentangled continuous representation that the VQ prosody encoder discretizes into prosody tokens."},{"cited_title":"Mega-TTS 2: Boosting prompting mechanisms for zero-shot speech synthesis,","cited_arxiv_id":null,"evidence_quote":"Provides the optimal-transport flow-matching objective that trains the acoustic generation transformer."},{"cited_title":"Enhancing expressive voice conversion with discrete pitch-conditioned flow matching model,","cited_arxiv_id":null,"evidence_quote":"Establishes the mask-and-predict in-context-learning recipe for generating masked acoustic features with a flow-matching transformer."},{"cited_title":"Reflow-tts: A rectified flow model for high-fidelity text-to- speech,","cited_arxiv_id":null,"evidence_quote":"Supplies the F0-supervised prosody modeling and factorized-codec design that the prosody encoder builds on."},{"cited_title":"Lafma: A latent flow matching model for text-to-audio generation,","cited_arxiv_id":null,"evidence_quote":"Provides the masked-token-modeling training and iterative unmasking procedure used by the prosody mask transformer."},{"cited_title":"Dddm-vc: Decoupled denoising diffusion models with disentangled representation and prior mixup for verified robust voice conversion,","cited_arxiv_id":null,"evidence_quote":"The main comparison system for controllable zero-shot voice conversion, serving as the baseline to beat."}],"review_version":1}