Pith. sign in

REVIEW 3 major objections 3 minor 41 references

DiffDSR: Dysarthric Speech Reconstruction Using Latent Diffusion Model

T0 review · 3 major / 3 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read DiffDSR uses a latent diffusion model, an SSL content encoder, and an in-context speaker prompt to reconstruct dysarthric speech with higher intelligibility and speaker similarity than prior DSR systems on UASpeech.

desk verdict A useful new DSR system with credible subjective gains, but the objective speaker-similarity metric is circular because it shares the SV embedding used inside the model. read the letter →

arxiv 2506.00350 v1 pith:5SCMLEPX submitted 2025-05-31 cs.SD eess.AS

Xueyuan Chen , Dongchao Yang , Wenxuan Wu , Minglin Wu , Jing Xu , Xixin Wu , Zhiyong Wu , Helen Meng This is my paper
classification cs.SDeess.AS
keywords dysarthricspeechreconstructionlatentdiffusionmodelself-supervisedlearningspeakeridentitypreservationin-contextneuralaudiocodecintelligibilityUASpeech
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DiffDSR is built to answer one question: can a latent diffusion model turn severely dysarthric speech into clearly understandable speech while keeping the patient's voice? The paper proposes a three-part pipeline: a frozen self-supervised speech model restores a phoneme embedding, a codec-based encoder with in-context attention extracts a speaker prompt from the same utterance, and a diffusion denoiser generates neural-codec latents conditioned on both. On four of the most impaired UASpeech speakers, the paper reports that this pipeline lowers phoneme error rate, raises human listening accuracy, and improves speaker-similarity MOS relative to the SV-DSR and CoLM-DSR baselines. The authors also compare Wav2Vec 2.0, HuBERT, and WavLM front-ends and find WavLM best for content restoration.

What carries the argument

The load-bearing mechanism is the latent diffusion model over neural-codec latents, following the NaturalSpeech 2 SDE formulation: a forward process turns a codec latent into Gaussian noise, and a WaveNet-based reverse process denoises it under conditions. The content condition is a phoneme probability embedding produced by a frozen pre-trained SSL model plus trainable convolution and CTC layers; the speaker condition comes from a speaker identity encoder that tokenizes denoised dysarthric speech with EnCodec, maps each token to the nearest normal-speech token in speaker-verification embedding space, then processes the mapped token sequence through Transformer layers. Q-K-V attention layers insert the speaker prompt into the duration/pitch predictors and the WaveNet. The pre-trained EnCodec decoder renders the final audio.

What would settle it

Recalculate the speaker-similarity table with an independently trained speaker-embedding model or a same-different voice discrimination test; if DiffDSR no longer beats CoLM-DSR, the reported identity gain is an artifact of using the same embedding for normalization and evaluation.

Watch

Extended reading notes

Core claim

The paper's central claim is that latent diffusion is a viable backbone for dysarthric speech reconstruction. With a WavLM-based content encoder, the latent diffusion generator produces reconstructed speech that human listeners recognize more accurately than the original dysarthric utterance and more accurately than the VGG-based SV-DSR; on the most severe speaker, listening accuracy goes from 7.4% for the original to 34.6% for DiffDSR. For speaker identity, the complete DiffDSR scores higher mean opinion scores than both the mel-based SV-DSR and the codec-LM CoLM-DSR across all four speakers, and the ablation without the codec normalizer keeps comparable identity MOS but hurts prosody and sound quality.

Load-bearing premise

The codec normalizer is assumed to remove dysarthric acoustic details while preserving the speaker, because it replaces each dysarthric codec frame with the nearest normal frame in speaker-verification embedding space; the objective speaker-similarity score is computed in that same embedding space, so the assumption is not independently checked.

Editorial extensions

If this is right

  • WavLM gives the best content restoration of the three SSL front-ends tested, with the lowest phoneme error rate on all four speakers.
  • Listening accuracy for the most severely affected speaker rises from 7.4% for the original speech to 34.6% for DiffDSR with WavLM.
  • The complete DiffDSR reaches speaker-similarity MOS near 4.0 on the least affected speaker, above CoLM-DSR and SV-DSR.
  • The normalizer-ablation results show the codec normalizer matters for prosody and sound quality, not for the MOS identity score itself.
  • Because the speaker prompt and content condition come from the same utterance, DiffDSR can reconstruct speech for a new impaired speaker without retraining the diffusion generator.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the objective speaker-similarity metric uses the same speaker-verification embedding that the codec normalizer optimizes, part of Table 4's improvement may reflect self-consistency rather than genuine timbre preservation; the human MOS results are independent but do not isolate the normalizer's contribution to identity.
  • Editorial inference: the four test speakers are the most severely impaired and all data are isolated words from UASpeech, so it remains an open extension whether the same pipeline holds for milder dysarthria, continuous sentences, or non-English accented speech.
  • Editorial inference: a direct test of the normalizer would be to ask listeners to identify the patient after hearing speech reconstructed with and without codec normalization; if identity preservation is unaffected while intelligibility changes, the normalizer's role would need re-ascribing.
  • Editorial inference: the architecture could be reused for other atypical speech, such as post-stroke or Parkinsonian speech, because both the content and speaker prompts come from the same utterance and no text transcription is needed at sampling time.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper introduces DiffDSR, a dysarthric speech reconstruction system combining three components: an SSL-based content encoder that produces phoneme embeddings, a speaker identity encoder that maps dysarthric EnCodec tokens to a normal-speech codebook via a speaker-verification embedding and then applies in-context transformer layers, and a latent diffusion speech generator conditioned on the restored phoneme content, variance predictions, and the speaker prompt. Experiments on the UASpeech corpus compare three SSL front-ends against a VGG-based baseline for intelligibility, and compare speaker similarity against SV-DSR and CoLM-DSR using human listening tests, MOS, PER, and L1 distances in a speaker-verification embedding space. The paper claims notable improvements in both intelligibility and speaker similarity for the proposed system.

Significance. If the results hold, the paper offers a useful demonstration that a latent diffusion generator with an in-context speaker prompt can improve dysarthric speech reconstruction relative to an older VGG-based baseline and a codec-LM baseline, and the systematic comparison of Wav2Vec 2.0, HuBERT, and WavLM for content restoration is informative. The subjective listening tests and the public audio samples are valuable assets. However, the objective speaker-similarity evaluation is weakened by a circular design: the metric uses the same speaker-verification embedding that selects the speaker prompt in the codec normalizer. The intelligibility claim also lacks significance testing and a comparison against the more recent CoLM-DSR baseline. These issues do not invalidate the subjective results, but they do limit the strength of the stated conclusions.

major comments (3)
  1. [§2.2 (Eq. 1) and §3.3 (Table 4)] The objective speaker-similarity metric in Table 4 computes L1 distances in the same speaker-verification embedding space f_SV that is used in Eq. (1) to map each dysarthric codec frame to the nearest normal codec token. Because the speaker prompt fed to the diffusion generator is explicitly selected to minimize distance to the input in that exact embedding space, the reconstruction inherits a built-in advantage on this metric, and Table 4 does not independently validate speaker identity preservation. Baselines SV-DSR and CoLM-DSR do not receive this same selection bias, so the comparison is not fair. I recommend replacing this objective measure with one based on a speaker encoder not used anywhere in the system, or with a speaker-discrimination test, and re-examining whether the ordering in Table 4 persists.
  2. [§3.2 (Table 3) and §3.3 (Figure 2)] The subjective evaluations use 5 listeners for the HLT and 10 raters for the MOS, with no significance tests, confidence intervals for HLT, or inter-rater agreement statistics. Many reported differences are small; for example, Table 3 M12 gives 34.6% for Diff-DSR (WavLM) versus 33.3% for Diff-DSR (Wav2Vec), and Figure 2 M12 gives 3.75 for Diff-DSR versus 3.77 for Diff-DSR(ab). Without paired significance tests (e.g., bootstrap or Wilcoxon signed-rank over listener scores), the claim of notable improvements for each speaker is not statistically supported.
  3. [§3.2 (Tables 2 and 3)] The intelligibility comparison includes only the VGG-based SV-DSR baseline; it does not include the more recent CoLM-DSR or Unit-DSR systems, despite the paper motivating its contribution partly by limitations of codec-LM methods. Since CoLM-DSR is evaluated later for speaker similarity, the authors could readily report its intelligibility on the same test words. Without that comparison, the central intelligibility claim is demonstrated only relative to an older baseline, which weakens the contribution.
minor comments (3)
  1. [§2.2, Eq. (1)] Eq. (1) has a variable mismatch: the argmin is written over the dysarthric codec \hat z_p ∈ Z, but the objective contains both \hat z_p and \tilde z_p. The intended expression is likely \tilde z_p = argmin_{\tilde z_p ∈ Z} |f_SV(\hat z_p) − f_SV(\tilde z_p)|; please correct the notation.
  2. [§3.3] The description of the objective speaker-similarity measure is underspecified: it is not stated whether the L1 distance is computed on utterance-level embeddings or frame-level embeddings, or how the SV embedding is aggregated over time. Please clarify the exact computation for reproducibility.
  3. [General] There are several typographical artifacts in the references and affiliations, such as “V ocal Engineering” and “V oice banking”, and the sentence beginning “which have shown significant recognition performance” in Section 2.1 should be capitalized and merged with the preceding sentence.

Circularity Check

1 steps flagged · score 6.0 of 10

Objective speaker-similarity metric reuses the same speaker-verification embedding that the codec normalizer explicitly minimizes, so Table 4 is partly self-fulfilling.

  1. other [Section 2.2 Eq. (1) and Section 3.3 / Table 4]
    "Following [16], we employ a normalization operation to map the dysarthric codecs ˆzp into corresponding normal codecs ˜zp within a pre-prepared nomal codec setZ. This operation is facilitated by a speaker verification (SV) estimator [28] with parametersθ SV through the nearest L1 distance. It can be formulated as: ˆzp → ˜zp = arg min ˆzp∈Z |f SV (ˆzp;θ SV )−f SV (˜zp;θ SV )| (1) ... Besides, we also employ the speaker verification model [28] as an objective measure to evaluate the speaker similarity."

    The same SV model [28] is load-bearing in both the model construction and the objective evaluation. Eq. (1) selects the normal-codec speaker prompt by minimizing L1 distance in the f_SV embedding between the dysarthric codec and a normal codec; Section 3.3 then measures speaker similarity as L1 distance in that same embedding between original dysarthric speech and reconstruction. Because the reconstruction is generated from a prompt explicitly chosen to be f_SV-close to the dysarthric input, any reconstruction that preserves its prompt inherits a small distance on the evaluation metric by construction.

full rationale

The content-intelligibility evaluation (PER and HLT) is self-contained: SSL content encoders are externally pretrained, and intelligibility is measured against ground-truth phoneme sequences or human transcription, with no overlap between the training objective and the evaluation metric. The subjective speaker-similarity MOS is also independent. The one significant circular step is the overlap between the codec normalizer in Section 2.2 and the objective speaker-similarity metric in Section 3.3: Eq. (1) explicitly minimizes L1 distance in the speaker-verification embedding f_SV to select the normal-codec prompt, and Table 4 reports L1 distance in the same embedding as evidence of speaker similarity. This makes the objective identity-preservation result partly self-fulfilling. The paper's reuse of its own prior work [16] for the phoneme-embedding design and codec normalizer is ordinary component reuse and is not itself load-bearing circularity. The central DiffDSR contribution—latent diffusion generation with in-context speaker prompting—has independent content, so the score is moderate rather than extreme.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim rests on several domain assumptions about the pre-trained SSL, EnCodec, and the speaker-verification-based normalizer. The only hand-chosen number affecting the results is the per-speaker finetuning step count. No new invented entities are introduced.

free parameters (1)
  • Per-speaker content encoder finetuning steps = 2000 iterations
    Hand-chosen in Section 3.1; all four speaker-dependent DSR systems finetune the SSL-CTC content encoder for 2k steps on the target speaker, which affects the reported PER and intelligibility.
assumptions (4)
  • domain assumption Pre-trained SSL models (WavLM, HuBERT, Wav2Vec 2.0) extract phoneme-discriminative features from dysarthric speech.
    Section 2.1 relies on frozen SSL features carrying robust linguistic content; only the CTC layer is trained, so the phoneme embedding quality depends entirely on the SSL model.
  • domain assumption Nearest-neighbor mapping in speaker-verification embedding space removes dysarthric details while preserving speaker identity.
    Section 2.2, Eq. (1): the codec normalizer selects the closest normal codec using L1 distance in the SV space; this is the load-bearing identity-preservation premise, and it is evaluated with the same SV space in Section 3.3.
  • domain assumption The pre-prepared normal codec set Z is comprehensive enough that the nearest neighbor is a valid normal rendering of the same content.
    Section 2.2: the construction of Z is not described, but the normalization output and thus speaker similarity depend on it.
  • domain assumption A latent diffusion generator trained only on LibriTTS normal speech generalizes to phoneme conditions and speaker prompts derived from dysarthric speech.
    Section 3.1: the generator sees no dysarthric data during training; the final reconstruction quality depends on this distribution match.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DiffDSR: Dysarthric Speech Reconstruction Using Latent Diffusion Model." pith.science (2026). https://pith.science/paper/5SCMLEPX

@misc{pith2026250600350,
  author       = {Pith},
  title        = {Pith review of: DiffDSR: Dysarthric Speech Reconstruction Using Latent Diffusion Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5SCMLEPX}},
  note         = {Machine review of arXiv:2506.00350}
}
read the original abstract

Dysarthric speech reconstruction (DSR) aims to convert dysarthric speech into comprehensible speech while maintaining the speaker's identity. Despite significant advancements, existing methods often struggle with low speech intelligibility and poor speaker similarity. In this study, we introduce a novel diffusion-based DSR system that leverages a latent diffusion model to enhance the quality of speech reconstruction. Our model comprises: (i) a speech content encoder for phoneme embedding restoration via pre-trained self-supervised learning (SSL) speech foundation models; (ii) a speaker identity encoder for speaker-aware identity preservation by in-context learning mechanism; (iii) a diffusion-based speech generator to reconstruct the speech based on the restored phoneme embedding and preserved speaker identity. Through evaluations on the widely-used UASpeech corpus, our proposed model shows notable enhancements in speech intelligibility and speaker similarity.

Figures

Figures reproduced from arXiv: 2506.00350 by the authors.

Figure 1
Figure 1. Diagram of the proposed Diff-DSR system, where (a) shows the overall architecture, (b), (c) and (d) show the model details of speech content encoder, speaker identity encoder and in-context learning mechanism respectively. speaker-aware representation prompt. The contributions of this paper include: • We propose a diffusion-based DSR system that combines a speech content encoder, a speaker identity encoder and the l… view at source ↗
Figure 2
Figure 2. Comparison results of MOS with 95% confidence in terms of speaker similarity. achieves significant similarity improvements for all the 4 speak￾ers. What’s more, the Diff-DSR shows comparable or better re￾sults with Diff-DSR(ab) in speaker similarity but significantly enhanced prosody and sound quality in practice, indicating the necessity of codec normalizer in our proposed model. Besides, we also employ the speaker… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

41 extracted references · 25 canonical work pages

  1. [1]

    Introduction Dysarthria is a motor speech disorder characterized by difficulty in articulating words due to weak, paralyzed, or uncoordinated muscles used for speech [1]. It can result in a notable decline in speech quality and voice characteristics compared to typi- cal speech patterns [2] and significantly impedes communica- tion for dysarthria patients...

    work page Pith review arXiv 2025

  2. [2]

    It comprises a speech content encoder, a speaker identity encoder and a latent diffusion-based speech generator

    Methodology The overall architecture of our proposed Diff-DSR model is il- lustrated in Figure 1 (a). It comprises a speech content encoder, a speaker identity encoder and a latent diffusion-based speech generator. The speech content encoder strives to extract robust phoneme embedding from dysarthric speech input to provide content condition. The speaker ...

    work page

  3. [3]

    Datasets and Training Details Experiments are conducted on the UASpeech [33], LibriSpeech [34], VCTK [35] and LibriTTS [36] datasets

    Experiments 3.1. Datasets and Training Details Experiments are conducted on the UASpeech [33], LibriSpeech [34], VCTK [35] and LibriTTS [36] datasets. Among them, the UASpeech corpus is a benchmark disordered speech corpus, which is recorded by an 8-channel microphone array with some background noise. It comprises recordings from 19 dysarthria speakers wi...

    work page

  4. [4]

    We explore and compare three widely used SSL speech foundation mod- els for content restoration

    Conclusion This paper proposes to leverage the latent diffusion model to enhance dysarthric speech reconstruction results. We explore and compare three widely used SSL speech foundation mod- els for content restoration. Additionally, we adopt a speaker identity encoder with in-context learning mechanism to facili- tate speaker-aware identity preservation....

    work page

  5. [5]

    Hilvoice: Human-in-the-loop style selection for elder-facing speech synthe- sis,

    X. Chen, Q. Huang, X. Wu, Z. Wu, and H. Meng, “Hilvoice: Human-in-the-loop style selection for elder-facing speech synthe- sis,” inISCSLP. IEEE, 2022, pp. 86–90

    work page 2022

  6. [6]

    Speech motor control is task-specific: Evidence from dysarthria and apraxia of speech,

    W. Ziegler, “Speech motor control is task-specific: Evidence from dysarthria and apraxia of speech,”Aphasiology, vol. 17, no. 1, pp. 3–36, 2003

    work page 2003

  7. [7]

    Multiple factors are involved in the dysarthria asso- ciated with parkinson’s disease: a review with implications for clinical practice and research,

    S. Sapir, “Multiple factors are involved in the dysarthria asso- ciated with parkinson’s disease: a review with implications for clinical practice and research,”Journal of Speech, Language, and Hearing Research, vol. 57, no. 4, pp. 1330–1343, 2014

    work page 2014

  8. [8]

    D. B. Freed,Motor speech disorders: diagnosis and treatment. plural publishing, 2023

    work page 2023

Show all 41 references
  1. [9]

    Speech synthesis technologies for individuals with vocal disabilities: V oice banking and reconstruction,

    J. Yamagishi, C. Veaux, S. King, and S. Renals, “Speech synthesis technologies for individuals with vocal disabilities: V oice banking and reconstruction,”Acoustical Science and Technology, vol. 33, no. 1, pp. 1–5, 2012

    2012

  2. [10]

    Un- supervised multi-scale expressive speaking style modeling with hierarchical context information for audiobook speech synthesis,

    X. Chen, S. Lei, Z. Wu, D. Xu, W. Zhao, and H. Meng, “Un- supervised multi-scale expressive speaking style modeling with hierarchical context information for audiobook speech synthesis,” inCOLING, 2022, pp. 7193–7202

    2022

  3. [11]

    Improving the intelligibility of dysarthric speech towards enhancing the effectiveness of speech therapy,

    S. A. Kumar and C. S. Kumar, “Improving the intelligibility of dysarthric speech towards enhancing the effectiveness of speech therapy,” in2016 International Conference on Advances in Com- puting, Communications and Informatics (ICACCI). IEEE, 2016, pp. 1000–1005

    2016

  4. [12]

    Phoneme-discriminative features for dysarthric speech conversion

    R. Aihara, T. Takiguchi, and Y . Ariki, “Phoneme-discriminative features for dysarthric speech conversion.” inInterspeech, 2017, pp. 3374–3378

    2017

  5. [13]

    End-to-end voice conversion via cross-modal knowledge dis- tillation for dysarthric speech reconstruction,

    D. Wang, J. Yu, X. Wu, S. Liu, L. Sun, X. Liu, and H. Meng, “End-to-end voice conversion via cross-modal knowledge dis- tillation for dysarthric speech reconstruction,” inICASSP 2020. IEEE, 2020, pp. 7744–7748

    2020

  6. [14]

    Learning explicit prosody models and deep speaker embeddings for atypical voice conversion,

    D. Wang, S. Liu, L. Sun, X. Wu, X. Liu, and H. Meng, “Learning explicit prosody models and deep speaker embeddings for atypical voice conversion,”arXiv preprint arXiv:2011.01678, 2020

    2011 arXiv

  7. [15]

    Target speech extraction with pre-trained av-hubert and mask-and-recover strat- egy,

    W. Wu, X. Chen, X. Wu, H. Li, and H. Meng, “Target speech extraction with pre-trained av-hubert and mask-and-recover strat- egy,”arXiv preprint arXiv:2403.16078, 2024

    2024 arXiv

  8. [16]

    Stylespeech: Self-supervised style enhancing with vq- vae-based pre-training for expressive audiobook speech synthe- sis,

    X. Chen, X. Wang, S. Zhang, L. He, Z. Wu, X. Wu, and H. Meng, “Stylespeech: Self-supervised style enhancing with vq- vae-based pre-training for expressive audiobook speech synthe- sis,” inICASSP 2024. IEEE, 2024, pp. 12 316–12 320

    2024

  9. [17]

    Unit-dsr: Dysarthric speech reconstruction system using speech unit nor- malization,

    Y . Wang, X. Wu, D. Wang, L. Meng, and H. Meng, “Unit-dsr: Dysarthric speech reconstruction system using speech unit nor- malization,”arXiv preprint arXiv:2401.14664, 2024

    2024 arXiv

  10. [18]

    Hubert: Self-supervised speech represen- tation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhutdi- nov, and A. Mohamed, “Hubert: Self-supervised speech represen- tation learning by masked prediction of hidden units,”IEEE/ACM transactions on audio, speech, and language processing, vol. 29, pp. 3451–3460, 2021

    2021

  11. [19]

    Exploiting audio-visual features with pretrained av- hubert for multi-modal dysarthric speech reconstruction,

    X. Chen, Y . Wang, X. Wu, D. Wang, Z. Wu, X. Liu, and H. Meng, “Exploiting audio-visual features with pretrained av- hubert for multi-modal dysarthric speech reconstruction,”arXiv preprint arXiv:2401.17796, 2024

    2024 arXiv

  12. [20]

    Naturalspeech 3: Zero-shot speech syn- thesis with factorized codec and diffusion models,

    Z. Ju, Y . Wang, K. Shen, X. Tan, D. Xin, D. Yang, Y . Liu, Y . Leng, K. Song, S. Tanget al., “Naturalspeech 3: Zero-shot speech syn- thesis with factorized codec and diffusion models,”arXiv preprint arXiv:2403.03100, 2024

    2024 arXiv

  13. [21]

    Colm-dsr: Leveraging neural codec language modeling for multi-modal dysarthric speech reconstruction,

    X. Chen, D. Yang, D. Wang, X. Wu, Z. Wu, and H. Meng, “Colm-dsr: Leveraging neural codec language modeling for multi-modal dysarthric speech reconstruction,”arXiv preprint arXiv:2406.08336, 2024

    2024 arXiv

  14. [22]

    Neural codec language mod- els are zero-shot text to speech synthesizers,

    C. Wang, S. Chen, Y . Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y . Liu, H. Wang, J. Liet al., “Neural codec language mod- els are zero-shot text to speech synthesizers,”arXiv preprint arXiv:2301.02111, 2023

    2023 arXiv

  15. [23]

    Vall-e 2: Neural codec language models are hu- man parity zero-shot text to speech synthesizers,

    S. Chen, S. Liu, L. Zhou, Y . Liu, X. Tan, J. Li, S. Zhao, Y . Qian, and F. Wei, “Vall-e 2: Neural codec language models are hu- man parity zero-shot text to speech synthesizers,”arXiv preprint arXiv:2406.05370, 2024

    2024 arXiv

  16. [24]

    Naturalspeech 2: Latent diffusion models are natu- ral and zero-shot speech and singing synthesizers,

    K. Shen, Z. Ju, X. Tan, Y . Liu, Y . Leng, L. He, T. Qin, S. Zhao, and J. Bian, “Naturalspeech 2: Latent diffusion models are natu- ral and zero-shot speech and singing synthesizers,”arXiv preprint arXiv:2304.09116, 2023

    2023 arXiv

  17. [25]

    Wavlm: Large-scale self- supervised pre-training for full stack speech processing,

    S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiaoet al., “Wavlm: Large-scale self- supervised pre-training for full stack speech processing,”IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, 2022

    2022

  18. [26]

    Simplespeech: Towards simple and efficient text-to-speech with scalar latent transformer diffusion models,

    D. Yang, D. Wang, H. Guo, X. Chen, X. Wu, and H. Meng, “Simplespeech: Towards simple and efficient text-to-speech with scalar latent transformer diffusion models,”arXiv preprint arXiv:2406.02328, 2024

    2024 arXiv

  19. [27]

    Scalable diffusion models with transform- ers,

    W. Peebles and S. Xie, “Scalable diffusion models with transform- ers,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 4195–4205

    2023

  20. [28]

    The L1 distances between the dysarthric speeches and corre- sponding reconstructed speeches are calculated and results are shown in Table 4

    as an objective measure to evaluate the speaker similarity. The L1 distances between the dysarthric speeches and corre- sponding reconstructed speeches are calculated and results are shown in Table 4. Similar to the MOS findings, our proposed model also achieves the best resul...

  21. [29]

    Video gen- eration models as world simulators,

    T. Brooks, B. Peebles, C. Holmes, W. DePue, Y . Guo, L. Jing, D. Schnurr, J. Taylor, T. Luhman, E. Luhmanet al., “Video gen- eration models as world simulators,” 2024

    2024

  22. [30]

    wav2vec 2.0: A framework for self-supervised learning of speech repre- sentations,

    A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech repre- sentations,”Advances in neural information processing systems, vol. 33, pp. 12 449–12 460, 2020

    2020

  23. [31]

    Improving automatic speech recogni- tion performance for low-resource languages with self-supervised models,

    J. Zhao and W.-Q. Zhang, “Improving automatic speech recogni- tion performance for low-resource languages with self-supervised models,”IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1227–1241, 2022

    2022

  24. [32]

    High fidelity neural audio compression,

    A. D ´efossez, J. Copet, G. Synnaeve, and Y . Adi, “High fidelity neural audio compression,”arXiv preprint arXiv:2210.13438, 2022

    2022 arXiv

  25. [33]

    Generalized end- to-end loss for speaker verification,

    L. Wan, Q. Wang, A. Papir, and I. L. Moreno, “Generalized end- to-end loss for speaker verification,” inICASSP 2018. IEEE, 2018, pp. 4879–4883

    2018

  26. [34]

    Zsvc: Zero-shot style voice conversion with disentangled la- tent diffusion models and adversarial training,

    X. Zhu, L. He, Y . Xiao, X. Wang, X. Tan, S. Zhao, and L. Xie, “Zsvc: Zero-shot style voice conversion with disentangled la- tent diffusion models and adversarial training,”arXiv preprint arXiv:2501.04416, 2025

    2025 arXiv

  27. [35]

    Score-based generative modeling through stochas- tic differential equations,

    Y . Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, “Score-based generative modeling through stochas- tic differential equations,”arXiv preprint arXiv:2011.13456, 2020

    2011 arXiv

  28. [36]

    Wavenet: A generative model for raw audio,

    A. Van Den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, K. Kavukcuogluet al., “Wavenet: A generative model for raw audio,”arXiv preprint arXiv:1609.03499, vol. 12, 2016

    2016 arXiv

  29. [37]

    Film: Visual reasoning with a general conditioning layer,

    E. Perez, F. Strub, H. De Vries, V . Dumoulin, and A. Courville, “Film: Visual reasoning with a general conditioning layer,” in Proceedings of the AAAI conference on artificial intelligence, vol. 32, no. 1, 2018

    2018

  30. [38]

    Dysarthric speech database for universal access research,

    H. Kim, M. Hasegawa-Johnson, A. Perlman, J. Gunderson, T. S. Huang, K. Watkin, and S. Frame, “Dysarthric speech database for universal access research,” inNinth Annual Conference of the International Speech Communication Association, 2008

    2008

  31. [39]

    Lib- rispeech: an asr corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Lib- rispeech: an asr corpus based on public domain audio books,” in2015 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2015, pp. 5206–5210

    2015

  32. [40]

    Superseded-cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit,

    C. Veaux, J. Yamagishi, K. MacDonaldet al., “Superseded-cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit,” 2016

    2016

  33. [41]

    Libritts: A corpus derived from librispeech for text- to-speech,

    H. Zen, V . Dang, R. Clark, Y . Zhang, R. J. Weiss, Y . Jia, Z. Chen, and Y . Wu, “Libritts: A corpus derived from librispeech for text- to-speech,”Interspeech 2019, 2019

    2019

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.