Pith. sign in

REVIEW 6 major objections 6 minor 59 references

FreeTalk:A plug-and-play and black-box defense against speech synthesis attacks

T0 review · 6 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A frequency-band noise mask can stop voice-cloning attacks without breaking speech transcription.

desk verdict A plausible frequency-domain voice-protection defense with a serious novelty conflict (overlapping prior work) and a transferability hole about encoder re-use inside TTS/VC models. read the letter →

arxiv 2509.00561 v1 pith:Y4JVMWGG submitted 2025-08-30 cs.CR

Yuwen Pu , Zhou Feng , Chunyi Zhou , Jiahao Chen , Chunqiang Hu , Haibo Hu , Shouling Ji This is my paper
classification cs.CR
keywords speechsynthesisattacksvoiceprivacyprotectionblack-boxdefensefrequency-domainperturbationtext-to-speechconversionspeakerverificationadversarial
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims to be the first plug-and-play, black-box defense against speech synthesis attacks. It adds an optimized noise pattern in the frequency domain to speech before it is uploaded, so human listeners and transcription systems still hear the words, but any attacker who feeds the protected audio to a text-to-speech or voice-conversion model produces synthetic speech that no speaker-verification system will match to the victim. On VoxCeleb1, across four cloning models and four verification models, the lowest mismatch rate exceeds 0.96 and the highest similarity is at most 0.15, while Whisper transcripts protected speech with zero character and word error. The method also learns one universal perturbation per speaker, applied frame-wise to recordings of any length, which cuts the per-utterance computational cost that limited earlier defenses. If the claim holds, voice assistants and social platforms could add a lightweight preprocessing step that protects voice identity without degrading speech services.

What carries the argument

The central mechanism is a learnable complex-noise mask in the short-time Fourier transform domain. Real and imaginary noise is initialized from a normal distribution and updated with Adam to maximize cosine distance between the original and protected speaker embeddings; a binary mask confines the perturbation to a middle frequency band (frequency-interval ratio alpha = 0.6) and to spectrogram bins above a minimum-magnitude threshold, and a 5-tap averaging filter smooths the noise. For identity-wise protection, one universal noise patch per speaker is learned with a PGN-inspired two-stage gradient interpolation and applied frame-wise with a random mask and a Wiener filter, which is what make

What would settle it

Take a zero-shot voice-cloning model trained to use the full spectrum or only the unperturbed low band (e.g., band-limited to 2 kHz), give it FreeTalk-protected VoxCeleb1 speech, and measure mismatch and similarity with ECAPA and CAM++. If the average similarity to the victim exceeds the reported roughly 0.15, or the mismatch rate drops below roughly 0.96, the claimed black-box transferability fails.

Watch

Extended reading notes

Core claim

FreeTalk claims that scrambling a speaker's identity in a selected middle frequency band of the spectrogram, optimized against a speaker-embedding extractor, is enough to make unknown cloning models fail. The noise is trained untargeted: it maximizes the cosine distance between the X-Vector embedding of the original audio and the embedding of the augmented, protected audio, while noise smoothing keeps the audio natural. After protection, four state-of-the-art TTS/VC models produce synthetic voices that four speaker-verification models no longer associate with the original speaker: on VoxCeleb1 the source-to-synthetic mismatch is at least 96% and the similarity at most 15%. The protected audi

Load-bearing premise

The defense assumes that pushing one speaker-recognition system's embedding away by scrambling a middle frequency band is enough to make every unknown cloning model produce a voice that other verification systems no longer match to the victim—and that the attacker never has an unmodified recording of the victim.

Editorial extensions

If this is right

  • If the claim holds, a voice-assistant platform can run FreeTalk before upload: users keep normal transcription (zero CER and WER on Whisper) while cloned speech fails speaker verification.
  • On VoxCeleb1, no combination of four cloning models and four verification models produces a synthetic voice that matches the victim: mismatch at least 0.96, similarity at most 0.15.
  • The identity-wise patch removes per-utterance optimization: one speaker-level perturbation protects recordings of any length, enabling deployment on resource-constrained devices.
  • The defense transfers across different speaker-verification models used as optimizers; cross-optimizer experiments keep the lowest mismatch above 0.72, supporting the black-box claim.
  • Both data augmentation and noise smoothing shape the protection-quality trade-off, and the method stays robust across noise levels and frequency intervals while preserving zero transcription error.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: The LibriSpeech results (lowest mismatch 0.67) suggest the defense is tuned to spontaneous, conversational audio; formal read speech may need a stronger noise budget or more per-speaker training samples.
  • Inference: Because the perturbation is confined to a middle frequency band, a cloning model that derives identity from the untouched low or high bands, or an adaptive attacker who retrains on protected samples, falls outside the paper's threat model and could plausibly bypass it.
  • Inference: The universal per-speaker patch is a natural candidate for a reusable 'voice cloak' across platforms; whether repeated public use weakens it is not tested.
  • Inference: The untargeted cosine loss only pushes embeddings away from the source; steering them toward a crowd of other speakers might harden transferability to unseen verification models.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 6 minor

Summary. The manuscript proposes FreeTalk, a frequency-domain perturbation defense against speech synthesis attacks. The method uses a pretrained speaker-verification (X-Vector) model as a proxy: it iteratively adds masked, smoothed complex noise to the STFT of an utterance so that the protected audio's speaker embedding is pushed away from the original, while aiming to preserve intelligibility. A second, identity-wise variant learns a universal perturbation patch that can be tiled over arbitrary-length utterances from the same speaker, with a frame-selection mask and a Wiener filter to reduce artifacts. The authors evaluate on LibriSpeech and VoxCeleb1 using five TTS/VC models (Glow-TTS, Speedy-Speech, Tacotron2-DCA, Tacotron2-DDC, YourTTS), four speaker-verification models (ECAPA, ResNet, ERes2Net, CAM++), and Whisper for ASR utility. The headline results on VoxCeleb1 are STCMR >= 0.96 and STCS <= 0.15 across synthesis/SV model combinations, with MOS around 3.2 and zero CER/WER. The paper also reports sensitivity analyses, ablations of augmentation and smoothing, and a transferability analysis of the identity-wise patch.

Significance. If the central transferability claim is valid, FreeTalk would be a practically relevant contribution: it is attack-agnostic, does not require querying the attack model, and the identity-wise patch has the potential to reduce per-utterance computational overhead. The paper is also commendable for including a relatively broad evaluation suite, a data augmentation strategy, and a noise-smoothing mechanism, and for reporting ASR utility metrics (CER/WER = 0) in addition to privacy metrics. However, the paper currently lacks any comparison to prior defenses, the evaluation may be confounded by the overlap between synthesis models' internal speaker encoders and the SV models used for evaluation, and the reported error bars are not statistically credible. These issues leave the central claim of black-box transferability to unknown synthesis models insufficiently supported, although the underlying idea is plausible and the identified problems appear addressable with additional experiments and analysis.

major comments (6)
  1. [Sec. V.B / Table I] No baseline comparison is provided. The introduction and Related Work (Sec. II.B) argue that existing methods suffer from low transferability, strong assumptions, and high computational overhead, yet the experiments evaluate only FreeTalk. Without reproducing or citing quantitative results for prior defenses such as AntiFake [3], VoiceGuard [39], Huang et al. [37], or Liu et al. [15] under the same threat model and metrics, the claimed advantage over the state of the art is not established. Please add comparative experiments or clearly reposition the paper as an evaluation-only study.
  2. [Sec. V.A.2 and Table I] The internal speaker-reference encoders of the five synthesis models are not disclosed. Several open-source zero-shot TTS/VC systems, including YourTTS, use pretrained speaker-verification encoders (e.g., ECAPA) to extract the reference speaker embedding. If any of the synthesis models relies on an encoder from the same family as the four SV models in Table I, then the corresponding STCMR values are not a held-out black-box transfer test: the perturbation displaces the exact representation that the synthesis model uses to condition generation. This would make the high STCMR an artifact of the evaluation setup rather than evidence of transfer to unknown attack models. Please (i) report the exact speaker encoder used by each synthesis model, (ii) evaluate at least one synthesis model whose reference encoder is architecturally unrelated (e.g., WavLM-TDNN), and (iii) directly measure how muc
  3. [Sec. V.A.5 and Algorithm 1] The evaluation metrics are closely aligned with the optimization objective. FreeTalk maximizes cosine distance between original and protected X-Vector embeddings (Sec. IV.A), while STCMR/STCS measure cosine similarity/mismatch between original and synthesized audio in SV embedding spaces. The four SV models in Table I are all from the SpeechBrain/3D-Speaker ecosystems and are conceptually similar to X-Vector. Thus the high STCMR may largely measure the success of the optimization, not holistic protection of speaker identity. Please add at least one truly held-out speaker representation model (e.g., a WavLM-based TDNN or a different framework) and, if the threat model claims deceiving human acquaintances, a human listening test or a proxy such as an open-set speaker identification task.
  4. [Tables I, II, and IV] All STCMR values are reported as x.xx±0.00. For a binary rate computed over a finite set of utterances, a zero standard error is not plausible; e.g., for STCMR = 0.96 on 100 test utterances the standard error would be about 0.02. The authors do not state the number of utterances per cell, whether the ± denotes standard deviation across repeated runs or standard error, or how the threshold for STCMR is chosen. This reporting makes the headline numbers difficult to trust. Please report raw counts, the number of trials, and confidence intervals or standard errors with meaningful precision.
  5. [Sec. III.B and Sec. III.A] The threat model assumes that the attacker obtains the victim's protected speech and uses it to synthesize. However, the paper also states that attackers may directly collect the victim's voice if they have prior acquaintance (Sec. III.A). If an attacker has even one unprotected recording of the victim, FreeTalk provides no protection, because the defense only perturbs the uploaded audio. This is a significant limitation that should be stated explicitly, and the paper should include an experiment where the attacker uses an unprotected reference utterance as input to TTS/VC, showing that the defense does not apply. Without this caveat, the practical claim that users can 'upload audio content freely and securely' is overstated.
  6. [Sec. V.E / Table IV] The identity-wise protection results contain clear failure cases: with frame length 120 and noise level 0.1, Test-STCMR for ECAPA is 0.13 and for CAM++ is 0.34, meaning the synthesized voice is often matched to the victim. The text acknowledges these lower values but still concludes that most results are above 0.8. Since the identity-wise method is one of the paper's three central contributions, these parameter-dependent failures need a more careful analysis, including whether such parameter settings are recommended and why. A universal perturbation that fails for large frame lengths at low noise levels requires explicit guidance for practitioners.
minor comments (6)
  1. [Eq. (5) and Algorithm 2] The symbol δ is used both for the complex noise and for the interpolation weight in the PGN-style update. In Algorithm 2, the line '∇(i) ← ∇(i) + (1− δ) · ∇(1) + δ · ∇(2)' uses δ as a scalar while δ elsewhere denotes the complex noise. Please rename the interpolation weight (e.g., β).
  2. [Sec. IV.D / Algorithm 2] The sample-wise algorithm (Algorithm 1) restricts noise to a frequency band using fs/fe and the Tmask threshold, while the identity-wise AddFreqNoise in Algorithm 2 applies λ·δ to full frames without such band limitation or masking. It should be clarified whether the identity-wise variant intentionally drops these restrictions, and if so, why.
  3. [Sec. V.A.6] The text says 'We set the frequency interval to 0.6' and then calls it 'around the middle frequency range.' Given the formula fs = F·(1−α)/2 and fe = F·(1+α)/2, α=0.6 means the middle 60% of the frequency bins, not a 0.6 kHz interval. Please clarify the terminology.
  4. [Sec. V.B] The paragraph introducing the experiments says 'Four speech synthesis models and four speech verification models are used to evaluate,' but Table I contains five synthesis models. This is a factual inconsistency that should be corrected.
  5. [Sec. IV.D] There is an incomplete sentence in the text: 'we employ a PGN-inspired two-stage interpolation strategy to better what .' Please complete or remove this phrase.
  6. [General] The paper claims to be 'the first plug-and-play and black-box defense against speech synthesis attacks,' but the Related Work cites Enkidu [38], which also proposes universal frequential perturbation for audio privacy protection. Please clarify the distinction or temper the novelty claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the defense objective is aligned with the evaluation metric but not identical to it, because the metric is measured on synthesized audio from held-out TTS/VC models, not on the directly optimized waveform.

full rationale

FreeTalk optimizes a frequency-domain perturbation to maximize the cosine distance between original and protected X-Vector embeddings, while evaluating protection via STCMR/STCS on TTS/VC-synthesized audio using four different speaker-verification models. This is objective-aligned evaluation, not circular derivation: STCS/STCMR are computed on synthetic audio produced by five synthesis models, not on the protected waveform whose embedding was directly optimized. The two-step transfer (shifting the protected utterance embedding causes the synthesized clone to be far from the victim identity) is an empirical assumption with real failure modes, e.g., if a synthesis model uses a different speaker encoder or the attacker has other victim recordings; but it is not a mathematical tautology from the paper's equations. In Table II, when the same SV model is used for optimization and evaluation (e.g., the ECAPA diagonal), those entries partly reflect the training objective, but the paper also reports off-diagonal results, and the main Table I optimizes with X-Vector while evaluating with ECAPA/ResNet/ERes2Net/CAM++, none of which is the optimizer. Identity-wise experiments evaluate on held-out test utterances from the same speaker, which is genuine out-of-sample testing. There is no load-bearing self-citation, no imported uniqueness theorem, and no fitted parameter renamed as a prediction. Concerns about undisclosed internal TTS speaker encoders are external-validity risks, not circularity.

Assumptions & free parameters 15 free parameters · 6 assumptions · 0 invented entities

FreeTalk introduces no new physical or conceptual entities; it combines existing building blocks (STFT, speaker embeddings, adversarial perturbation, universal adversarial noise) into a new defense pipeline. The load-bearing assumptions are the transferability of embedding-space perturbation to unknown synthesis models and the restricted attacker access to unprotected voice samples. Many hyperparameters are chosen by hand or left undefined, which limits the claim's independence from author tuning.

free parameters (15)
  • Frequency interval ratio alpha = 0.6
    Controls the middle-frequency band [fs, fe] where noise is added; chosen to balance privacy and quality via sensitivity analysis.
  • Noise level lambda (sample-wise) = 0.05
    Controls perturbation magnitude; sensitivity analysis shows privacy scales with lambda but quality declines.
  • Noise level lambda (identity-wise) = 0.1 to 0.5
    Scaled noise range for the universal patch in Table IV; results are non-monotonic across lambda.
  • Frame length l (identity-wise) = 30 and 120
    Patch granularity in the frame-wise noise application; shorter frames give higher STCMR in the experiments.
  • Mask ratio r (identity-wise) = not reported
    Probability of keeping a frame in the random mask (Eq. 6); never specified for the reported experiments.
  • PGN interpolation weight delta (Eq. 5) = undefined
    The interpolation weight between first- and second-stage gradients is named delta, which collides with the noise variable and is never assigned a value.
  • PGN step size alpha (Eq. 5 context) = undefined
    Forward projection step for the PGN-inspired update is never specified.
  • Inner-loop noise scale epsilon (Algorithm 2) = undefined
    Neighborhood size for sampling delta-prime is undefined.
  • Optimization steps N = 100
    Fixed iteration count for both sample-wise and identity-wise optimization.
  • Learning rate = 0.1
    Adam learning rate for noise updates.
  • Smoothing kernel size K = 5
    Averaging kernel length for noise smoothing (Section IV.C).
  • Augmentation noise variance = 0.012 in Sec IV.B; 0.01 in Sec V.A.6
    Gaussian noise injection level; the paper is internally inconsistent about the value.
  • Time shift range = plus/minus 10 percent
    Random shift range k sampled as floor of U(-0.1L, 0.1L) in Section IV.B.
  • Volume scaling range = U(0.8, 1.2)
    Random amplitude scaling factor kappa in Section IV.B.
  • STFT configuration = FFT 1024, hop 512, Hann window
    Fixed time-frequency resolution for all experiments (Section V.A.6).
assumptions (6)
  • standard math STFT and ISTFT form an invertible pair for the chosen parameters, so frequency-domain noise maps to valid waveforms.
    Section IV.A relies on applying noise in the spectral domain and then inverting via ISTFT.
  • domain assumption Speaker identity is predominantly encoded in the middle frequency band selected by alpha=0.6.
    Section IV.A cites [45]-[47] for this claim, but the choice is not validated for the specific TTS and SV models used in the evaluation.
  • domain assumption Maximizing cosine distance between original and protected X-Vector embeddings transfers to held-out SV models and to TTS and VC outputs.
    This is the core mechanism; it is only tested empirically on a limited set of models, not derived.
  • domain assumption The attacker only has access to protected speech, not other recordings of the victim.
    Threat model in Section III.B states the defender processes all shared speech; the paper does not discuss bypass via prior acquaintance or other sources.
  • domain assumption The TTS and VC models consume the protected audio as reference features and cannot easily invert or remove the perturbation.
    Needed for the perturbation to affect synthesized voices; not explicitly tested.
  • ad hoc to paper The masking threshold Tmask = min(|s|[fs:fe]) restricts noise to significant spectral components without hurting intelligibility.
    Eq. 1; this design choice is not justified beyond the stated purpose and is specific to FreeTalk.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FreeTalk:A plug-and-play and black-box defense against speech synthesis attacks." pith.science (2026). https://pith.science/paper/Y4JVMWGG

@misc{pith2026250900561,
  author       = {Pith},
  title        = {Pith review of: FreeTalk:A plug-and-play and black-box defense against speech synthesis attacks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y4JVMWGG}},
  note         = {Machine review of arXiv:2509.00561}
}
read the original abstract

Recently, speech assistant and speech verification have been used in many fields, which brings much benefit and convenience for us. However, when we enjoy these speech applications, our speech may be collected by attackers for speech synthesis. For example, an attacker generates some inappropriate political opinions with the characteristic of the victim's voice by obtaining a piece of the victim's speech, which will greatly influence the victim's reputation. Specifically, with the appearance of some zero-shot voice conversion methods, the cost of speech synthesis attacks has been further reduced, which also brings greater challenges to user voice security and privacy. Some researchers have proposed the corresponding privacy-preserving methods. However, the existing approaches have some non-negligible drawbacks: low transferability and robustness, high computational overhead. These deficiencies seriously limit the existing method deployed in practical scenarios. Therefore, in this paper, we propose a lightweight, robust, plug-and-play privacy preservation method against speech synthesis attacks in a black-box setting. Our method generates and adds a frequency-domain perturbation to the original speech to achieve privacy protection and high speech quality. Then, we present a data augmentation strategy and noise smoothing mechanism to improve the robustness of the proposed method. Besides, to reduce the user's defense overhead, we also propose a novel identity-wise protection mechanism. It can generate a universal perturbation for one speaker and support privacy preservation for speech of any length. Finally, we conduct extensive experiments on 5 speech synthesis models, 5 speech verification models, 1 speech recognition model, and 2 datasets. The experimental results demonstrate that our method has satisfying privacy-preserving performance, high speech quality, and utility.

Figures

Figures reproduced from arXiv: 2509.00561 by the authors.

Figure 1
Figure 1. The overview of speech synthesis attack. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Threat model of FreeTalk. enhance the presented method. Furthermore, we present a novel identity-wise privacy protection method that can reduce computational overhead and support any length from the same speaker. Algorithm 1 Sample-wise Protection Input: Source waveform x; Frequency interval ratio α; Noise level λ; Number of iterations N; Output: Protected waveform xˆ function FREQUENCYBANDPROTECTION(x) e ← ExtractE… view at source ↗
Figure 3
Figure 3. Privacy-preserving performance of FreeTalk against YourTTS with different noise levels on LibriSpeech. 0.2 0.4 0.6 0.8 1.0 Frequency Interval 0.0 0.2 0.4 0.6 0.8 1.0 STCMR ( ) 0.2 0.4 0.6 0.8 1.0 Frequency Interval 0.0 0.2 0.4 0.6 0.8 1.0 STCS ( ) ECAPA ResNet ERes2Net CAM++ (a) Privacy Metric. 0.2 0.4 0.6 0.8 1.0 Frequency Interval 2.9 3.0 3.1 3.2 3.3 3.4 3.5 3.6 MOS ( ) 0.2 0.4 0.6 0.8 1.0 Frequency Interval 0.010… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Privacy-preserving performance of FreeTalk on different frequency intervals on YourTTS and LibriSpeech. As shown in [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Temporal and frequential domain comparison [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

59 extracted references · 48 canonical work pages

  1. [38]

    Enkidu: Universal frequential perturbation for real-time audio privacy protection against voice deepfakes,

    Z. Feng, J. Chen, C. Zhou, Y . Pu, Q. Li, T. Du, and S. Ji, “Enkidu: Universal frequential perturbation for real-time audio privacy protection against voice deepfakes,” arXiv preprint arXiv:2507.12932 , 2025

    work page arXiv 2025

  2. [3]

    Antifake: Using adversarial audio to prevent unauthorized speech synthesis,

    Z. Yu, S. Zhai, and N. Zhang, “Antifake: Using adversarial audio to prevent unauthorized speech synthesis,” inProceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security , 2023, pp. 460–474

    work page 2023

  3. [39]

    V oice guard: Protecting voice privacy with strong and imperceptible adversarial perturbation in the time domain

    J. Li, D. Ye, L. Tang, C. Chen, and S. Hu, “V oice guard: Protecting voice privacy with strong and imperceptible adversarial perturbation in the time domain.” in IJCAI, 2023, pp. 4812–4820

    work page 2023

  4. [37]

    Defending your voice: Adversarial attack on voice conversion,

    C.-y. Huang, Y . Y . Lin, H.-y. Lee, and L.-s. Lee, “Defending your voice: Adversarial attack on voice conversion,” in2021 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2021, pp. 552–559

    work page 2021

  5. [15]

    Protecting your voice from speech synthesis attacks,

    Z. Liu, Y . Zhang, and C. Miao, “Protecting your voice from speech synthesis attacks,” in Proceedings of the 39th Annual Computer Security Applications Conference, 2023, pp. 394–408

    work page 2023

  6. [1]

    Any-to-many voice conversion with location-relative sequence-to-sequence modeling,

    S. Liu, Y . Cao, D. Wang, X. Wu, X. Liu, and H. Meng, “Any-to-many voice conversion with location-relative sequence-to-sequence modeling,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 29, pp. 1717–1728, 2021

    work page 2021

  7. [2]

    Face-driven zero- shot voice conversion with memory-based face-voice alignment,

    Z.-Y . Sheng, Y . Ai, Y .-N. Chen, and Z.-H. Ling, “Face-driven zero- shot voice conversion with memory-based face-voice alignment,” in Proceedings of the 31st ACM International Conference on Multimedia , 2023, pp. 8443–8452

    work page 2023

  8. [4]

    Pmvc: Data augmentation-based prosody modeling for expressive voice conversion,

    Y . Deng, H. Tang, X. Zhang, J. Wang, N. Cheng, and J. Xiao, “Pmvc: Data augmentation-based prosody modeling for expressive voice conversion,” in Proceedings of the 31st ACM International Conference on Multimedia, 2023, pp. 184–192

    work page 2023

Show all 59 references
  1. [5]

    Empathetic speech synthesis and testing for healthcare robots,

    J. James, B. Balamurali, C. I. Watson, and B. MacDonald, “Empathetic speech synthesis and testing for healthcare robots,” International Journal of Social Robotics , vol. 13, no. 8, pp. 2119–2137, 2021

    2021

  2. [6]

    Direct speech-to-speech translation with a sequence-to-sequence model (2019),

    Y . Jia, R. J. Weiss, F. Biadsy, W. Macherey, M. Johnson, Z. Chen, and Y . Wu, “Direct speech-to-speech translation with a sequence-to-sequence model (2019),” arXiv preprint arXiv:1904.06037 , vol. 21, 1904

    2019 arXiv

  3. [7]

    Quality of life of patients after total laryngectomy: the struggle against stigmatization and social ex- clusion using speech synthesis,

    J. Mertl, E. ˇZ´aˇckov´a, and B. ˇRepov´a, “Quality of life of patients after total laryngectomy: the struggle against stigmatization and social ex- clusion using speech synthesis,” Disability and Rehabilitation: Assistive Technology, vol. 13, no. 4, pp. 342–352, 2018

    2018

  4. [8]

    Overcoming the language barrier with speech translation technology,

    S. Nakamura, “Overcoming the language barrier with speech translation technology,” Science & Technology Trends-Quarterly Review , vol. 31, 2009

    2009

  5. [9]

    Adver- sarial speech for voice privacy protection from personalized speech generation,

    S. Chen, L. Chen, J. Zhang, K. Lee, Z. Ling, and L. Dai, “Adver- sarial speech for voice privacy protection from personalized speech generation,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2024, pp. 11 411–11 415

    2024

  6. [10]

    A proactive and dual prevention mechanism against illegal song covers empowered by singing voice conversion,

    G. Chen, Y . Zhang, F. Song, T. Wang, X. Du, and Y . Liu, “A proactive and dual prevention mechanism against illegal song covers empowered by singing voice conversion,” arXiv preprint arXiv:2401.17133 , 2024

    2024 arXiv

  7. [11]

    Naturalspeech 2: Latent diffusion models are natural and zero- shot speech and singing synthesizers,

    K. Shen, Z. Ju, X. Tan, Y . Liu, Y . Leng, L. He, T. Qin, S. Zhao, and J. Bian, “Naturalspeech 2: Latent diffusion models are natural and zero- shot speech and singing synthesizers,” arXiv preprint arXiv:2304.09116, 2023

    2023 arXiv

  8. [12]

    Generspeech: Towards style transfer for generalizable out-of-domain text-to-speech,

    R. Huang, Y . Ren, J. Liu, C. Cui, and Z. Zhao, “Generspeech: Towards style transfer for generalizable out-of-domain text-to-speech,” Advances 13 in Neural Information Processing Systems , vol. 35, pp. 10 970–10 983, 2022

    2022

  9. [13]

    End-to-end zero-shot voice conversion with location-variable convolutions,

    W. Kang, M. Hasegawa-Johnson, and D. Roy, “End-to-end zero-shot voice conversion with location-variable convolutions,” arXiv preprint arXiv:2205.09784, 2022

    2022 arXiv

  10. [14]

    Fraudsters used ai to mimic ceo’s voice in unusual cybercrime case,

    C. Stupp, “Fraudsters used ai to mimic ceo’s voice in unusual cybercrime case,” The Wall Street Journal , vol. 30, no. 08, 2019

    2019

  11. [16]

    V oice conversion based on maximum-likelihood estimation of spectral parameter trajectory,

    T. Toda, A. W. Black, and K. Tokuda, “V oice conversion based on maximum-likelihood estimation of spectral parameter trajectory,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 15, no. 8, pp. 2222–2235, 2007

    2007

  12. [17]

    Robust processing techniques for voice conversion,

    O. Turk and L. M. Arslan, “Robust processing techniques for voice conversion,” Computer Speech & Language , vol. 20, no. 4, pp. 441– 467, 2006

    2006

  13. [18]

    V oice conversion using deep bidirectional long short-term memory based recurrent neural networks,

    L. Sun, S. Kang, K. Li, and H. Meng, “V oice conversion using deep bidirectional long short-term memory based recurrent neural networks,” in 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2015, pp. 4869–4873

    2015

  14. [19]

    V oice conversion with transformer network,

    R. Liu, X. Chen, and X. Wen, “V oice conversion with transformer network,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2020, pp. 7759–7759

    2020

  15. [20]

    V oice conver- sion using partial least squares regression,

    E. Helander, T. Virtanen, J. Nurminen, and M. Gabbouj, “V oice conver- sion using partial least squares regression,” IEEE Transactions on Audio, Speech, and Language Processing , vol. 18, no. 5, pp. 912–921, 2010

    2010

  16. [21]

    V oice conversion from non-parallel corpora using variational auto-encoder,

    C.-C. Hsu, H.-T. Hwang, Y .-C. Wu, Y . Tsao, and H.-M. Wang, “V oice conversion from non-parallel corpora using variational auto-encoder,” in 2016 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA) . IEEE, 2016, pp. 1–6

    2016

  17. [22]

    One-shot voice conversion by vector quan- tization,

    D.-Y . Wu and H.-y. Lee, “One-shot voice conversion by vector quan- tization,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2020, pp. 7734–7738

    2020

  18. [23]

    Non-parallel training in voice conversion using an adaptive restricted boltzmann machine,

    T. Nakashika, T. Takiguchi, and Y . Minami, “Non-parallel training in voice conversion using an adaptive restricted boltzmann machine,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 24, no. 11, pp. 2032–2045, 2016

  19. [24]

    Non-parallel voice conversion using variational autoencoders conditioned by phonetic pos- teriorgrams and d-vectors,

    Y . Saito, Y . Ijima, K. Nishida, and S. Takamichi, “Non-parallel voice conversion using variational autoencoders conditioned by phonetic pos- teriorgrams and d-vectors,” in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2018, pp...

    2018

  20. [25]

    Again-vc: A one- shot voice conversion using activation guidance and adaptive instance normalization,

    Y .-H. Chen, D.-Y . Wu, T.-H. Wu, and H.-y. Lee, “Again-vc: A one- shot voice conversion using activation guidance and adaptive instance normalization,” in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2021, pp. 5954–5958

    2021

  21. [26]

    Nvc-net: End-to-end adversarial voice conversion,

    B. Nguyen and F. Cardinaux, “Nvc-net: End-to-end adversarial voice conversion,” in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2022, pp. 7012–7016

    2022

  22. [27]

    Lm-vc: Zero-shot voice conversion via speech generation based on language models,

    Z. Wang, Y . Chen, L. Xie, Q. Tian, and Y . Wang, “Lm-vc: Zero-shot voice conversion via speech generation based on language models,” IEEE Signal Processing Letters , 2023

    2023

  23. [28]

    Transfer learning from speaker verification to multispeaker text-to-speech synthesis,

    Y . Jia, Y . Zhang, R. Weiss, Q. Wang, J. Shen, F. Ren, P. Nguyen, R. Pang, I. Lopez Moreno, Y . Wu et al. , “Transfer learning from speaker verification to multispeaker text-to-speech synthesis,” Advances in neural information processing systems , vol. 31, 2018

    2018

  24. [29]

    Fastspeech: Fast, robust and controllable text to speech,

    Y . Ren, Y . Ruan, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y . Liu, “Fastspeech: Fast, robust and controllable text to speech,” Advances in neural information processing systems , vol. 32, 2019

    2019

  25. [30]

    Neural text-to-speech adaptation from low quality public recordings,

    Q. Hu, E. Marchi, D. Winarsky, Y . Stylianou, D. Naik, and S. Kajarekar, “Neural text-to-speech adaptation from low quality public recordings,” in Speech Synthesis Workshop, vol. 10, 2019

    2019

  26. [31]

    Neural codec language models are zero-shot text to speech synthesizers,

    C. Wang, S. Chen, Y . Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y . Liu, H. Wang, J. Li et al., “Neural codec language models are zero-shot text to speech synthesizers,” arXiv preprint arXiv:2301.02111 , 2023

    2023 arXiv

  27. [32]

    V oid: A fast and light voice liveness detection system,

    M. E. Ahmed, I.-Y . Kwak, J. H. Huh, I. Kim, T. Oh, and H. Kim, “V oid: A fast and light voice liveness detection system,” in29th USENIX Security Symposium (USENIX Security 20) , 2020, pp. 2685–2702

    2020

  28. [33]

    Detecting ai-synthesized speech using bispectral analysis

    E. A. AlBadawy, S. Lyu, and H. Farid, “Detecting ai-synthesized speech using bispectral analysis.” in CVPR workshops, 2019, pp. 104–109

    2019

  29. [34]

    Who are you (i really wanna know)? detecting audio {DeepFakes} through vocal tract reconstruction,

    L. Blue, K. Warren, H. Abdullah, C. Gibson, L. Vargas, J. O’Dell, K. Butler, and P. Traynor, “Who are you (i really wanna know)? detecting audio {DeepFakes} through vocal tract reconstruction,” in 31st USENIX Security Symposium (USENIX Security 22) , 2022, pp. 2691– 2708

    2022

  30. [35]

    De- tecting voice cloning attacks via timbre watermarking,

    C. Liu, J. Zhang, T. Zhang, X. Yang, W. Zhang, and N. Yu, “De- tecting voice cloning attacks via timbre watermarking,” arXiv preprint arXiv:2312.03410, 2023

    2023 arXiv

  31. [36]

    Hiddenspeaker: Generate imperceptible unlearnable audios for speaker verification system,

    Z. Zhang and P. Huang, “Hiddenspeaker: Generate imperceptible unlearnable audios for speaker verification system,” arXiv preprint arXiv:2405.15655, 2024

    2024 arXiv

  32. [40]

    Active defense against voice conversion through generative adversarial network,

    S. Dong, B. Chen, K. Ma, and G. Zhao, “Active defense against voice conversion through generative adversarial network,” IEEE Signal Processing Letters, 2024

    2024

  33. [41]

    Wavefuzz: A clean-label poisoning attack to protect your voice,

    Y . Ge, Q. Wang, J. Zhang, J. Zhou, Y . Zhang, and C. Shen, “Wavefuzz: A clean-label poisoning attack to protect your voice,” arXiv preprint arXiv:2203.13497, 2022

    2022 arXiv

  34. [42]

    https://www.bbc.com/news/technology-60780142,

    “https://www.bbc.com/news/technology-60780142,” 2022

    2022

  35. [43]

    Privacy-preserving and privacy- attacking approaches for speech and audio–a survey,

    Y . Liu, A. Kapadia, and D. Williamson, “Privacy-preserving and privacy- attacking approaches for speech and audio–a survey,” arXiv preprint arXiv:2309.15087, 2023

    2023 arXiv

  36. [44]

    Personal voice assistant security and pri- vacy—a survey,

    P. Cheng and U. Roedig, “Personal voice assistant security and pri- vacy—a survey,”Proceedings of the IEEE, vol. 110, no. 4, pp. 476–507, 2022

    2022

  37. [45]

    Yourtts: Towards zero-shot multi-speaker tts and zero-shot voice conversion for everyone,

    E. Casanova, J. Weber, C. D. Shulby, A. C. Junior, E. G ¨olge, and M. A. Ponti, “Yourtts: Towards zero-shot multi-speaker tts and zero-shot voice conversion for everyone,” in International Conference on Machine Learning. PMLR, 2022, pp. 2709–2720

    2022

  38. [46]

    Glow-tts: A generative flow for text-to-speech via monotonic alignment search,

    J. Kim, S. Kim, J. Kong, and S. Yoon, “Glow-tts: A generative flow for text-to-speech via monotonic alignment search,” Advances in Neural Information Processing Systems , vol. 33, pp. 8067–8077, 2020

    2020

  39. [47]

    Tacotron: Towards end-to-end speech synthesis,

    Y . Wang, R. J. Skerry-Ryan, D. Stanton, Y . Wu, R. J. Weiss, N. Jaitly, Z. Yang, Y . Xiao, Z. Chen, S. Bengio, Q. V . Le, Y . Agiomyrgiannakis, R. Clark, and R. A. Saurous, “Tacotron: Towards end-to-end speech synthesis,” in Proc. Interspeech. ISCA, 2017, pp. 4006–4010

    2017

  40. [48]

    Boosting adversarial transferability by achieving flat local maxima,

    Z. Ge, X. Wang, H. Liu, F. Shang, and Y . Liu, “Boosting adversarial transferability by achieving flat local maxima,” in Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, D...

    2023

  41. [49]

    Librispeech: an asr corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an asr corpus based on public domain audio books,” in2015 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2015, pp. 5206–5210

    2015

  42. [50]

    V oxceleb: a large-scale speaker identification dataset,

    A. Nagrani, J. S. Chung, and A. Zisserman, “V oxceleb: a large-scale speaker identification dataset,” arXiv preprint arXiv:1706.08612 , 2017

    2017 arXiv

  43. [51]

    https://github.com/coqui-ai/tts,

    “https://github.com/coqui-ai/tts,” 2023

    2023

  44. [52]

    Zero-shot long-form voice cloning with dynamic convolution attention,

    A. Gorodetskii and I. Ozhiganov, “Zero-shot long-form voice cloning with dynamic convolution attention,” arXiv preprint arXiv:2201.10375 , 2022

    2022 arXiv

  45. [53]

    Speedyspeech: Efficient neural speech synthe- sis,

    J. Vainer and O. Du ˇsek, “Speedyspeech: Efficient neural speech synthe- sis,” arXiv preprint arXiv:2008.03802 , 2020

    2008 arXiv

  46. [54]

    Natural tts synthesis by conditioning wavenet on mel spectrogram predictions,

    J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y . Zhang, Y . Wang, R. Skerrv-Ryan et al. , “Natural tts synthesis by conditioning wavenet on mel spectrogram predictions,” in 2018 IEEE international conference on acoustics, speech and signal processin...

    2018

  47. [55]

    Speech- brain: A general-purpose speech toolkit,

    T. Parcollet, M. Ravanelli, P. Plantinga, A. Rouhe, S. Cornell, L. Lu- gosch, C. Subakan, N. Dawalatabad, A. Heba, J. Zhong et al., “Speech- brain: A general-purpose speech toolkit,” 2022

    2022

  48. [56]

    https://github.com/speechbrain/speechbrain,

    “https://github.com/speechbrain/speechbrain,” 2024

    2024

  49. [57]

    3d-speaker-toolkit: An open source toolkit for multi-modal speaker verification and diarization,

    Y . Chen, S. Zheng, H. Wang, L. Cheng, T. Zhu, C. Song, R. Huang, Z. Ma, Q. Chen, S. Zhang et al. , “3d-speaker-toolkit: An open source toolkit for multi-modal speaker verification and diarization,” arXiv preprint arXiv:2403.19971, 2024

    2024 arXiv

  50. [58]

    https://github.com/modelscope/3d-speaker,

    “https://github.com/modelscope/3d-speaker,” 2024

    2024

  51. [59]

    https://openai.com/index/whisper/,

    “https://openai.com/index/whisper/,” 2022

    2022

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.