REVIEW 3 major objections 4 minor 1 cited by
V2S attack: building DNN-based voice conversion from automatic speaker verification
T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read A leaked speaker-verification model can train a voice impersonator
desk verdict Novel idea for training VC without target spectral data, but the missing ASV attack metric and the hidden F0 leakage undercut the headline claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is a feed-forward voice-conversion network $G$ trained with the composite loss $L(x,\hat{y},l_y)=L_{\mathrm{SCE}}(l_y,V(\hat{y}))+\omega L_{\mathrm{MSE}}(R(x),R(\hat{y}))$, where $V$ is the white-boxed ASV model, $R$ the pretrained ASR model, $x$ the source features, $\hat{y}=G(x)$ the converted features, and $l_y$ the one-hot target speaker label. The first term backpropagates a softmax cross-entropy gradient through $V$ so $G$ learns spectral patterns the verifier attributes to the target; the second uses phonetic posteriorgrams from $R$ to keep the utterance's linguistic content unchanged. The ASV model is a d-vector-based classifier, a network whose frame-level posteriors encode speaker identity, and the ASR model outputs a 56-dimension phoneme posterior per frame.
What would settle it
Train the V2S converter against a white-box ASV model while leaving F0 unconverted or using the source speaker's F0, then test ASV acceptance and listener target identification; if accepting converted voices collapses to chance, the claim that spectral conversion alone reproduces the target speaker's individuality is falsified.
Extended reading notes
Core claim
The central claim is that speaker identity stored in a white-box ASV model is itself a usable training signal for voice conversion. The converter is trained to minimize $L = L_{\mathrm{SCE}}(l_y, V(\hat{y})) + \omega L_{\mathrm{MSE}}(R(x), R(\hat{y}))$, where $V$ is the ASV model, $l_y$ the one-hot target speaker label, $R$ a pretrained ASR model, $x$ the source speaker's acoustic features, and $\hat{y}$ the converted features. Minimizing the first term shapes the converted spectrum so the verifier attributes it to the target; minimizing the second preserves the phonetic content of the utterance. No parallel target speech is used to train the spectral VC model, and the reported preference tests show naturalness and speaker individuality comparable to parallel VC trained on a very small number of utterances, with the caveat that target F0 statistics are still taken from target voice data.
Load-bearing premise
The load-bearing premise is that a spectral envelope trained only to fool a frame-level ASV classifier, with phonetic content pinned by an ASR model, carries enough target identity to be usable, and as the paper admits the target's F0 statistics are still taken from the target's voice data, so the no-target-voice claim covers only the spectrum.
Editorial extensions
If this is right
- If a deployed ASV model leaks, an attacker can convert arbitrary source utterances toward an enrolled target's voice without a parallel corpus, making the secrecy of enrollment data a security boundary.
- The attack's effectiveness is tied to how much speaker identity the verifier's frame-level posteriors encode, so a more discriminative ASV model gives attackers a stronger training signal.
- Spoofing countermeasures must treat the verifier itself as a possible training oracle, not only as a scoring function to be evaded at run time.
- The reported parity with five-utterance parallel VC sets a low-data baseline for evaluating future impersonation defenses.
Reading between the lines
- Editorial inference: because the target's F0 statistics are still measured from target voice data, the no-target-voice result is narrower than the title suggests; a fully data-free attack would need a separate method for pitch transfer, and the experiments so far demonstrate only that the spectral envelope can be stolen from an ASV model.
- Editorial inference: testing the attack on speakers held out of ASV training would clarify whether the method generalizes to arbitrary enrollments or only to voices whose statistics the verifier has memorized.
- Editorial inference: the same two-loss recipe could be combined with an end-to-end differentiable vocoder or waveform generator, which would test whether the attack remains effective when the entire acoustic signal, not just the spectral envelope, is synthesized from the ASV gradient.
- Editorial inference: the attack suggests a concrete defense criterion: verifiers whose frame-level posteriors are differentiable and backpropagatable are more exposed, whereas non-differentiable or ensemble scoring would blunt this training signal.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces the 'verification-to-synthesis (V2S) attack,' in which a deep-neural-network voice conversion (VC) model is trained to impersonate an enrolled speaker without using that target speaker's acoustic features for the spectral model. The training objective in Eq. (4) combines a softmax cross-entropy term that deceives a white-boxed automatic speaker verification (ASV) model into classifying converted frames as the target speaker with an ASR-based mean-squared-error term that encourages the converted speech to preserve the phonetic content of the source utterance. Experiments compare the converted speech of the proposed method with one-to-one parallel VC trained on 5, 10, and 30 parallel utterances and with non-parallel d-vector VC, using AB preference tests for naturalness and XAB preference tests for speaker individuality. The results indicate that V2S is comparable to parallel VC trained on 5 utterances in some conditions, and the authors conclude that the method can synthesize voice with naturalness and individuality comparable to a very small-data parallel VC. The key caveat noted in the paper is that F0 statistics (mean and variance) are taken from the targeted speaker's voice data, so the claim of not using target voice data applies only to spectral-envelope conversion.
Significance. If the central claim were fully supported, the paper would demonstrate a practically important attack pathway: a leaked or hacked ASV model, combined with a generic ASR model, could allow an attacker to build a usable voice impersonation system for an enrolled speaker while needing almost no target-speaker speech data. The paper's strengths include a clearly stated threat model, a concrete training objective, baseline comparisons with p-values, and a candid discussion of its own limitations, including the F0 data requirement and the white-box assumption. However, the significance is currently limited because the paper never measures whether converted voices actually deceive the ASV system, and because the evaluation uses a single source speaker and target speakers whose labels were present in the ASV training data. The result is best read as a proof-of-concept that ASV-guided spectral conversion can produce voices perceived as similar to a target; the specific claim that this constitutes an impersonation attack against ASV remains unverified.
major comments (3)
- [Section 4.2, Tables 1-4; Eq. (3)] The stated purpose of the V2S attack is to impersonate a speaker in an automatic speaker verification system, and the training objective in Eq. (4) is explicitly designed to deceive the white-boxed ASV model. However, the evaluation reports only human AB/XAB preference tests on naturalness and speaker individuality. There is no false-acceptance rate, no ASV decision-score comparison, and not even a measurement of whether the ASV's target-speaker posterior increases on converted speech. Without such a metric, the central attack claim is unsupported: converted speech that sounds similar to human listeners may or may not shift an ASV system's verification decision, and conversely an ASV might be fooled by speech that human listeners do not find particularly similar. Adding an objective ASV evaluation with converted utterances is a straightforward extension and is load-bearing for the paper's title and abstract.
- [Section 4.1, 'For F0 conversion' paragraph] The paper explicitly states that 'the targeted speaker's F0 is not observed in the V2S attack. This problem is not solved in this paper. Therefore, we calculated F0 statistics (mean and variance) from the targeted speaker's voice data, and performed linear conversion of F0.' This is a direct use of target-speaker voice data, contradicting the abstract's claim of training 'without the targeted speaker's voice data.' Since F0 is a major carrier of speaker identity, this is not a minor implementation detail; the claim in the title and abstract should be refined to specify that the spectral model is built without target voice data, while the F0 component still depends on target statistics, or the method should be extended to estimate F0 statistics from the ASV/ASR models.
- [Section 4.1, experimental setup and Tables 1-4] The experimental design is too narrow to support the paper's general conclusion. Only one source speaker is used, and the four target speakers are explicitly drawn from the 260 speakers used to train the ASV model, because the attack is white-box. This means the evaluation cannot show whether the method generalizes to target speakers enrolled after ASV model training, which is the common enrollment scenario, nor whether results depend on the particular source speaker. Reporting per-target results or an additional condition with a held-out target (e.g., a new speaker added to the ASV system after training) would materially strengthen the central claim.
minor comments (4)
- [Section 3.1 and Section 3.4] There are several typographical errors: 'syth speaker' should be 'target speaker' in Section 3.1, and 'an white-boxed ASV model' should be 'a white-boxed ASV model' in Section 3.4.
- [Section 4.1, acoustic features] The description '39-dimensional (1st-through-39th) mel-cepstral coefficients and their delta features' is ambiguous as written; a 39-dimensional vector usually consists of 13 static, 13 delta, and 13 delta-delta coefficients, so the sentence should state exactly which coefficients are included rather than implying both 39 coefficients and separate delta features.
- [Section 4.2, 'The total number of listeners'] The sentence 'The total number of listeners was 2 (AB or XAB) × 2 (male-to-male or male-to-female) × 4 (reference methods) × 40 (listeners) = 640' appears to count listener judgments rather than unique listeners; if 40 listeners participated in each of the 16 conditions, the number of unique listeners is 40, not 640, and the text should be reworded accordingly.
- [Section 3.3, Eq. (4)] The observation that Eq. (4) does not include the target speaker's acoustic features y is correct for spectral features, but the sentence should also acknowledge that the later F0 conversion step does use target statistics, to avoid a misleading contrast with the Section 4.1 caveat.
Circularity Check
No significant circularity: the training loss is independent of the evaluation metric, and the F0 data dependency is a validity gap, not a circular reduction.
full rationale
The paper's derivation chain is not circular. The V2S training objective in Eq. (4) combines a softmax cross-entropy term that deceives a white-boxed ASV model and an MSE term that matches ASR phonetic posteriorgrams; neither term is fitted to the reported evaluation outcome. The central claim of comparable naturalness and speaker individuality is tested by external human preference tests (AB and XAB) against natural target voices, not by re-using the ASV posteriors that appear in the loss. No equation reduces to a fitted parameter renamed as a prediction, and no conclusion is forced by definition. The self-citations (Saito et al. [6], [9]) are related-work references and a baseline configuration, not load-bearing justification for the proposed method. The acknowledged use of the targeted speaker's F0 statistics in Section 4.1 is a data-dependency that weakens the 'without target voice data' claim, but it is not circular reasoning. The absence of an ASV false-acceptance metric is a validation gap in the attack claim, not a circular step. Under the stated hard rules, no specific reduction can be exhibited, so the circularity score is 0.
Assumptions & free parameters
free parameters (2)
- omega (loss weight) =
0.01
- Target speaker F0 statistics (mean and variance) =
Not reported (per-target mean and variance)
assumptions (4)
- domain assumption Attack has white-box access to the ASV model and the targeted speaker's label
- domain assumption ASV and ASR models were trained on the same 260-speaker corpus that includes the target speakers
- domain assumption Frame-level ASV posterior probabilities encode sufficient speaker identity to transfer to human perception after conversion
- domain assumption The ASR model's phonetic posteriorgrams are a faithful content representation
Cite this review
Pith. "Pith review of V2S attack: building DNN-based voice conversion from automatic speaker verification." pith.science (2026). https://pith.science/paper/RPQ5NUWD
@misc{pith2026190801454,
author = {Pith},
title = {Pith review of: V2S attack: building DNN-based voice conversion from automatic speaker verification},
year = {2026},
howpublished = {\url{https://pith.science/paper/RPQ5NUWD}},
note = {Machine review of arXiv:1908.01454}
}
read the original abstract
This paper presents a new voice impersonation attack using voice conversion (VC). Enrolling personal voices for automatic speaker verification (ASV) offers natural and flexible biometric authentication systems. Basically, the ASV systems do not include the users' voice data. However, if the ASV system is unexpectedly exposed and hacked by a malicious attacker, there is a risk that the attacker will use VC techniques to reproduce the enrolled user's voices. We name this the ``verification-to-synthesis (V2S) attack'' and propose VC training with the ASV and pre-trained automatic speech recognition (ASR) models and without the targeted speaker's voice data. The VC model reproduces the targeted speaker's individuality by deceiving the ASV model and restores phonetic property of an input voice by matching phonetic posteriorgrams predicted by the ASR model. The experimental evaluation compares converted voices between the proposed method that does not use the targeted speaker's voice data and the standard VC that uses the data. The experimental results demonstrate that the proposed method performs comparably to the existing VC methods that trained using a very small amount of parallel voice data.
Figures
Forward citations
Cited by 1 Pith paper
-
Evaluating Synthetic Command Attacks on Smart Voice Assistants
Unit-selection diphone synthesis from limited unrelated speech can produce Alexa commands that are recognized with 93.8% accuracy and often receive the highest speaker-similarity confidence, but the headline 30-second...
Reference graph
Works this paper leans on
-
[1]
Introduction Automatic speaker verification (ASV), which offers natural and flexible biometric authentication systems, has been active ly studied in recent decades [1, 2]. Because the ASV systems identify the speaker of the input voice without using other b io- metrics, they are preferred for use in keyword spotting [3] a nd voice search implemented in smar...
-
[2]
Building voice conversion using targeted speaker’s voice This section describes standard VC techniques using a targe ted speaker’s voice: parallel and non-parallel VC (shown in the up- per half of Fig. 1). They are references to evaluate the perfo r- mances of the proposed V2S attack. 2.1. One-to-one parallel VC [8] Let G(·) be a VC model (a.k.a., an acou...
-
[3]
V2S attack: building voice conversion without using targeted speaker’s voice This section proposes a novel voice impersonation attack named a V2S attack (shown in the lower half of Fig. 1). Unlike methods described in Section 2, the VC model is trained from a white-boxed ASV model but without the targeted speaker’s voice data. Besides the ASV model, we us...
-
[4]
Experimental evaluation 4.1. Experimental setup Acoustic features used in VC, ASV , and ASR were 39- dimensional (1st-through-39th) mel-cepstral coefficient s and their delta features. The STRAIGHT [24] vocoder systems were used for the feature extraction. The frame shift was 5 ms. We set two types of conversion: male-to-male and male-to-fema le. One male ...
-
[5]
Bold indicates the method preferred more with p-value < 0
Conclusion This paper presents a new voice impersonation attack us- ing voice conversion (VC), named the verification-to-synth esis Table 1: Results of preference AB tests on naturalness (male-to- male). Bold indicates the method preferred more with p-value < 0. 05. A Scores p-value B ParaVC ( 5 utts) 0.388 vs. 0.612 1.221× 10−10 V2S ParaVC (10 utts) 0.475...
-
[6]
Front-end factor analysis for speaker verification,
N. Dehak, P . J. Kenny, R. Dehak, P . Dumouchel, and P . Ouell et, “Front-end factor analysis for speaker verification,” IEEE Trans- actions on Audio, Speech, and Language Processing , vol. 14, no. 4, pp. 788–798, 2011
work page 2011
-
[7]
Deep neural networks for small footprint text- dependent speaker verification,
E. V ariani, X. Lei, E. McDermott, I. L. Moreno, and J. Gonz alez- Dominguez, “Deep neural networks for small footprint text- dependent speaker verification,” in Proc. ICASSP, Florence, Italy, May 2014, pp. 4080–4084
work page 2014
-
[8]
R. Prabhavalkar, R. Alvarez, C. Parada, P . Nakkiran, and T. Sainath, “Automatic gain control and multi-style training for ro- bust small-footprint keyword spotting with deep neural networks,” in Proc. ICASSP, Brisbane, Australia, Apr. 2015, pp. 4704–4708
work page 2015
Show all 32 references
-
[9]
Continuous probabilis- tic transform for voice conversion,
Y . Stylianou, O. Capp´ e, and E. Moulines, “Continuous probabilis- tic transform for voice conversion,” IEEE Transactions on Speech and Audio Processing, vol. 6, no. 2, pp. 131–142, Mar. 1998
1998
-
[10]
V oice conversion bas ed on maximum likelihood estimation of spectral parameter traje ctory,
T. Toda, A. W. Black, and K. Tokuda, “V oice conversion bas ed on maximum likelihood estimation of spectral parameter traje ctory,” IEEE Transactions on Audio, Speech, and Language Processin g, vol. 15, no. 8, pp. 2222–2235, 2007
2007
-
[11]
Statistical parametric speech synthesis incorporating generative adversarial ne tworks,
Y . Saito, S. Takamichi, and H. Saruwatari, “Statistical parametric speech synthesis incorporating generative adversarial ne tworks,” IEEE/ACM Transactions on Audio, Speech, and Language Pro- cessing, vol. 26, no. 1, pp. 755–767, Jun. 2018
2018
-
[12]
Phonetic post e- riorgrams for many-to-one voice conversion without parall el data training,
L. Sun, K. Li, H. Wang, S. Kang, and H. Meng, “Phonetic post e- riorgrams for many-to-one voice conversion without parall el data training,” in Proc. ICME, Seattle, U.S.A., Jul. 2016
2016
-
[13]
V oice conversion using artificial neural net works,
S. Desai, E. V . Raghavendra, B. Y egnanarayana, A. W. Black, and K. Prahallad, “V oice conversion using artificial neural net works,” in Proc. ICASSP, Taipei, Taiwan, Apr. 2009, pp. 3893–3896. Table 3: Results of preference XAB tests on speaker individu- ality (male-to-male). ...
2009
-
[14]
Non-pa rallel voice conversion using variational autoencoders conditio ned by phonetic posteriorgrams and d-vectors,
Y . Saito, Y . Ijima, K. Nishida, and S. Takamichi, “Non-pa rallel voice conversion using variational autoencoders conditio ned by phonetic posteriorgrams and d-vectors,” in Proc. ICASSP , Cal- gary, Canada, Apr. 2018, pp. 5274–5278
2018
-
[15]
Synthetic speech de tec- tion using temporal modulation feature,
Z. Wu, X. Xiao, E. S. Chng, and H. Li, “Synthetic speech de tec- tion using temporal modulation feature,” in Proc. ICASSP, V an- couver, Canada, May. 2013, pp. 7234–7238
2013
-
[16]
Vulnerability in speaker verifi- cation - a study of technical impostor techniques,
J. Lindberg and M. Blomberg, “Vulnerability in speaker verifi- cation - a study of technical impostor techniques,” in Proc. EU- ROSPEECH, Budapest, Hungary, Mar. 1999, pp. 1211–1214
1999
-
[17]
Can we use speaker recognition technology to attack itself ? en- hancing mimicry attacks using automatic target speaker sel ec- tion,
T. Kinnunen, R. G. Hautamki, V . V estman, and M. Sahidull ah, “Can we use speaker recognition technology to attack itself ? en- hancing mimicry attacks using automatic target speaker sel ec- tion,” in Proc. ICASSP, Brighton, United Kingdom, May 2019
2019
-
[18]
Robust audio adversarial exam ple for a physical attack,
H. Y akura and J. Sakuma, “Robust audio adversarial exam ple for a physical attack,” arXiv:1810.11793, 2018
2018 arXiv
-
[19]
Explainin g and har- nessing adversarial examples,
I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explainin g and har- nessing adversarial examples,” arXiv:1412.6572, 2014
2014 arXiv
-
[20]
Classifier-to-generator atta ck: Esti- mation of training data distribution from classifier,
K. Kusano and J. Sakuma, “Classifier-to-generator atta ck: Esti- mation of training data distribution from classifier,” 2018 . [On- line]. Available: https://openreview.net/forum?id=SJOl4DlCZ
2018
-
[21]
Membership inference at- tacks on sequence-to-sequence models,
S. Hisamoto, M. Post, and K. Duh, “Membership inference at- tacks on sequence-to-sequence models,” arXiv:1904.05506, 2019
1904 arXiv
-
[22]
Transfer learnin g from speaker verification to multispeaker text-to-speech s ynthe- sis,
Y . Jia, Y . Zhang, R. J. Weiss, Q. Wang, J. Shen, F. Ren, Z. C hen, P . Nguyen, R. Pang, I. L. Moreno, and Y . Wu, “Transfer learnin g from speaker verification to multispeaker text-to-speech s ynthe- sis,” vol. abs/1806.04558, 2018
2018 arXiv
-
[23]
WaveNet: A generative model for raw audio,
A. v. d. Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. W. Senior, and K. Kavukcuoglu, “WaveNet: A generative model for raw audio, ” vol. abs/1609.03499, 2016
2016 arXiv
-
[24]
End-to-end text-independen t speaker verification with flexibility in utterance duration,
C. Zhang and K. Koishida, “End-to-end text-independen t speaker verification with flexibility in utterance duration,” in Proc. ASRU, Okinawa, Japan, Dec. 2017, pp. 584–590
2017
-
[25]
Towards end-to-end speech recognition w ith deep convolutional neural networks,
Y . Zhang, M. Pezeshki, P . Brakel, S. Zhang, C. L. Y . Bengi o, and A. Courville, “Towards end-to-end speech recognition w ith deep convolutional neural networks,” arXiv, vol. abs/1701.02720, 2017
2017 arXiv
-
[26]
Generative adversarial nets,
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. War deFar- ley, S. Ozair, A. Courville, and Y . Bengio, “Generative adversarial nets,” in Proc. NIPS , Montreal, Canada, Dec. 2014, pp. 2672– 2680
2014
-
[27]
MetricGAN : Gen- erative adversarial networks based black-box metric score s opti- mization for speech enhancement,
S.-W. Fu, C.-F. Liao, Y . Tsao, and S.-D. Lin, “MetricGAN : Gen- erative adversarial networks based black-box metric score s opti- mization for speech enhancement,” vol. abs/1905.04874, 20 19
1905 arXiv
-
[28]
DNN-based source enhancement self-optimized by reinforc e- ment learning using sound quality measurements,
Y . Koizumi, K. Niwa, Y . Hioka, K. Kobayashi, and Y . Haned a, “DNN-based source enhancement self-optimized by reinforc e- ment learning using sound quality measurements,” in Proc. ICASSP, Orleans, U.S.A., Mar. 2017, pp. 81–85
2017
-
[29]
Kawahara, I
H. Kawahara, I. Masuda-Katsuse, and A. D. Cheveigne, “R e- structuring speech representations using a pitch-adaptiv e time- frequency smoothing and an instantaneous-frequency-base d F0 extraction: Possible role of a repetitive structure in soun ds,” Speech Communication, vol. ...
1999
-
[30]
Rectified linear units improve restricted boltzmann machines,
V . Nair and G. E. Hinton, “Rectified linear units improve restricted boltzmann machines,” in Proc. ICML , Haifa, Israel, June 2010, pp. 807–814
2010
-
[31]
Adaptive subgradien t meth- ods for online learning and stochastic optimization,
J. Duchi, E. Hazan, and Y . Singer, “Adaptive subgradien t meth- ods for online learning and stochastic optimization,” Journal of Machine Learning Research, vol. 12, pp. 2121–2159, Jul. 2011
2011
-
[32]
Auto-encoding variationa l bayes,
D. P . Kingma and M. Welling, “Auto-encoding variationa l bayes,” arXiv, vol. abs/1312.6114, 2013
2013 arXiv
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.