Pith. sign in

REVIEW 3 major objections 5 minor 43 references

DQ-Data2vec: Decoupling Quantization for Multilingual Speech Recognition

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DQ-Data2vec adds two online K-means quantizers—one with 9 clusters for language, one with 174 for phonemes—to data2vec's teacher–student masked prediction, and reports relative PER reductions of 9.51% (self-supervised) and 18.09% (weakly…

desk verdict Controlled gains from decoupling quantizers in data2vec are credible, but the count-matching mechanism is never ablated, leaving the core claim unproven. read the letter →

arxiv 2501.13497 v1 pith:LZXSH2YA submitted 2025-01-23 cs.SD cs.CLeess.AS

Qijie Shao , Linhao Dong , Kun Wei , Sining Sun , Lei Xie This is my paper · ORCID
classification cs.SDcs.CLeess.AS
keywords self-supervisedspeechlearningmultilingualASRvectorquantizationdecouplingdata2veconlineK-meansCommonVoice
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a self-supervised speech model can be told, without labels, which information to keep for multilingual speech recognition. Its answer is a modified data2vec that adds two online K-means quantizers: one assigned to shallow teacher layers and given exactly as many codebook entries as there are languages (9), the other assigned to middle layers and given as many entries as there are phonemes (174). The quantizers turn the teacher's averaged representations into separate language and phoneme targets, and the student is trained to reconstruct both alongside the standard data2vec target. On the CommonVoice setup, the self-supervised version cuts average phoneme error rate from 7.99% to 7.23% and word error rate by 11.58% relative to data2vec; adding language labels and English phoneme labels cuts PER to 6.93%. If the result holds, it suggests that choosing codebook size by the number of categories one wants to isolate is a cheap way to inject task knowledge into self-supervised speech pretraining.

What carries the argument

Central machinery: two online K-means vector quantizers with codebook sizes set to the number of languages (9) and phonemes (174), attached to hand-picked teacher layers (shallow 4–6 for language, middle 7–9 for phonemes). The K-means update (MSE loss with stop-gradient and a 0.25 commitment weight) forces the codewords toward the teacher layer outputs, while a grouped Conv1D (two groups, simulating product quantization) makes the quantizer input learnable without adding context capacity that would collapse training. The quantized codes become additional masked-prediction targets through a contrastive loss that also supplies negative examples (intra-utterance frames for phonemes, inter-utterance for language); in the weakly supervised variant, cross-entropy on the language code mixture and CTC on the phoneme code mixture are added. The design assumption is that matching codebook size to the target category count steers the quantizer to isolate that category from the many other factors (speakers, recording conditions, words) present in the same layer.

What would settle it

Measure the speaker purity of the language quantizer's codewords on the CommonVoice test set: if codewords align with speakers almost as strongly as with languages when the codebook has 9 entries, the count-matching heuristic is not isolating language. Alternatively, run the same pretraining with the language codebook size set to, say, 50 and check whether PER and language purity change; if neither experiment degrades the reported gains, the decoupling story is not doing the work claimed.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that the failure of data2vec for multilingual speech is that its teacher target averages the top 8 layers, entangling language, phoneme, speaker, and word information into a single continuous representation. DQ-Data2vec instead selects the teacher's shallow layers (4–6) for utterance-level language quantization and middle layers (7–9) for frame-level phoneme quantization, with L2 normalization and temporal pooling for the former and instance normalization for the latter. Two improved online K-means quantizers—one with 9 codewords, one with 174—map these inputs to discrete codes, and contrastive losses push the student's matching layers toward the codes; in the weakly supervised 'deep decoupling' variant, language and phoneme labels from non-target high-resource languages add CE and CTC losses on mixtures of student outputs and codes. The reported outcome is that the decoupled targets are complementary: removing either quantizer hurts PER, and the full model beats the data2vec and UniData2vec baselines in both scenarios.

Load-bearing premise

The whole method leans on the premise that the layer-wise information split seen in monolingual speech SSL—shallow layers carrying language and speaker cues, middle layers carrying phoneme and word cues—holds in a multilingual data2vec teacher, and that a codebook whose size equals the number of languages or phonemes will latch onto those categories rather than onto other correlated factors such as speakers or recording conditions.

Editorial extensions

If this is right

  • In the self-supervised setting, adding both quantizers reduces average PER from 7.99% to 7.23% over the data2vec baseline, and WER by 11.58% relative, on the 8-language CommonVoice test set.
  • In the weakly supervised setting, DQ-Data2vec reaches 6.93% average PER, beating the rerun UniData2vec baseline (7.20%) and the published UniData2vec without data balancing (8.46%), while needing only single-stage pretraining.
  • Ablations show both quantizers contribute: removing the language quantizer or the phoneme quantizer raises PER by 0.32–0.40 points in the self-supervised scenario, and the language quantizer alone accounts for most of the gain.
  • The quantizer analysis shows the language quantizer reaches LNMI of 0.34 without labels and 0.95 with labels, while the phoneme quantizer's PNMI improves to 0.49–0.62 when measured per language, indicating the codes carry the intended information.
  • Because the method separates language and phoneme targets in pretraining, its learned representations should transfer beyond ASR to tasks such as language identification and code-switched speech processing.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural testable extension—not run in the paper—is ablating the codebook sizes themselves (for example, 4, 9, and 18 for language; 50, 174, and 500 for phonemes). Without such an ablation, it is unresolved whether the count-matching heuristic or merely the presence of an extra quantization oracle drives the gains.
  • The same count-matching recipe could be applied to other utterance-level targets (speaker, dialect, emotion) or frame-level targets (tone, stress) by picking layer positions and codebook sizes accordingly; the paper gestures at this generality but does not test it.
  • The language quantizer's unsupervised LNMI of 0.34 suggests the learned codes could double as a zero-shot language identification signal, a downstream use beyond ASR that the paper does not pursue.
  • Because the phoneme quantizer's PNMI drops under CTC while its purity rises, the weakly supervised objective appears to trade coverage for confidence; an explicit codebook-usage penalty might recover the lost tail codewords and push PER lower.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes DQ-Data2vec, a multilingual ASR self-supervised learning method built on the data2vec teacher-student backbone. Two online K-means quantizers are added to selected teacher layers: a language quantizer applied to temporally pooled shallow layers with a codebook of size 9, and a phoneme quantizer applied to middle layers with a codebook of size 174. In the self-supervised 'shallow decoupling' setting, only contrastive and K-means losses are used; in the weakly supervised 'deep decoupling' setting, language labels and English phoneme labels are added through CE and CTC losses. On CommonVoice 6.0 with nine languages and 1-hour fine-tuning, the paper reports PER reductions from 7.99 to 7.23 over data2vec and from 7.20 to 6.93 over a rerun UniData2vec, along with WER improvements, and it presents quantizer purity metrics and ablations over quantizer type, group number, normalization, and trainable parameters.

Significance. If the central mechanism held, the paper would provide a simple and practical way to inject language and phoneme structure into SSL pre-training for low-resource multilingual ASR, with the notable engineering advantage of single-stage pre-training. The controlled experimental setup is a genuine strength: the main baselines are rerun under the same one-stage pre-training, data balance, and learning rate, and the comparison against the rerun UniData2vec is a fairer test than comparing only against published numbers. The ablation study in Table VI also covers several design dimensions and documents training collapse in informative settings. However, the paper's headline contribution is the count-matching codebook heuristic, and that specific mechanism is not tested, which limits the significance of the results as currently presented.

major comments (3)
  1. [III.B, Eq. (5); Table VI] The central claim is that setting the codebook size N to the number of languages (9) or phonemes (174) decouples language and phoneme information from irrelevant factors. No experiment varies N. Table VI ablates quantizer type, group number, normalization, and trainable parameters, but B1-B4 and S1-S7 all keep N fixed at 9 and 174, and the w/o PQT and w/o LQT rows in Table II are not codebook-size ablations. As a result, the observed gains over data2vec and UniData2vec could come entirely from adding extra quantized prediction targets and contrastive losses, with N acting as a capacity knob rather than as the count-matching decoupling device. This is load-bearing because the abstract and conclusion identify cluster-number alignment as the core contribution. Please add ablations that vary N for both quantizers, e.g., language codebooks of size 4, 9, 16, 32, 64 and phoneme codebooks of size 64, 128, 174, 256, 512, and report PER/WER together with LP/LNMI/PP/PNMI for each setting.
  2. [IV.A, Eqs. (12)-(13); Table IV; Figs. 2-3] The mechanism validation is partly self-referential in the deep decoupling setting. The CE loss in Eq. (12) uses language labels Y_l and the CTC loss in Eq. (13) uses phoneme labels Y_p, and the LP/LNMI/PP/PNMI metrics in Table IV are computed against the same label inventory. The large increases in the deep decoupling rows, e.g., LNMI from 0.34 to 0.95 and LP from 0.54 to 0.88, are therefore expected by construction as the supervised losses directly pull the quantizer outputs toward those labels. These numbers do not independently establish that the K=9 and K=174 K-means quantizers, rather than the supervised losses, are what decouple the information. Please provide an evaluation of the quantizer on held-out language/phoneme labels, or compare against a variant with supervised losses but non-matching codebook sizes, so that the count-matching mechanism can be separated from the effect of supervision.
  3. [Tables II and III] All headline reductions are reported from single training runs, with no standard deviations, multiple seeds, or significance tests. The absolute differences are small in several cases, e.g., 6.93 vs 7.20 PER in deep decoupling, and per-language results are noisy, with WER for Swedish worsening from 46.94 to 47.29 in shallow decoupling and from 45.38 to 49.35 in deep decoupling. Given that fine-tuning uses only 1 hour of labeled data per language, run-to-run variation could plausibly be comparable to some of the claimed gains. Please report means and variances over at least a few pre-training/fine-tuning seeds for the main comparisons, or otherwise justify that the differences are above the noise floor.
minor comments (5)
  1. [III.B, Eq. (10)] Eq. (10) uses L^l_qt and L^p_qt but these are not explicitly defined; please state that each is L_qt = L_ctr + L_km for the corresponding quantizer.
  2. [Table III] Rows for UniData2vec w/o DB report only Avg3; please report all per-language values or clearly mark them as unavailable, since Avg8 is used for other rows.
  3. [Table VI, rows S5-S6] The terms 'quick collapse' and 'slow collapse' are not quantified; please define the collapse criterion and provide representative training curves or codebook activity statistics for these rows.
  4. [Figs. 2 and 3] The captions use ACN and AGN without defining them; please define these abbreviations in the captions or refer explicitly to the definitions in Section V.C.
  5. [Abstract] The phrase 'compared to data2vec and UniData2vec' attaches two different baselines to the two relative reductions; please make clear in the abstract that the 9.51%/11.58% reductions are against data2vec and the 18.09%/1.55% reductions are against UniData2vec.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: held-out ASR evaluation keeps the derivation independent; missing codebook-size ablation is an evidential gap, not a circular step.

full rationale

The paper's central ASR claims are evaluated by fine-tuning on 1-hour labeled data and reporting held-out PER/WER in Tables II and III, so the reported gains are not defined in terms of the proposed quantizer targets. The codebook sizes (9 for languages, 174 for phonemes) and layer selections are design choices based on external layer-analysis work (Pasad et al., ASRU 2021, cited as [16]) and on the known label inventory; no equation reduces a reported error rate to the count-matching heuristic. The deep-decoupling language purity and LNMI metrics are partly self-referential because the CE loss in Eq. 12 directly supervises the language quantizer with language labels, but the paper uses those metrics only as diagnostics for the quantizer, not as the evidence for the held-out ASR improvement. The absence of an ablation varying codebook size is a genuine threat to the mechanistic interpretation: the gains could come from adding extra quantized prediction targets generally rather than from setting K equal to the language or phoneme count. That is missing evidence or a correctness risk, not circular derivation. I found no instance where a predicted quantity is equivalent by construction to a fitted input, no load-bearing self-citation chain, and no imported uniqueness theorem; the central derivation is self-contained against the external ASR benchmarks.

Assumptions & free parameters 8 free parameters · 5 assumptions · 0 invented entities

The method leans on prior layer-wise analyses and on hand-picked hyperparameters; the central mechanism (cluster count matching) is not ablated. No new physical entities are introduced.

free parameters (8)
  • gamma_1 (language quantizer loss weight) = 0.1
    Loss weight for language quantizer contrastive and K-means losses in Eq. 10; chosen by hand and not ablated.
  • gamma_2 (phoneme quantizer loss weight) = 0.2
    Loss weight for phoneme quantizer losses in Eq. 10; chosen by hand and not ablated.
  • gamma_3 (supervised loss weight) = 0.1
    Weight for CE and CTC losses in deep decoupling (Eq. 14); chosen by hand.
  • language cluster count = 9 (design choice, not fitted)
    Codebook size for language quantizer set to number of languages; core design choice, not ablated against other K values.
  • phoneme cluster count = 174 (design choice, not fitted)
    Codebook size for phoneme quantizer set to dictionary size (169 phonemes plus 5 special symbols); core design choice.
  • language layer set y_l = {4,5,6}
    Teacher layers used for language quantization, based on prior layer-wise analysis; not re-validated on this multilingual data.
  • phoneme layer set y_p = {7,8,9}
    Teacher layers used for phoneme quantization, based on prior layer-wise analysis; not re-validated.
  • contrastive temperature kappa = 0.1
    Temperature in contrastive loss Eq. 8; default from prior work, not varied.
assumptions (5)
  • domain assumption data2vec shallow layers encode speaker and language information, middle layers encode phoneme and word information, deep layers handle reconstruction.
    Used in Section III.B to pick y_l and y_p; sourced from [16] and [18], but not demonstrated for the multilingual data2vec model used here.
  • ad hoc to paper Matching the K-means codebook size to the target type count (9 languages, 174 phonemes) steers clustering toward those targets and away from other features such as speakers.
    Core heuristic in Section III.B; no ablation over different cluster counts is provided to isolate this effect.
  • domain assumption Online K-means quantization with stop-gradient and commitment loss maintains stable SSL training and meaningful codebooks.
    Inherited from vq-wav2vec [11,19]; the paper relies on this stability property, and its own ablations show collapse when trainable parameters are added.
  • domain assumption Contrastive learning with in-batch negatives prevents representation collapse during SSL pre-training.
    Used in Section III.B for quantizer learning; based on prior contrastive SSL results [10,40].
  • domain assumption Product quantization by splitting features into two groups improves codebook utilization.
    Design choice in Section III.B inspired by wav2vec 2.0 [10]; supported by ablation group=1 vs group=2 in Table VI.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DQ-Data2vec: Decoupling Quantization for Multilingual Speech Recognition." pith.science (2026). https://pith.science/paper/LZXSH2YA

@misc{pith2026250113497,
  author       = {Pith},
  title        = {Pith review of: DQ-Data2vec: Decoupling Quantization for Multilingual Speech Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LZXSH2YA}},
  note         = {Machine review of arXiv:2501.13497}
}
read the original abstract

Data2vec is a self-supervised learning (SSL) approach that employs a teacher-student architecture for contextual representation learning via masked prediction, demonstrating remarkable performance in monolingual ASR. Previous studies have revealed that data2vec's shallow layers capture speaker and language information, middle layers encode phoneme and word features, while deep layers are responsible for reconstruction. Language and phoneme features are crucial for multilingual ASR. However, data2vec's masked representation generation relies on multi-layer averaging, inevitably coupling these features. To address this limitation, we propose a decoupling quantization based data2vec (DQ-Data2vec) for multilingual ASR, which includes a data2vec backbone and two improved online K-means quantizers. Our core idea is using the K-means quantizer with specified cluster numbers to decouple language and phoneme information for masked prediction. Specifically, in the language quantization, considering that the number of languages is significantly different from other irrelevant features (e.g., speakers), we assign the cluster number to match the number of languages, explicitly decoupling shallow layers' language-related information from irrelevant features. This strategy is also applied to decoupling middle layers' phoneme and word features. In a self-supervised scenario, experiments on the CommonVoice dataset demonstrate that DQ-Data2vec achieves a relative reduction of 9.51% in phoneme error rate (PER) and 11.58% in word error rate (WER) compared to data2vec and UniData2vec. Moreover, in a weakly-supervised scenario incorporating language labels and high-resource language text labels, the relative reduction is 18.09% and 1.55%, respectively.

Figures

Figures reproduced from arXiv: 2501.13497 by the authors.

Figure 1
Figure 1. The framework overview of our proposed DQ-Data2vec. The symbol [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The conditional probability 𝑃(𝑙𝑎𝑛𝑔|𝑐𝑜𝑑𝑒) on CommonVoice test set. The y-axis is the language set sorted by the number of occurrences, and the x-axis is the active codewords sorted by the most correlated language. In the figure, ACN denotes the active codeword number, while AGN indicates the active clusters within the two clustering groups. LP signifies language purity, and LNMI refers to language-normalized mutual i… view at source ↗
Figure 3
Figure 3. (b), with the supervised losses introduced into the deep decoupling scenario, the numbers of active codewords decline. This is reasonable as the certainty of active codewords is higher. In summary, while a great number of active codewords does not always guarantee superior clustering, too few quan￾tities indicate clustering collapse. Evidently, our clustering results do not fall into the latter scenario. Moreover, i… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 41 canonical work pages

  1. [1]

    Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio,

    G. Chen, S. Chai, G. Wang, J. Du, W. Q. Zhang, C. Weng, D. Su, D. Povey, J. Trmal, J. Zhang et al., “Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio,” in Proceedings of the Annual Conference of the International Speech Communication Association (INTERSPEECH), 2021, pp. 4376–4380

    work page 2021

  2. [2]

    Wenetspeech: A 10000+ hours multi-domain mandarin corpus for speech recognition,

    B. Zhang, H. Lv, P. Guo, Q. Shao, C. Yang, L. Xie, X. Xu, H. Bu, X. Chen, C. Zeng et al., “Wenetspeech: A 10000+ hours multi-domain mandarin corpus for speech recognition,” in Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, pp. 6182–6186

    work page 2022

  3. [3]

    Katzner and K

    K. Katzner and K. Miller, The languages of the world . Routledge, 2002

    work page 2002

  4. [4]

    XLS-R: Self- supervised cross-lingual speech representation learning at scale,

    A. Babu, C. Wang, A. Tjandra, K. Lakhotia, Q. Xu, N. Goyal, K. Singh, P. von Platen, Y . Saraf, J. Pino, A. Baevski, A. Conneau, and M. Auli, “XLS-R: Self- supervised cross-lingual speech representation learning at scale,” in Proceedings of the Annual Conference JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 12 of the International Speech Commu...

    work page 2015

  5. [5]

    Self- supervised learning with random-projection quantizer for speech recognition,

    C.-C. Chiu, J. Qin, Y . Zhang, J. Yu, and Y . Wu, “Self- supervised learning with random-projection quantizer for speech recognition,” in Proceedings of the International Conference on Machine Learning (ICML) , 2022, pp. 3915–3924

    work page 2022

  6. [6]

    Unispeech: Unified speech rep- resentation learning with labeled and unlabeled data,

    C. Wang, Y . Wu, Y . Qian, K. Kumatani, S. Liu, F. Wei, M. Zeng, and X. Huang, “Unispeech: Unified speech rep- resentation learning with labeled and unlabeled data,” in Proceedings of the International Conference on Machine Learning (ICML), 2021, pp. 10 937–10 947

    work page 2021

  7. [7]

    Tranusr: Phoneme-to-word transcoder based unified speech representation learning for cross-lingual speech recognition,

    H. Xue, Q. Shao, P. Chen, P. Guo, L. Xie, and J. Liu, “Tranusr: Phoneme-to-word transcoder based unified speech representation learning for cross-lingual speech recognition,” in Proceedings of the Annual Conference of the International Speech Communication Association (INTERSPEECH), 2023, pp. 216–220

    work page 2023

  8. [8]

    Data2vec: A general framework for self- supervised learning in speech, vision and language,

    A. Baevski, W.-N. Hsu, Q. Xu, A. Babu, J. Gu, and M. Auli, “Data2vec: A general framework for self- supervised learning in speech, vision and language,” in Proceedings of the International Conference on Machine Learning (ICML), 2022, pp. 1298–1312

    work page 2022

Show all 43 references
  1. [9]

    Mo- mentum contrast for unsupervised visual representation learning,

    K. He, H. Fan, Y . Wu, S. Xie, and R. Girshick, “Mo- mentum contrast for unsupervised visual representation learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2020, pp. 9729–9738

    2020

  2. [10]

    wav2vec 2.0: A framework for self-supervised learning of speech representations,

    A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” Advances in Neural Infor- mation Processing Systems (NIPS) , vol. 33, pp. 12 449– 12 460, 2020

    2020

  3. [11]

    Vq-wav2vec: Self-supervised learning of discrete speech representa- tions,

    A. Baevski, S. Schneider, and M. Auli, “Vq-wav2vec: Self-supervised learning of discrete speech representa- tions,” in Proceedings of the International Conference on Learning Representations (ICLR) , 2020, pp. 1–12

    2020

  4. [12]

    SUPERB: Speech processing universal performance benchmark,

    S.-w. Yang, P.-H. Chi, Y .-S. Chuang, C.-I. J. Lai, K. Lakhotia, Y . Y . Lin, A. T. Liu, J. Shi, X. Chang, G.-T. Lin et al., “SUPERB: Speech processing universal performance benchmark,” in Proceedings of the Annual Conference of the International Speech Communication Associati...

    2021

  5. [13]

    Mas- sively multilingual ASR: 50 languages, 1 model, 1 billion parameters,

    V . Pratap, A. Sriram, P. Tomasello, A. Hannun, V . Liptchinsky, G. Synnaeve, and R. Collobert, “Mas- sively multilingual ASR: 50 languages, 1 model, 1 billion parameters,” in Proceedings of the Annual Conference of the International Speech Communication Association (INTERSPEE...

    2020

  6. [14]

    A unified system for multilingual speech recognition and language iden- tification,

    D. Liu, J. Xu, P. Zhang, and Y . Yan, “A unified system for multilingual speech recognition and language iden- tification,” Speech Communication, vol. 127, pp. 17–28, 2021

    2021

  7. [15]

    Improving massively multilingual ASR with auxiliary CTC objectives,

    W. Chen, B. Yan, J. Shi, Y . Peng, S. Maiti, and S. Watanabe, “Improving massively multilingual ASR with auxiliary CTC objectives,” in Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2023, pp. 1–5

    2023

  8. [16]

    Layer-wise anal- ysis of a self-supervised speech representation model,

    A. Pasad, J.-C. Chou, and K. Livescu, “Layer-wise anal- ysis of a self-supervised speech representation model,” in Proceedings of the IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) , 2021, pp. 914– 921

    2021

  9. [17]

    What all do audio transformer models hear? probing acoustic representations for language delivery and its structure,

    J. Shah, Y . K. Singla, C. Chen, and R. R. Shah, “What all do audio transformer models hear? probing acoustic representations for language delivery and its structure,” arXiv preprint arXiv:2101.00387 , 2021

    2021 arXiv

  10. [18]

    Sshr: Leveraging self-supervised hierarchical represen- tations for multilingual automatic speech recognition,

    H. Xue, Q. Shao, K. Huang, P. Chen, L. Xie, and J. Liu, “Sshr: Leveraging self-supervised hierarchical represen- tations for multilingual automatic speech recognition,” arXiv preprint arXiv:2309.16937 , 2023

    2023 arXiv

  11. [19]

    Neural discrete representation learning,

    A. Van Den Oord, O. Vinyals et al. , “Neural discrete representation learning,” Advances in Neural Information Processing Systems (NIPS) , vol. 30, pp. 6306–6315, 2017

    2017

  12. [20]

    Common V oice: A massively-multilingual speech corpus,

    R. Ardila, M. Branson, K. Davis, M. Kohler, J. Meyer, M. Henretty, R. Morais, L. Saunders, F. Tyers, and G. Weber, “Common V oice: A massively-multilingual speech corpus,” in Proceedings of the International Con- ference on Language Resources and Evaluation (LREC) , 2020, pp. ...

    2020

  13. [21]

    Hubert: Self- supervised speech representation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self- supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP) , vol. 29, pp. 3451–3460, 2021

    2021

  14. [22]

    WavLM: Large- scale self-supervised pre-training for full stack speech processing,

    S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao et al., “WavLM: Large- scale self-supervised pre-training for full stack speech processing,” IEEE Journal of Selected Topics in Signal Processing (J-STSP) , vol. 16, no. 6, pp. 1505–1518, 2022

    2022

  15. [23]

    W2v-BERT: Combining con- trastive learning and masked language modeling for self- supervised speech pre-training,

    Y .-A. Chung, Y . Zhang, W. Han, C.-C. Chiu, J. Qin, R. Pang, and Y . Wu, “W2v-BERT: Combining con- trastive learning and masked language modeling for self- supervised speech pre-training,” in Proceedings of the IEEE Automatic Speech Recognition and Understanding Workshop (ASR...

    2021

  16. [24]

    wav2vec: Unsupervised pre-training for speech recog- nition,

    S. Schneider, A. Baevski, R. Collobert, and M. Auli, “wav2vec: Unsupervised pre-training for speech recog- nition,” in Proceedings of the Annual Conference of the International Speech Communication Association (IN- TERSPEECH), 2019, pp. 3465–3469

    2019

  17. [25]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of the Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologie...

    2019

  18. [26]

    Categorical reparame- terization with Gumbel-Softmax,

    E. Jang, S. Gu, and B. Poole, “Categorical reparame- terization with Gumbel-Softmax,” in Proceedings of the International Conference on Learning Representations (ICLR), 2016, pp. 1–13

    2016

  19. [27]

    A* sampling,

    C. J. Maddison, D. Tarlow, and T. Minka, “A* sampling,” Advances in Neural Information Processing Systems JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 13 (NIPS), vol. 4, pp. 3086–3094, 2014

    2015

  20. [28]

    Joint unsupervised and super- vised training for multi-lingual ASR,

    J. Bai, B. Li, Y . Zhang, A. Bapna, N. Siddhartha, K. C. Sim, and T. N. Sainath, “Joint unsupervised and super- vised training for multi-lingual ASR,” in Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2022, pp. 6402–6406

    2022

  21. [29]

    Cross-lingual self-training to learn multi-lingual representation for low-resource speech recognition,

    Z.-Q. Zhang, Y . Song, M.-H. Wu, X. Fang, I. McLough- lin, and L.-R. Dai, “Cross-lingual self-training to learn multi-lingual representation for low-resource speech recognition,” Circuits, Systems, and Signal Processing (CSSP), vol. 41, no. 12, pp. 6827–6843, 2022

    2022

  22. [30]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in Neural Infor- mation Processing Systems (NIPS) , vol. 30, 2017

    2017

  23. [31]

    Connectionist temporal classification: labelling unseg- mented sequence data with recurrent neural networks,

    A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Connectionist temporal classification: labelling unseg- mented sequence data with recurrent neural networks,” in Proceedings of the International Conference on Machine Learning (ICML), 2006, pp. 369–376

    2006

  24. [32]

    Sequence transduction with recurrent neural networks,

    A. Graves, “Sequence transduction with recurrent neural networks,” in Workshop of the International Conference on Machine Learning (ICML) , 2012

    2012

  25. [33]

    Exploring wav2vec 2.0 on speaker verification and language identification,

    Z. Fan, M. Li, S. Zhou, and B. Xu, “Exploring wav2vec 2.0 on speaker verification and language identification,” in Proceedings of the Annual Conference of the In- ternational Speech Communication Association (INTER- SPEECH), 2021, pp. 896–900

    2021

  26. [34]

    Non-contrastive self-supervised learning for utterance- level information extraction from speech,

    J. Cho, J. Villalba, L. Moro-Velazquez, and N. Dehak, “Non-contrastive self-supervised learning for utterance- level information extraction from speech,” IEEE Journal of Selected Topics in Signal Processing (JSTSP), vol. 16, no. 6, pp. 1284–1295, 2022

    2022

  27. [35]

    Unispeech-sat: Uni- versal speech representation learning with speaker aware pre-training,

    S. Chen, Y . Wu, C. Wang, Z. Chen, Z. Chen, S. Liu, J. Wu, Y . Qian, F. Wei, J. Li et al., “Unispeech-sat: Uni- versal speech representation learning with speaker aware pre-training,” in Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processin...

    2022

  28. [36]

    Trinet: stabilizing self-supervised learning from complete or slow collapse,

    L. Cao, J. Wang, B. Yang, D. Su, and D. Yu, “Trinet: stabilizing self-supervised learning from complete or slow collapse,” in Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023, pp. 1–5

    2023

  29. [37]

    Instance nor- malization: The missing ingredient for fast stylization,

    D. Ulyanov, A. Vedaldi, and V . Lempitsky, “Instance nor- malization: The missing ingredient for fast stylization,” arXiv preprint arXiv:1607.08022 , 2016

    2016 arXiv

  30. [38]

    Goodfellow, Y

    I. Goodfellow, Y . Bengio, and A. Courville, Deep learn- ing. MIT press, 2016

    2016

  31. [39]

    Product quanti- zation for nearest neighbor search,

    H. Jegou, M. Douze, and C. Schmid, “Product quanti- zation for nearest neighbor search,” IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI) , vol. 33, no. 1, pp. 117–128, 2010

    2010

  32. [40]

    Exploring simple siamese rep- resentation learning,

    X. Chen and K. He, “Exploring simple siamese rep- resentation learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 15 750–15 758

    2021

  33. [41]

    Fairseq: A fast, extensible toolkit for sequence modeling,

    M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng, D. Grangier, and M. Auli, “Fairseq: A fast, extensible toolkit for sequence modeling,” in Proceedings of the Conference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Language Tech...

    2019

  34. [42]

    Adam: A method for stochas- tic optimization,

    D. P. Kingma and J. L. Ba, “Adam: A method for stochas- tic optimization,” in Proceedings of the International Conference on Learning Representations (ICLR) , 2015, pp. 1–15

    2015

  35. [43]

    Unsupervised cross-lingual representation learning for speech recognition,

    A. Conneau, A. Baevski, R. Collobert, A. Mohamed, and M. Auli, “Unsupervised cross-lingual representation learning for speech recognition,” in Proceedings of the Annual Conference of the International Speech Commu- nication Association (INTERSPEECH) , 2021, pp. 346– 350

    2021

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.