Pith. sign in

REVIEW 4 major objections 5 minor 45 references

Continual Learning with Embedding Layer Surgery and Task-wise Beam Search using Whisper

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Per-language embedding copies reduce catastrophic forgetting when Whisper learns new languages.

desk verdict The embedding-surgery idea is timely and the experiments look honest, but the headline gain rests on a single run and an unstated OOV/fallback mechanism for the per-language embedding tables. read the letter →

arxiv 2501.07875 v1 pith:SEGSO7LN submitted 2025-01-14 cs.CL cs.AI

Chin Yuen Kwok , Jia Qi Yip , Eng Siong Chng This is my paper · ORCID
classification cs.CLcs.AI
keywords continuallearningcatastrophicforgettingmultilingualASRWhispertokenembeddingslanguageidentificationbeamsearchlow-resourcespeech
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the main hidden source of catastrophic forgetting when a multilingual ASR model like Whisper learns new languages is the decoder's token embedding lookup table, which gets overwritten with new-language semantics. It proposes Embedding Layer Surgery: give each new language its own copy of the token embeddings, keep the original embeddings for old languages, and split each copy into a shared special-token part (used for language identification) and a language-specific vocabulary part. Because language ID errors would then select the wrong embedding table, it adds task-wise beam search that decodes in the top two candidate languages and picks the hypothesis with the highest ASR score. In experiments adapting Whisper-small and Whisper-large-v2 to ten unseen Common Voice languages, the method lowers the average WER on pre-trained languages from 14.2% to 11.9% relative to Experience Replay, with no loss on the new languages. A sympathetic reader would take this as evidence that continual expansion of language-agnostic speech models can be made cheaper and safer by isolating the embedding layer rather than only replaying old data.

What carries the argument

Embedding Layer Surgery: for each new language $\ell$, create a separate token-embedding lookup table $\hat{A}_\ell \in \mathbb{R}^{E \times J_\ell}$ with $J_\ell < U$, containing only the tokens used by that language, and keep the original matrix $A$ for old languages; split the table into a language-shared special-token part and a language-specific vocabulary part. Task-wise beam search: after the model emits a language ID, decode in the top $N=2$ scoring languages, score each path by summed log probabilities, and choose the best hypothesis, disabling the search when a path has fewer than $M_{\text{len}}$ words or more than $M_{\text{overlap}}$ overlaps. The embedding surgery does the work of preventing overwriting; the beam search does the work of preventing LID errors from selecting the wrong embedding table.

What would settle it

Decode a new-language test set deliberately constructed to be rich in tokens absent from that language's 10-hour training set, then measure old-language word error rate before and after adaptation; if out-of-vocabulary tokens are routed through the shared embedding and old-language WER degrades, the surgery does not fully isolate old languages.

Watch

Extended reading notes

Core claim

The central claim is that catastrophic forgetting in language-agnostic multilingual ASR is substantially caused by updating the shared token embedding matrix at the decoder, and that this can be prevented by maintaining a separate embedding lookup table for each newly added language. Each new table stores only the subset of tokens that appeared in that language's 10-hour training set; the original table remains untouched for old languages. The lookup table is partitioned so that special tokens such as language IDs stay shared, which keeps the model able to identify the input language and then automatically route decoding to the correct language-specific vocabulary embeddings. Task-wise beam search then fixes LID mistakes by keeping the top-N language hypotheses and selecting the final transcription by ASR score, with pruning rules for blank or overlapping hypotheses. The result is a language-agnostic continual learning system in which adding ten low-resource languages improves average WER on old languages from 14.2% to 11.9% compared with Experience Replay, without hurting new-language WER.

Load-bearing premise

The per-language embedding table contains only the tokens that happened to appear in that language's 10-hour training set, and the paper does not state how tokens outside that set are handled during inference; if such tokens fall back to the shared old embeddings, old-language semantics could still be overwritten.

Editorial extensions

If this is right

  • The embedding layer is a primary site of forgetting, so isolating it should be part of any continual learning recipe for multilingual ASR.
  • The model stays language-agnostic because special tokens, including language IDs, remain shared, so no manual language selection is required.
  • Task-wise beam search reduces LID confusion errors by more than 40% for ER and 60% for ER-E in the ablation, which directly improves language-agnostic WER.
  • Adding ten unseen languages one after another improves average WER on old languages from 14.2% to 11.9% compared with Experience Replay while new-language WER is unchanged.
  • The method applies to both Whisper-small and Whisper-large-v2, suggesting the benefit does not depend on a specific model scale.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the per-language tables cover only 0.4% to 7.9% of Whisper's vocabulary, so the reported results rely on an unstated policy for out-of-vocabulary tokens; specifying one is the next necessary step.
  • Editorial inference: if embedding surgery keeps old semantics intact without large replay buffers, it may allow the replay size to be reduced below one hour per language, which the paper does not test.
  • Editorial inference: the surgery component should transfer to any encoder-decoder ASR with a shared subword embedding, such as other multilingual speech models, where it could be tested independently of the beam-search component.
  • Editorial inference: task-wise beam search is a general LID-error correction scheme and could improve any language-agnostic decoder, even without per-language embedding surgery.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper addresses catastrophic forgetting in language-agnostic massively multilingual ASR when continually adapting Whisper to unseen languages. It proposes Embedding Layer Surgery, where separate per-language token embedding lookup tables are created and split into a language-shared special-token part and a language-specific vocabulary part, and Task-wise Beam Search, which keeps the top-N language hypotheses during decoding and selects by ASR score. Experiments on Common Voice subsets with 10 unseen and 10 seen languages report that the method reduces the average WER of pre-trained languages from 14.2% to 11.9% compared with Experience Replay, without degrading WER on the newly added languages.

Significance. If the reported results are reliable, the paper makes a useful contribution to continual learning for multilingual ASR: it identifies the token embedding lookup table as a source of catastrophic forgetting in language-agnostic models, proposes a simple architectural remedy, and augments decoding to correct language-identification errors. The use of the CL-MASR benchmark, comparison with several standard CL baselines, and ablation studies are strengths. However, the central quantitative claim is currently conditional on an unspecified out-of-vocabulary mechanism and on the absence of statistical verification, so the significance cannot be fully assessed without revision.

major comments (4)
  1. [Section 2.3, Table 1] The per-language embedding table A_hat in R^{E x J} is constructed from tokens appearing in the 10-hour training set, and Table 1 reports J/U as low as 0.4% and at most 7.9% of Whisper's vocabulary for the ten new languages. The paper never states what happens during inference when a new-language utterance contains a token outside A_hat. This is load-bearing for the 'without compromising the AWER of the unseen languages' claim: the new-language WERs in Table 2 and Figure 4 depend on whether such tokens are covered by the test set, mapped to a fallback embedding from the original shared table, or suppressed during decoding. Each option changes the meaning of the result and its effect on old-language isolation. Please report token coverage on the test sets, specify the fallback mechanism, or justify that out-of-table tokens cannot occur.
  2. [Section 3.1, Section 3.2, Figure 4] All experimental results appear to be single runs: no repeated seeds, no error bars, and no significance tests are reported. The headline improvement in Figure 4, reducing AWER from 14.2% to 11.9% for whisper-large-v2, is a 2.3-point difference over Experience Replay, which could plausibly lie within run-to-run variance. In addition, the paper states that hyper-parameters are swept 'for all methods' but does not report the sweep ranges, the selection criterion, or whether selection used the validation set rather than the test set. The task-wise beam search thresholds N=2, Mlen=5, and Moverlap=3 are fixed without sensitivity analysis. Please provide seeds, variance or significance tests, the complete hyperparameter search protocol, and an ablation of the decoding thresholds.
  3. [Section 2.5] The stability condition for Task-wise Beam Search is underspecified. The text says that decoding is disabled when a path has fewer than Mlen words or more than Moverlap overlapped words with other paths, but it does not say what decoding is performed instead: does the system fall back to the single best LID path, use the original beam search, or suppress the hypothesis? This affects the reproducibility of all language-agnostic results, including Figure 4 and Table 3, and should be stated precisely.
  4. [Section 2.4] The construction of the split lookup table is not fully specified. It is not explained how new language ID tokens are inserted into the language-shared special token block while the original language ID embeddings for the pre-trained languages are kept frozen, nor how the model selects which per-language vocabulary table A_hat to use during the language-agnostic LID phase. Since the language-agnostic claim depends on this mechanism, please provide the exact table construction, the embedding initializations, and the selection rule.
minor comments (5)
  1. [Abstract and Section 1, Index Terms] The index terms include 'speech emotion recognition (SER)' and 'instruction-tuning', which are not topics of this paper and should be corrected.
  2. [Table 2] The entries in the 'avg' column, such as '41.0−00.0%', appear malformed and are not explained; please define whether the column reports average WER or relative improvement, and clarify the notation.
  3. [Section 3.2, Figure 4] Figure 4 plots AWER over time for whisper-large-v2 but does not give numerical values in the text; please report the exact AWER numbers for all ten languages at the end of sequential adaptation.
  4. [Section 3.3] The ablations in Tables 3 and 4 are performed only for whisper-small on the language pair Esperanto/English, so it is unclear whether the conclusions transfer to whisper-large-v2 and to the full ten-language setting; please add at least one large-v2 ablation or discuss the limitation.
  5. [Throughout] There are several typos and formatting issues, including 'V oice' in the abstract, 'langauge' and 'becuase' in Section 2.4, 'forgeting' in Table 2, 'Germen' in Table 2 header, and 'topN = 2scoring' in Figure 3; these should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims are empirical comparisons on held-out test sets, not derivations that reduce to their inputs.

full rationale

The paper proposes two architectural/decoding components — Embedding Layer Surgery (separate per-language token embedding lookups) and Task-wise Beam Search (decoding under top-N language hypotheses) — and evaluates them on Common Voice subsets with held-out test partitions. The claimed reduction in AWER from 14.2% to 11.9% is an experimental result comparing ER-E-B with Experience Replay, not a quantity that is defined in terms of itself. The per-language embedding table is constructed from the new-language training tokens, but the reported WERs are measured on separate test utterances, so the evaluation is not forced by the construction. The only self-citation (reference [37]) is a prior beam-search-related paper by the authors and is not load-bearing: the task-wise beam search mechanism is described independently in Section 2.5 and its effect is isolated by ablations in Section 3.3. Hyper-parameters such as N, Mlen, Moverlap and beta are manually chosen, which is a tuning concern rather than a circular reduction, especially since all compared methods are tuned in the same way. The paper does leave an engineering detail unspecified — how tokens outside the per-language A_hat table are handled at inference — but this is an incompleteness/reproducibility concern, not an equivalence between input and output. No equation, fitted parameter, or self-citation chain makes the headline result true by construction.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim relies on the pretrained Whisper model, the CL-MASR benchmark data, and several manually chosen thresholds. The per-language embedding surgery depends on the assumption that token embeddings are a dominant forgetting source; the task-wise beam search depends on the assumption that the true language survives the top-N LID pruning; and the practical system depends on an unspecified treatment of out-of-vocabulary tokens.

free parameters (4)
  • beta (replay loss weight) = not reported
    Balances old-task replay loss and new-task loss in Eq. (1); tuned via hyperparameter sweep but its value or range is not stated.
  • N (top LID candidates) = 2
    Task-wise beam search keeps only top N=2 language hypotheses; manually set in Section 3.1, no sensitivity analysis.
  • Mlen (minimum words per path) = 5
    Threshold below which a decoding path is discarded; manually set, no sensitivity analysis.
  • Moverlap (maximum word overlap) = 3
    Threshold above which a decoding path is discarded as redundant; manually set, no sensitivity analysis.
assumptions (5)
  • domain assumption Whisper's decoder token embedding lookup table is a significant contributor to catastrophic forgetting.
    The method's core motivation, stated in Section 1 and 2.3, relies on the premise that overwriting shared token embeddings causes forgetting; this is supported by citation [31] but not re-established in this paper's setting.
  • domain assumption The correct language is among the top N=2 LID hypotheses after adaptation.
    Task-wise beam search can only recover from LID errors if the true language is in the pruned set; no analysis of LID confidence distribution is provided beyond a confusion matrix for one language pair (Figure 5).
  • domain assumption Summing per-token log probabilities across languages yields comparable ASR scores.
    Section 2.5 ranks hypotheses by summed log probabilities without length normalization; this can bias toward shorter outputs and is not justified.
  • ad hoc to paper Unseen tokens during inference are either absent or handled by an unspecified mechanism.
    The per-language lookup table A_hat has only J<U columns for tokens seen in the 10-hour training set (Table 1); the paper never states how inference on OOV tokens proceeds.
  • domain assumption The pretrained Whisper encoder can be frozen without losing adaptation quality.
    Section 3.1 freezes the encoder following [23]; if encoder adaptation were needed, the method would not work as described.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Continual Learning with Embedding Layer Surgery and Task-wise Beam Search using Whisper." pith.science (2026). https://pith.science/paper/SEGSO7LN

@misc{pith2026250107875,
  author       = {Pith},
  title        = {Pith review of: Continual Learning with Embedding Layer Surgery and Task-wise Beam Search using Whisper},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SEGSO7LN}},
  note         = {Machine review of arXiv:2501.07875}
}
read the original abstract

Current Multilingual ASR models only support a fraction of the world's languages. Continual Learning (CL) aims to tackle this problem by adding new languages to pre-trained models while avoiding the loss of performance on existing languages, also known as Catastrophic Forgetting (CF). However, existing CL methods overlook the adaptation of the token embedding lookup table at the decoder, despite its significant contribution to CF. We propose Embedding Layer Surgery where separate copies of the token embeddings are created for each new languages, and one of the copies is selected to replace the old languages embeddings when transcribing the corresponding new language. Unfortunately, this approach means LID errors also cause incorrect ASR embedding selection. Our Task-wise Beam Search allows self-correction for such mistakes. By adapting Whisper to 10 hours of data for each of 10 unseen languages from Common Voice, results show that our method reduces the Average WER (AWER) of pre-trained languages from 14.2% to 11.9% compared with Experience Replay, without compromising the AWER of the unseen languages.

Figures

Figures reproduced from arXiv: 2501.07875 by the authors.

Figure 4
Figure 4. Our method ER-E-B outperforms the best CL base [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 29 canonical work pages

  1. [1]

    INTRODUCTION Recent advancements in training speech models involve uti- lizing millions of hours of multilingual ASR and translation labeled data, leading to the development of models supporting Massively Multilingual ASR (MMASR) which can transcribe over 50 languages [1]. Additionally, newer models such as Whisper [2] and MMS [3] are language-agnostic [4...

    arXiv 2025

  2. [2]

    PROPOSED METHOD We use Whisper [2] as the pre-trained MMASR model and adapt it to the ASR and LID datasets of the new languages. LID adaptation is needed as it allows the model to transcribe in a language-agnostic setting [4], where instead of manually prompting Whisper to control the transcribed language, the model can use LID to automatically determine ...

    work page

  3. [3]

    Un- adapted

    EXPERIMENT AND DISCUSSION 3.1. Dataset and Model Details We implement our CL methods based on the popular Speech- Brain [39] toolkit and CL-MASR [23]. Following previous works [21, 23], we evaluate our method on a subset of the widely used large-scale Com- monV oice dataset [40]. We follow CL-MASR [23] to extract the data subsets. They consists of ten lan...

    work page

  4. [4]

    CONCLUSION To conclude, we present ER-E-B, a CL method that outper- forms ER methods in mitigating CF and ablation study has shown the effectiveness of our proposed language-agnostic dynamic architecture, task-wise beam search and separate to- ken embedding

    work page

  5. [5]

    Mas- sively multilingual asr: 50 languages, 1 model, 1 billion parameters,

    V . Pratap, A. Sriram, P. Tomasello, A. Hannun, V . Liptchinsky, G. Synnaeve, and R. Collobert, “Mas- sively multilingual asr: 50 languages, 1 model, 1 billion parameters,” arXiv preprint arXiv:2007.03001, 2020

    arXiv 2007

  6. [6]

    Robust speech recogni- tion via large-scale weak supervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recogni- tion via large-scale weak supervision,” in International Conference on Machine Learning . PMLR, 2023, pp. 28 492–28 518

    work page 2023

  7. [7]

    Scaling speech technology to 1,000+ languages,

    V . Pratap, A. Tjandra, B. Shi, P. Tomasello, A. Babu, S. Kundu, A. Elkahky, Z. Ni, A. Vyas, M. Fazel-Zarandi et al., “Scaling speech technology to 1,000+ languages,” arXiv preprint arXiv:2305.13516, 2023

    arXiv 2023

  8. [8]

    Language-agnostic multilingual modeling,

    A. Datta, B. Ramabhadran, J. Emond, A. Kannan, and B. Roark, “Language-agnostic multilingual modeling,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2020, pp. 8239–8243

    work page 2020

Show all 45 references
  1. [9]

    Katzner and K

    K. Katzner and K. Miller, The languages of the world . Routledge, 2002

    2002

  2. [10]

    Investigating the catastrophic forgetting in multimodal large language models,

    Y . Zhai, S. Tong, X. Li, M. Cai, Q. Qu, Y . J. Lee, and Y . Ma, “Investigating the catastrophic forgetting in multimodal large language models,” arXiv preprint arXiv:2309.10313, 2023

    2023 arXiv

  3. [11]

    A simple baseline that questions the use of pretrained-models in continual learning,

    P. Janson, W. Zhang, R. Aljundi, and M. Elho- seiny, “A simple baseline that questions the use of pretrained-models in continual learning,” arXiv preprint arXiv:2210.04428, 2022

    2022 arXiv

  4. [12]

    Overcoming catastrophic forgetting in neural networks,

    J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ra- malho, A. Grabska-Barwinska et al. , “Overcoming catastrophic forgetting in neural networks,”Proceedings of the national academy of sciences, vol. 114, no. 13, pp. 3521–3...

    2017

  5. [13]

    Memory aware synapses: Learning what (not) to forget,

    R. Aljundi, F. Babiloni, M. Elhoseiny, M. Rohrbach, and T. Tuytelaars, “Memory aware synapses: Learning what (not) to forget,” in Proceedings of the European confer- ence on computer vision (ECCV) , 2018, pp. 139–154

    2018

  6. [14]

    Learning without forgetting,

    Z. Li and D. Hoiem, “Learning without forgetting,” IEEE transactions on pattern analysis and machine in- telligence, vol. 40, no. 12, pp. 2935–2947, 2017

    2017

  7. [15]

    Improving replay sample selection and storage for less forgetting in continual learning,

    D. Brignac, N. Lobo, and A. Mahalanobis, “Improving replay sample selection and storage for less forgetting in continual learning,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 3540–3549

    2023

  8. [16]

    Dark experience for general continual learning: a strong, simple baseline,

    P. Buzzega, M. Boschini, A. Porrello, D. Abati, and S. Calderara, “Dark experience for general continual learning: a strong, simple baseline,” Advances in neu- ral information processing systems, vol. 33, pp. 15 920– 15 930, 2020

    2020

  9. [17]

    Efficient lifelong learning with a-gem,

    A. Chaudhry, M. Ranzato, M. Rohrbach, and M. El- hoseiny, “Efficient lifelong learning with a-gem,” arXiv preprint arXiv:1812.00420, 2018

    2018 arXiv

  10. [18]

    Piggyback: Adapting a single network to multiple tasks by learn- ing to mask weights,

    A. Mallya, D. Davis, and S. Lazebnik, “Piggyback: Adapting a single network to multiple tasks by learn- ing to mask weights,” in Proceedings of the European conference on computer vision (ECCV) , 2018, pp. 67– 82

    2018

  11. [19]

    Online con- tinual learning in keyword spotting for low-resource de- vices via pooling high-order temporal statistics,

    U. Michieli, P. P. Parada, and M. Ozay, “Online con- tinual learning in keyword spotting for low-resource de- vices via pooling high-order temporal statistics,” arXiv preprint arXiv:2307.12660, 2023

    2023 arXiv

  12. [20]

    An inves- tigation of the combination of rehearsal and knowledge distillation in continual learning for spoken language understanding,

    U. Cappellazzo, D. Falavigna, and A. Brutti, “An inves- tigation of the combination of rehearsal and knowledge distillation in continual learning for spoken language understanding,” in Proceedings of INTERSPEECH 2023, 2023, pp. 735–739

    2023

  13. [21]

    Rehearsal-free online contin- ual learning for automatic speech recognition,

    S. Vander Eeckt et al. , “Rehearsal-free online contin- ual learning for automatic speech recognition,” arXiv e- prints, pp. arXiv–2306, 2023

    2023

  14. [22]

    Dual-memory multi- modal learning for continual spoken keyword spotting with confidence selection and diversity enhancement,

    Z. Yang, D. Ng, X. Li, C. Zhang, R. Jiang, W. Xi, Y . Ma, C. Ni, J. Zhao, B. Ma et al. , “Dual-memory multi- modal learning for continual spoken keyword spotting with confidence selection and diversity enhancement,” in Proc. INTERSPEECH, 2023

    2023

  15. [23]

    Clrl-tuning: A novel continual learning approach for automatic speech recog- nition

    Z. Wang, F. Hou, and R. Wang, “Clrl-tuning: A novel continual learning approach for automatic speech recog- nition.”

  16. [24]

    Massively multilingual asr: A lifelong learning solu- tion,

    B. Li, R. Pang, Y . Zhang, T. N. Sainath, T. Strohman, P. Haghani, Y . Zhu, B. Farris, N. Gaur, and M. Prasad, “Massively multilingual asr: A lifelong learning solu- tion,” in ICASSP 2022-2022 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP)....

    2022

  17. [25]

    Master-asr: achieving multilingual scal- ability and low-resource adaptation in asr with modu- lar learning,

    Z. Yu, Y . Zhang, K. Qian, C. Wan, Y . Fu, Y . Zhang, and Y . C. Lin, “Master-asr: achieving multilingual scal- ability and low-resource adaptation in asr with modu- lar learning,” in International Conference on Machine Learning. PMLR, 2023, pp. 40 475–40 487

    2023

  18. [26]

    Weight averaging: A simple yet effective method to overcome catastrophic forget- ting in automatic speech recognition,

    S. V . Eeckt et al. , “Weight averaging: A simple yet effective method to overcome catastrophic forget- ting in automatic speech recognition,” arXiv preprint arXiv:2210.15282, 2022

    2022

  19. [27]

    Cl-masr: A continual learn- ing benchmark for multilingual asr,

    L. Della Libera, P. Mousavi, S. Zaiem, C. Sub- akan, and M. Ravanelli, “Cl-masr: A continual learn- ing benchmark for multilingual asr,” arXiv preprint arXiv:2310.16931, 2023

    2023 arXiv

  20. [28]

    Lookup-table recurrent lan- guage models for long tail speech recognition,

    W. R. Huang, T. N. Sainath, C. Peyser, S. Kumar, D. Ry- bach, and T. Strohman, “Lookup-table recurrent lan- guage models for long tail speech recognition,” arXiv preprint arXiv:2104.04552, 2021

    2021 arXiv

  21. [29]

    Comparison of multilingual self- supervised and weakly-supervised speech pre-training for adaptation to unseen languages,

    A. Rouditchenko, S. Khurana, S. Thomas, R. Feris, L. Karlinsky, H. Kuehne, D. Harwath, B. Kings- bury, and J. Glass, “Comparison of multilingual self- supervised and weakly-supervised speech pre-training for adaptation to unseen languages,” arXiv preprint arXiv:2305.12606, 2023

    2023 arXiv

  22. [30]

    Unsupervised cross-lingual representa- tion learning for speech recognition,

    A. Conneau, A. Baevski, R. Collobert, A. Mohamed, and M. Auli, “Unsupervised cross-lingual representa- tion learning for speech recognition,” arXiv preprint arXiv:2006.13979, 2020

    2006 arXiv

  23. [31]

    Wavlm: Large-scale self-supervised pre-training for full stack speech processing,

    S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao et al. , “Wavlm: Large-scale self-supervised pre-training for full stack speech processing,” IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, 2022

    2022

  24. [32]

    Language identification networks for multi- lingual everyday recordings,

    K. Praveen, B. Radhakrishnan, K. Sabu, A. Pandey, and M. Shaik, “Language identification networks for multi- lingual everyday recordings,” 2023

    2023

  25. [33]

    Xls-r: Self-supervised cross-lingual speech representation learning at scale,

    A. Babu, C. Wang, A. Tjandra, K. Lakhotia, Q. Xu, N. Goyal, K. Singh, P. von Platen, Y . Saraf, J. Pino et al. , “Xls-r: Self-supervised cross-lingual speech representation learning at scale,” arXiv preprint arXiv:2111.09296, 2021

    2021 arXiv

  26. [34]

    Salmonn: Towards generic hear- ing abilities for large language models,

    C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. Ma, and C. Zhang, “Salmonn: Towards generic hear- ing abilities for large language models,” arXiv preprint arXiv:2310.13289, 2023

    2023 arXiv

  27. [35]

    Does an lstm for- get more than a cnn? an empirical study of catastrophic forgetting in nlp,

    G. Arora, A. Rahimi, and T. Baldwin, “Does an lstm for- get more than a cnn? an empirical study of catastrophic forgetting in nlp,” inProceedings of the The 17th Annual Workshop of the Australasian Language Technology As- sociation, 2019, pp. 77–86

    2019

  28. [36]

    How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings,

    K. Ethayarajh, “How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings,” arXiv preprint arXiv:1909.00512, 2019

    1909 arXiv

  29. [37]

    Introduction to word embedding and word2vec,

    D. Karani, “Introduction to word embedding and word2vec,” Towards Data Science, vol. 1, 2018

    2018

  30. [38]

    End to end spoken language diarization with wav2vec embeddings

    J. Mishra, J. N. Patil, A. Chowdhury, and S. M. Prasanna, “End to end spoken language diarization with wav2vec embeddings.”

  31. [39]

    Self-supervised learning representation based accent recognition with persistent accent mem- ory

    R. Li, Z. Xie, H. Xu, Y . Peng, H. Liu, H. Huang, and E. S. Chng, “Self-supervised learning representation based accent recognition with persistent accent mem- ory.”

  32. [40]

    Multi- resolution approach to identification of spoken lan- guages and to improve overall language diarization sys- tem using whisper model,

    B. Vachhani, D. Singh, and R. Lawyer, “Multi- resolution approach to identification of spoken lan- guages and to improve overall language diarization sys- tem using whisper model,” in Proc. INTERSPEECH , vol. 2023, 2023, pp. 1993–1997

    2023

  33. [41]

    Im- proved alignment for score combination of rnn-t and ctc decoder for online decoding,

    C. Y . Kwok, J. Q. Yip, and E. S. Chng, “Im- proved alignment for score combination of rnn-t and ctc decoder for online decoding,” in Text, Speech, and Dialogue: 27th International Confer- ence, TSD 2024, Brno, Czech Republic, September 9–13, 2024, Proceedings, Part II . Berli...

    2024 doi

  34. [42]

    Diverse beam search: Decoding diverse solutions from neu- ral sequence models,

    A. K. Vijayakumar, M. Cogswell, R. R. Selvaraju, Q. Sun, S. Lee, D. Crandall, and D. Batra, “Diverse beam search: Decoding diverse solutions from neu- ral sequence models,”arXiv preprint arXiv:1610.02424, 2016

    2016 arXiv

  35. [43]

    Speechbrain: A general-purpose speech toolkit,

    M. Ravanelli, T. Parcollet, P. Plantinga, A. Rouhe, S. Cornell, L. Lugosch, C. Subakan, N. Dawalatabad, A. Heba, J. Zhong et al. , “Speechbrain: A general-purpose speech toolkit,” arXiv preprint arXiv:2106.04624, 2021

    2021 arXiv

  36. [44]

    Common voice: A massively-multilingual speech corpus,

    R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, J. Meyer, R. Morais, L. Saunders, F. M. Tyers, and G. Weber, “Common voice: A massively-multilingual speech corpus,” arXiv preprint arXiv:1912.06670, 2019

    1912 arXiv

  37. [45]

    Decoupled weight decay regularization,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.