REVIEW 3 major objections 6 minor 36 references
AsyncSwitch: Asynchronous Text-Speech Adaptation for Code-Switched ASR
T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper claims that training Whisper's decoder on 38M text-only utterances before reconnecting audio improves code-switched Malay-English ASR by 9.02% relative WER over a speech-only baseline.
desk verdict Useful three-stage recipe for text-only ASR adaptation, but the paper never isolates Stage 1, so the headline gain may come from fine-tuning and merging alone. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing trick is Stage 1's zeroed encoder input ($x=0$): with cross-attention frozen and the encoder output absent, the decoder's self-attention, feedforward, and output layers are trained as a conditional language model on code-switched text. Stage 2 then unfreezes only the cross-attention layers on 5k hours of paired speech-text to reconnect the encoder, and Stage 3 fine-tunes all parameters. A final parameter-merge step interpolates the fine-tuned model with the original Whisper at ratio 0.4, chosen to preserve code-switched gains without sacrificing English. The evaluation also uses a combined Malay-English language prompt across test sets.
What would settle it
Train Whisper-Large-v3 on the same 5k hours of paired speech-text data with no Stage 1 or Stage 2, exactly as the paper's WHISPER-5K baseline, and evaluate on the same Malay-English code-switched test sets; if its WER is not worse than the reported 17.04% (e.g., within one standard deviation), the claimed text-stage benefit is not real.
Extended reading notes
Core claim
The central discovery is that a Whisper decoder can be adapted to code-switched language patterns using text alone, before it ever sees audio, and this text-stage transfers to speech once cross-attention is realigned. The paper calls this asynchronous because language adaptation and speech-text alignment happen in separate stages rather than jointly. The final model, WHISPER-38M-5K, reports 18.22% average WER across Singlish, Malay, and code-switched Malay-English test sets versus 23.77% for the original Whisper and 21.20% for a model fine-tuned on the same 5k hours of speech without the text stage; on code-switched Malay-English specifically it reaches 17.04% versus 18.73% for the speech-only baseline.
Load-bearing premise
The claim rests on the premise that training the decoder on text alone, with the encoder silenced, leaves code-switched language patterns that survive once cross-attention is retrained and the whole model is fine-tuned; if those text-stage patterns do not transfer to speech, the three-stage design has no advantage over ordinary speech fine-tuning.
Editorial extensions
If this is right
- Large unpaired text corpora can transfer into ASR gains for low-resource code-switched languages without generating synthetic audio.
- The three-stage progression gives incremental gains, with the full fine-tuning stage producing the best performance across all domains.
- The approach prevents catastrophic forgetting, improving English recognition on the OpenASR Leaderboard by 5.37% relative to the original model.
- The method outperforms a 6x larger SpeechLLM on the Southeast Asian test sets and beats commercial APIs on Malay and code-switched Malay-English, though not on Singlish.
- Text scale and diversity both matter: a smaller 1.7M-utterance Malay-focused corpus helps Malay and code-switching more but hurts Singlish, while the larger 38M corpus gives the best overall average.
Reading between the lines
- If decoder text-only adaptation transfers this cleanly, the same three-stage recipe should work for other encoder-decoder ASR families and other code-switched language pairs where text is abundant but paired speech is scarce; a direct test would be Tamil-English or Mandarin-Malay with the same 5k-hour setup.
- The text stage could be combined with synthetic speech augmentation rather than treated as an alternative, potentially reducing the amount of synthetic audio needed to reach a target WER.
- The ablation suggests that Stage 1 plus Stage 2 alone may already be useful for purely text-driven domain adaptation when no paired speech is available at all.
- A sweep of paired-speech hours (e.g., 500h, 1k, 2k, 5k) could quantify how much scarce speech data the text stage can replace, which would clarify the method's value beyond the specific 5k-hour setting.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes AsyncSwitch, a three-stage adaptation framework for code-switched speech recognition. The approach first fine-tunes the decoder self-attention, feed-forward, and output layers of a Whisper model on 38M text utterances with the encoder output zeroed (Stage 1), then fine-tunes only cross-attention on 5k hours of paired speech-text data (Stage 2), and finally fine-tunes the full model on the same paired data (Stage 3). The final model is linearly merged with the original Whisper at a ratio of 0.4. Evaluations on Malay-English code-switched, Malay, Singlish, and English test sets report a 9.02% relative WER reduction over a same-data speech-only baseline (WHISPER-5K) and improvements over commercial systems and a SpeechLLM baseline.
Significance. If the results hold, the paper shows a computationally cheap way to use unpaired text to improve low-resource code-switched ASR, which is a practically valuable contribution. The evaluation is fairly comprehensive, with multiple test sets, comparisons to strong baselines, and an ablation of the three stages. The paper also reports that the method improves English performance, addressing catastrophic forgetting. However, the central mechanism (the text-only Stage 1) is not isolated in the ablation, and the merging ratio is chosen on the test sets, so the significance is currently conditional on additional experiments.
major comments (3)
- [Section III-B, Stage 1 and Table II] The claim that text-only adaptation transfers to speech is not directly tested. The paper compares WHISPER-38M-5K (Stages 1+2+3) with WHISPER-5K (Stage 3 only), which differs by both Stage 1 and Stage 2. Table II shows that Stage 1 alone (Phase1-0.4) yields worse WER than the original on Malay (31.32 vs 29.82) and code-switch (23.41 vs 20.94), and Stage 1+2 (Phase2-0.4) is still worse. Only after Stage 3 does the model improve. Without an experiment that runs Stages 2+3 without Stage 1 (or replaces Stage 1 with a non-code-switched or random text), the observed gains cannot be attributed to the asynchronous text adaptation, which is the core contribution. This is a load-bearing gap.
- [Table V and Section IV-B] The merging ratio of 0.4 is selected based on test-set performance ('We choose a merging ratio of 0.4 for best code-switching performance'). This is test-set selection and can overfit. The paper should either use a validation split not used in any test set, or report results across the full range of ratios. Additionally, it is not clear whether the baselines (e.g., WHISPER-5K) are also merged at the same ratio, and whether the ratio was tuned for those models. Without this, the comparison may be unfair to the baselines.
- [Table I] The reported gains are not accompanied by confidence intervals or significance tests. Some differences are small (e.g., OpenASR 7.04 vs 7.44; NLB 22.75 vs 22.52), and WER measurements on limited test sets have non-negligible variance. Adding significance tests or error bars would substantially strengthen the central claim.
minor comments (6)
- [Section I] The phrase 'Conventional finetune approaches' should be 'Conventional fine-tuning approaches'.
- [Section IV-B] The sentence '1k-hours phrase-mixed [8] - all are same as [8]' is awkward and should be rephrased.
- [References] Reference [8] has inconsistent capitalization ('asr' should be 'ASR', 'singapore' should be 'Singapore'); reference [13] is a Hugging Face URL with spaces that should be corrected.
- [Table I] In the model column, 'WHISPERTURBO-V3' is missing a space; 'MERALION-AUDIOLLM-WHISPER-SEA-LION' is written with a hyphen inconsistency.
- [Section V-A] The percentages '23.35%' and '14.05%' should be explicitly labeled as relative WER reductions to avoid ambiguity.
- [Section III-B, Stage 1] The description 'we zero out the encoder output (x = 0)' is slightly confusing because x was earlier defined as audio encoder features; clarify that the cross-attention layer receives zero-valued key/value vectors.
Circularity Check
The headline 9.02% code-switch WER gain is partially circular: the merging ratio 0.4 is selected using the same test sets that produce the reported 17.04 CS WER, so the central improvement is a fit to the evaluation labels rather than an independent prediction.
-
fitted input called prediction
[Section V-B3 (Optimal Merging Ratio, Table V); also Section IV-B2]
"We choose a merging ratio of 0.4 for best code-switching performance (17.04) while maintaining acceptable Singlish and Malay results. Although 0.8 achieves the best overall average (16.77), it degrades performance on all other scenarios (Singlish, CS, OpenASR Leaderboard with diverse English)."
The final system WHISPER-38M-5K is defined by merging ratio 0.4, chosen by comparing WERs on the evaluation test sets themselves (Table V). The headline claim of 9.02% relative code-switch improvement over WHISPER-5K is computed from the very 17.04 CS WER that the ratio grid was selected to minimize (18.73 -> 17.04). Thus the central reported number is the best of the ratios tried on the test labels, not an out-of-sample prediction. No held-out validation split is used for this selection, so the improvement is partly manufactured by the selection procedure.
full rationale
The three-stage loss chain (L1, L2, L3) is not definitionally circular: each stage optimizes a standard cross-entropy objective, and the stages are not defined in terms of the final WER. The evaluation is also anchored to fixed test sets, and the comparison against Whisper-5K is a real empirical measurement once a configuration is fixed. However, the configuration is not fixed independently: the model-merging coefficient is selected on the test sets, and the specific code-switch WER used in the abstract is the one optimized by that selection. This is a partial circularity in the evaluation dimension. Separately, the claim that Stage 1 text-only adaptation transfers to speech (Section III-B, L1) is unsupported by an ablation removing Stage 1; Table II shows Phase1 and Phase2 are worse than the original on Malay and code-switch, so only Stage 3 full fine-tuning reaches the reported gains. This is an omitted proof rather than a circular step. The benchmarks and 20% of the training speech come from the authors' own prior work [8], which adds a self-referential burden but does not by itself make the derivation circular. The stated limitation that the bilingual Singapore/Malaysian setting may limit generalizability is acknowledged in Section VI and is consistent with the narrow external evidence.
Assumptions & free parameters
free parameters (2)
- model merging ratio =
0.4
- 5-gram LM interpolation coefficients alpha and beta =
alpha in [0, 0.1], beta in [-0.2, 0.2]
assumptions (4)
- domain assumption Whisper decoder can be treated as a conditional language model when encoder output is zeroed, and updating only self-attention, feedforward, and output layers in Stage 1 does not break later speech alignment.
- domain assumption The 38M text utterances are representative enough of target code-switched and monolingual speech domains.
- domain assumption The 5k-hour paired speech-text corpus, including 1k hours of phrase-mixed synthetic audio from prior work, is a fair shared basis for comparing WHISPER-5K and WHISPER-38M-5K.
- domain assumption Merging the fine-tuned model with the original Whisper weights via linear interpolation is a valid way to trade off domain adaptation and general English performance.
Cite this review
Pith. "Pith review of AsyncSwitch: Asynchronous Text-Speech Adaptation for Code-Switched ASR." pith.science (2026). https://pith.science/paper/KXGIQUK7
@misc{pith2026250614190,
author = {Pith},
title = {Pith review of: AsyncSwitch: Asynchronous Text-Speech Adaptation for Code-Switched ASR},
year = {2026},
howpublished = {\url{https://pith.science/paper/KXGIQUK7}},
note = {Machine review of arXiv:2506.14190}
}
read the original abstract
Developing code-switched ASR systems is challenging due to language ambiguity and limited exposure to multilingual, code-switched data, while collecting such speech is costly. Prior work generates synthetic audio from text, but these methods are computationally intensive and hard to scale. We introduce AsyncSwitch, a novel asynchronous adaptation framework that leverages large-scale, text-rich web data to pre-expose ASR models to diverse code-switched domains before fine-tuning on paired speech-text corpora. Our three-stage process (1) trains decoder self-attention and feedforward layers on code-switched text, (2) aligns decoder and encoder via cross-attention using limited speech-text data, and (3) fully fine-tunes the entire model. Experiments with Whisper on Malay-English code-switching demonstrate a 9.02% relative WER reduction, while improving monolingual performance in Singlish, Malay, and other English variants.
Figures
Reference graph
Works this paper leans on
-
[5]
Whisper- lm: Improving asr models with language models for low-resource languages,
X. de Zuazo, E. Navas, I. Saratxaga, and I. H. Rioja, “Whisper- lm: Improving asr models with language models for low-resource languages,” 2025. [Online]. Available: https://arxiv.org/abs/2503.23542
arXiv 2025
-
[8]
T. Nguyen and H. D. Tran, “Can we train asr systems on code-switch without real code-switch data? case study for singapore’s languages,” in Interspeech 2025 , 2025
work page 2025
-
[1]
Wikipedia contributors, “Malay language,” https://en.wikipedia.org/wiki/ Malay language, n.d., accessed: 2025-05-12
work page 2025
-
[2]
Robust speech recognition via large-scale weak supervision,
A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” 2022. [Online]. Available: https://arxiv.org/abs/2212.04356
arXiv 2022
-
[3]
Meta- transfer learning for code-switched speech recognition,
G. I. Winata, S. Cahyawijaya, Z. Lin, Z. Liu, P. Xu, and P. Fung, “Meta- transfer learning for code-switched speech recognition,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault, Eds. Online: Association for Computational Linguistics, Jul. 2020, pp. 3770–
work page 2020
-
[4]
A semi- supervised complementary joint training approach for low-resource speech recognition,
Y .-Q. Du, J. Zhang, X. Fang, M.-H. Wu, and Z.-W. Yang, “A semi- supervised complementary joint training approach for low-resource speech recognition,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 31, pp. 3908–3921, 2023
work page 2023
-
[6]
Language Bias in Self-Supervised Learning For Automatic Speech Recognition
E. Storey, N. Harte, and P. Bell, “Language bias in self-supervised learning for automatic speech recognition,” 2025. [Online]. Available: https://arxiv.org/abs/2501.19321
work page Pith review arXiv 2025
-
[7]
Careless whisper: Speech-to-text hallucination harms,
A. Koenecke, A. S. G. Choi, K. X. Mei, H. Schellmann, and M. Sloane, “Careless whisper: Speech-to-text hallucination harms,” in The 2024 ACM Conference on Fairness, Accountability, and Transparency , ser. FAccT ’24. ACM, Jun. 2024, p. 1672–1681. [Online]. Available: http://dx.doi.org/10.1145/3630106.3658996
arXiv 2024
Show all 36 references
-
[9]
Seamless: Multi- lingual expressive and streaming speech translation,
S. Communication, L. Barrault, Y .-A. Chung et al. , “Seamless: Multi- lingual expressive and streaming speech translation,” 2023
2023
-
[10]
A survey on speech large language models,
J. Peng, Y . Wang, Y . Xi, X. Li, X. Zhang, and K. Yu, “A survey on speech large language models,” 2025. [Online]. Available: https://arxiv.org/abs/2410.18908
2025
-
[11]
I2r astar advanced code-switch api,
A. ALI-ACS I2R, “I2r astar advanced code-switch api,” 2025. [Online]. Available: https://demo.kkode.com
2025
-
[12]
[Online]
Azure, 2025. [Online]. Available: https://azure.microsoft.com/en-us/ products/ai-services/ai-speech
2025
-
[13]
Open automatic speech recognition leaderboard,
V . Srivastav, S. Majumdar, N. Koluguri, A. Moumen, S. Gandhi et al. , “Open automatic speech recognition leaderboard,” https://huggingface. co/spaces/hf-audio/open asr leaderboard, 2023
2023
-
[14]
Text-only domain adaptation using unified speech-text representation in transducer,
L. Huang, B. Li, J. Zhang, L. Lu, and Z. Ma, “Text-only domain adaptation using unified speech-text representation in transducer,” in Interspeech 2023 , 2023, pp. 386–390
2023
-
[15]
Internal language model estimation for domain-adaptive end-to-end speech recognition,
Z. Meng, S. Parthasarathy, E. Sun, Y . Gaur, N. Kanda, L. Lu, X. Chen, R. Zhao, J. Li, and Y . Gong, “Internal language model estimation for domain-adaptive end-to-end speech recognition,” in 2021 IEEE Spoken Language Technology Workshop (SLT) , 2021, pp. 243–250
2021
-
[16]
AdaBERT-CTC: Leveraging BERT-CTC for text-only domain adaptation in ASR,
T. Vuong, K. Mundnich, D. Bekal, V . Elluru, S. Ronanki, and S. Bodapati, “AdaBERT-CTC: Leveraging BERT-CTC for text-only domain adaptation in ASR,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track, M. Wang and I. Zitoun...
2023
-
[17]
Building a unified code-switching asr system for south african lan- guages,
E. Yılmaz, A. Biswas, E. van der Westhuizen, F. de Wet, and T. Niesler, “Building a unified code-switching asr system for south african lan- guages,” in Interspeech 2018 , 2018, pp. 1923–1927
2018
-
[18]
Improving code-switched ASR with linguistic information,
J. Chi and P. Bell, “Improving code-switched ASR with linguistic information,” in Proceedings of the 29th International Conference on Computational Linguistics , N. Calzolari, C.-R. Huang, H. Kim, J. Pustejovsky, L. Wanner, K.-S. Choi, P.-M. Ryu, H.-H. Chen, L. Donatelli, H. J...
2022
-
[19]
Improving low resource code-switched asr using augmented code-switched tts,
Y . Sharma, B. Abraham, K. Taneja, and P. Jyothi, “Improving low resource code-switched asr using augmented code-switched tts,” in Interspeech 2020 , 2020, pp. 4771–4775
2020
-
[20]
Speechlm: Enhanced speech pre-training with unpaired textual data,
Z. Zhang, S. Chen, L. Zhou, Y . Wu, S. Ren, S. Liu, Z. Yao, X. Gong, L. Dai, J. Li, and F. Wei, “Speechlm: Enhanced speech pre-training with unpaired textual data,” IEEE/ACM Trans. Audio, Speech and Lang. Proc. , vol. 32, p. 2177–2187, Mar. 2024. [Online]. Available: https://d...
2024
-
[21]
Data augmentation for end-to-end code-switching speech recognition,
C. Du, H. Li, Y . Lu, L. Wang, and Y . Qian, “Data augmentation for end-to-end code-switching speech recognition,” in 2021 IEEE Spoken Language Technology Workshop (SLT) . IEEE, Jan. 2021, p. 194–200. [Online]. Available: http://dx.doi.org/10.1109/slt48900.2021.9383620
2021
-
[22]
NLB Dataset - Discover Our Collections,
National Library Board, Singapore, “NLB Dataset - Discover Our Collections,” https://www.nlb.gov.sg/main/discover-and-learn/ discover-our-collections/National-Library-Datasets, 2025, accessed: 2025-05-27
2025
-
[23]
Building the singapore english national speech corpus,
J. X. Koh, A. Mislan, K. Khoo, B. Ang, W. Ang, C. Ng, and Y .-Y . Tan, “Building the singapore english national speech corpus,” in Interspeech 2019, 2019, pp. 321–325
2019
-
[24]
Malay conversational speech corpus (asr-malcsc),
“Malay conversational speech corpus (asr-malcsc),” https: //magichub.com/datasets/malay-conversational-speech-corpus/, 2025, accessed: 2025-05-27
2025
-
[25]
Meralion-audiollm: Bridging audio and language with large language models,
Y . He, Z. Liu, S. Sun, B. Wang, W. Zhang, X. Zou, N. F. Chen, and A. T. Aw, “Meralion-audiollm: Bridging audio and language with large language models,” 2025. [Online]. Available: https://arxiv.org/abs/2412.09818
2025 arXiv
-
[26]
Sea-pile v1 dataset,
AI Singapore, “Sea-pile v1 dataset,” https://huggingface.co/datasets/ aisingapore/SEA-PILE-v1, 2023, accessed: 2025-05-06
2023
-
[27]
Librispeech: An asr corpus based on public domain audio books,
V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: An asr corpus based on public domain audio books,” in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015, pp. 5206–5210
2015
-
[28]
Language identification model – lang-id-voxlingua107- ecapa,
SpeechBrain, “Language identification model – lang-id-voxlingua107- ecapa,” https://huggingface.co/speechbrain/lang-id-voxlingua107-ecapa, 2021, accessed: 2025-05-06
2021
-
[29]
Fasttext language identification model,
Facebook AI, “Fasttext language identification model,” https://huggingface.co/facebook/fasttext-language-identification, 2020, accessed: 2025-05-06
2020
-
[30]
Leave no knowledge behind during knowledge distillation: Towards practical and effective knowledge distillation for code-switching asr using realistic data,
L.-H. Tseng, Z.-C. Chen, W.-S. Chang, C.-K. Lee, T.-R. Huang, and H. yi Lee, “Leave no knowledge behind during knowledge distillation: Towards practical and effective knowledge distillation for code-switching asr using realistic data,” 2024. [Online]. Available: https://arxiv....
2024 arXiv
-
[31]
Specaugment: A simple data augmentation method for automatic speech recognition,
D. S. Park, W. Chan, Y . Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V . Le, “Specaugment: A simple data augmentation method for automatic speech recognition,” in Interspeech 2019 , 2019, pp. 2613– 2617
2019
-
[32]
MUSAN: A Music, Speech, and Noise Corpus,
D. Snyder, G. Chen, and D. Povey, “MUSAN: A Music, Speech, and Noise Corpus,” 2015, arXiv:1510.08484v1
2015 arXiv
-
[33]
Adapting openai’s whisper for speech recognition on code-switch mandarin- english seame and asru2019 datasets,
Y . Yang, Y . Peng, X. Zhong, H. Huang, and E. S. Chng, “Adapting openai’s whisper for speech recognition on code-switch mandarin- english seame and asru2019 datasets,” 2023. [Online]. Available: https://arxiv.org/abs/2311.17382
2023 arXiv
-
[34]
Comparing the level of code-switching in corpora,
B. Gamb ¨ack and A. Das, “Comparing the level of code-switching in corpora,” in Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16) , N. Calzolari, K. Choukri, T. Declerck, S. Goggi, M. Grobelnik, B. Maegaard, J. Mariani, H. Mazo, A...
2016
-
[35]
Linear interpolation in parameter space is good enough for fine-tuned language models,
M. Rofin, N. Balagansky, and D. Gavrilov, “Linear interpolation in parameter space is good enough for fine-tuned language models,” 2022. [Online]. Available: https://arxiv.org/abs/2211.12092
2022 arXiv
-
[3776]
Available: https://aclanthology.org/2020.acl-main.348/
[Online]. Available: https://aclanthology.org/2020.acl-main.348/
2020
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.