Pith. sign in

REVIEW 4 major objections 4 minor 47 references

Analyzing Mitigation Strategies for Catastrophic Forgetting in End-to-End Training of Spoken Language Models

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read End-to-end speech adaptation destroys most of a text LLM's knowledge; experience replay keeps nearly all of it intact.

desk verdict A useful but confounded case study: replay wins, but the D0 proxy overlaps the eval distribution, so the headline claim overreaches. read the letter →

arxiv 2505.17496 v1 pith:IVUV5NFI submitted 2025-05-23 cs.CL cs.AIcs.LGcs.SDeess.AS

Chi-Yuan Hsiao , Ke-Han Lu , Kai-Wei Chang , Chih-Kai Yang , Wei-Chih Chen , Hung-yi Lee This is my paper · ORCID
classification cs.CLcs.AIcs.LGcs.SDeess.AS
keywords spokenlanguagemodelcatastrophicforgettingcontinuallearningexperiencereplaymergingLoRAscalingfactorquestionansweringinstructionfollowing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Spoken language models are usually built by taking a pre-trained text LLM and fine-tuning it in stages on speech recognition, speech synthesis, and spoken question answering. The paper shows that this multi-stage process causes catastrophic forgetting: on the LLaMA-Questions benchmark, text question-answering accuracy drops from 70.0 to 14.3 without any mitigation. Among three mitigation families—model merging, discounting the LoRA scaling factor, and experience replay—experience replay is the most effective, bringing the same benchmark back to 66.3 while still allowing the new speech abilities to be learned. Combining replay with model merging or with a lower LoRA scaling factor yields further improvements on some speech-to-text settings. The practical upshot is that a simple training-time replay buffer can make SLM pipelines both more knowledgeable and more stable.

What carries the argument

The central object is the replay-augmented training set. At each stage $i$, the dataset is $$D'_i = D_i \cup \bigcup_{j=0}^{i-1} \mathrm{Sample}(D_j, s|D_i|),$$ which adds random samples from every earlier stage to the current stage's data, with sampling ratio $s = 0.005$. The original text dataset $D_0$ is taken to be Magpie-Air, a synthetic instruction-tuning set generated by Llama-3-8B, which the paper assumes closely approximates the original Llama-3.2-11B training distribution. This augmentation converts each training stage into multi-task learning and is the mechanism that carries the argument: it is the only one of the three compared strategies that substantially stops text-knowledge loss while preserving newly learned speech capabilities. The other strategies—merging weights across stages and lowering the LoRA scaling factor $\alpha$ from 16 to 15 or 14 at inference—are also evaluated both alone and on top of replay.

What would settle it

Run the same three-stage pipeline with experience replay where $D_0$ is a held-out sample of the original pretraining corpus, or where the replay buffer is deliberately chosen to avoid anything similar to the evaluation tasks. If text QA retention stays near 66.3, the central claim holds; if it collapses toward the no-mitigation 14.3, the replay advantage is an artifact of overlapping distributions between Magpie-Air and the test sets.

Watch

Extended reading notes

Core claim

The paper's central claim is that experience replay, not model merging or LoRA scaling discount, is the mitigation strategy that carries the load in end-to-end SLM training. In the authors' pipeline, a Llama-3.2-11B backbone is adapted with a speech encoder and vocoder and trained through ASR, TTS, and SQA stages. Without mitigation, text QA on LLaMA-Questions falls to 14.3 from an original 70.0; replay at sampling ratio 0.005 retains 66.3 and also produces the best S2T and S2S SQA scores among single strategies, with 50.3 and 28.7 on the same benchmark. Replay is the only strategy that remains robust in the speech-to-speech setting, and applying model merging or LoRA scaling after replay can push some S2T numbers higher, though not uniformly the S2S numbers. The authors conclude that replay should be a central component of SLM training recipes.

Load-bearing premise

The replay result rests on treating the synthetic Magpie-Air instruction set as a stand-in for Llama-3.2's original training data; if that approximation fails, the retention numbers may show the model relearning Magpie-Air rather than preserving its original knowledge.

Editorial extensions

If this is right

  • Multi-stage SLM training without mitigation loses roughly 80% of original text QA performance, while replay retains about 95% on LLaMA-Questions (66.3 vs 70.0).
  • Experience replay is the only single strategy that keeps speech-to-speech QA robust; model merging and LoRA scaling alone leave S2S accuracy near zero.
  • Adding replay to model merging or to LoRA scaling can improve speech-to-text SQA further, with the best reported S2T numbers reaching the mid-50s.
  • The largest forgetting occurs between the ASR and TTS stages, so retention efforts should focus on the transition into speech generation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because $D_0$ is a synthetic instruction set rather than the true pretraining corpus, part of the measured retention may be relearning Magpie-Air itself; testing replay with a buffer drawn from the actual pretraining mixture would separate genuine knowledge retention from test-set overlap.
  • Replay's success suggests SLM training should be treated explicitly as a continual-learning problem, where buffer selection, sampling schedules, and forgetting-aware task ordering become first-class design choices.
  • A natural next experiment is to check replay on longer-form, multilingual, and reasoning-heavy abilities, which are likely more fragile than the short QA and instruction-following tasks reported here.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper studies catastrophic forgetting during multi-stage end-to-end training of a spoken language model built on Llama-3.2-11B-Vision-Instruct, with ASR, TTS, and spoken question answering (SQA) stages. It compares three mitigation strategies: model merging (linear, TIES, DARE), discounting the LoRA scaling factor at inference time, and experience replay. The central claim is that experience replay is the most effective strategy, substantially reducing knowledge loss on text QA and instruction-following tasks while maintaining speech QA performance, and that combining replay with the other strategies yields further, albeit modest, gains. The experiments report accuracies on LLaMA-Questions, Spoken WebQuestions, Audio Trivia QA, and IFEval under T2T, S2T, and S2S settings.

Significance. If the main result holds, the paper provides a practically useful comparison of forgetting-mitigation strategies for a realistic multi-stage SLM training pipeline, a question of growing importance as SLMs are increasingly built by adapting text LLMs. The work covers three families of mitigation methods and evaluates both text and speech modalities, which is a useful breadth. The paper also ships a concrete case study with an 11B-parameter model, so the empirical findings are directly informative for practitioners. Its main strength is the systematic side-by-side comparison; its main weaknesses are the reliance on a single synthetic proxy dataset for experience replay and the absence of statistical validation, both of which currently limit the strength of the claims that can be drawn.

major comments (4)
  1. [Section 3.2.1 and Section 3.3.1 (Eq. (3))] The choice of Magpie-Air as D0 is load-bearing for the central claim, but its validity is only assumed, not established. The paper writes, "We assume that its distribution closely approximates that of the original training data" (Section 3.2.1), and later justifies the choice by saying Magpie-Air comes from the same Llama family (Section 3.3.1). No evidence is provided for the approximation. Because D0 is replayed at every training stage and the evaluation tasks (LLaMA-Questions, WebQuestions, Trivia QA, IFEval) are all QA or instruction-following benchmarks, the large replay advantage seen in Table 1 (e.g., LLaMA T2T 66.3 vs. 14.3) may reflect that the replay buffer closely matches the evaluation task distribution rather than that replay preserves knowledge from the original pretraining corpus. This concern is compounded by the fact that the SQA training stage itself is built from Magpie-Air, so the replay data, the training data, and the evaluation data share a common source. A concrete test would be to rerun replay with a different D0 (e.g., a general text corpus or the actual pretraining data if available) or to evaluate on tasks that are not instruction-following in format; if the advantage disappears, the paper's conclusion must be substantially qualified.
  2. [Table 1 and Section 4.2] All results come from single runs with no error bars, confidence intervals, or significance tests. This is particularly problematic for the paper's second claim that mixed strategies provide "further gains" over replay alone. The differences are small and inconsistent across metrics: for example, Merge (Linear) w/ R improves LLaMA T2T from 66.3 to 68.0 but decreases Trivia S2T from 25.2 to 16.3, while Scaling (α=15) w/ R improves some S2T numbers but lowers IFEval Prompt from 47.5 to 43.7. Without repeated runs or statistical testing, these fluctuations cannot be distinguished from training noise, and the robustness claim in Section 4.2 item (3) is not supported. At minimum, the authors should provide multiple seeds and report variance or conduct a paired significance test on the replay-vs-mixed comparisons.
  3. [Section 3.2.1] The model merging hyperparameters are ad hoc and may confound the comparison between merging and replay. Linear, TIES, and DARE all use weight = [0.02, 0.03, 0.05, 0.9] with density 0.9, giving the final SQA-stage model a dominant 0.9 weight. Consequently, the merged model is close to the already-forgetting SQA model, which likely explains why merging alone performs poorly in Table 1 (e.g., LLaMA T2T 19.3 for linear merge vs. 14.3 for no mitigation). No sensitivity analysis or standard merging recipe is provided, so the paper's conclusion that merging is less effective than replay may be an artifact of this parameter choice. The authors should justify the weights or sweep them (e.g., equal weights, or task-vector-based merging) before drawing comparative conclusions.
  4. [Section 3.2.1 (Discounting LoRA-scaling factor)] The LoRA discounting intervention is nearly inert, making the comparison unfair. The model is trained with α=16, yet the paper only evaluates α=15 and α=14. The reduction in the adapter contribution is tiny, and indeed Table 1 shows that Scaling (α=15) performs almost identically to the no-mitigation baseline (e.g., LLaMA T2T 15.0 vs. 14.3). This does not constitute a meaningful test of the strategy. The authors should evaluate substantially smaller α values (e.g., α=8, α=4, or even α=1) before concluding that scaling is inferior to replay. The current result is not informative about the strategy's potential.
minor comments (4)
  1. [Table 1] The table formatting is broken: many numbers are concatenated without separators, such as "7.38.0", "3.61.50.8", and "25.411.2". This makes the table difficult to parse and should be fixed.
  2. [Section 4.1] There is a typo: "question anawering" should read "question answering".
  3. [Section 1] There is a missing space in "exploringspoken".
  4. [Section 3.2.1] The terminology around D0 is inconsistent: Eq. (3) defines D0 as the "original training data," but the text later says "we select a text instruction-tuning dataset generated by the same LLM." Clarify whether D0 is meant to be the pretraining corpus or a proxy, and align the wording accordingly.

Circularity Check

1 steps flagged · score 4.0 of 10

Experience-replay advantage is partly forced by using Magpie-Air, an instruction-response corpus similar to the text QA/IF evaluation, as the replay buffer D0.

  1. other [Section 3.2.1 (Experience replay), Eq. (3); Section 3.3.1 (Training datasets); Section 3.3.2 (Evaluation)]
    "For the initial dataset D0, the LLM's original training data, we select a text instruction-tuning dataset generated by the same LLM. We assume that its distribution closely approximates that of the original training data. ... When applying experience replay, we use Magpie-Air as D0, randomly sampling text instruction-response pairs for training. Since Magpie-Air is constructed by prompting Llama 3 8B, an LLM from the same series as ours, it serves as a suitable dataset for experience replay. ..."

    Equation (3) makes every replay stage contain Sample(D0, s|Di|), and D0 is set to Magpie-Air, a Llama-generated instruction-response corpus. The paper's forgetting metric is text QA and instruction-following (LLaMA-Questions, Spoken WebQuestions, Audio Trivia QA, IFEval), which is the same task family as Magpie-Air. Thus the replay rows in Table 1 are trained, by construction, on data resembling the evaluation distribution, while model merging and LoRA scaling are not. The large retention gain (LLaMA T2T 66.3 vs. 14.3; IFEval 47.5 vs. 9.2) therefore reflects in part direct training on eval-like instruction data rather than verified preservation of the original pretraining knowledge.

full rationale

No equation-level identity or fitted-parameter-as-prediction was found, and there is no load-bearing self-citation. The partial circularity is experimental: the paper's own Eq. (3) injects D0 = Magpie-Air into every replay stage, and the forgetting benchmark is text QA/instruction-following, which is the same data family as Magpie-Air. This makes the replay advantage in the text-modality retention numbers partly self-fulfilling. The paper is transparent about the D0 assumption, and the SQA speech results still require genuine training, so the circularity is partial rather than total. Score 4 reflects one central confound that bears on the main claim.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

No invented entities. The central comparison rests on several hand-picked hyperparameters (merging weights, LoRA discount values, replay ratio) and on the assumption that a synthetic instruction dataset approximates the model's original pretraining distribution. These choices, rather than a closed-form derivation, drive the reported outcome.

free parameters (5)
  • Linear combination merge weights = [0.02, 0.03, 0.05, 0.9]
    Chosen ad hoc; heavily weights the final SQA-stage model, limiting the ability of merging to recover earlier-stage knowledge.
  • TIES/DARE merge weights and density = weights [0.04, 0.06, 0.9], density 0.9
    Ad hoc selection; not tuned on a validation set; affects the conclusion that merging underperforms replay.
  • LoRA discount factors = alpha = 15, 14 (training alpha = 16)
    Values chosen close to the training scaling factor, making the intervention weak; a fairer comparison would test smaller alpha values.
  • Experience replay sampling ratio = s = 0.005
    A single value selected without a reported sensitivity analysis; the amount of replayed data could affect the magnitude of the effect.
  • Training hyperparameters = LoRA r=64, alpha=16; LR 1e-5; epochs 2/2/1; batch sizes 4/4/1
    Standard choices, but they determine the severity of forgetting and the effectiveness of each mitigation; not varied.
assumptions (3)
  • domain assumption Magpie-Air (Llama-3-Magpie-Air-3M) approximates the original pretraining distribution of Llama-3.2-11B
    The only basis for replaying D0 as a proxy for the model's original text knowledge; stated in Section 3.2.2.
  • domain assumption Word-level text-speech alignment from Whisper-Timestamped is accurate
    Used to construct interleaved TTS and SQA training targets; alignment errors would degrade the training signal.
  • domain assumption The chosen evaluation suites (Spoken WebQuestions, LLaMA-Questions, Audio Trivia QA, IFEval) measure the capabilities the pipeline is meant to retain
    The forgetting and mitigation conclusions are drawn from these metrics; they may not cover the full range of original LLM abilities.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Analyzing Mitigation Strategies for Catastrophic Forgetting in End-to-End Training of Spoken Language Models." pith.science (2026). https://pith.science/paper/IVUV5NFI

@misc{pith2026250517496,
  author       = {Pith},
  title        = {Pith review of: Analyzing Mitigation Strategies for Catastrophic Forgetting in End-to-End Training of Spoken Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IVUV5NFI}},
  note         = {Machine review of arXiv:2505.17496}
}
read the original abstract

End-to-end training of Spoken Language Models (SLMs) commonly involves adapting pre-trained text-based Large Language Models (LLMs) to the speech modality through multi-stage training on diverse tasks such as ASR, TTS and spoken question answering (SQA). Although this multi-stage continual learning equips LLMs with both speech understanding and generation capabilities, the substantial differences in task and data distributions across stages can lead to catastrophic forgetting, where previously acquired knowledge is lost. This paper investigates catastrophic forgetting and evaluates three mitigation strategies-model merging, discounting the LoRA scaling factor, and experience replay to balance knowledge retention with new learning. Results show that experience replay is the most effective, with further gains achieved by combining it with other methods. These findings provide insights for developing more robust and efficient SLM training pipelines.

Figures

Figures reproduced from arXiv: 2505.17496 by the authors.

Figure 1
Figure 1. Continual training of a spoken language model using multi-stage speech processing tasks. LLM through a projection network for representation align￾ment. The projection network are then trained optionally along with the LLM to make it familiar with the speech modality, as seen in models like Qwen-Audio [17, 18], SALMONN [14], and DeSTA[15, 16]. While these SLMs demonstrate strong speech understanding capabilities, th… view at source ↗
Figure 2
Figure 2. The architecture of a Spoken Language Model (SLM), which consists of a backbone LLM, a speech encoder that con￾verts speech into speech tokens, and a vocoder that synthesizes the speech tokens into a speech waveform. learning is applied when training with the augmented dataset D ′ i . Notably, the number of random samples is according to the size of i-th dataset |Di|. 3. Experimental setup 3.1. Spoken language model… view at source ↗
Figure 3
Figure 3. Evaluation results on instruction-following and ques￾tion answering. LLaMA, Web, and Trivia denote LLaMA￾Questions, Spoken WebQuestions, and Audio Trivia QA. IFEval-P and IFEval-I stand for IFEval in prompt-level and instruction-level. w/ R means with experience replay. text responses, and speech-to-speech (S2S), where the speech responses is first transcribed with Whisper-large-v3 and consid￾ered correct if the tra… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 26 canonical work pages

  1. [1]

    Introduction Inspired by the remarkable success of large language mod- els (LLMs) [1, 2, 3, 4] in natural language processing (NLP), researchers have begun exploringspoken language models (SLMs) 1 as powerful solutions for speech processing tasks. For instance, textless SLMs [5] perform speech continuation with- out text supervision, while task-specific S...

    arXiv 2025

  2. [2]

    Mitigation strategies In this section, we present three common strategies for miti- gating catastrophic forgetting in LLMs and SLMs, which are the focus of this paper: (1) model merging [26, 27], (2) dis- counting the LoRA scaling factor [32, 15], and (3) experience replay [29, 30, 31]. 2.1. Model merging Consider an SLM training process withNstages, wher...

    work page

  3. [3]

    model merging after experience replay

    Experimental setup 3.1. Spoken language model 3.1.1. Model architecture As shown in Figure 2, our SLM comprises three main com- ponents: a speech encoder, a LLM backbone, and a vocoder. The speech encoder extracts speech features from speech wave- forms, subsequently quantized into discrete speech tokens via k- means clustering. These tokens are incorpora...

    work page

  4. [4]

    Catastrophic forgetting Fig.3 shows the evaluation results on instruction-following and question answering in each training stage on T2T setting

    Results 4.1. Catastrophic forgetting Fig.3 shows the evaluation results on instruction-following and question answering in each training stage on T2T setting. For SLM without any mitigation strategy, it is obvious that catas- trophic forgetting appear during training. As training stage moves on, the accuracy of both evaluation tasks decrease in dif- feren...

    work page

  5. [5]

    The results demonstrate that expe- rience replay is the most effective method, with further perfor- mance gains achievable by combining it with other techniques

    Conclusion This paper investigates mitigation strategies for continual learn- ing in developing spoken language models (SLMs) from large language models (LLMs). The results demonstrate that expe- rience replay is the most effective method, with further perfor- mance gains achievable by combining it with other techniques. Through a case study, we highlight...

    work page

  6. [6]

    Gpt-4 technical report,

    OpenAI, “Gpt-4 technical report,”arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  7. [7]

    Gemini: a family of highly capable multimodal models,

    G. Teamet al., “Gemini: a family of highly capable multimodal models,”arXiv preprint arXiv:2312.11805, 2023

    arXiv 2023

  8. [8]

    The llama 3 herd of models,

    A. Dubeyet al., “The llama 3 herd of models,”arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

Show all 47 references
  1. [9]

    Qwen2.5 technical report,

    A. Yanget al., “Qwen2.5 technical report,”arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  2. [10]

    On generative spoken language modeling from raw audio,

    K. Lakhotiaet al., “On generative spoken language modeling from raw audio,” vol. 9, pp. 1336–1354, 2021

    2021

  3. [11]

    Neural codec language models are zero-shot text to speech synthesizers,

    S. Chenet al., “Neural codec language models are zero-shot text to speech synthesizers,”IEEE Transactions on Audio, Speech and Language Processing, vol. 33, pp. 705–718, 2025

    2025

  4. [12]

    Seamless: Multilingual expressive and stream- ing speech translation,

    L. Barraultet al., “Seamless: Multilingual expressive and stream- ing speech translation,”arXiv preprint arXiv:2312.05187, 2023

    2023 arXiv

  5. [13]

    Dynamic-superb phase-2: A collaboratively expanding benchmark for measuring the capabilities of spoken language models with 180 tasks,

    C. yu Huanget al., “Dynamic-superb phase-2: A collaboratively expanding benchmark for measuring the capabilities of spoken language models with 180 tasks,” 2024. [Online]. Available: https://arxiv.org/abs/2411.05361

    2024 arXiv

  6. [14]

    Dynamic-superb: Towards a dynamic, col- laborative, and comprehensive instruction-tuning benchmark for speech,

    C.-Y . Huanget al., “Dynamic-superb: Towards a dynamic, col- laborative, and comprehensive instruction-tuning benchmark for speech,” inICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024, pp. 12 136–12 140

    2024

  7. [15]

    On the landscape of spoken language models: A comprehensive survey,

    S. Arora, K.-W. Changet al., “On the landscape of spoken language models: A comprehensive survey,”arXiv preprint arXiv:2504.08528, 2025

    2025 arXiv

  8. [16]

    UniverSLU: Universal spoken language under- standing for diverse tasks with natural language instructions,

    S. Aroraet al., “UniverSLU: Universal spoken language under- standing for diverse tasks with natural language instructions,” in Proceedings of the 2024 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Lan- guage Technologies (V...

    2024

  9. [17]

    ESPnet-SpeechLM: An open speech language model toolkit,

    J. Tianet al., “ESPnet-SpeechLM: An open speech language model toolkit,” inProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Com- putational Linguistics: Human Language Technologies (System Demonstrations), Apr. 2025, pp. 116–124

    2025

  10. [18]

    Joint audio and speech understanding,

    Y . Gonget al., “Joint audio and speech understanding,” in2023 IEEE Automatic Speech Recognition and Understanding Work- shop (ASRU). IEEE, 2023

    2023

  11. [19]

    SALMONN: Towards generic hearing abilities for large language models,

    C. Tanget al., “SALMONN: Towards generic hearing abilities for large language models,” inThe Twelfth International Conference on Learning Representations, 2024

    2024

  12. [20]

    Desta: Enhancing speech language models through descriptive speech-text alignment,

    K.-H. Luet al., “Desta: Enhancing speech language models through descriptive speech-text alignment,” inProc. Interspeech 2024, 2024, pp. 4159–4163

    2024

  13. [21]

    Developing instruction- following speech language model without speech instruction- tuning data,

    K.-H. Lu, Z. Chen, S.-W. Fu, C.-H. H. Yang, J. Balam, B. Gins- burg, Y .-C. F. Wang, and H.-y. Lee, “Developing instruction- following speech language model without speech instruction- tuning data,”arXiv preprint arXiv:2409.20007, 2024

    2024 arXiv

  14. [22]

    Qwen-audio: Advancing universal audio under- standing via unified large-scale audio-language models,

    Y . Chuet al., “Qwen-audio: Advancing universal audio under- standing via unified large-scale audio-language models,”arXiv preprint arXiv:2311.07919, 2023

    2023 arXiv

  15. [23]

    Qwen2-audio technical report,

    Y . Chu, J. Xu, Q. Yang, H. Wei, X. Wei, Z. Guo, Y . Leng, Y . Lv, J. He, J. Linet al., “Qwen2-audio technical report,”arXiv preprint arXiv:2407.10759, 2024

    2024 arXiv

  16. [24]

    SpeechGPT: Empowering large language mod- els with intrinsic cross-modal conversational abilities,

    D. Zhanget al., “SpeechGPT: Empowering large language mod- els with intrinsic cross-modal conversational abilities,” inFind- ings of the Association for Computational Linguistics: EMNLP 2023, Dec. 2023, pp. 15 757–15 773

    2023

  17. [25]

    Audiolm: A language modeling approach to au- dio generation,

    Z. Borsoset al., “Audiolm: A language modeling approach to au- dio generation,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 31, pp. 2523–2533, 2023

    2023

  18. [26]

    Moshi: a speech-text foundation model for real-time dialogue,

    A. D ´efossezet al., “Moshi: a speech-text foundation model for real-time dialogue,”arXiv preprint arXiv:2410.00037, 2024

    2024 arXiv

  19. [27]

    Mini-omni: Language models can hear, talk while thinking in streaming,

    Z. Xie and C. Wu, “Mini-omni: Language models can hear, talk while thinking in streaming,”arXiv preprint arXiv:2408.16725, 2024

    2024 arXiv

  20. [28]

    Glm-4-voice: Towards intelligent and human-like end-to-end spoken chatbot,

    A. Zenget al., “Glm-4-voice: Towards intelligent and human-like end-to-end spoken chatbot,”arXiv preprint arXiv:2412.02612, 2024

    2024 arXiv

  21. [29]

    Building a taiwanese mandarin spoken lan- guage model: A first attempt,

    C.-K. Yanget al., “Building a taiwanese mandarin spoken lan- guage model: A first attempt,”arXiv preprint arXiv:2411.07111, 2024

    2024 arXiv

  22. [30]

    An empirical investigation of catastrophic forgetting in gradient-based neural networks,

    I. J. Goodfellowet al., “An empirical investigation of catastrophic forgetting in gradient-based neural networks,”arXiv preprint arXiv:1312.6211, 2013

    2013 arXiv

  23. [31]

    Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities,

    E. Yanget al., “Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities,”arXiv preprint arXiv:2408.07666, 2024

    2024 arXiv

  24. [32]

    Mitigating the alignment tax of rlhf,

    Y . Linet al., “Mitigating the alignment tax of rlhf,” inProceedings of the 2024 Conference on Empirical Methods in Natural Lan- guage Processing, 2024, pp. 580–606

    2024

  25. [33]

    Desta: Enhancing speech language models through descriptive speech-text alignment,

    K.-H. Luet al., “Desta: Enhancing speech language models through descriptive speech-text alignment,” inInterspeech 2024, 2024, pp. 4159–4163

    2024

  26. [34]

    Experience replay for continual learning,

    D. Rolnicket al., “Experience replay for continual learning,”Ad- vances in neural information processing systems, vol. 32, 2019

    2019

  27. [35]

    Lifelong learning of large language model based agents: A roadmap,

    J. Zhenget al., “Lifelong learning of large language model based agents: A roadmap,”arXiv preprint arXiv:2501.07278, 2025

    2025

  28. [36]

    Vqacl: A novel visual question answering con- tinual learning setting,

    X. Zhanget al., “Vqacl: A novel visual question answering con- tinual learning setting,” inProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, 2023, pp. 19 102–19 112

    2023

  29. [37]

    Salmonn: Towards generic hearing abilities for large language models,

    C. Tanget al., “Salmonn: Towards generic hearing abilities for large language models,” inThe Twelfth International Conference on Learning Representations

  30. [38]

    Ties-merging: Resolving interference when merging models,

    P. Yadav, D. Tamet al., “Ties-merging: Resolving interference when merging models,” inAdvances in Neural Information Pro- cessing Systems, A. Ohet al., Eds., vol. 36. Curran Associates, Inc., 2023, pp. 7093–7115

    2023

  31. [39]

    Language models are super mario: Absorbing abili- ties from homologous models as a free lunch,

    L. Yuet al., “Language models are super mario: Absorbing abili- ties from homologous models as a free lunch,” inForty-first Inter- national Conference on Machine Learning

  32. [40]

    Unsupervised cross-lingual representation learning for speech recognition,

    A. Conneauet al., “Unsupervised cross-lingual representation learning for speech recognition,” inInterspeech 2021, 2021, pp. 2426–2430

    2021

  33. [41]

    Genetic k-means algo- rithm,

    K. Krishna and M. Narasimha Murty, “Genetic k-means algo- rithm,”IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), vol. 29, no. 3, pp. 433–439, 1999

    1999

  34. [42]

    Hifi-gan: Generative adversarial networks for ef- ficient and high fidelity speech synthesis,

    J. Konget al., “Hifi-gan: Generative adversarial networks for ef- ficient and high fidelity speech synthesis,”Advances in neural in- formation processing systems, vol. 33, pp. 17 022–17 033, 2020

    2020

  35. [43]

    Librispeech: An asr corpus based on pub- lic domain audio books,

    V . Panayotovet al., “Librispeech: An asr corpus based on pub- lic domain audio books,” in2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015, pp. 5206–5210

    2015

  36. [44]

    Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing,

    Z. Xuet al., “Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing,”arXiv preprint arXiv:2406.08464, 2024

    2024 arXiv

  37. [45]

    SpeechT5: Unified-modal encoder- decoder pre-training for spoken language processing,

    J. Ao and R. a. Wang, “SpeechT5: Unified-modal encoder- decoder pre-training for spoken language processing,” inProceed- ings of the 60th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), May 2022, pp. 5723– 5738

    2022

  38. [46]

    Spoken question answering and speech continuation using spectrogram-powered llm,

    E. Nachmaniet al., “Spoken question answering and speech continuation using spectrogram-powered llm,”arXiv preprint arXiv:2305.15255, 2023

    2023 arXiv

  39. [47]

    Instruction-following evaluation for large language models,

    J. Zhouet al., “Instruction-following evaluation for large language models,”arXiv preprint arXiv:2311.07911, 2023

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.