The reviewed record of science sign in
Pith

arxiv: 2606.09342 · v1 · pith:IP2QSK5D · submitted 2026-06-08 · eess.AS

Parameter-Efficient Continual Learning for Automatic Speech Recognition

Steven Vander Eeckt , Hugo Van hamme This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-06-27 15:06 UTCgrok-4.3pith:IP2QSK5Drecord.jsonopen to challenge →

classification eess.AS
keywords parameter-efficient continual learningautomatic speech recognitioncatastrophic forgettingsingular value decompositionweight averagingspeech foundation models
0
0 comments X

The pith

Restricting adaptation to low-energy subspaces reduces forgetting in sequential ASR fine-tuning.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces a parameter-efficient continual learning method for automatic speech recognition. Pretrained weight matrices are partitioned into head and tail subspaces based on singular values, with changes restricted to approximate rotations only in the low-energy tail. This keeps the dominant pretrained components intact. Rotations from multiple tasks are then combined through weight averaging to improve retention. On two benchmarks the method shows less forgetting and higher overall performance than recent baselines.

Core claim

By partitioning pretrained weight matrices into head and tail subspaces according to singular values and restricting adaptation to approximate rotations within the low-energy tail subspace, while combining rotations via weight averaging for subsequent tasks, the method preserves dominant components and reduces catastrophic forgetting in continual learning for ASR.

What carries the argument

Head-tail subspace partition of weight matrices by singular values, with adaptation restricted to rotations in the tail subspace.

Load-bearing premise

Partitioning weight matrices into head and tail subspaces according to singular values and restricting adaptation to approximate rotations in the tail subspace preserves dominant components and reduces catastrophic forgetting across sequential tasks.

What would settle it

Running the method on the same ASR task sequences and observing forgetting rates equal to or higher than standard fine-tuning or other PECL baselines.

Figures

Figures reproduced from arXiv: 2606.09342 by Hugo Van hamme, Steven Vander Eeckt.

Figure 1
Figure 1. Figure 1: Overview of CSSVD for a single linear layer (bias omitted) when learning task Ti+1. Right: (1) current weights Wi are decomposed via SVD into head and tail subspaces; (2) an approximate rotation matrix Gi+1 is introduced within the tail and optimized on the new task to obtain W˜ i+1; (3) Wi and W˜ i+1 are merged through averaging to produce Wi+1. Left: schematic view of the decomposition for an input x ∈ R… view at source ↗
read the original abstract

Speech foundation models enable strong general-purpose ASR and are attractive for downstream adaptation. However, their size and the catastrophic forgetting induced by sequential fine-tuning demand parameter-efficient and regularized training methods, motivating parameter-efficient continual learning (PECL). While PECL has been widely studied in NLP and vision, it has received less attention in ASR. In this paper, we propose a simple yet effective PECL method based on recent advances in parameter-efficient fine-tuning for ASR. We partition pretrained weight matrices into head and tail subspaces according to singular values and restrict adaptation to approximate rotations within the low-energy tail subspace, preserving dominant components and reducing forgetting. For subsequent tasks, rotations are combined via weight averaging to further improve retention. Experiments on two benchmarks demonstrate reduced forgetting and superior overall performance compared to recent PECL baselines.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

0 major / 3 minor

Summary. The manuscript proposes a parameter-efficient continual learning (PECL) method for ASR foundation models. Pretrained weight matrices are partitioned into head and tail subspaces by singular values; adaptation is restricted to approximate rotations in the low-energy tail subspace (to preserve dominant components and reduce forgetting), with rotations from subsequent tasks combined by weight averaging. Experiments on two benchmarks are reported to show reduced forgetting and superior overall performance relative to recent PECL baselines.

Significance. If the empirical results hold, the work supplies a simple, SVD-based PECL recipe tailored to ASR that could enable efficient sequential adaptation of large speech models while mitigating catastrophic forgetting. This addresses a documented gap (PECL less studied in ASR than in NLP/vision) and, if reproducible, offers a practical baseline for the community.

minor comments (3)
  1. The abstract states that experiments were run on 'two benchmarks' but does not name them or give data-split details; the introduction or experimental section should explicitly identify the corpora, task sequence, and evaluation protocol.
  2. The description of 'approximate rotations' in the tail subspace lacks a precise algorithmic statement (e.g., how the rotation matrix is parameterized or optimized). A short pseudocode block or equation in the method section would improve clarity and reproducibility.
  3. No mention is made of the singular-value cutoff or rank-selection procedure used to define the tail subspace; an ablation or sensitivity plot on this hyper-parameter should be added to the experimental section.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for the positive summary, recognition of the significance of addressing PECL for ASR, and the recommendation for minor revision. No specific major comments were provided in the report.

Circularity Check

0 steps flagged

No significant circularity

full rationale

The paper proposes an empirical PECL method for ASR: SVD-based head/tail partitioning of weight matrices, adaptation restricted to approximate rotations in the low-energy tail, and weight averaging across tasks. No equations, derivations, or self-citations are presented that reduce any claimed result to the inputs by construction. The central claims rest on experimental outcomes measured against external benchmarks rather than internal fitting or self-referential uniqueness theorems. This matches the default case of a self-contained empirical proposal with no load-bearing circular steps.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

Based on abstract only. The method invokes standard linear algebra (SVD) and introduces at least one tunable cutoff for the tail subspace; no new physical entities are postulated.

free parameters (1)
  • singular-value cutoff or rank for tail subspace
    The division between head and tail subspaces must be chosen or tuned; this choice directly controls which parameters are allowed to adapt.
axioms (1)
  • standard math Singular value decomposition exists and can be computed for the pretrained weight matrices
    Invoked to partition each weight matrix into head and tail subspaces.

pith-pipeline@v0.9.1-grok · 5658 in / 1300 out tokens · 29931 ms · 2026-06-27T15:06:53.238008+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

52 extracted references · 4 canonical work pages · 3 internal anchors

  1. [1]

    More recently, speech foundation models trained on massive and multilingual datasets have emerged as powerful general-purpose models [1,2]

    Introduction Automatic Speech Recognition (ASR) has undergone remark- able progress over the past decade. More recently, speech foundation models trained on massive and multilingual datasets have emerged as powerful general-purpose models [1,2]. These models capture general speech knowledge, making them attrac- tive for adaptation to specific downstream t...

  2. [2]

    Parameter-Efficient Continual Learning for Automatic Speech Recognition

    Problem Formulation Let an initial model with parametersθ 0 ∈R N be trained on an initial set of tasksT 0, whose data(X,y)∈ D 0, with X∈R F×d s the input utterance (consisting ofFframes of dimensiond s) andy∈R w the corresponding set ofwground truth tokens, is no longer available. In parameter-efficient con- tinual learning (PECL), this model is sequentia...

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  3. [3]

    Our Method: Continual SSVD 3.1. Learning the first task LetW∈R dout×din be the weight matrix of a linear layer of the initial modelθ 0 whose output ish=W xfor an inputx∈ Rdin (we omit the biasb∈R dout for simplicity). Its singular value decomposition (SVD) is given as: W=UΣV ⊤ (1) withU∈R dout×d andV∈R din×d, the left and right singular vectors, resp., an...

  4. [4]

    More detailed informa- tion and code are available at our Github repository 1

    Experiments Experiments are done in ESPnet2 [31]. More detailed informa- tion and code are available at our Github repository 1. Model.We use Open Whisper-style Speech Model (OWSM) v3.2 small [2], comprising nine E-Branchformer [32] encoder and nine Transformer decoder layers. A CTC branch is used only during training. The model has a vocabulary of50,000 ...

  5. [5]

    Results Table 1 shows the results of both experiments. 5.1. Experiment 1 CSSVD achieves the lowest Average WER among all meth- ods. While LoRA, SSVD, OPLoRA, and MiLoRA successfully learn the new tasks, they suffer from catastrophic forgetting, with the WER on previous tasks increasing by more than 30 points—exceeding FFT. For LoRA and SSVD, this behavior...

  6. [6]

    CSSVD 18.33 -1.9 2.→Keep initial head-tail separation 18.40 b -1.9 3.→Do not average, i.e. skip Eq. (7) 19.16 a -3.3 4.→Train rotation + rescaling 18.27 b -1.8

  7. [7]

    SSVD + FTA 19.22 a -2.7

  8. [8]

    b No significant difference with respect to the reference method

    OPLoRA [k OP = 461] 31.61 a -20.2 a Significant deterioration with respect to the reference method. b No significant difference with respect to the reference method. mance, with the latter offering a simpler implementation. • Row 3 demonstrates that skipping the averaging step substan- tially increases forgetting and Average WER. Nevertheless, this varian...

  9. [9]

    Conclusion We study parameter-efficient continual learning for ASR, a set- ting that has received considerable attention in NLP and vision but remains underexplored in ASR. Building on SSVD, a re- cent PEFT method for ASR, we propose CSSVD, a PECL ap- proach that decomposes linear weight matrices into head and tail subspaces based on singular values and l...

  10. [10]

    Acknowledgments Research supported by Research Foundation Flanders (FWO) under grant S004923N of the SBO programme

  11. [11]

    All scientific content, ex- periments, and conclusions were developed by the authors, who take full responsibility for the manuscript

    Generative AI Use Disclosure Generative AI tools were used to assist with minor language editing and phrasing improvements. All scientific content, ex- periments, and conclusions were developed by the authors, who take full responsibility for the manuscript

  12. [12]

    Robust Speech Recognition via Large-Scale Weak Supervision

    A. Radfordet al., “Robust speech recognition via large- scale weak supervision,” 2022. [Online]. Available: https: //arxiv.org/abs/2212.04356

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  13. [13]

    Owsm v3.1: Better and faster open whisper-style speech models based on e-branchformer,

    Y . Penget al., “Owsm v3.1: Better and faster open whisper-style speech models based on e-branchformer,” inInterspeech, 2024, pp. 352–356

    2024

  14. [14]

    Catastrophic interference in connectionist networks: The sequential learning problem,

    M. McCloskeyet al., “Catastrophic interference in connectionist networks: The sequential learning problem,” ser. Psychology of Learning and Motivation, 1989, vol. 24, pp. 109–165

    1989

  15. [15]

    Sapt: A shared attention framework for parameter-efficient continual learning of large language models,

    W. Zhao, S. Wang, Y . Hu, Y . Zhao, B. Qin, X. Zhang, Q. Yang, D. Xu, and W. Che, “Sapt: A shared attention framework for parameter-efficient continual learning of large language models,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024, pp. 11 641–11 661

    2024

  16. [16]

    CorDA: Context-oriented decomposition adaptation of large language models for task-aware parameter- efficient fine-tuning,

    Y . Yanget al., “CorDA: Context-oriented decomposition adaptation of large language models for task-aware parameter- efficient fine-tuning,” inProc. NeurIPS 2024, 2024. [Online]. Available: https://openreview.net/forum?id=Gi00NVru6n

    2024

  17. [17]

    MiLoRA: Harnessing minor singular components for parameter-efficient LLM finetuning,

    H. Wang, Y . Li, S. Wang, G. Chen, and Y . Chen, “MiLoRA: Harnessing minor singular components for parameter-efficient LLM finetuning,” inProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), L. Chiruzzo, A. Ritter, and L. Wang, Eds. ...

    2025

  18. [18]

    Oplora: Orthogonal projection lora prevents catastrophic forgetting during parameter-efficient fine- tuning,

    Y . Xiong and X. Xie, “Oplora: Orthogonal projection lora prevents catastrophic forgetting during parameter-efficient fine- tuning,” 2025. [Online]. Available: https://arxiv.org/abs/2510. 13003

    2025

  19. [19]

    Sculpting subspaces: Constrained full fine-tuning in LLMs for continual learning,

    N. S. Nayak, K. Killamsetty, L. Han, A. Bhandwaldar, P. Chanda, K. Xu, O. Silkin, M. Eyceoz, H. Wang, A. Pareja, and A. Srivastava, “Sculpting subspaces: Constrained full fine-tuning in LLMs for continual learning,” inThe Fourteenth International Conference on Learning Representations, 2026. [Online]. Available: https://openreview.net/forum?id=vQcyqsGJDw

    2026

  20. [20]

    Merge before forget: A single loRA continual learning via continual merging,

    F. Qiao and M. Mahdavi, “Merge before forget: A single loRA continual learning via continual merging,” inThe Fourteenth In- ternational Conference on Learning Representations, 2026. [On- line]. Available: https://openreview.net/forum?id=i1Rj7yU6eF

    2026

  21. [21]

    Task arithmetic with loRA for continual learning,

    R. Chitale, A. Vaidya, A. Kane, and A. S. Ghotkar, “Task arithmetic with loRA for continual learning,” inWorkshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (WANT@NeurIPS 2023),

    2023

  22. [22]

    Available: https://openreview.net/forum?id= 4CLNFKi12w

    [Online]. Available: https://openreview.net/forum?id= 4CLNFKi12w

  23. [23]

    Bilora: Almost- orthogonal parameter spaces for continual learning,

    H. Zhu, Y . Zhang, J. Dong, and P. Koniusz, “Bilora: Almost- orthogonal parameter spaces for continual learning,” in2025 IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), 2025, pp. 25 613–25 622

    2025

  24. [24]

    Cl-lora: Continual low-rank adap- tation for rehearsal-free class-incremental learning,

    J. He, Z. Duan, and F. Zhu, “Cl-lora: Continual low-rank adap- tation for rehearsal-free class-incremental learning,”Proceed- ings of the Computer Vision and Pattern Recognition Conference (CVPR), pp. 30 534–30 544, June 2025

    2025

  25. [25]

    Clora: Parameter- efficient continual learning with low-rank adaptation,

    S. Muralidhara, D. Stricker, and R. Schuster, “Clora: Parameter- efficient continual learning with low-rank adaptation,” inProceed- ings of the Conference on Lifelong Learning Agents (CoLLAs), 2025

    2025

  26. [26]

    KeeploRA: Continual learning with residual gradient adaptation,

    M.-L. Luo, Z.-H. Zhou, Y .-L. Zhang, Y . Wan, T. Wei, and M.-L. Zhang, “KeeploRA: Continual learning with residual gradient adaptation,” inThe Fourteenth International Conference on Learning Representations, 2026. [Online]. Available: https: //openreview.net/forum?id=T3Vc5fkTzV

    2026

  27. [27]

    Revisiting weight regularization for low-rank continual learning,

    Y . Zheng, Y . Zhang, J. van de Weijer, G. M. van de Ven, S. Du, X. Zhang, and Z. Tian, “Revisiting weight regularization for low-rank continual learning,” inThe Fourteenth International Conference on Learning Representations, 2026. [Online]. Available: https://openreview.net/forum?id=pZj2DhfaVD

    2026

  28. [28]

    LoRA: Low-rank adaptation of large language models,

    E. J. Huet al., “LoRA: Low-rank adaptation of large language models,” inInternational Conference on Learning Representations, 2022. [Online]. Available: https://openreview. net/forum?id=nZeVKeeFYf9

    2022

  29. [29]

    Towards rehearsal-free multilingual asr: A lora- based case study on whisper,

    T. Xuet al., “Towards rehearsal-free multilingual asr: A lora- based case study on whisper,” inInterspeech 2024, 2024, pp. 2534–2538

    2024

  30. [30]

    Orthogonal subspace learning for language model continual learning,

    X. Wang, T. Chen, Q. Ge, H. Xia, R. Bao, R. Zheng, Q. Zhang, T. Gui, and X. Huang, “Orthogonal subspace learning for language model continual learning,” inFindings of the Association for Computational Linguistics: EMNLP 2023, H. Bouamor, J. Pino, and K. Bali, Eds. Singapore: Association for Computational Linguistics, Dec. 2023, pp. 10 658–10 671. [Online]...

    2023

  31. [31]

    Adaptive budget allocation for parameter- efficient fine-tuning,

    Q. Zhanget al., “Adaptive budget allocation for parameter- efficient fine-tuning,” in11th International Conference on Learning Representations, 2023. [Online]. Available: https: //openreview.net/forum?id=lq62uWRJjiY

    2023

  32. [32]

    Weight Factorization and Centralization for Continual Learning in Speech Recognition,

    E. Ugan, N.-Q. Pham, and A. Waibel, “Weight Factorization and Centralization for Continual Learning in Speech Recognition,” in Interspeech 2025, 2025, pp. 2200–2204

    2025

  33. [33]

    Weight averaging: A sim- ple yet effective method to overcome catastrophic forgetting in automatic speech recognition,

    S. Vander Eeckt and H. Van hamme, “Weight averaging: A sim- ple yet effective method to overcome catastrophic forgetting in automatic speech recognition,” inICASSP 2023, 2023

    2023

  34. [34]

    Parameter-efficient transfer learning for NLP,

    N. Houlsbyet al., “Parameter-efficient transfer learning for NLP,” inProceedings of the 36th International Conference on Machine Learning, 2019, pp. 2790–2799. [Online]. Available: https://proceedings.mlr.press/v97/houlsby19a.html

    2019

  35. [35]

    Using adapters to overcome catastrophic forgetting in end-to-end automatic speech recogni- tion,

    S. Vander Eeckt and H. Van hamme, “Using adapters to overcome catastrophic forgetting in end-to-end automatic speech recogni- tion,” inICASSP 2023, 2023

    2023

  36. [36]

    Towards Lifelong Learning of End-to-End ASR,

    H.-J. Changet al., “Towards Lifelong Learning of End-to-End ASR,” inProc. Interspeech 2021, 2021, pp. 2551–2555

    2021

  37. [37]

    Continual learning for monolingual end-to-end automatic speech recognition,

    S. Vander Eeckt and H. Van hamme, “Continual learning for monolingual end-to-end automatic speech recognition,” in2022 30th European Signal Processing Conference (EUSIPCO), 2022

    2022

  38. [38]

    Efficient rehearsal for continual learning in asr via singular value tuning,

    ——, “Efficient rehearsal for continual learning in asr via singular value tuning,”IEEE Transactions on Audio, Speech and Language Processing, vol. 34, pp. 978–991, 2026

    2026

  39. [39]

    Updating only encoders prevents catas- trophic forgetting of end-to-end ASR models,

    Y . Takashimaet al., “Updating only encoders prevents catas- trophic forgetting of end-to-end ASR models,” inProc. Inter- speech 2022. ISCA, 2022, pp. 2218–2222

    2022

  40. [40]

    Clrl-tuning: A novel contin- ual learning approach for automatic speech recognition,

    Z. Wang, F. Hou, and R. Wang, “Clrl-tuning: A novel contin- ual learning approach for automatic speech recognition,” inInter- speech 2023, 2023, pp. 1279–1283

    2023

  41. [41]

    Inverse-hessian regular- ization for continual learning in asr,

    S. Vander Eeckt and H. Van hamme, “Inverse-hessian regular- ization for continual learning in asr,” inProc. IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), 2026

    2026

  42. [42]

    Ssvd: Structured svd for parameter-efficient fine-tuning and benchmarking under domain shift in asr,

    P. Wang, S. Watanabe, and H. Van hamme, “Ssvd: Structured svd for parameter-efficient fine-tuning and benchmarking under domain shift in asr,” inProceedings of the IEEE ASRU Workshop,

  43. [43]

    Ssvd: Structured svd for parameter-efficient fine-tuning and benchmarking under domain shift in asr,

    [Online]. Available: https://arxiv.org/abs/2509.02830

    work page arXiv

  44. [44]

    ESPnet: End-to-end speech processing toolkit,

    S. Watanabeet al., “ESPnet: End-to-end speech processing toolkit,” inProceedings of Interspeech, 2018, pp. 2207–2211

    2018

  45. [45]

    E-branchformer: Branchformer with enhanced merging for speech recognition,

    K. Kimet al., “E-branchformer: Branchformer with enhanced merging for speech recognition,” in2022 IEEE Spoken Language Technology Workshop (SLT), 2023, pp. 84–91

    2023

  46. [46]

    Adam: A Method for Stochastic Optimization

    D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in3rd International Conference on Learning Representations, 2015. [Online]. Available: http://arxiv.org/abs/ 1412.6980

    work page internal anchor Pith review Pith/arXiv arXiv 2015

  47. [47]

    Common voice: A massively-multilingual speech corpus,

    R. Ardilaet al., “Common voice: A massively-multilingual speech corpus,” inProceedings of the 12th Conference on Lan- guage Resources and Evaluation (LREC 2020), 2020

    2020

  48. [48]

    The spoken dutch corpus: Overview and first eval- uation,

    N. Oostdijk, “The spoken dutch corpus: Overview and first eval- uation,”Proceedings of LREC-2000, Athens, vol. 2, 01 2000

    2000

  49. [49]

    Gesproken corpus van de zuidelijk-nederlandse dialecten,

    A. Breitbarth, M. Farasyn, A.-S. Ghyselen, L. Hellebaut, F. Lamsens, K. Depuydt, J. De Does, J. Niestadt, and K. Mertens, “Gesproken corpus van de zuidelijk-nederlandse dialecten,” 2024, 1st release, October 2024. [Online]. Available: https://hdl.handle.net/10032/tm-a2-z8

    2024

  50. [50]

    Overcoming catastrophic forgetting in neural networks,

    J. Kirkpatricket al., “Overcoming catastrophic forgetting in neural networks,”Proceedings of the National Academy of Sciences, vol. 114, no. 13, pp. 3521–3526, 2017. [Online]. Available: https://www.pnas.org/content/114/13/3521

    2017

  51. [51]

    Comparing the recognition performance of csrs: in search of an adequate met- ric and statistical significance test,

    H. Strik, C. Cucchiarini, and J. M. Kessens, “Comparing the recognition performance of csrs: in search of an adequate met- ric and statistical significance test,” inINTERSPEECH, 2000

    2000

  52. [52]

    Continuous Learning for Children’s ASR: Overcoming Catas- trophic Forgetting with Elastic Weight Consolidation and Synap- tic Intelligence,

    E. Ahadzi, V . Pratap Singh, T. Kinnunen, and V . Hautamaki, “Continuous Learning for Children’s ASR: Overcoming Catas- trophic Forgetting with Elastic Weight Consolidation and Synap- tic Intelligence,” inInterspeech 2025, 2025, pp. 2880–2884

    2025