{"id":"cb552e47-defe-4cb0-9916-b8937202ec75","arxiv_id":"2506.16574","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Periodically averaging per-dataset LoRA adapters into a fixed Whisper base reduces catastrophic forgetting in rehearsal-free continual learning for code-switched ASR, with modest gains over a weight-averaging plus distillation baseline.","lead":"This paper tests a simple way to stop a speech recognition model from forgetting old languages while it keeps learning new, code-switched audio: train small per-dataset adapters, then periodically average them back into the main model. The method beats a distillation-based continual learning baseline on six code-switching test sets and even slightly improves some held-out languages.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 1's accumulation rule double-counts earlier adapters after the first centralization, because later adapters are trained on the already-merged base but then averaged with old adapters as if all were relative to θ0; the 2nd-centralization results may not follow from the stated procedure.","rationale":"The reader's weakest_assumption already noted that the accumulation rule in Algorithm 1 is ambiguous about double-counting after a merge, so this is a partial agreement. I elevate it to the primary concern because it is more load-bearing than the conceded failure of the Gaussian-prior premise: even if adapter weights are not zero-mean Gaussians, simple averaging could still work empirically, but if the accumulation rule as written double-counts old adapters, the reported second-centralization numbers are not a clean test of the proposed mechanism and the method is not reproducible from the text. The concern is concrete and checkable: the algebra above shows an explicit 3/2 vs 1 weighting of the first-block average under the as-written algorithm. It does not require assuming the authors ran anything other than what is described; it only requires that the description be internally consistent. I keep the reader's CONDITIONAL verdict rather than moving to REJECT because the ambiguity is addressable in a revision and the first-centralization results are consistent with the stated mechanism, while the second-centralization numbers could still survive a corrected implementation. The paper would also benefit from releasing code and providing variance information, but the double-counting question is the single most decisive check.","tokens_in":8813,"tokens_out":6881,"duration_ms":71171,"concrete_test":"Run two variants on the same six code-switching datasets with K=3: (a) Algorithm 1 exactly as written, averaging all adapters up to t with weight 1/t and merging into θ_{t−1}; (b) a corrected variant that maintains a running total of deltas expressed relative to θ0, or equivalently merges only the current block of K adapters each time. Recompute the backward WER rows in Table 2. If only variant (a) reproduces the '2nd Cent.' numbers, the forgetting-prevention result is partly an artifact of double-counting old adapters; if variant (b) reproduces them, the mechanism survives but Algorithm 1 and the efficient-implementation note must be rewritten to match what was actually run.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In §3.2, Algorithm 1 first centralizes at t=K with Δavg_K = (1/K)Σ_{j=1}^K Δθ_j and θ_K = θ0 + Δavg_K. For t>K, line 4 trains Δθ_t on θ_{t−1}=θ_K, and the paper's own note after Algorithm 1 says adapters learned after centralization are conditioned on previously learned adapters. Thus second-block adapters are deltas relative to θ_K, not to θ0. At t=2K, line 8 averages all 2K stored Δθ_j with equal weight 1/t, treating first-block deltas (relative to θ0) and second-block deltas (relative to θ_K) as comparable, then line 9 adds the result to θ_K. Writing A1=(1/K)Σ_{j=1}^K Δθ_j and B1=(1/K)Σ_{j=K+1}^{2K} Δθ_j, the as-written rule gives θ_{2K}=θ0+A1+(1/2)A1+(1/2)B1, i.e., θ0+(3/2)A1+(1/2)B1, whereas the non-double-counting cumulative average of all deltas expressed relative to θ0 is θ0+A1+(1/2)B1. Old adapters are therefore weighted 1.5× instead of 1×. The '2nd Cent.' row of Table 2, the main evidence for forgetting prevention at the second consolidation, may then reflect re-adding old adapter weights rather than the variance-reduction mechanism claimed in §3.2. Moreover, the Gaussian independence argument does not apply to the second merge: second-block adapters are trained on a base that already contains the first-block average, so their updates are not independent draws around θ0. The efficient-implementation note ('storing the Δavg after each centralization, which is rescaled and added to the new adapters') does not specify the rescale factor or show that the double count is avoided. This is a correctness and reproducibility issue in the central algorithm, not just a missing theoretical justification.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a rehearsal-free continual learning method for ASR foundation models. In the factorization stage, the incoming stream of datasets is scattered across per-dataset LoRA adapters. In the centralization stage, after every K=3 datasets all stored adapters are averaged and merged into the Whisper base model via a LoRA merge. Experiments on six code-switching datasets compare the centralized model with a fine-tuned ceiling and with an SWA-plus-distillation baseline (SW ADT). The paper reports forward WER improvements (average 39.4 to 28.7) and backward error-rate recovery from severe forgetting caused by individual adapters; the second centralization row in Table 2 is the main evidence that the method outperforms SW ADT on average backward metrics. The proposed mechanism is motivated by a Gaussian variance-reduction argument, but the paper concedes in Section 3.2 that this premise is violated in practice. The contribution is primarily empirical, with the method advertised as simple and scalable.","tokens_in":9305,"tokens_out":6223,"duration_ms":63374,"significance":"If the experimental results hold after a corrected implementation and with proper statistical support, the method would be a lightweight, rehearsal-free alternative for continually adapting foundation ASR models to streams of code-switched data. The reported backward recovery is striking: for example, Arabic WER moves from 49.0 after a single adapter to 25.4 after centralization, and Turkish from 43.5 to 15.4 in Table 1. The comparison against SW ADT is appropriate, and the method's small hyperparameter footprint (one centralization size K, no rehearsal data) is a practical strength. However, the central claim is currently supported by a single set of numbers, with no error bars, no significance tests, no ablation of K, and an accumulation rule that is internally inconsistent at the second centralization. The significance is therefore conditional on corrections that the manuscript itself needs to provide.","major_comments":[{"comment":"The accumulation rule in Algorithm 1 double-counts the first block of adapters at the second centralization, so the '2nd Cent.' row of Table 2 does not follow from the stated procedure. After the first centralization at t=K, adapters Δθ_j for j=K+1,...,2K are trained on θ_K and are therefore deltas relative to θ_K, not to θ0. Writing A1=(1/K)Σ_{j=1}^K Δθ_j and B1=(1/K)Σ_{j=K+1}^{2K} Δθ_j, line 8 with t=2K gives θ_{2K}=θ0+A1+(1/2)A1+(1/2)B1, i.e., an effective weight of 1.5 on A1, whereas a non-double-counting cumulative average expressed relative to θ0 would be θ0+A1+(1/2)B1. The efficient variant described after Algorithm 1, which stores Δavg after each centralization and 'rescaled and added to the new adapters', does not specify the rescale factor or show that later adapters are expressed relative to θ0. Please state the actually implemented rule, derive its per-block weighting, and re-report the second-centralization results under a non-double-counting rule.","section":"Algorithm 1, lines 7-9; Table 2"},{"comment":"The Gaussian independence argument is not supported as a justification for forgetting prevention. The paper itself states that 'in practice, however, model weights have different standard deviation because the regularization term is dominated by the main cross entropy loss', which directly concedes that the zero-mean Gaussian premise is violated. In addition, adapters trained after the first centralization are conditioned on previously learned adapters, so their updates are not independent draws around θ0 at the second merge. The abstract's claim that centralization 'can effectively prevent catastrophic forgetting' therefore rests on an empirical regularity that is neither derived nor directly validated. Please provide direct evidence for the claimed mechanism, such as per-layer and per-adapter norm/sparsity statistics across both centralization blocks, an ablation of the number of adapters averaged, and a comparison with a properly anchored averaging baseline that does not re-weight earlier adapters.","section":"Section 3.2"},{"comment":"The experimental comparison against SW ADT is reported without error bars, confidence intervals, or significance tests. Many differences in Table 2 are small in absolute terms (e.g., English 4.8 vs 5.0, Mandarin CER 6.9 vs 8.4, Spanish 5.8 vs 6.7), and the paper does not state how many independent training runs were performed. Because ASR test sets are fixed, the relevant uncertainty is training-seed variance; please report the number of runs, mean and standard deviation over seeds, and a paired significance test (or per-seed tables) for both forward and backward metrics. Without this information, the claim that the proposed method outperforms SW ADT is not yet established.","section":"Tables 1 and 2; Section 4.1"},{"comment":"The paper calls K the 'only hyper-parameter to select' but provides no ablation or selection procedure for it. All experiments use K=3, and no results are shown for K=1, K=2, or larger K. Since K directly controls how many adapters are averaged and how often the base model is modified, the robustness of the forgetting-prevention effect to K is central to the method's claim. Please add an ablation over K (at least K in {1,2,3,6}) and report whether the backward recovery persists across these values.","section":"Section 4.1"},{"comment":"Essential LoRA hyperparameters are not reported, and some setup details are missing. In particular, the rank r, scaling factor α, learning rate, number of update steps per adapter, weight decay coefficient, and the LoRA target layers (query and key are mentioned, but not whether projection layers were also adapted) are not given. The 'Fine-tuned' ceiling row in Table 2 also lacks details such as how many adapters or steps were used and whether it was trained on all six datasets jointly. Without these details the method is not reproducible, and the claimed simplicity of the approach cannot be assessed.","section":"Sections 3.1 and 4.1"}],"minor_comments":[{"comment":"There are several typos, including 'contitnual' and 'framwework' in the Table 2 caption, 'serendeipitously' in the Section 4.2 heading, and 'robost' in the Introduction. Please proofread the manuscript.","section":"Various"},{"comment":"The indexing in Equation (1) is ambiguous: Nt and Nj are used without a clear definition, and the sums over previously seen datasets are not written consistently. Please define T, N_t, and the cumulative risk more explicitly.","section":"Equation (1)"},{"comment":"The sentence after Algorithm 1 describing the efficient implementation says the stored Δavg is 'rescaled and added to the new adapters' but gives no rescale factor. Please specify the formula so that the rule can be checked against the reported experiments.","section":"Algorithm 1 note"},{"comment":"The paper mentions measuring sparsity of Δθ_t and states that 'the averaged model has much higher sparsity than any other adapter', but no quantitative sparsity values are reported. Please add a table or figure with these statistics.","section":"Section 4.2"},{"comment":"Some references are incomplete, e.g., reference [17] lacks full bibliographic information and reference [26] does not give a complete venue. Please complete the bibliography.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about Algorithm 1 is accurate and is the main technical issue in the manuscript. The second-centralization numbers should not be interpreted as evidence of the proposed mechanism until the accumulation rule is corrected and re-run. The paper would also benefit from a statistically grounded comparison against SW ADT, as several backward differences are small. This is a promising empirical idea, but in its current form it needs a corrected algorithm, re-reported results, and basic reproducibility details."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful idea here is straightforward: instead of one stream of adapters, assign each new dataset its own LoRA, then periodically average the adapters and merge them into the frozen base. The first centralization (after three code-switched datasets) recovers backward WERs from 50-150% error back to near base levels, and the forward average after two centralizations beats the SW ADT baseline (28.7 vs. 29.7 WER) while also improving backward metrics (9.8 vs. 11.3). That is a real, practically relevant observation for anyone adapting ASR foundation models without rehearsal. The paper is clearly written, uses appropriate datasets, and does not overclaim the theory: it explicitly concedes that the Gaussian variance argument is violated by the cross-entropy loss. Good credit where due.\n\nThe soft spots are real. The entire evaluation is a single run with no error bars, no significance tests, no ablations of K or LoRA rank, and no code. The theoretical motivation is post-hoc and openly weakened in Section 3.2. More importantly, the stress-test note about Algorithm 1 is correct: after the first centralization, new adapters are trained on the merged base, so their deltas are relative to theta_K, not theta_0. At t=2K, the algorithm averages all 2K stored deltas with equal weight 1/t, which gives theta_0 + 1.5A1 + 0.5B1 instead of theta_0 + A1 + 0.5B1. The efficient-implementation note - that delta_avg is stored, rescaled, and added to new adapters - does not specify the rescale factor or show that the double count is avoided. Thus the '2nd Cent.' row in Table 2, the main evidence for improvement after a second consolidation, may be partly an artifact of re-adding old adapter weights, not of the variance-reduction story. The first centralization result is unaffected by this issue, so the core forgetting-mitigation claim has some support, but the second-round numbers are suspect.\n\nThis paper deserves a serious referee - sending it to review is right, not a desk reject. A referee should ask for the exact accumulation rule, code, multiple seeds, and ablations. For a reading group, it is a nice case study in how an elegant practical recipe can hide a subtle algorithmic inconsistency, but I would not cite it in my own work until the double-counting is resolved and the results are reproduced with proper statistics.","headline":"A simple adapter-averaging recipe shows a genuine forgetting-mitigation effect when merging into Whisper, but the paper's centralization algorithm double-counts old adapters after the first merge, so the second-round results are not clearly supported.","tokens_in":773,"tokens_out":972,"would_cite":false,"duration_ms":57140,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A continual-learning recipe that averages per-dataset low-rank adapters and merges them into a foundation model prevents catastrophic forgetting while still learning new code-switching speech tasks.","keywords":["continual learning","catastrophic forgetting","speech recognition","code-switching","low-rank adapters","weight averaging","Whisper","rehearsal-free"],"falsifier":"Measure the actual distribution of entries in the averaged adapter update $\\Delta_{\\text{avg}}$ after centralization on a different set of task streams (for example, languages outside the paper's six code-switching corpora) and with different values of $K$; if the average is not near zero or the backward error on held-out languages rises substantially after the merge, the centralization claim would be refuted. Concretely, run the same recipe on a second sequence of datasets and check whether per-language WER after centralization stays within a few points of the base model, as it does for German in Table 1.","tokens_in":8605,"feed_emoji":"🎙️","tokens_out":5538,"duration_ms":47713,"temperature":0.7,"pith_summary":"This paper claims that a continual-learning recipe built from two phases can stop a large multilingual speech model from forgetting its original languages when it is adapted to new code-switched data. In the factorization phase, each incoming dataset is learned through its own low-rank adapter. In the centralization phase, the accumulated adapters are averaged and merged into the base model. The authors report that this keeps backward error rates near the original model while still improving on the new tasks, and that it beats a stochastic-weight-averaging-plus-distillation baseline. The underlying reason proposed is that averaging many small adapter updates concentrates the merged weights near zero, so the merge disturbs the base model only mildly.","feed_headline":"Averaging adapters stops speech models from forgetting old languages","feed_subtitle":"A wake-sleep-style two-phase method learns new code-switching datasets while keeping backward error near the base model.","key_machinery":"The mechanism is the periodic averaging of LoRA update matrices. Each dataset learns a low-rank update $\\Delta W = (\\alpha/r) A B$ on query and key projections of the Whisper transformer. Periodically (every $K$ datasets), all adapters seen so far are averaged into $\\Delta_{\\text{avg}}$ and merged into the base weights. The paper's stated justification is that independent zero-mean Gaussian weights average to a smaller variance $\\sigma^2/N$, so the merged update stays sparse and close to zero, limiting the drift of the knowledge base. The merge rule is the central object that is supposed to turn many individually destructive adapters into a nearly harmless cumulative update.","core_discovery":"The central claim is that centrally merging multiple scattered low-rank adapters into a frozen foundation model prevents catastrophic forgetting without rehearsal. Concretely, after fine-tuning one LoRA adapter per code-switching dataset, the authors average all adapters and merge the average into the Whisper base model; repeating this periodically yields a model whose backward word-error rate on held-out languages stays close to the original (for example, German moves from 23.5% to 25.4% WER, and after the second merge Arabic improves from 23.5% to 19.9%), while forward average WER on six code-switching corpora improves from 39.4% to 28.7%. The same recipe outperforms a stochastic-weight-averaging with distillation baseline on both forward and backward metrics. The paper frames the effect as a Gaussian prior: averaging independent zero-mean weight updates reduces variance by a factor of the number of adapters, making the merged update sparse and near zero.","pith_inferences":["If the averaging rationale holds generally, the same factorization-and-centralization recipe should transfer to other foundation-model adaptation problems, such as adapting vision or language models to a sequence of domains with LoRA-style adapters.","The paper's Gaussian justification is violated by the data-dependent cross-entropy loss; a more direct test would be to measure the distribution of per-entry adapter weights after training and check how sparsity of the merged update correlates with backward stability across different task orders and $K$ values.","The observed positive backward transfer suggests the merging acts as a form of implicit regularization that may favor flatter minima; whether this is specific to speech or general to merged low-rank updates could be tested by comparing loss landscapes before and after merging.","Algorithm 1's averaging step with weight $1/t$ is ambiguous about whether previously merged adapters are double-counted; a clean reformulation that tracks the running average explicitly would make the method more reproducible."],"forward_implications":["A speech foundation model can be adapted to a sequence of code-switching datasets without access to the original training data or replay.","The recipe needs only one hyperparameter, the number of adapters per centralization ($K$), making it easy to apply in practice.","The centralized model delivers positive backward transfer, improving on some held-out languages relative to the base model rather than merely recovering them.","The method can be implemented with constant extra memory by storing the running average of adapters rather than retaining all of them.","The approach reaches only about 58% of the improvement of an oracle fine-tune on all datasets, leaving room for better merging strategies."],"supporting_citations":[{"why":"Supplies the Whisper foundation model used as the knowledge base in all experiments.","marker":"[1]"},{"why":"Introduces the low-rank adapter (LoRA) mechanism used for the factorization stage.","marker":"[9]"},{"why":"Provides the averaging motivation and forms the core of the baseline's stochastic weight averaging.","marker":"[16]"},{"why":"Defines the SW ADT baseline that the proposed method is compared against.","marker":"[26]"},{"why":"Supplies the distillation component used in the SW ADT baseline.","marker":"[44]"}],"fun_headline_variants":["Averaging adapters stops speech model forgetting","Merging low-rank adapters blocks speech forgetting","Wake-sleep merging shields speech knowledge","Centralized adapters keep speech models current","Adapter averaging boosts multilingual speech retention"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The safety of merging averaged adapters into the base model rests on the empirical claim that the averaged update is sparse and close to zero, but the paper concedes its variance-reduction justification does not hold in practice because the cross-entropy loss dominates the weight-decay regularization.","fun_headline_variants_meta":{"raw":{"variants":["Averaging adapters stops speech model forgetting","Merging low-rank adapters blocks speech forgetting","Wake-sleep merging shields speech knowledge","Centralized adapters keep speech models current","Adapter averaging boosts multilingual speech retention"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00019,"raw_usage":{"total_tokens":1302,"prompt_tokens":870,"completion_tokens":432,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":486,"completion_tokens_details":{"reasoning_tokens":366}},"tokens_in":486,"tokens_out":432,"duration_ms":4533,"temperature":1.0,"reasoning_tokens":366,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:23:10.888116+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the actual distribution of entries in the averaged adapter update $\\Delta_{\\text{avg}}$ after centralization on a different set of task streams (for example, languages outside the paper's six code-switching corpora) and with different values of $K$; if the average is not near zero or the backward error on held-out languages rises substantially after the merge, the centralization claim would be refuted. Concretely, run the same recipe on a second sequence of datasets and check whether per-language WER after centralization stays within a few points of the base model, as it does for German in Table 1.","supporting_citations":[{"cited_title":"Continual learning through synaptic intelligence,","cited_arxiv_id":null,"evidence_quote":"Defines the SW ADT baseline that the proposed method is compared against."},{"cited_title":"Ted-lium 3: Twice as much data and corpus repartition for experiments on speaker adaptation,","cited_arxiv_id":null,"evidence_quote":"Supplies the distillation component used in the SW ADT baseline."}],"review_version":2}