REVIEW 3 major objections 4 minor 2 cited by
Continual Learning in Vision-Language Models via Aligned Model Merging
T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read This paper claims that catastrophic forgetting in continual learning stems from sequential fine-tuning's bias toward the latest task, and that training each task in a temporary LoRA and merging it into an evolving global LoRA, with…
desk verdict PAM is a simple and effective merging-plus-reinitialization recipe for VLM continual learning, but the paper never isolates the sign criterion, so the mechanism claim outruns the evidence. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the pair of LoRA adapters together with the sign-based alignment rule that keeps them mergeable. A global LoRA $W_G$ holds the accumulated knowledge of all previous tasks; for each new task a temporary LoRA $W'$ is cloned from it, fine-tuned on the task's data, and merged back by element-wise averaging $W_G \leftarrow (W_G + W')/2$. The alignment rule, applied every 100 training steps, uses the previously published observation that weight sign is a meaningful alignment signal: any temporary weight $w'$ whose sign differs from the corresponding global weight $w_G$ is re-initialized to $w_G$ whenever $|w_G|$ ranks among the top $p\%$ of magnitudes. The percentage $p$ controls the stability-plasticity trade-off, with 50% the default, and re-initializing with the global value rather than zero favors stability. Because the pretrained PaliGemma weights stay frozen and only one LoRA evolves, the model's parameter count does not grow with the number of tasks and no task identity is needed at inference time.
What would settle it
Train a deliberately interfering task pair (the paper identifies ScienceQA and RSVQA-lr as dissimilar, interfering tasks) with PAM and with plain averaging, and log the current task's training loss at every re-initialization step. If re-initializing sign-conflicting weights systematically raises the current task's loss, and if on such sequences PAM's final accuracy on the new task is no better than plain averaging's, then the sign criterion is discarding task-specific signal rather than interference, and the central claim fails.
Extended reading notes
Core claim
The central claim is that the stability-plasticity dilemma of continual learning is best addressed by changing the learning paradigm rather than by patching sequential fine-tuning: each task is learned in a temporary LoRA initialized from a global LoRA that accumulates all prior knowledge, and after training the two are fused by element-wise averaging. Merging alone already reduces forgetting, but plain averaging is suboptimal because task-specific updates interfere, so PAM aligns parameters during training. Every 100 steps it compares the signs of the temporary weights with the global weights and re-initializes any temporary weight whose sign conflicts with a global weight in the top $p\%$ of magnitudes, resetting it to the global value, with $p=50\%$ by default acting as the stability-plasticity dial. Empirically, PAM preserves per-task plasticity (accuracy on the just-learned task stays at 76.3, matching plain fine-tuning), cuts backward-task forgetting on the six-task CoIN benchmark from $-39.5$ to $-19.5$ across three task orders, raises forward transfer to unseen tasks from 7.7 to 11.1, and beats the post-training merging methods TIES and TALL; combined with TALL's post-training localization it reaches the best reported accuracy (51.97) and lowest forgetting ($-9.27$). The same alignment step also improves sequential fine-tuning itself, and adding PAM to the regularization method LWF matches experience replay without storing any old data.
Load-bearing premise
The argument depends on weight sign being a trustworthy signal of conflict: if the temporary weights that PAM re-initializes mid-training carry useful task-specific information rather than harmful interference, the alignment step would erase part of the new task's learning and PAM's advantage over plain averaging would disappear.
Editorial extensions
If this is right
- A single fixed-size LoRA suffices for a whole task stream: merging folds each task into one global adapter, so parameter count stops growing and no task identifier is needed at test time, removing two practical obstacles for deploying continual VLMs.
- Task-order robustness: on CoIN, PAM's final accuracy varies only $\pm 1.66$ across three task orders versus $\pm 8.18$ for sequential fine-tuning, meaning deployed systems become far less sensitive to the order in which data arrives.
- Better generalization: forward transfer on CoIN rises from 7.7 to 11.1, and on the extended ten-task and video benchmarks PAM also beats fine-tuning on unseen-task accuracy, suggesting merging preserves transferable knowledge that sequential updates erase.
- Complementarity with existing continual learning methods: PAM improves both a regularization method (LWF) and experience replay, and PAM+LWF approaches replay's performance without storing any old data, so the paradigm shift can be layered onto current techniques rather than replacing them.
- A tunable stability-plasticity dial: raising the alignment percentage from 30% to 70% monotonically reduces forgetting but slightly lowers plasticity, giving practitioners an explicit control for their forgetting-versus-adaptation requirements.
Reading between the lines
- An implication the paper leaves implicit: the sign-alignment criterion is a generic tool, so the same during-training alignment could be applied to other parameter-efficient families (adapters, prefixes, or prompt tokens) whenever their weights can be element-wise averaged; the paper's gains on LoRA suggest the mechanism, not the parameterization, carries the benefit.
- The paper's strongest combined result, TALL+PAM, hints that during-training alignment and post-training localization address different sources of interference; a natural extension the authors do not pursue is per-task or per-layer masks on top of the evolving global LoRA, which could target the stated limitation that a single merged adapter may struggle when new tasks differ sharply from previous
- If merging really desensitizes learning to task order, the result also bears on evaluation practice: continual learning results are often reported on one fixed sequence, and PAM-style merging would make such single-sequence comparisons more reliable, though the paper does not discuss this.
- The near-match of PAM+LWF to experience replay, achieved without storing data, suggests a testable scaling claim: on longer task streams where replay buffers saturate, alignment-based regularization may exceed replay's ceiling; this is an extrapolation beyond the paper's six- to ten-task experiments.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a continual learning paradigm for vision-language models based on model merging rather than sequential fine-tuning. A temporary LoRA is fine-tuned on each new task and then merged by element-wise averaging into a single evolving global LoRA; the proposed PAM method adds a during-training step that periodically re-initializes temporary LoRA weights whose signs conflict with the important (top-p% by magnitude) weights of the global LoRA. The authors evaluate on the CoIN benchmark and an extended CoIN sequence plus a video QA benchmark using PaliGemma, reporting improved accuracy, reduced backward transfer (forgetting), improved forward transfer, and greater robustness to task order relative to fine-tuning and several merging and continual learning baselines.
Significance. If the central claims hold, the paper offers a simple and parameter-efficient alternative to replay- and regularization-based continual learning for large VLMs: a single evolving LoRA that does not require task identities at inference. The authors provide a broad empirical study, including three task orders, an extended ten-task sequence, video QA, ablations of the alignment percentage, schedule, re-initialization target, LoRA rank, and initialization strategy, and combinations with LWF and ER. These are real strengths. However, the quantitative advantage over the simple Average baseline is modest (about 2.5 ACC points on CoIN), the gains are attributed to a sign-based alignment mechanism that is not isolated by a control experiment, and the main results use one seed per task order with hyperparameters selected on the same benchmark. The central idea is plausible and worth publishing after the mechanism is properly validated and the statistical basis is strengthened.
major comments (3)
- [§4, Algorithm 1; §5.3, Table 2] The paper attributes PAM's improvement over plain averaging to the sign-based alignment criterion, but no experiment varies the selection rule while holding the reinitialization operation fixed. Table 2 compares PAM (sign-based reinitialization) with Average (no reinitialization), and Table 4 varies only the reinitialization target (global weights vs. zero), not the criterion for choosing which weights to reset. The observed gain could therefore come from a generic periodic shrink toward the global LoRA: reinitializing any matched subset to the global values reduces the magnitude of the temporary update and may mitigate forgetting. Please add ablations that reinitialize a matched random subset, or a magnitude-based subset, with the same schedule and percentage. If those match PAM's accuracy and backward transfer, then the sign criterion is not the operative mechanism and the novelty claim in §4 would need to be revised.
- [§5.3, Table 3; Appendix A] The main hyperparameters (alignment percentage p=50%, schedule=100 steps) are selected using the same ACC/BWT metrics on the same CoIN benchmark that produces the headline results, and the reported standard deviations are across three task orders with one seed each, not across training runs. This makes the quantitative gains difficult to interpret: PAM's ACC advantage over Average in Table 2 (49.89 vs. 47.41) is smaller than the order-induced variability, and Table 7's three-seed check is on a different three-task sequence rather than the six-task CoIN setup. Please provide per-order paired results, confirm the main comparisons with multiple seeds on the full benchmark, and state whether any validation split was used for hyperparameter selection.
- [§5.3, Fig. 4] The 'Finetune + align' condition is not precisely defined. If alignment is applied to the sequential fine-tuning baseline, it is unclear what global weights serve as the sign reference and whether this condition is simply the PAM regularizer without merging. Without this definition, the claim that alignment helps fine-tuning itself is hard to interpret and cannot be used as evidence for the mechanism.
minor comments (4)
- [Algorithm 1] The loop in Algorithm 1 starts at t=2, but the first task must also be trained and merged; please state explicitly that task 1 is handled before the loop, or initialize the global LoRA accordingly.
- [Algorithm 1] The term 'Re-initialize' in Algorithm 1 leaves the replacement value unspecified; Table 4 later reveals that the main variant uses global weights, but the pseudo-code should state the default replacement value.
- [Appendix A, Eq. (5)] The FWT definition sums i from 2 to T-1 and divides by T-1, which counts only T-2 terms; please check the index range (it should likely be i=2 to T with denominator T-1, or i=1 to T-1).
- [Appendix B, Fig. 8] The left panel of Fig. 8 is described in the text as comparing 'randomly initializing' with initialization from the global LoRA, but the legend refers to 'Independent WG,t-1'; please clarify the terminology.
Circularity Check
No significant circularity: PAM is an empirically evaluated method with no claimed derivation from first principles.
full rationale
The paper's central claims are empirical demonstrations on external benchmarks (CoIN, extended CoIN, and video QA), not predictions derived from a fitted model or a named first-principles result. The sign-alignment criterion is imported from prior external work (Chen et al.; Davari and Belilovsky; Yadav et al.) and is not defined in terms of the outcome metrics. The merging step is an explicit algorithm, and the reported ACC/BWT/FWT/At values are measurements rather than quantities forced by construction. The ablations compare PAM with and without alignment, with different re-initialization targets, and with varying alignment percentages, providing ordinary empirical support rather than a derivation. Self-citations such as Kleiman et al., Sharma et al., and Frankle et al. appear in related-work or background contexts and are not load-bearing for PAM's design, validity, or evaluation. Hyperparameter selection on the same benchmark, and the absence of a control for the sign-selection rule, are potential external-validity concerns, but they are not circularity: no equation or fitted parameter reduces a reported result to an input of the method.
Assumptions & free parameters
free parameters (5)
- alignment_percentage =
50%
- alignment_schedule =
100 steps
- LoRA_rank =
32
- learning_rate =
5e-4
- reinitialization_strategy =
global weights W_G,t-1
assumptions (3)
- domain assumption Element-wise averaging of LoRA weights preserves and combines knowledge from different tasks.
- domain assumption Weight sign is a valid measure of alignment between LoRA weights.
- ad hoc to paper Re-initializing sign-misaligned weights during training does not prevent the current task from being learned.
Cite this review
Pith. "Pith review of Continual Learning in Vision-Language Models via Aligned Model Merging." pith.science (2026). https://pith.science/paper/WDHWFHIP
@misc{pith2026250603189,
author = {Pith},
title = {Pith review of: Continual Learning in Vision-Language Models via Aligned Model Merging},
year = {2026},
howpublished = {\url{https://pith.science/paper/WDHWFHIP}},
note = {Machine review of arXiv:2506.03189}
}
read the original abstract
Continual learning is conventionally tackled through sequential fine-tuning, a process that, while enabling adaptation, inherently favors plasticity over the stability needed to retain prior knowledge. While existing approaches attempt to mitigate catastrophic forgetting, a bias towards recent tasks persists as they build upon this sequential nature. In this work we present a new perspective based on model merging to maintain stability while still retaining plasticity. Rather than just sequentially updating the model weights, we propose merging newly trained task parameters with previously learned ones, promoting a better balance. To maximize the effectiveness of the merging process, we propose a simple mechanism that promotes learning aligned weights with previous ones, thereby avoiding interference when merging. We evaluate this approach on large Vision-Language Models (VLMs), and demonstrate its effectiveness in reducing forgetting, increasing robustness to various task orders and similarities, and improving generalization.
Forward citations
Cited by 2 Pith papers
-
Task Alignment: A Simple Proxy for Practical Model Merging Across Diverse Vision Tasks
Task alignment serves as an efficient proxy for hyperparameter selection in model merging, accelerating the process by orders of magnitude while preserving performance in vision models with heterogeneous decoders.
-
Towards Human-like Physical Intelligence: Lifelong Vision-Language-Action Learning for Robotic Manipulation
LifelongVLA pairs dual-timescale LoRA gating with stochastic cached-prefix replay to cut catastrophic forgetting in VLA policies, reporting 83.2% average success and 11.4% forgetting on a 10-task LIBERO stream.
Reference graph
Works this paper leans on
-
[3]
Z. Chen, J. Ngiam, Y. Huang, T. Luong, H. Kretzschmar, Y. Chai, and D. Anguelov. Just pick a sign: Optimizing deep multitask models with gradient sign dropout.Advances in Neural Information Processing Systems, 33:2039–2050,
-
[4]
L. Choshen, E. Venezian, N. Slonim, and Y. Katz. Fusing finetuned models for better pretraining.arXiv preprint arXiv:2204.03044,
-
[7]
How to Merge Your Multimodal Models Over Time?
S. Dziadzio, V. Udandarao, K. Roth, A. Prabhu, Z. Akata, S. Albanie, and M. Bethge. How to merge your multimodal models over time?arXiv preprint arXiv:2412.06712,
-
[10]
A. Kembhavi, M. Salvato, E. Kolve, M. Seo, H. Hajishirzi, and A. Farhadi. A diagram is worth a dozen images. InComputer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, pages 235–251. Springer,
work page 2016
-
[13]
J. Liu, J. Wu, J. Liu, and Y. Duan. Learning attentional mixture of loras for language model continual learning. arXiv preprint arXiv:2409.19611,
-
[14]
A. Prabhakar, Y. Li, K. Narasimhan, S. Kakade, E. Malach, and S. Jelassi. Lora soups: Merging loras for practical skill composition tasks.arXiv preprint arXiv:2410.13025,
-
[16]
The Non-Local Model Merging Problem: Permutation Symmetries and Variance Collapse
E. Sharma, D. M. Roy, and G. K. Dziugaite. The non-local model merging problem: Permutation symmetries and variance collapse.arXiv preprint arXiv:2410.12766,
-
[17]
K. Wang, N. Dimitriadis, G. Ortiz-Jimenez, F. Fleuret, and P. Frossard. Localizing task information for improved model merging and compression. InForty-first International Conference on Machine Learning, 2024a. L. Wang, X. Zhang, H. Su, and J. Zhu. A comprehensive survey of continual learning: theory, method and application.IEEE Transactions on Pattern An...
work page 2023
Show all 22 references
-
[18]
L. Xu, H. Xie, S.-Z. J. Qin, X. Tao, and F. L. Wang. Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment.arXiv preprint arXiv:2312.12148,
-
[19]
We used the best hyperparameters reported by Beyer et al
15 Continual Learning in Vision-Language Models via Aligned Model Merging Table 6| Hyperparameters used in training each task. We used the best hyperparameters reported by Beyer et al. (2024) on fine-tuning on each task independently. Task E poc h s Batc h s i z e We igh t Dec...
2024
-
[20]
Baselines We used a LoRA rank of 32 for all baselines
Consistent with previous findings, we observe that the standard deviation across three run is very small. Baselines We used a LoRA rank of 32 for all baselines. For TIES. (Yadav et al., 2024), we tuned the percentage of weights retained before alignment (i.e., during the pruni...
2024
-
[21]
At each time step, we report the average forgetting on seen tasks so far (lower is better)
for the second (left) and third (right) orders. At each time step, we report the average forgetting on seen tasks so far (lower is better). Evaluation metrics We do extensive evaluation by assessing four metrics to measure model capabil- ities. (1) Average accuracy (ACC): Let𝐴...
2017
-
[22]
In contrast, merging demonstrates greater robustness to these task variations
Consistent with previous observations, fine-tuning is more prone to forgetting and the performance of a task varies over time; some tasks have a greater impact on previous task performance than others, likely due to varying degrees of task similarity. In contrast, merging demo...
2022
-
[2017]
Kleiman, G
A. Kleiman, G. K. Dziugaite, J. Frankle, S. Kakade, and M. Paul. Soup to go: mitigating forgetting during continual learning with model averaging.arXiv preprint arXiv:2501.05559,
-
[2018]
Don’tforget, thereismorethanforgetting: new metrics for continual learning
N.Díaz-Rodríguez, V.Lomonaco, D.Filliat, D.Maltoni, etal. Don’tforget, thereismorethanforgetting: new metrics for continual learning. InContinual Learning Workshop at NeurIPS 2018, pages 1–7,
2018
-
[2019]
Ilharco, M
G. Ilharco, M. T. Ribeiro, M. Wortsman, S. Gururangan, L. Schmidt, H. Hajishirzi, and A. Farhadi. Editing models with task arithmetic.arXiv preprint arXiv:2212.04089,
-
[2020]
Chaudhry, M
A. Chaudhry, M. Rohrbach, M. Elhoseiny, T. Ajanthan, P. K. Dokania, P. H. Torr, and M. Ranzato. On tiny episodic memories in continual learning.arXiv preprint arXiv:1902.10486,
1902 arXiv
-
[2021]
icarl: Incrementalclassifierandrepresentation learning
S.-A.Rebuffi,A.Kolesnikov,G.Sperl,andC.H.Lampert. icarl: Incrementalclassifierandrepresentation learning. InProceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001–2010,
2001
-
[2022]
Izmailov, D
P. Izmailov, D. Podoprikhin, T. Garipov, D. Vetrov, and A. G. Wilson. Averaging weights leads to wider optima and better generalization.arXiv preprint arXiv:1803.05407,
-
[2023]
X. Li, W. Ren, W. Qin, L. Wang, T. Zhao, and R. Hong. Analyzing and reducing catastrophic forgetting in parameter efficient tuning. InICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE,
2025
-
[2024]
ISSN 2835-8856. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,
1901
-
[2025]
J. Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805,
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.