PASs-MoE: Mitigating Misaligned Co-drift among Router and Experts via Pathway Activation Subspaces for Continual Learning

Haiyun Guo; Haokai Ma; Jinqiao Wang; Yandu Sun; Yonghui Yang; Zhiyan Hou

REVIEW 4 major objections 6 minor 6 references

PASs-MoE: Mitigating Misaligned Co-drift among Router and Experts via Pathway Activation Subspaces for Continual Learning

T0 review · 4 major / 6 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read In fixed-capacity MoE-LoRA continual instruction tuning, replacing the learned router with each expert's own low-rank activation energy and stabilizing historically important rank directions reduces forgetting and lifts final average accura

desk verdict A plausible self-routing MoE-LoRA method with a real confound in the parameter budget and thin statistics; worth refereeing with requests for matched comparisons and multiple seeds. read the letter →

arxiv 2601.13020 v2 pith:XHY625QC submitted 2026-01-19 cs.LG cs.AI

Zhiyan Hou , Haiyun Guo , Haokai Ma , Yandu Sun , Yonghui Yang , Jinqiao Wang This is my paper

classification cs.LGcs.AI

keywords continualinstructiontuningmixtureofexpertsLoRApathwayactivationsubspacecatastrophicforgettingroutingmultimodallargelanguagemodelsrankstabilization

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that in fixed-capacity MoE-LoRA continual instruction tuning, the router and experts drift in misaligned ways: the router reroutes old inputs while expert updates change how experts respond, blurring task specialization and worsening forgetting. It proposes to drop the learned router entirely and derive routing from the activation energy of each expert's own low-rank pathway, defined via the pathway activation subspace (PASs) spanned by the LoRA down-projection. A second component, PASs-aware rank stabilization, uses activation-based importance scores to selectively freeze the rank directions that previous tasks depended on, while leaving others free to adapt. On a seven-task multimodal continual benchmark, the method beats the strongest comparison by 5.1% average accuracy and roughly halves forgetting, without adding any parameters. The contribution is a reproducible recipe: one subspace signal for both routing and preservation.

What carries the argument

Pathway activation subspace (PASs): for each LoRA expert e, S_e = span(A_e^T), the row space of the down-projection A_e. The low-rank response z_e = A_e h is an r-dimensional vector of activation coefficients, one per rank direction. From it the paper derives two signals: activation energy s_e = (1/r)||A_e h||^2 for softmax routing (PASs-RW), and rank importance I_{e,k} = E[π_e(h)(a_{e,k}^T h)^2] for weighted stabilization of the rows of A (PASs-RS). The same subspace thus anchors both routing and preservation, eliminating the separate learned router.

What would settle it

Take a trained MoE-LoRA and artificially rescale one expert's A rows or its input so that activation energies rise without changing the expert's output function; if PASs-RW routes more inputs to that expert despite no change in capability, the energy proxy is tracking scale, not specialization. Concretely, compute the correlation between s_e(h) and input norm on held-out examples: if high-energy experts are simply the ones with larger input norms, the assumption fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that the low-rank activation A_e h of a LoRA expert already encodes enough information to do both routing and forgetting protection, so the separate learned router is not just unnecessary but actively harmful under sequential training. The pathway activation subspace S_e = span(A_e^T) is the row space of the down-projection; the low-rank response z_e = A_e h gives a capability-aligned coordinate system. PASs-RW sets expert mixture weights to softmax of (1/r)||A_e h||^2, tying routing to each expert's functional response. PASs-RS accumulates importance I_{e,k} = E[π_e(h)(a_{e,k}^T h)^2] over past tasks and adds weighted L2 stabilization on rows of A to protect his

Load-bearing premise

Routing and importance both rely on the premise that the magnitude of A_e h reflects which expert (and which rank direction) genuinely specializes for that input — if activation energy is dominated by input scale, task hardness, or outlier directions rather than by learned specialization, both the reweighting and the stabilization will target the wrong places.

Editorial extensions

If this is right

Learned routers can be replaced by activation-based self-routing in fixed-capacity MoE-LoRA, removing a source of drift and adding no parameters.
Selective rank-level stabilization guided by activation importance yields better retention than uniform or random regularization of LoRA parameters.
The method keeps the same fixed expert set across all tasks, so model capacity does not grow with the task stream.
The gains are not tied to a single task ordering: a reversed task sequence still improves average performance by 9.46% over the second-best baseline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

(Editorial inference) If activation energy genuinely tracks task-relevant specialization, the same signal could serve as a diagnostic: inspecting which experts and rank directions fire on a new task could predict how much forgetting that task will cause before training begins.
(Editorial inference) Router-free routing simplifies deployment because inference no longer needs a separately stored routing network; routing becomes a deterministic function of the expert parameters themselves.
(Editorial inference) The core idea may transfer to other parameter-efficient fine-tuning forms (adapters, prefix tuning) whenever a parameter-induced subspace and activation norm can be defined, though the paper notes that extending it would require redefining the coordinate system.
(Editorial inference) A cleaner ablation would isolate the two components against a single-LoRA baseline and against PASs-RS alone; the paper's ablations always combine both on top of a softmax MoE baseline, so the individual contribution of each to forgetting reduction is not fully separated.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The core idea is simple and sensible: use the norm of the LoRA down-projection activation as a router signal, and regularize per-rank directions by accumulated activation importance. That combination is not in the cited MoE-LoRA works, and it's cleanly motivated by the factorization of LoRA as B(Ah). The paper does a decent job of showing that both components matter: the ablation in Table 2 and the random-regularization comparison in Table 7 support the importance signal, and the alternative task order in the appendix is a good check. The limitations section is candid about the proxy nature of the activation energy and the noise in importance estimates. That's real credit.

The soft spots are empirical, not conceptual. The largest is the parameter-budget confound. The MoELoRA baselines include a learned router; PASs-RW removes it. The paper claims a fixed MoE-LoRA parameter budget, but the router parameters are not accounted for. So the comparison is not parameter-matched, and the ablation in Table 2 conflates removing the router with adding the new routing signal. The router is small relative to the LoRA parameters, so the effect is probably modest, but it is a genuine confound and the paper should either match trainable parameters (add a dummy router or bump LoRA rank) or explicitly argue why the router size is negligible. The paper doesn't do that.

Second, all numbers come from a single seed, with no standard deviations or significance tests. The hyperparameters lambda and E are tuned on the evaluation benchmark itself, and the benchmark is co-authored by the same group. That combination weakens the headline 5.1% AP gain. These are addressable with multiple seeds and ideally an independent benchmark, but as is, the empirical support is thinner than the abstract suggests.

The conceptual assumption—that activation energy tracks expert specialization rather than input scale or outlier directions—is acknowledged in the limitations, and I buy that it is a real limitation rather than a fatal flaw. The method is internally coherent and the ablations are informative. I don't see a load-bearing mathematical error.

Who this is for: people working on MoE-LoRA for continual instruction tuning. It's a within-subfield improvement, not a paradigm shift. But it deserves a serious referee. I'd send it to peer review with the request that the authors fix the parameter budget comparison and report multiple seeds. The idea is worth engaging with; the current evidential base just needs to be tightened.

Referee Report

4 major / 6 minor

Summary. The paper proposes PASs-MoE, a continual instruction tuning method for MLLMs under a fixed-capacity MoE-LoRA setting. The method defines a Pathway Activation Subspace (PASs) as span(A_e^T) from each LoRA down-projection, and uses the low-rank activation energy s_e=(1/r)||A_e h||^2 to compute routing weights (PASs-RW), replacing the learned router. A second component, PASs-aware Rank Stabilization (PASs-RS), computes rank-level importance I_{e,k}=E[π_e(h)(a_{e,k}^T h)^2] and applies a weighted quadratic penalty on changes of the corresponding LoRA rows. Experiments on MLLM-CTBench with LLaVA-1.5-7B report that the method outperforms MoE-LoRA baselines and standard continual learning methods in both AP and BWT, with a claimed 5.1% AP improvement over the second-best method. Ablations and analyses on routing stability, importance sparsity, and update-vs-importance patterns are provided.

Significance. If the empirical claims hold, the contribution is conceptually interesting and practically useful: routing is tied to the expert's low-rank pathway rather than a separately learned router, and the same activation signal is reused for rank-level regularization. The method is clearly specified, the code is promised, and the paper includes a random-regularization ablation and several diagnostic plots that go beyond a simple accuracy comparison. However, the force of the empirical claim is weakened by the fact that the MoE-LoRA baselines include a learned router while PASs-RW removes it, so the 'fixed MoE-LoRA parameter budget' is not actually matched; by the use of a single random seed; and by hyperparameters λ and E selected on the same benchmark and reported at their best. These issues must be addressed before the claimed improvements can be attributed to the proposed components.

major comments (4)

[Section 5.3, Tables 2 and 3] The central comparison is not parameter-matched. The MoELoRA (softmax) baseline contains a learned router linear layer; PASs-RW removes that layer entirely, so the comparison conflates the routing mechanism with a reduction in trainable parameters. The text claims a 'fixed MoE-LoRA parameter budget' without accounting for router parameters. Please either (a) add a baseline that keeps the router (or an equal-size linear layer) while using the PASs energy for reweighting, or (b) increase the expert rank of the MoE-LoRA baseline so that total trainable parameters match PASs-MoE, and re-run the main tables. Without such a matched comparison, the 5.1% AP gain in Table 1 cannot be attributed to the proposed components.
[Section 5.1, Tables 1–6] All experiments are reported for a single fixed random seed (42), and no standard deviations or significance tests are given. For a continual learning paper that claims consistent improvements, this is insufficient. Please report mean±std over at least three seeds for the key comparisons (Tables 1, 2, and the Order-B results in Table 6), and state whether the improvements are statistically significant. This is particularly important because the hyperparameters λ and E appear to be tuned on the same benchmark.
[Section 5.3.2, Fig. 4 and Table 3] The hyperparameters λ and E are selected on MLLM-CTBench and the paper reports the best configuration (E=6, λ=5e-4). Since Fig. 4 shows a non-monotonic AP curve and Table 3 shows non-monotonic behavior in E, the reported gains may reflect selection on the test set. Please provide a validation-split or a robustness analysis (e.g., a small grid of λ and E) showing that the conclusion is not an artifact of choosing the best values on the test benchmark. At minimum, disclose that these values were chosen on the test set.
[Section 5.1 and References (MLLM-CTBench)] The evaluation benchmark MLLM-CTBench is cited as Guo et al. 2025b, which includes several of the same authors as this paper (Haiyun Guo, Jinqiao Wang). Evaluating on a self-authored benchmark without external validation is a concern, especially since the paper compares to many methods from other groups. Please either provide the exact splits and evaluation scripts to allow third-party reproduction, or supplement the main evaluation with an external benchmark (e.g., COIN or another established multimodal CIT benchmark) to demonstrate generality. The current paper does not mention this overlap or address potential bias.

minor comments (6)

[Abstract and Section 5.2] The phrase 'without increasing model parameters' is technically true but potentially misleading: PASs-MoE actually has fewer trainable parameters than MoE-LoRA because the router is removed. Please rephrase to 'with no additional parameters beyond the existing expert pool' or 'with a fixed parameter budget excluding the router', and clarify the router parameter counts.
[Section 5.2] The claim 'our final AP exceeds that of the second-best method by 5.1%' refers to an absolute difference of 5.10 percentage points (48.46 vs 43.36). Please use '5.1 points' or '5.1 percentage points' to avoid ambiguity.
[References] There is a typo in the Radford et al. reference: 'PmLR' should be 'PMLR'. Also, 'HiDe-llava' in Tables 1 and 6 should be 'HiDe-LLaVA' for consistency.
[Table 1 caption] The caption says 'Best and second-best results for Acc and AP' but the table reports both Acc and Forget. Please clarify which columns the bold/underline refers to (likely Acc and AP).
[Fig. 3] The stability metric is defined as '−MeanJS ↑' but the precise computation (which layers, which inputs, how the gating distributions are compared) is not fully described in the main text. Please add a short explanation or refer to the appendix.
[Appendix A.2, Table 6] Table 6 presents results under Order-B but has the same caption as Table 1. Please retitle it as 'Results under the alternative task order (Order-B)' to avoid confusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the routing and rank-stabilization signals are defined from the same LoRA matrices that are trained, but the paper's central claims are empirical comparisons on a fixed public benchmark, not a derivation that assumes its conclusion.

full rationale

The paper's formal chain is definitional rather than derivational: S_e = span(A_e^T) (Eq. 5), s_e = (1/r)||A_e h||^2 (Eq. 6), π_e = softmax(s_e) (Eq. 7), I_{e,k} = E[π_e(h)(a_{e,k}^T h)^2] (Eq. 9), and L_stab (Eq. 11) all operate on the same matrices A_e that are being trained. This is the intended control loop, not an input covertly renamed as an output. The load-bearing claim is empirical: PASs-RW and PASs-RS improve AP and BWT in Tables 1-3 and 6-7. The evaluation uses MLLM-CTBench, cited to Guo et al. 2025b with overlapping authors. That is a self-citation, but the benchmark is a fixed public protocol with specified tasks, orders, and evaluation metrics, so it is externally falsifiable and counts as real evidence under the reviewing rules; it does not make the derivation circular. The Limitations section explicitly flags that low-rank energy is a proxy for compatibility and that activation-derived importance can be noisy; these are robustness/calibration caveats, not admissions of circularity. No equation is shown to be identical to another by construction, no fitted parameter is relabeled as a prediction, and no uniqueness theorem or prior ansatz is imported from the authors' own work to force the design. The 'pathway activation subspace' is a new name for the standard row-space span(A_e^T), but the paper uses it to construct a regularizer rather than to claim that a known empirical pattern has been derived. Therefore no significant circularity is present.

Assumptions & free parameters 2 free parameters · 5 assumptions · 2 invented entities

The method introduces two ad hoc modeling premises—activation energy as compatibility and activation importance as task criticality—plus a conceptual failure mode. It also relies on a benchmark from the same group. All main results depend on these premises; the paper acknowledges the first two as limitations.

free parameters (2)

stabilization coefficient lambda = 5e-4
Chosen via hyperparameter ablation on MLLM-CTBench (Fig. 4); controls the stability-plasticity trade-off and directly determines the L_stab contribution.
number of experts E = 6
Selected from {2,4,6,8} ablation on the same benchmark (Table 3); best AP at E=6.

assumptions (5)

ad hoc to paper The LoRA low-rank response z_e = A_e h and its squared norm indicate an expert's capability/input-expert compatibility.
Central premise for PASs-RW; the authors themselves flag it as a proxy in Limitations.
ad hoc to paper Catastrophic forgetting in MoE-LoRA is driven by 'Misaligned Co-drift' between router preferences and expert pathways.
Introduced and motivated in Section 1; no formal definition or independent causal test.
ad hoc to paper Aggregated activation importance I_agg identifies rank directions that should be preserved.
Used by PASs-RS; support is the paper's own non-uniformity analysis (Fig. 7), and Limitations note the estimate can be noisy.
domain assumption MLLM-CTBench and its two task orders are a valid and representative evaluation of CIT.
The benchmark is cited from the same research group (Guo et al., 2025b); no independent benchmark is used in the paper.
domain assumption Frozen pretrained backbone plus fixed pool of E LoRA experts is the right capacity setting.
Stated in Section 4.1; the paper explicitly limits conclusions to this setting in Limitations.

invented entities (2)

Pathway Activation Subspace (PASs)
purpose: Named subspace S_e = span(A_e^T) used as the coordinate system for routing energy and rank importance.
It is a re-labeling of the row space of the LoRA down-projection A_e, with no falsifiable handle outside the paper's own method.
Misaligned Co-drift
purpose: Postulated failure mode of MoE-LoRA continual learning that the method is designed to mitigate.
Never formally quantified; Fig. 3 shows router drift via an unspecified JS divergence, but no causal test distinguishes co-drift from other sources of forgetting.

how reviews work

0 comments

Cite this review

Pith. "Pith review of PASs-MoE: Mitigating Misaligned Co-drift among Router and Experts via Pathway Activation Subspaces for Continual Learning." pith.science (2026). https://pith.science/paper/XHY625QC

@misc{pith2026260113020,
  author       = {Pith},
  title        = {Pith review of: PASs-MoE: Mitigating Misaligned Co-drift among Router and Experts via Pathway Activation Subspaces for Continual Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XHY625QC}},
  note         = {Machine review of arXiv:2601.13020}
}

read the original abstract

Continual instruction tuning (CIT) requires multimodal large language models (MLLMs) to adapt to a stream of tasks without forgetting prior capabilities. A common strategy is to isolate updates by routing inputs to different LoRA experts. However, existing LoRA-based Mixture-of-Experts (MoE) methods often jointly update the router and experts in an indiscriminate way, causing the router's preferences to co-drift with experts' adaptation pathways and gradually deviate from early-stage input--expert specialization. We term this as Misaligned Co-drift, which blurs expert responsibilities and exacerbates forgetting. To address this, we introduce the pathway activation subspace (PASs), a LoRA-induced subspace that reflects which low-rank pathway directions an input activates in each expert, providing a capability-aligned coordinate system for routing and preservation. Based on PASs, we propose a fixed-capacity PASs-based MoE--LoRA method with two components: PAS-guided Reweighting, which calibrates routing using each expert's pathway activation signals, and PAS-aware Rank Stabilization, which selectively stabilizes rank directions important to previous tasks. Experiments on a CIT benchmark show that our approach consistently outperforms a range of conventional continual learning baselines and MoE--LoRA variants in both accuracy and resistance to forgetting, without increasing model parameters. Our code is publicly available at https://github.com/yueluoshuangtian/PASs-MoE.

Figures

Figures reproduced from arXiv: 2601.13020 by the authors.

**Figure 1.** Illustration of the Misaligned Co-drift in existing MoE-LoRA methods, where the thickness of arrow indicates the sampling probability of router. Here, the drift in router assignments and the internal drift within expert parameters for the same task jointly drive this issue, thereby exacerbating catastrophic forgetting. quentially adapt MLLMs on a stream of tasks with supervised instruction–response data while retai… view at source ↗

**Figure 2.** Overview of the proposed PASs-based MoE-LoRA method. We consider continual instruction tuning with a fixed set of E LoRA experts. (Bottom) Each input x t generates a low-rank response ze and activation energy se via the Pathway Activation Subspace (PASs) of each expert. (Right) PASs-guided Reweighting (PASs-RW) leverages se to compute mixture weights, eliminating the need for an independent router. (Top) PASs-aware … view at source ↗

**Figure 3.** Comparison of final performance and router stability. Top and bottom panels display the final accuracy and the corresponding router stability across the task stream, respectively. Stability is quantified as the negative Mean Jensen–Shannon (JS) divergence. an effective basis for expert reweighting under sequential training. This observation aligns with our motivation in the Introduction that routing should be groun… view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Ablation results on the regularization strength [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: Update magnitude versus old-task importance across representative layers. Each plot bins coefficients by the previous task’s aggregated importance Iagg (low → high) and reports the median (with interquartile range) of the log update norm log ∥∆W∥ within each bin [PITH…

**Figure 6.** Figure 6: Training loss curves across tasks [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

6 extracted references · 5 linked inside Pith

[6]

InThe Eleventh International Con- ference on Learning Representations

Progressive prompts: Continual learning for language models. InThe Eleventh International Con- ference on Learning Representations. David Rolnick, Arun Ahuja, Jonathan Schwarz, Tim- othy Lillicrap, and Gregory Wayne. 2019a. Expe- rience replay for continual learning.Advances in neural information processing systems, 32. David Rolnick, Arun Ahuja, Jonathan...

arXiv 2023
[2017]

Zhizhong Li and Derek Hoiem

Overcoming catastrophic forgetting in neural networks.Proceedings of the National Academy of Sciences, 114(13):3521–3526. Zhizhong Li and Derek Hoiem. 2017a. Learning with- out forgetting.IEEE transactions on pattern analysis and machine intelligence, 40(12):2935–2947. Zhizhong Li and Derek Hoiem. 2017b. Learning with- out forgetting.Preprint, arXiv:1606....

arXiv 2024
[2018]

InInternational conference on machine learn- ing, pages 2668–2677

Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). InInternational conference on machine learn- ing, pages 2668–2677. PMLR. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Ag- nieszka Grabska-Barwinska, Demi...
[2022]

ArXiv, abs/2210.11416

Scaling instruction-finetuned language models. ArXiv, abs/2210.11416. Yawen Cui, Jian Zhao, Zitong Yu, Rizhao Cai, Xun Wang, Lei Jin, Alex C Kot, Li Liu, and Xuelong Li. 2025. Cmoa: Contrastive mixture of adapters for generalized few-shot continual learning.IEEE Transactions on Multimedia. Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. 2020. Ort...

arXiv 2025
[2023]

Jinghan He, Haiyun Guo, Kuan Zhu, Zihan Zhao, Ming Tang, and Jinqiao Wang

Continual instruction tuning for large multi- modal models.arXiv preprint arXiv:2311.16206. Jinghan He, Haiyun Guo, Kuan Zhu, Zihan Zhao, Ming Tang, and Jinqiao Wang. 2024. Seekr: Selective attention-guided knowledge retention for continual 9 learning of large language models.arXiv preprint arXiv:2411.06171. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyua...

arXiv 2024
[2025]

Haiyang Guo, Fanhu Zeng, Ziwei Xiang, Fei Zhu, Da-Han Wang, Xu-Yao Zhang, and Cheng-Lin Liu

Dynamic mixture of curriculum lora experts for continual multimodal instruction tuning.arXiv preprint arXiv:2506.11672. Haiyang Guo, Fanhu Zeng, Ziwei Xiang, Fei Zhu, Da-Han Wang, Xu-Yao Zhang, and Cheng-Lin Liu. 2025a. Hide-llava: Hierarchical decoupling for con- tinual instruction tuning of multimodal large lan- guage model.arXiv preprint arXiv:2503.129...

arXiv

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.

[1] [6]

InThe Eleventh International Con- ference on Learning Representations

Progressive prompts: Continual learning for language models. InThe Eleventh International Con- ference on Learning Representations. David Rolnick, Arun Ahuja, Jonathan Schwarz, Tim- othy Lillicrap, and Gregory Wayne. 2019a. Expe- rience replay for continual learning.Advances in neural information processing systems, 32. David Rolnick, Arun Ahuja, Jonathan...

arXiv 2023

[2] [2017]

Zhizhong Li and Derek Hoiem

Overcoming catastrophic forgetting in neural networks.Proceedings of the National Academy of Sciences, 114(13):3521–3526. Zhizhong Li and Derek Hoiem. 2017a. Learning with- out forgetting.IEEE transactions on pattern analysis and machine intelligence, 40(12):2935–2947. Zhizhong Li and Derek Hoiem. 2017b. Learning with- out forgetting.Preprint, arXiv:1606....

arXiv 2024

[3] [2018]

InInternational conference on machine learn- ing, pages 2668–2677

Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). InInternational conference on machine learn- ing, pages 2668–2677. PMLR. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Ag- nieszka Grabska-Barwinska, Demi...

[4] [2022]

ArXiv, abs/2210.11416

Scaling instruction-finetuned language models. ArXiv, abs/2210.11416. Yawen Cui, Jian Zhao, Zitong Yu, Rizhao Cai, Xun Wang, Lei Jin, Alex C Kot, Li Liu, and Xuelong Li. 2025. Cmoa: Contrastive mixture of adapters for generalized few-shot continual learning.IEEE Transactions on Multimedia. Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. 2020. Ort...

arXiv 2025

[5] [2023]

Jinghan He, Haiyun Guo, Kuan Zhu, Zihan Zhao, Ming Tang, and Jinqiao Wang

Continual instruction tuning for large multi- modal models.arXiv preprint arXiv:2311.16206. Jinghan He, Haiyun Guo, Kuan Zhu, Zihan Zhao, Ming Tang, and Jinqiao Wang. 2024. Seekr: Selective attention-guided knowledge retention for continual 9 learning of large language models.arXiv preprint arXiv:2411.06171. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyua...

arXiv 2024

[6] [2025]

Haiyang Guo, Fanhu Zeng, Ziwei Xiang, Fei Zhu, Da-Han Wang, Xu-Yao Zhang, and Cheng-Lin Liu

Dynamic mixture of curriculum lora experts for continual multimodal instruction tuning.arXiv preprint arXiv:2506.11672. Haiyang Guo, Fanhu Zeng, Ziwei Xiang, Fei Zhu, Da-Han Wang, Xu-Yao Zhang, and Cheng-Lin Liu. 2025a. Hide-llava: Hierarchical decoupling for con- tinual instruction tuning of multimodal large lan- guage model.arXiv preprint arXiv:2503.129...

arXiv