Pith. sign in

REVIEW 4 major objections 4 minor 3 cited by

Enhancing Multimodal Continual Instruction Tuning with BranchLoRA

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read BranchLoRA claims that splitting LoRA into a shared trunk and task branches reduces forgetting and beats MoELoRA on the CoIN benchmark.

desk verdict Plausible, well-motivated LoRA variant for MCIT with consistent wins, but missing error bars, a HydraLoRA baseline, and a thorough look at the auto-selector. read the letter →

arxiv 2506.02041 v1 pith:OKKNWPZK submitted 2025-05-31 cs.CL cs.AI

Duzhen Zhang , Yong Ren , Zhong-Zhi Li , Yahan Yu , Jiahua Dong , Chenxing Li , Zhilong Ji , Jinfeng Bai This is my paper
classification cs.CLcs.AI
keywords multimodalcontinualinstructiontuningcatastrophicforgettingLoRAmixture-of-expertstask-specificroutersparameter-efficientfine-tuninglargelanguagemodelslearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

BranchLoRA is an attempt to make multimodal continual instruction tuning both cheaper and less forgetful by restructuring the Mixture-of-Experts LoRA approach. The paper argues that in MoELoRA, the low-rank $A$ matrices of all experts converge to nearly the same shared patterns while the $B$ matrices stay distinct, so maintaining many full experts is parameter-inefficient. BranchLoRA therefore keeps one shared $A$ matrix as a trunk and lets each task live in separate $B$ branches, selected sparsely by task-specific routers, with previously trained branches frozen to protect old knowledge. A learned key-based task selector routes test inputs to the right router without needing task identity. If the reported results hold, this is a path to updating a multimodal model across sequential instruction tasks with higher average accuracy, less backward forgetting, and fewer trainable parameters than the previous state of the art.

What carries the argument

The load-bearing object is the asymmetric BranchLoRA decomposition of a LoRA adapter into a single shared matrix $A$ (the trunk) and many task-specific matrices $B$ (the branches), used inside every feed-forward layer of the MLLM. Around this decomposition, three mechanisms carry the argument: a top-$k$ sparse selection that replaces summation over all experts; a flexible tuning-freezing rule that freezes the most-activated branches after each task so later tasks can read but not overwrite them; and per-task routers with learned image and text keys that select the correct router automatically during inference. Together they convert the paper's observation about $A$/$B$ divergence into an architecture whose branches specialize per task while the shared trunk transfers what is common.

What would settle it

Re-run the CoIN experiment with an oracle task selector that always routes each test sample to its true task router. If oracle-routed ACC, MAA, or BWT move materially away from 44.20, 49.94, and -20.98 on LLaVA-1.5-7B, the automatic selector is carrying part of the reported gain; reporting a per-task routing confusion matrix on held-out data would settle whether the 95.8% average hides systematic failures.

Watch

Extended reading notes

Core claim

The paper's central claim is that the parameter-inefficiency it observes in MoELoRA is real and fixable: after continual tuning, all expert $A$ matrices converge onto task-invariant features while $B$ matrices carry task-specific knowledge, so an asymmetric decomposition is the right inductive bias. BranchLoRA replaces the summed-expert aggregation with a shared $A$ matrix and multiple $B$ branches, activates only the top-$k$ branches, freezes the most activated branches after each task, and adds one router per task to avoid drifting toward the most recent task. It further claims that learned image and text keys can select the correct router automatically at test time. On the CoIN benchmark with LLaVA-1.5-7B, the paper reports ACC, MAA, and BWT of 44.20, 49.94, and -20.98, compared with 37.13, 42.76, and -25.91 for MoELoRA, and reports similar gains at 13B scale with 222M trainable parameters versus 350M for MoELoRA.

Load-bearing premise

The load-bearing premise is that the cosine-aligned task keys learned by Eq. (4) keep selecting the correct router on held-out test samples; the paper reports only a single average selector accuracy of 95.8%, with no per-task routing accuracy and no oracle-routing comparison, so if routing errors are more frequent on some tasks the headline ACC/MAA/BWT numbers would not be reproducible.

Editorial extensions

If this is right

  • If the reported gains hold, continual instruction tuning of multimodal models can be made more parameter-efficient: 222M trainable parameters instead of 350M, with higher average accuracy and a less negative backward-transfer score.
  • A model updated this way can be deployed without requiring task identity at inference, because the key-based selector picks the router; this removes a major obstacle to real-world continual deployment.
  • The component-level ablations attribute the final ACC/MAA/BWT to each addition—shared $A$, sparse selection, tuning-freezing, task-specific routers—so each mechanism contributes, but none alone matches the full framework.
  • The pattern extends to a larger backbone: BranchLoRA also outperforms MoELoRA on LLaVA-1.5-13B, so the benefit is not an artifact of a small model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference beyond the paper: the frozen-branch design suggests a modular memory; one could inspect, merge, or remove individual branches after training to update or unlearn a task, but the paper does not demonstrate any such post-hoc manipulation.
  • Inference beyond the paper: the selector's reported 95.8% average accuracy is an aggregate; comparing auto-routed results against an oracle that always chooses the true task router would isolate how much of the 44.20 ACC depends on the selector staying correct.
  • Inference beyond the paper: if the $A$-converges/$B$-diverges pattern is generic, the same asymmetric decomposition could reduce forgetting in non-multimodal continual instruction tuning, since instruction-following models may share language-level patterns across tasks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper identifies a parameter-redundancy pattern in MoELoRA for Multimodal Continual Instruction Tuning (MCIT): during sequential training, the low-rank A matrices across experts converge while the B matrices remain distinct. Based on this observation, the authors propose BranchLoRA, an asymmetric LoRA architecture with a shared A matrix, multiple task-specific B experts, a dynamic top-k sparse selection rule, a flexible tuning-freezing mechanism, task-specific routers, and a learned automatic task selector based on image/text keys. Experiments on the CoIN benchmark with LLaVA-1.5-7B and LLaVA-1.5-13B report consistent improvements over MoELoRA in ACC, MAA, and BWT, together with reductions in trainable parameters and training time. An ablation study attributes gains to each proposed component.

Significance. If the empirical claims hold, BranchLoRA is a meaningful contribution to MCIT: it is more parameter-efficient than MoELoRA (222M vs 350M trainable parameters on LLaVA-7B), reports better average accuracy and less catastrophic forgetting on a standard eight-task benchmark, and the ablation in Table 3 is consistent with the proposed design. The paper also provides a clearly stated motivation with t-SNE evidence (Figure 1) and reports results on two model sizes. However, the significance is currently conditional on two pieces of missing evidence: a proper validation of the automatic task selector and statistical support for the 'significantly outperforms' claim. Without per-task routing accuracy, an oracle comparison, and variance information, the magnitude of the reported margins over MoELoRA is not fully established.

major comments (4)
  1. [§6.3 with Tables 1 and 4] The central no-task-identity inference claim rests on the automatic task selector, but the paper gives only one aggregate number, an average accuracy of 95.8%, and does not state whether the main results in Tables 1 and 4 were produced with the auto-selector or with true task IDs. This is load-bearing because the CoIN tasks have incompatible output formats: a Grounding sample misrouted to a QA router can receive zero credit, while a misrouted VQA sample may receive credit only by chance. Please state explicitly which routing mode produced Tables 1 and 4, report per-task selector accuracy and a confusion matrix, and provide an oracle-router comparison together with an analysis of how misrouted samples change ACC, MAA, and BWT.
  2. [Abstract and §6.1] The phrase 'significantly outperforms' is used without any statistical support. No standard deviations, number of seeds, or significance tests are reported for Table 1, Table 3, or Table 4. For example, the ACC difference between BranchLoRA and MoELoRA on LLaVA-7B is 7.07 points and the BWT difference is 4.93 points, but without variance estimates we cannot tell whether these margins are stable across runs. Please report results over multiple seeds with mean and standard deviation, and either perform significance testing or temper the wording.
  3. [§4.1 and §6.1] The paper acknowledges in Section 4.1 that HydraLoRA reported a similar asymmetric A/B pattern, yet HydraLoRA is not included in any experimental comparison. Since the main architectural novelty of BranchLoRA is precisely the asymmetric decomposition, omitting the closest asymmetric-LoRA baseline leaves the incremental contribution unquantified and makes the 'new SOTA' claim difficult to evaluate. Please add HydraLoRA as a baseline or justify its exclusion in the experimental setup.
  4. [Eq. (4) and §4.2.2] The automatic task selector is not specified well enough to be reproduced or audited. Equation (4) aligns trainable keys with image and text embeddings, but the paper does not define which embeddings are used (e.g., CLIP vision features, LLM text features, pooled token representations), the dimensionality and initialization of the keys, how the image and text similarities are combined into a single routing decision, or how the reported 95.8% routing accuracy was computed. Without these details, the key-based routing mechanism cannot be independently implemented, and the sensitivity of the final metrics to routing errors cannot be assessed.
minor comments (4)
  1. [§5.4] The paper should state how hyperparameters k=2, lambda=1.0, and the freezing criterion were chosen. If they were selected using CoIN test-set performance, a held-out validation split or a sensitivity analysis is needed to avoid selection on the test set.
  2. [Limitations and Abstract] The Limitations section honestly acknowledges that experiments use a single benchmark, but the abstract and Section 1 say 'extensive experiments' and 'various MLLM sizes' when only CoIN and two sizes are used; please calibrate these claims.
  3. [Figure 4 caption] The caption contains the typo 'Ony Frozen' and should read 'Only Frozen'.
  4. [Appendix D, Table 7] In the 10Type column for OCR-VQA, the listed templates appear to be the ScienceQA multiple-choice templates rather than free-form VQA templates; if this is not a formatting artifact, it should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central performance claims are independent empirical comparisons on the external CoIN benchmark; no equation in the paper reduces to its fitted targets.

full rationale

BranchLoRA's central claim is that its asymmetric LoRA variant, flexible tuning-freezing, task-specific routers, and auto-selector outperform MoELoRA on the CoIN benchmark (Tables 1 and 4). These ACC, MAA, and BWT numbers are measured against held-out test sets using the same metrics as the benchmark, and no quantity that is fit during training is reported as a prediction. The t-SNE observation of matrices A converging and B remaining distinct motivates the architecture but is an empirical analysis, not an input that forces the reported outcome. Self-citations appear only in related-work and method-motivation contexts (e.g., prior continual learning papers by the same authors) and are not load-bearing: the comparison against MoELoRA is made through the authors' own re-implementation and reported numbers, not through a uniqueness theorem or cited result. The auto-selector's 95.8% routing accuracy is a measured auxiliary result; its incompleteness (no per-task breakdown or oracle comparison) is a validation weakness and a robustness risk, not a circular reduction. No equation in Sections 4-6 is fitted to the headline metrics, so no self-definitional or fitted-input-as-prediction pattern is present. The central derivation chain is self-contained with respect to the external CoIN benchmark.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The main uncharged premises are the qualitative t-SNE observation, the reliability of learned keys for task routing, the stability of router activations for freezing, and the standard trainability of LoRA-MoE modules. Hyperparameters top-k, lambda, N, rank, and alpha are fixed choices with little to no sensitivity analysis.

free parameters (4)
  • top-k selection count k = 2
    Number of experts selected by the sparse router and frozen after each task; chosen by hand, no sensitivity analysis reported.
  • alignment loss coefficient lambda = 1.0
    Weight for key-sample alignment loss in Eq. 5; chosen by hand, no ablation over lambda.
  • number of experts N = 8
    Inherited from MoELoRA for fair comparison; still a design choice that affects capacity of the expert pool.
  • LoRA rank r and alpha = r=128, alpha=256
    Standard LoRA hyperparameters adopted from prior work and kept fixed across methods; not explored in this paper.
assumptions (4)
  • domain assumption The t-SNE visualization in Figure 1 reliably indicates that MoELoRA's A matrices converge across tasks while B matrices remain task-specific.
    The entire asymmetric design is motivated by this qualitative observation; no quantitative similarity metric or statistical test is provided.
  • domain assumption The learned task keys, aligned via cosine similarity in Eq. 4, generalize from training embeddings to test samples well enough to select the correct router.
    The no-task-ID inference claim depends on key-based routing; only a single average accuracy of 95.8% is reported without per-task breakdown or oracle comparison.
  • domain assumption Freezing the top-k most activated experts after each task preserves their task-specific knowledge while still allowing inter-task collaboration.
    The tuning-freezing mechanism assumes router activation statistics are a stable proxy for expert importance; no analysis of distribution shift is provided.
  • standard math Standard LoRA low-rank updates and softmax top-k routing are trainable with standard backpropagation.
    Implicitly assumed when finetuning the model; standard in the literature.
invented entities (1)
  • task-specific image and text keys
    purpose: Automatically route test samples to the correct task-specific router during inference without task identity.
    The keys are trained with the cosine alignment loss in Eq. 4 and evaluated only through the single 95.8% average selector accuracy; there is no external falsifiable handle outside this paper's benchmark.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enhancing Multimodal Continual Instruction Tuning with BranchLoRA." pith.science (2026). https://pith.science/paper/OKKNWPZK

@misc{pith2026250602041,
  author       = {Pith},
  title        = {Pith review of: Enhancing Multimodal Continual Instruction Tuning with BranchLoRA},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OKKNWPZK}},
  note         = {Machine review of arXiv:2506.02041}
}
read the original abstract

Multimodal Continual Instruction Tuning (MCIT) aims to finetune Multimodal Large Language Models (MLLMs) to continually align with human intent across sequential tasks. Existing approaches often rely on the Mixture-of-Experts (MoE) LoRA framework to preserve previous instruction alignments. However, these methods are prone to Catastrophic Forgetting (CF), as they aggregate all LoRA blocks via simple summation, which compromises performance over time. In this paper, we identify a critical parameter inefficiency in the MoELoRA framework within the MCIT context. Based on this insight, we propose BranchLoRA, an asymmetric framework to enhance both efficiency and performance. To mitigate CF, we introduce a flexible tuning-freezing mechanism within BranchLoRA, enabling branches to specialize in intra-task knowledge while fostering inter-task collaboration. Moreover, we incrementally incorporate task-specific routers to ensure an optimal branch distribution over time, rather than favoring the most recent task. To streamline inference, we introduce a task selector that automatically routes test inputs to the appropriate router without requiring task identity. Extensive experiments on the latest MCIT benchmark demonstrate that BranchLoRA significantly outperforms MoELoRA and maintains its superiority across various MLLM sizes.

Figures

Figures reproduced from arXiv: 2506.02041 by the authors.

Figure 1
Figure 1. MoELoRA parameter analysis during MCIT across 4 sequential tasks: matrices = layers × experts. human intent, enabling the creation of versatile models with general-purpose capabilities. In prac￾tical scenarios, MLLMs are often required to adapt to new instructions to support evolving function￾alities as knowledge and societal needs advance (Zheng et al., 2024a). However, current MLLMs remain static, limiting their a… view at source ↗
Figure 2
Figure 2. Diagram of MoELoRA and BranchLoRA. a shared router to modulate their contributions. However, experiments on the MCIT benchmark have revealed a critical limitation of MoELoRA related to parameter inefficiency. As visualized in [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The overall framework of the proposed BranchLoRA. The shared matrix [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Diagram of the Flexible Tuning-Freezing Mechanism. 3 distinct expert combinations for an input during new task ti training: (a) Only Tunable, (b) Tun￾able and Frozen, and (c) Ony Frozen, with the shared matrix A and task-specific Router ti being tunable. cally selects …
Figure 5
Figure 5. Figure 5: Task-wise performance comparison on the LLaVA-1.5-7B model. Our BranchLoRA method con￾sistently outperforms the previous MCIT baselines, LoRA and MoELoRA, in all task-wise evaluations. with MoELoRA, selecting the top-2 experts, with the loss coefficient λ set to 1.0. A…

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Text as Partial Constraint: Core-Residual Alignment for Robust Vision-Language Learning

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Aligning images to multi-view caption cores while suppressing orthogonal residual text and disagreement-aware temperature improves robust zero-shot recognition and LVLM transfer.

  2. Harmonious Parameter Adaptation in Continual Visual Instruction Tuning for Safety-Aligned MLLMs

    cs.CV 2025-11 conditional novelty 6.0 of 10

    HPA is a post-training parameter-selection method that keeps safety-aligned multimodal LLMs safe and reduces forgetting during continual visual instruction tuning.

  3. Continual Learning for Generative AI: From LLMs to MLLMs and Beyond

    cs.LG 2025-06 conditional novelty 4.0 of 10

    A survey that categorizes continual learning methods for generative models into architecture-based, regularization-based, and replay-based paradigms across four model families.

Reference graph

Works this paper leans on

59 extracted references · 41 canonical work pages · cited by 3 Pith papers

  1. [1]

    Anton Alexandrov, Veselin Raychev, Mark Mueller, Ce Zhang, Martin Vechev, and Kristina Toutanova. 2024. Mitigating Catastrophic Forgetting in Language Transfer via Model Merging . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 17167--17186

    work page 2024

  2. [2]

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A frontier large vision-language model with versatile abilities . arXiv preprint arXiv:2308.12966

    arXiv 2023

  3. [3]

    Cheng Chen, Junchen Zhu, Xu Luo, Heng Tao Shen, Jingkuan Song, and Lianli Gao. 2024. https://openreview.net/forum?id=EpnsUQavJA Co IN : A benchmark of continual instruction tuning for multimodel large language models . In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track

    work page 2024

  4. [4]

    Fei-Long Chen, Du-Zhen Zhang, Ming-Lun Han, Xiu-Yi Chen, Jing Shi, Shuang Xu, and Bo Xu. 2023 a . Vlp: A survey on vision-language pre-training . Machine Intelligence Research, 20(1):38--56

    work page 2023

  5. [5]

    Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. 2023 b . ShareGPT4V: Improving Large Multi-Modal Models with Better Captions . arXiv preprint arXiv:2311.12793

    arXiv 2023

  6. [6]

    Model Merging by Uncertainty-Based Gradient Matching

    Nico Daheim, Thomas M \"o llenhoff, Edoardo Ponti, Iryna Gurevych, and Mohammad Emtiyaz Khan. Model Merging by Uncertainty-Based Gradient Matching . In The Twelfth International Conference on Learning Representations

    work page

  7. [7]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. 2023. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning . In Thirty-seventh Conference on Neural Information Processing Systems

    work page 2023

  8. [8]

    Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ale s Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. 2021. A continual learning survey: Defying forgetting in classification tasks . IEEE transactions on pattern analysis and machine intelligence, 44(7):3366--3385

    work page 2021

Show all 59 references
  1. [9]

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database . In 2009 IEEE conference on computer vision and pattern recognition, pages 248--255. Ieee

    2009

  2. [10]

    Jiahua Dong, Hongliu Li, Yang Cong, Gan Sun, Yulun Zhang, and Luc Van Gool. 2024. https://doi.org/10.1109/TPAMI.2023.3334213 No One Left Behind: Real-World Federated Class-Incremental Learning . IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(4):2054--2070

    2024

  3. [11]

    Jiahua Dong, Lixu Wang, Zhen Fang, Gan Sun, Shichao Xu, Xiao Wang, and Qi Zhu. 2022. Federated Class-Incremental Learning . In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    2022

  4. [12]

    Jiahua Dong, Duzhen Zhang, Yang Cong, Wei Cong, Henghui Ding, and Dengxin Dai. 2023. Federated incremental semantic segmentation . In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3934--3943

    2023

  5. [13]

    Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Jun Zhao, Wei Shen, Yuhao Zhou, Zhiheng Xi, Xiao Wang, Xiaoran Fan, et al. 2023. Loramoe: Revolutionizing mixture of experts for maintaining world knowledge in language model alignment . arXiv preprint arXiv:2312.09979, 4(7)

    2023 arXiv

  6. [14]

    William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity . Journal of Machine Learning Research, 23(120):1--39

    2022

  7. [15]

    Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. 2013. An empirical investigation of catastrophic forgetting in gradient-based neural networks . arXiv preprint arXiv:1312.6211

    2013 arXiv

  8. [16]

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering . In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904--6913

    2017

  9. [17]

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. 2018. Vizwiz grand challenge: Answering visual questions from blind people . In Proceedings of the IEEE conference on computer vision and pattern recognition, pages ...

    2018

  10. [18]

    Jinghan He, Haiyun Guo, Ming Tang, and Jinqiao Wang. 2023. Continual instruction tuning for large multimodal models . arXiv preprint arXiv:2311.16206

    2023 arXiv

  11. [19]

    Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. LoRA: Low-Rank Adaptation of Large Language Models . In International Conference on Learning Representations

    2022

  12. [20]

    Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering . In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700--6709

    2019

  13. [21]

    Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannaneh Hajishirzi, Simon Kornblith, Ali Farhadi, and Ludwig Schmidt. 2022. Patching open-vocabulary models by interpolating weights . Advances in Neural Information Processing Systems, 35:29262--29277

    2022

  14. [22]

    Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. 2014. Referitgame: Referring to objects in photographs of natural scenes . In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 787--798

    2014

  15. [23]

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017. Overcoming catastrophic forgetting in neural networks . Proceedings of the national academy o...

    2017

  16. [24]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. 2023. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models . In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , page...

    2023

  17. [25]

    Zhizhong Li and Derek Hoiem. 2017. Learning without forgetting . IEEE transactions on pattern analysis and machine intelligence, 40(12):2935--2947

    2017

  18. [26]

    Zhong-Zhi Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jie Wang, Xiuyi Chen, et al. 2025. From system 1 to system 2: A survey of reasoning large language models . arXiv preprint arXiv:2502.17419

    2025 arXiv

  19. [27]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023 a . Improved Baselines with Visual Instruction Tuning . In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following

    2023

  20. [28]

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024. https://llava-vl.github.io/blog/2024-01-30-llava-next/ LLaVA-NeXT: Improved reasoning, OCR, and world knowledge

    2024

  21. [29]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023 b . Visual Instruction Tuning . In Thirty-seventh Conference on Neural Information Processing Systems

    2023

  22. [30]

    Qidong Liu, Xian Wu, Xiangyu Zhao, Yuanshao Zhu, Derong Xu, Feng Tian, and Yefeng Zheng. 2023 c . Moelora: An moe-based parameter efficient fine-tuning method for multi-task medical applications . arXiv preprint arXiv:2310.18339

    2023 arXiv

  23. [31]

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering . Advances in Neural Information Processing Systems, 35:...

    2022

  24. [32]

    Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. 2016. Generation and comprehension of unambiguous object descriptions . In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 11--20

    2016

  25. [33]

    Michael McCloskey and Neal J Cohen. 1989. Catastrophic interference in connectionist networks: The sequential learning problem . In Psychology of learning and motivation, volume 24, pages 109--165. Elsevier

    1989

  26. [34]

    Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. 2019. Ocr-vqa: Visual question answering by reading text in images . In 2019 international conference on document analysis and recognition (ICDAR), pages 947--952. IEEE

    2019

  27. [35]

    Dhananjay Ram, Aditya Rawal, Momchil Hardalov, Nikolaos Pappas, and Sheng Zha. 2024. DEM: Distribution Edited Model for Training with Mixed Data Distributions . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 19287--19301

    2024

  28. [36]

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards vqa models that can read . In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317--8326

    2019

  29. [37]

    Sainbayar Sukhbaatar, Olga Golovneva, Vasu Sharma, Hu Xu, Xi Victoria Lin, Baptiste Rozi \`e re, Jacob Kahn, Daniel Li, Wen-tau Yih, Jason Weston, et al. 2024. Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM . arXiv preprint arXiv:2403.07816

    2024 arXiv

  30. [38]

    Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Cheng-Zhong Xu. 2024. Hydralora: An asymmetric lora architecture for efficient fine-tuning . Advances in Neural Information Processing Systems, 37:9565--9584

    2024

  31. [39]

    Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of machine learning research, 9(11)

    2008

  32. [40]

    Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. 2022 a . Dualprompt: Complementary prompting for rehearsal-free continual learning . In European Conference on Computer Vision, pages 631--6...

    2022

  33. [41]

    Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. 2022 b . Learning to prompt for continual learning . In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages...

    2022

  34. [42]

    Tongtong Wu, Linhao Luo, Yuan-Fang Li, Shirui Pan, Thuy-Trang Vu, and Gholamreza Haffari. 2024. Continual Learning for Large Language Models: A Survey . arXiv preprint arXiv:2402.01364

    2024 arXiv

  35. [43]

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. 2023. Ties-merging: Resolving interference when merging models . Advances in Neural Information Processing Systems, 36:7093--7115

    2023

  36. [44]

    Yahan Yu, Duzhen Zhang, Xiuyi Chen, and Chenhui Chu. 2024. Flexible Weight Tuning and Weight Fusion Strategies for Continual Named Entity Recognition . In Findings of the Association for Computational Linguistics ACL 2024, pages 1351--1358

    2024

  37. [45]

    Yuexiang Zhai, Shengbang Tong, Xiao Li, Mu Cai, Qing Qu, Yong Jae Lee, and Yi Ma. 2023. Investigating the catastrophic forgetting in multimodal large language models . arXiv preprint arXiv:2309.10313

    2023 arXiv

  38. [46]

    Yuexiang Zhai, Shengbang Tong, Xiao Li, Mu Cai, Qing Qu, Yong Jae Lee, and Yi Ma. 2024. Investigating the catastrophic forgetting in multimodal large language model fine-tuning . In Conference on Parsimony and Learning, pages 202--227. PMLR

    2024

  39. [47]

    Duzhen Zhang, Wei Cong, Jiahua Dong, Yahan Yu, Xiuyi Chen, Yonggang Zhang, and Zhen Fang. 2023 a . Continual Named Entity Recognition without Catastrophic Forgetting . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8186--8197

    2023

  40. [48]

    Duzhen Zhang, Hongliu Li, Wei Cong, Rongtao Xu, Jiahua Dong, and Xiuyi Chen. 2023 b . Task relation distillation and prototypical pseudo label for incremental named entity recognition . In Proceedings of the 32nd ACM International Conference on Information and Knowledge Manage...

    2023

  41. [49]

    Duzhen Zhang, Yahan Yu, Chenxing Li, Jiahua Dong, Dan Su, Chenhui Chu, and Dong Yu. 2024 a . Mm-llms: Recent advances in multimodal large language models . In Findings of the Association for Computational Linguistics ACL 2024

    2024

  42. [50]

    Duzhen Zhang, Yahan Yu, Chenxing Li, Jiahua Dong, and Dong Yu. 2025. Federated Incremental Named Entity Recognition . IEEE Transactions on Audio, Speech and Language Processing

    2025

  43. [51]

    Hengyuan Zhang, Yanru Wu, Dawei Li, Zacc Yang, Rui Zhao, Yong Jiang, and Fei Tan. 2024 b . Balancing Speciality and Versatility: a Coarse to Fine Framework for Supervised Fine-tuning Large Language Model . arXiv e-prints, pages arXiv--2404

    2024

  44. [52]

    Xuanle Zhao, Xuexin Liu, Haoyue Yang, Xianzhen Luo, Fanhu Zeng, Jianling Li, Qi Shi, and Chi Chen. 2025 a . ChartEdit: How Far Are MLLMs From Automating Chart Analysis? Evaluating MLLMs' Capability via Chart Editing . arXiv preprint arXiv:2505.11935

    2025 arXiv

  45. [53]

    Xuanle Zhao, Xianzhen Luo, Qi Shi, Chi Chen, Shuo Wang, Wanxiang Che, Zhiyuan Liu, and Maosong Sun. 2025 b . ChartCoder: Advancing Multimodal Large Language Model for Chart-to-Code Generation . arXiv preprint arXiv:2501.06598

    2025 arXiv

  46. [54]

    Junhao Zheng, Qianli Ma, Zhen Liu, Binquan Wu, and Huawen Feng. 2024 a . Beyond Anti-Forgetting: Multimodal Continual Instruction Tuning with Positive Forward Transfer . arXiv preprint arXiv:2401.09181

    2024 arXiv

  47. [55]

    Junhao Zheng, Shengjie Qiu, Chengming Shi, and Qianli Ma. 2024 b . Towards Lifelong Learning of Large Language Models: A Survey . arXiv preprint arXiv:2406.06391

    2024 arXiv

  48. [56]

    Junhao Zheng, Chengming Shi, Xidi Cai, Qiuke Li, Duzhen Zhang, Chenxing Li, Dong Yu, and Qianli Ma. 2025. Lifelong Learning of Large Language Model based Agents: A Roadmap . arXiv preprint arXiv:2501.07278

    2025

  49. [57]

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models . arXiv preprint arXiv:2304.10592

    2023 arXiv

  50. [58]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  51. [59]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.