Pith. sign in

REVIEW 4 major objections 5 minor 40 references

This paper argues that splitting LoRA weight updates by their sign—keeping sign-aligned 'general' parameters and sign-conflicted 'specific' parameters separate—reduces negative interference in multimodal continual instruction tuning, and sh

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 08:38 UTC pith:U2INBSXN

load-bearing objection Solid SOTA MCIT paper with a real contribution, but the sign-decoupling mechanism is not yet isolated from magnitude pruning and regularization effects. the 4 major comments →

arxiv 2607.20511 v1 pith:U2INBSXN submitted 2026-07-05 cs.AI

SiGMA: Sign-Guided Merging and Adaptation for Multimodal Continual Instruction Tuning

Keonhee Park , Gunhee Kim This is my paper
classification cs.AI
keywords continual instruction tuningnegative interferenceLoRAsign-based weight decouplingmultimodal large language modelscatastrophic forgettingparameter-efficient fine-tuningweight merging
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper is trying to establish that negative interference—not just catastrophic forgetting—is the key obstacle when a multimodal model is instruction-tuned on a sequence of tasks, and that the sign of individual LoRA parameter updates provides a practical way to tell which parameters carry transferable, task-invariant knowledge and which carry task-specific knowledge that will collide during weight consolidation. It proposes a training-time step (sign-guided adaptive tuning) that pulls new task updates toward the sign-aligned 'general' subspace of prior LoRA weights, and an inference-time step (sign-guided merging) that selects and scales the sign-conflicted 'specific' parameters so they do not cancel each other out. If correct, the method offers a parameter-efficient way to keep old task performance high without auxiliary text encoders or Mixture-of-Experts routing, and it opens the possibility that task-boundary information can be encoded simply in the sign pattern of low-rank updates. The reported gains on two benchmarks—higher average accuracy across all previously seen tasks and lower forgetting than state-of-the-art baselines—are the concrete evidence the paper offers.

Core claim

The central claim is that negative interference in MCIT is a sign-conflict phenomenon: when two LoRA updates disagree in sign at a parameter, merging cancels magnitude and degrades performance; when they agree, the combination reinforces knowledge. SiGMA partitions LoRA weights into a general subspace (signs aligned with a reference) and a specific subspace (signs opposed), uses the general subspace during training to keep new updates near prior knowledge, and at inference merges only a magnitude-selected subset of the specific subspace, scaled by a cosine-distance factor. The paper reports the best average accuracy and lowest forgetting on both UCIT and DCL, with large gains on hard tasks s

What carries the argument

The core mechanism is a sign-based decoupling of LoRA updates. Element-wise sign comparison between a current update and a prior (or anchor) update yields two binary masks: the general mask marks parameters with matching signs, the specific mask marks parameters with opposing signs. These masks partition the parameter space, so summing the masked weights exactly reconstructs the original update. At training time, the sign-aligned slice of the aggregated previous LoRA update is added to the current update, acting as a soft regularizer that keeps the new task close to prior knowledge. At inference, each task's specific subspace is filtered by a magnitude threshold relative to the first task's

Load-bearing premise

The load-bearing assumption is that the sign of a LoRA parameter update, relative to prior updates, is a stable and reliable indicator of whether that parameter carries transferable general knowledge or conflicting task-specific knowledge; the paper supports this with empirical case studies, not a theoretical derivation.

What would settle it

Train two independent LoRA modules on a pair of tasks, then merge only the sign-conflicted parameters (no sign-aligned ones) exactly as in the paper's Figure 3 and measure performance on both tasks. If the resulting performance is not substantially worse than merging the full updates, or if randomly flipping signs of the 'specific' parameters produces the same behavior, then sign conflict is not the causal mechanism and the SiGMA gains would not transfer.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the sign-decoupling claim holds, continual instruction tuning can be improved without auxiliary models or task-specific routers; a simple overlap of LoRA updates is enough to decide what to share and what to protect.
  • Because the method explicitly reduces negative interference, its benefit should grow as the number of tasks increases, since sign-conflicted subspaces are filtered and scaled instead of simply summed.
  • Since the mechanism operates on LoRA matrices, it should apply to any model using low-rank adaptation, not just vision-language architectures.
  • The reported VizWiz gain (+4.85% over the second-best method) suggests the approach is most valuable on tasks that are otherwise heavily degraded by consolidation.
  • The paper's layer-depth analysis shows the top-1 output layer should be excluded from sign-guided tuning, indicating the method's gains depend on not injecting invariant knowledge at the final prediction head.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the paper's reasoning is right, the same sign-split logic could be tested outside LoRA: applying it to full-parameter fine-tuning or to adapters and prompts should produce similar interference reduction, providing a way to check how general the sign mechanism really is.
  • Because the first task serves as the fixed anchor, the method's robustness may depend on that task being representative; using an averaged anchor across several initial tasks could be more stable, but that is a possible extension the paper does not explore.
  • The sign-decoupling view implies that interference could be predicted before merging by computing the fraction of sign-conflicted parameters between two tasks, which could guide task ordering or routing decisions.
  • The paper's observation that sign-aligned parameters can exceed individually trained performance hints that this merging acts as a form of positive transfer; testing whether this holds across other dataset pairs would clarify the boundary between interference mitigation and beneficial knowledge sharing.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes SiGMA, a framework for multimodal continual instruction tuning that uses the sign of LoRA parameter updates to decouple weights into 'general' (sign-aligned) and 'specific' (sign-conflicted) subspaces. During training, sign-guided adaptive tuning (S-Tuning) injects the sign-aligned portion of aggregated prior LoRA weights into the forward pass to reduce drift; at inference, sign-guided merging (S-Merge) masks and amplifies salient sign-conflicted parameters relative to a first-task anchor before consolidating LoRAs. The method is evaluated on the UCIT and DCL benchmarks with a LLaVA-v1.5-7B backbone, reporting improved Avg. All and Forgets over existing MoE and expansion–merge baselines, with ablations on components, layer depth, and the temperature λ. The paper also includes qualitative examples and a small case study in Figure 3 to motivate the sign-based decomposition.

Significance. If the sign-guided mechanism is genuinely responsible for the reported gains, the paper makes a useful empirical contribution: it addresses negative interference at inference time, a real limitation of expansion–merge continual learning, and it does so without auxiliary text encoders or architectural overhead beyond masking and scaling. The code release and evaluations on two standard benchmarks are strengths, as is the ablation coverage of the individual components. However, the central attribution to the sign signal is not yet established: the sign mask is never ablated against non-sign baselines, no variance or significance information is reported, and key hyperparameters are selected on the same benchmark used for the headline numbers. These issues are load-bearing because the paper's mechanism, title, and contributions all rest on sign-based decoupling being the cause of the improvement.

major comments (4)
  1. [§3.5, Eq. (11), Table 3b] The sign mask is never ablated. In S-Merge, Eq. (11) combines a sign-conflict mask with a magnitude threshold λ|ΔW_1|, but Table 3b only removes the cosine-distance scaling d_k while keeping both the sign condition and magnitude selection. In S-Tuning, Eq. (8)-(9) applies a sign-alignment mask, yet there is no comparison to a magnitude-based mask or a random mask of equal sparsity. Consequently, the observed gains could be due to magnitude-based pruning or soft regularization rather than to sign agreement/conflict. Please add ablations that isolate the sign signal: (a) S-Merge with M^s_k replaced by a top-magnitude mask without the sign condition; (b) S-Tuning with the sign-alignment mask replaced by a magnitude or random mask of the same sparsity; (c) S-Merge using the sign mask alone without the magnitude threshold. This is essential to support the paper's central claim.
  2. [§3.5, Eq. (10)-(12)] The signature step d_k is not well defined. The text says d_k = δ(ΔW^s_1, ΔW^s_k), but ΔW^s_1 is never defined: if M^s_1 is computed by Eq. (6) using ΔW_1 as both old and new, then M^s_1 = 0 and ΔW^s_1 = 0, making the cosine distance undefined. Also, the claim that d_k ≥ 1 'empirically' is not supported by any numbers or analysis. Please define δ and ΔW^s_1 explicitly, and report the observed distribution of d_k across tasks. Without this, the amplification step in Eq. (12) is not reproducible.
  3. [§4.1, Table 1] No standard deviations or error bars are reported anywhere. The abstract and Section 4.2 state that SiGMA 'significantly reduces negative interference,' but all numbers appear to come from a single run. Several comparisons are close (e.g., DCL Avg. All: 62.30 vs 60.42; Table 3b λ=0.8 vs λ=0.9 differ by 0.18 in Avg. All). Please report results over multiple seeds or another valid variance estimate, and avoid the word 'significantly' unless a statistical test is performed.
  4. [§4.3, Tables 3a and 3b] The hyperparameters λ=0.8 and k=1 (top layers excluded from S-Tuning) are selected by scanning on the UCIT benchmark and then used to produce the UCIT headline results in Table 1. If no separate validation split is used, then part of the reported UCIT improvement may reflect tuning to the test set. Please clarify the selection procedure, or evaluate the chosen settings on a held-out validation split and report the main results with the validation-selected hyperparameters.
minor comments (5)
  1. [Eq. (3)] The definition of Rconf uses index i on the left side and j on the right; this should be consistent (both should be the same parameter index, e.g., j).
  2. [Throughout] Typos and inconsistent naming: 'ArixivQA' vs 'ArxivQA', 'Hide-LLaVA' vs 'HiDe-LLaVA'/'HiDE-LLaVA', 'behcnmarks', 'Adapative Tuning', 'consolidated representaion'. Please copyedit.
  3. [Figure 3] The caption says 'Walig represents merging only sign-aligned weights' and 'Wconf represents merging only sign-conflicted weights,' but it is not stated which pair of tasks the signs are compared against. Please clarify the experimental setup, including how the three separately trained LoRA modules are paired for sign comparison.
  4. [Table 3b] The λ=0 row reports Avg. All 54.32, Avg. Each 70.69, and Forgets -19.65. It is surprising that Avg. Each is highest while Forgets is catastrophic; please explain what λ=0 means mechanically and why it produces this pattern.
  5. [Eq. (7)] The text says 'consolidating A and B is more effective than consolidating full LoRA weights Δ when SWD is used,' but the final model still stores aggregated A and B matrices. Please clarify how the decomposition in Eq. (10) is applied to the merged low-rank factors, since sign masks are defined on the full product ΔW rather than on A or B separately.

Circularity Check

0 steps flagged

No significant circularity: SiGMA's sign-based components are evaluated against external benchmarks with independent baselines; no prediction reduces to its inputs by construction.

full rationale

The paper's derivation chain is empirical rather than definitional: (i) it observes that sign-conflicted parameters cancel under simple addition (Eq. 4-5, an arithmetic identity, not a fitted prediction); (ii) it validates this with the Figure 3 merging study; (iii) it builds S-Tuning (Eq. 8-9) and S-Merge (Eq. 10-12) from sign masks; and (iv) it measures the resulting method on the UCIT and DCL benchmarks against external baselines. No load-bearing self-citation is present, and no equation or fitted hyperparameter is defined in terms of the headline metric such that the result is forced by construction. The main weaknesses are validity concerns rather than circularity: the sign mask is never ablated against a magnitude-only or random mask of equal sparsity, so the causal attribution to sign conflict is under-identified; and lambda and the top-k layer exclusion are tuned on the same UCIT benchmark used for the headline result, which is a test-set-selection caveat. The paper's own limitation about the general subspace becoming sparse is likewise a robustness concern, not a circular step.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The method introduces no new physical or parametric entities; it introduces a new way to partition existing LoRA parameters (general vs specific subspaces) and rules for selecting/scaling them. The main assumptions are empirical: sign agreement is a stable proxy for transferable knowledge, and the first task works as an anchor. The free parameters are standard engineering choices, except λ and k which are tuned on the evaluation benchmark (mild post-hoc selection).

free parameters (4)
  • λ (temperature factor for anchor weights) = 0.8
    Selected by ablation on UCIT (Table 3b). The paper states 'we set λ=0.8 because it best reflects overall knowledge retention', which is a hand-tuned choice on the evaluation benchmark, not derived from first principles.
  • k (number of top layers excluded from S-Tuning) = 1
    Selected by ablation on UCIT (Table 3a): excluding the top-1 layer gives the best Avg. All. This is a post-hoc architectural choice tuned on the same benchmark.
  • LoRA rank r = 16
    Set to 16 'following the LoRA fine-tuning setup from LLaVA'. This is a hyperparameter carried from a prior setup, not derived in the paper.
  • batch size = 32 (UCIT), 8 (DCL)
    Stated in implementation details; chosen by the authors, not justified by analysis.
axioms (4)
  • domain assumption Negative interference is well characterized by parameter sign conflicts: merging opposite-sign parameters cancels magnitudes and degrades performance (Eq. 3-5).
    Core premise of the method, motivated by Figure 3's case study. The paper does not prove this; it assumes sign opposition is the dominant mechanism for negative interference.
  • domain assumption The sign of LoRA weights is meaningful relative to magnitude; using sign comparisons across different tasks with arbitrary scales is a valid similarity/conflict measure.
    Invoked in Eq. 3, 6, 9, 11. LoRA updates can have arbitrary scaling; the paper does not analyze scale invariance or normalization of updates before sign comparison.
  • ad hoc to paper The first task's LoRA weights provide a stable anchor for defining general/specific subspaces across all tasks.
    Stated in Section 3.5: 'we use the LoRA weights from the first task ΔW1 as the anchor weights'. This is a design choice; its validity for arbitrary task orderings is untested.
  • domain assumption LLaVA-v1.5-7B is a representative backbone and results on UCIT/DCL generalize to other MLLMs and task sequences.
    All experiments use one backbone. The method's sign-based claims are presented as general but are only demonstrated on this model.

pith-pipeline@v1.3.0-alltime-deepseek · 12623 in / 8081 out tokens · 67306 ms · 2026-08-02T08:38:24.504203+00:00 · methodology

0 comments
read the original abstract

Multimodal Continual Instruction Tuning (MCIT) is crucial for adapting Multimodal Large Language Models (MLLMs) to evolving a sequence of downstream tasks. Prior methods mostly utilize Mixture of Experts or expansion merge approach, primarily focusing on catastrophic forgetting, yet they still suffer from negative interference during inference, where newly learned updates overwrite useful prior knowledge and degrade overall performance. To address this, we propose SiGMA (Sign Guided Merging and Adaptation), a simple yet effective framework that mitigates negative interference with two components: sign guided adaptive tuning during training and sign guided merging at inference. Sign guided adaptive tuning reduces collisions with past knowledge and learns the current task with minimal drift, mitigating severe forgetting. Sign guided merging further improves consolidation by selectively scaling salient parameters to preserve and amplify useful task specific knowledge. Experiments on UCIT and DCL benchmarks show that SiGMA significantly reduces negative interference and outperforms state of the art MCIT methods. Our code is available at SiGMA.

Figures

Figures reproduced from arXiv: 2607.20511 by Gunhee Kim, Keonhee Park.

Figure 1
Figure 1. Figure 1: Comparison of overall performance on UCIT benchmark. We report a holistic metric over the continual instruction tuning process, computed as the average accuracy on all previously seen tasks after training each task. While existing MoE and expan￾sion–merge approaches gradually suffer from performance degradation due to negative interference and severe forgetting, SiGMA, which explicitly addresses negative i… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of SiGMA. During training, sign-guided adaptive tuning (left) de￾composes prior weights into general and task-specific subspaces to convey transferable knowledge. During inference, sign-guided LoRA merging (right) consolidates weights by selecting and scaling salient task-specific parameters, preserving crucial knowledge. CoIN [5] introduces an MCIT benchmark of eight datasets and proposes MoE￾LoR… view at source ↗
Figure 3
Figure 3. Figure 3: Impact of sign-based subspaces on continual learning performance. We train separate LoRA modules for ImageNet-R, ArxivQA, and CLEVR to preserve task￾specific knowledge, and then merge them at inference. Walig represents merging only sign-aligned weights, while Wconf represents merging only sign-conflicted weights. In the sign-aligned subspace, weights share the same sign, so their magnitudes tend to accumu… view at source ↗
Figure 4
Figure 4. Figure 4: Performance variation under the UCIT benchmark. We report changes in single￾task performance (e.g., ArxivQA and VizWiz) as the model sequentially learns new tasks. “Individual” denotes the performance achieved when the model is trained only on the corresponding task. As shown in Table 3b, increasing the temperature λ improves overall performance up to 0.8. Although the best Avg. Each and Forgets report at … view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative examples from ImageNet-R (left) and ArxivQA (right) on the UCIT benchmark. Given the same image and instruction, we compare responses from CL￾MoE, HiDe-LLaVA, and SiGMA, shown both after task-specific training and after sequential training on all tasks. Red text denotes the task-specific instruction format that the model must follow to generate a response. LLaVA. All methods follow instructions… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

40 extracted references · 9 linked inside Pith

  1. [1]

    arXiv preprint arXiv:2303.08774 (2023)

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    Pith/arXiv arXiv 2023

  2. [2]

    Advances in neural information processing systems32(2019)

    Aljundi, R., Belilovsky, E., Tuytelaars, T., Charlin, L., Caccia, M., Lin, M., Page- Caccia, L.: Online continual learning with maximal interfered retrieval. Advances in neural information processing systems32(2019)

    2019

  3. [3]

    arXiv preprint arXiv:2308.12966 (2023)

    Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., Zhou, J.: Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966 (2023)

    Pith/arXiv arXiv 2023

  4. [4]

    arXiv preprint arXiv:2211.08545 (2022)

    Chang, S., Palzer, D., Li, J., Fosler-Lussier, E., Xiao, N.: Mapqa: A dataset for question answering on choropleth maps. arXiv preprint arXiv:2211.08545 (2022)

    Pith/arXiv arXiv 2022

  5. [5]

    Advances in Neural Information Processing Systems37, 57817–57840 (2024)

    Chen, C., Zhu, J., Luo, X., Shen, H.T., Song, J., Gao, L.: Coin: A benchmark of continual instruction tuning for multimodel large language models. Advances in Neural Information Processing Systems37, 57817–57840 (2024)

    2024

  6. [6]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al.: Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 24185–24198 (2024)

    2024

  7. [7]

    Advances in neural information processing systems36, 49250–49267 (2023)

    Dai, W., Li, J., Li, D., Tiong, A., Zhao, J., Wang, W., Li, B., Fung, P.N., Hoi, S.: Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances in neural information processing systems36, 49250–49267 (2023)

    2023

  8. [8]

    In: Che, W., Nabende, J., Shutova, E., Pilehvar, M.T

    Guo, H., Zeng, F., Xiang, Z., Zhu, F., Wang, D.H., Zhang, X.Y., Liu, C.L.: HiDe- LLaVA: Hierarchical decoupling for continual instruction tuning of multimodal large language model. In: Che, W., Nabende, J., Shutova, E., Pilehvar, M.T. (eds.) Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp...

    doi:10.18653/v1/ 2025

  9. [9]

    In: Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision

    Guo, H., Zeng, F., Zhu, F., Liu, W., Wang, D.H., Xu, J., Zhang, X.Y., Liu, C.L.: Federated continual instruction tuning. In: Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision. pp. 1325–1335 (2025)

    2025

  10. [10]

    In: Findings of the Association for Computational Lin- guistics: ACL 2025

    Guo, Z., Zhang, R., Chen, H., Gao, J., Jiang, D., Wang, J., Heng, P.A.: Sciverse: Unveiling the knowledge comprehension and visual reasoning of lmms on multi- modal scientific problems. In: Findings of the Association for Computational Lin- guistics: ACL 2025. pp. 19683–19704 (2025) 16 Keonhee park et al

    2025

  11. [11]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Gurari, D., Li, Q., Stangl, A.J., Guo, A., Lin, C., Grauman, K., Luo, J., Bigham, J.P.: Vizwiz grand challenge: Answering visual questions from blind people. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3608–3617 (2018)

    2018

  12. [12]

    arXiv preprint arXiv:2003.10286 (2020)

    He, X., Zhang, Y., Mou, L., Xing, E., Xie, P.: Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003.10286 (2020)

    Pith/arXiv arXiv 2003

  13. [13]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., et al.: The many faces of robustness: A critical analysis of out-of-distribution generalization. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 8340–8349 (2021)

    2021

  14. [14]

    In: International Con- ference on Learning Representations (2022),https://openreview.net/forum?id= nZeVKeeFYf9

    Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: LoRA: Low-rank adaptation of large language models. In: International Con- ference on Learning Representations (2022),https://openreview.net/forum?id= nZeVKeeFYf9

    2022

  15. [15]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Huai, T., Zhou, J., Wu, X., Chen, Q., Bai, Q., Zhou, Z., He, L.: Cl-moe: Enhanc- ing multimodal large language model with dual momentum mixture-of-experts for continual visual question answering. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 19608–19617 (2025)

    2025

  16. [16]

    Neural computation3(1), 79–87 (1991)

    Jacobs, R.A., Jordan, M.I., Nowlan, S.J., Hinton, G.E.: Adaptive mixtures of local experts. Neural computation3(1), 79–87 (1991)

    1991

  17. [17]

    ArXivabs/2310.06825(2023),https://api

    Jiang, A.Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D.S., de Las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L.R., Lachaux, M.A., Stock, P., Scao, T.L., Lavril, T., Wang, T., Lacroix, T., Sayed, W.E.: Mistral 7b. ArXivabs/2310.06825(2023),https://api. semanticscholar.org/CorpusID:263830494

    Pith/arXiv arXiv 2023

  18. [18]

    In: European conference on computer vision

    Kembhavi, A., Salvato, M., Kolve, E., Seo, M., Hajishirzi, H., Farhadi, A.: A di- agram is worth a dozen images. In: European conference on computer vision. pp. 235–251. Springer (2016)

    2016

  19. [19]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern recognition

    Kembhavi, A., Seo, M., Schwenk, D., Choi, J., Farhadi, A., Hajishirzi, H.: Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. In: Proceedings of the IEEE Conference on Computer Vision and Pattern recognition. pp. 4999–5007 (2017)

    2017

  20. [20]

    In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    Li, L., Wang, Y., Xu, R., Wang, P., Feng, X., Kong, L., Liu, Q.: Multimodal arxiv: A dataset for improving scientific comprehension of large vision-language models. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 14369–14387 (2024)

    2024

  21. [21]

    In: proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Li, Z., Yang, B., Liu, Q., Ma, Z., Zhang, S., Yang, J., Sun, Y., Liu, Y., Bai, X.: Monkey: Image resolution and text label are important things for large multi- modal models. In: proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 26763–26773 (2024)

    2024

  22. [22]

    Lindström, A., Abraham, S.: Clevr-math: A dataset for compositional language, visualandmathematicalreasoning.In:Proceedingsofthe16thInternationalWork- shop on Neural-Symbolic Learning and Reasoning. vol. 3212, pp. 155–170 (2022)

    2022

  23. [23]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Liu, H., Li, C., Li, Y., Lee, Y.J.: Improved baselines with visual instruction tun- ing. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 26296–26306 (2024)

    2024

  24. [24]

    Advances in neural information processing systems36, 34892–34916 (2023)

    Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. Advances in neural information processing systems36, 34892–34916 (2023)

    2023

  25. [25]

    IEEE Transactions on Geoscience and Remote Sensing 58(12), 8555–8566 (2020) SiGMA 17

    Lobry, S., Marcos, D., Murray, J., Tuia, D.: Rsvqa: Visual question answering for remote sensing data. IEEE Transactions on Geoscience and Remote Sensing 58(12), 8555–8566 (2020) SiGMA 17

    2020

  26. [26]

    In: The Twelfth International Conference on Learning Representations (2024)

    Lu, P., Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng, H., Chang, K.W., Galley, M., Gao, J.: Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In: The Twelfth International Conference on Learning Representations (2024)

    2024

  27. [27]

    arXiv preprint arXiv:2110.13214 (2021)

    Lu, P., Qiu, L., Chen, J., Xia, T., Zhao, Y., Zhang, W., Yu, Z., Liang, X., Zhu, S.C.: Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. arXiv preprint arXiv:2110.13214 (2021)

    Pith/arXiv arXiv 2021

  28. [28]

    McCloskey, M., Cohen, N.J.: Catastrophic interference in connectionist networks: Thesequentiallearningproblem.In:Psychologyoflearningandmotivation,vol.24, pp. 109–165. Elsevier (1989)

    1989

  29. [29]

    In: Proceedings of the IEEE international conference on computer vision

    Plummer, B.A., Wang, L., Cervantes, C.M., Caicedo, J.C., Hockenmaier, J., Lazeb- nik, S.: Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In: Proceedings of the IEEE international conference on computer vision. pp. 2641–2649 (2015)

    2015

  30. [30]

    In: International conference on machine learning

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PmLR (2021)

    2021

  31. [31]

    In: International Conference on Learning Representations (2017)

    Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., Dean, J.: Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In: International Conference on Learning Representations (2017)

    2017

  32. [32]

    In: European conference on computer vision

    Sima, C., Renz, K., Chitta, K., Chen, L., Zhang, H., Xie, C., Beißwenger, J., Luo, P., Geiger, A., Li, H.: Drivelm: Driving with graph visual question answering. In: European conference on computer vision. pp. 256–274. Springer (2024)

    2024

  33. [33]

    Tang, B., Boggust, A., Satyanarayan, A.: Vistext: A benchmark for semantically richchartcaptioning.In:Proceedingsofthe61stAnnualMeetingoftheAssociation for Computational Linguistics (Volume 1: Long Papers). pp. 7268–7298 (2023)

    2023

  34. [34]

    arXiv preprint arXiv:2307.09288 (2023)

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bash- lykov, N., Batra, S., Bhargava, P., Bhosale, S., et al.: Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)

    Pith/arXiv arXiv 2023

  35. [35]

    In: Findings of the Association for Computational Linguistics: EMNLP 2023

    Wang, X., Chen, T., Ge, Q., Xia, H., Bao, R., Zheng, R., Zhang, Q., Gui, T., Huang, X.J.: Orthogonal subspace learning for language model continual learning. In: Findings of the Association for Computational Linguistics: EMNLP 2023. pp. 10658–10671 (2023)

    2023

  36. [36]

    arXiv preprint arXiv:2308.01430 (2023)

    Wang, Z., Li, Y., Wu, J., Soon, J., Zhang, X.: Finvis-gpt: A multimodal large language model for financial chart analysis. arXiv preprint arXiv:2308.01430 (2023)

    Pith/arXiv arXiv 2023

  37. [37]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Wang, Z., Zhang, Z., Lee, C.Y., Zhang, H., Sun, R., Ren, X., Su, G., Perot, V., Dy, J., Pfister, T.: Learning to prompt for continual learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 139–149 (2022)

    2022

  38. [38]

    In: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing

    Zeng, F., Zhu, F., Guo, H., Zhang, X.Y., Liu, C.L.: Modalprompt: Towards efficient multimodal continual instruction tuning with dual-modality guided prompt. In: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. pp. 12137–12152 (2025)

    2025

  39. [39]

    arXiv preprint arXiv:2506.05453 (2025)

    Zhao, H., Zhu, F., Guo, H., Wang, M., Wang, R., Meng, G., Zhang, Z.: Mllm- cl: Continual learning for multimodal large language models. arXiv preprint arXiv:2506.05453 (2025)

    arXiv 2025

  40. [40]

    arXiv preprint arXiv:2304.10592 (2023)

    Zhu, D., Chen, J., Shen, X., Li, X., Elhoseiny, M.: Minigpt-4: Enhancing vision- language understanding with advanced large language models. arXiv preprint arXiv:2304.10592 (2023)

    Pith/arXiv arXiv 2023