Pith. sign in

REVIEW 5 major objections 5 minor 64 references

Contrastive Regularization over LoRA for Multimodal Biomedical Image Incremental Learning

T0 review · 5 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Contrastive LoRA branches beat separate models on biomedical imaging

desk verdict A plausible, code-released method for modality-incremental learning on medical LVLMs, but the headline result rests on a single favorable task order and no error bars; worth a serious referee, not yet a citable result. read the letter →

arxiv 2508.11673 v1 pith:BYU7DYZO submitted 2025-08-08 cs.LG cs.AIcs.CVcs.MM

Haojie Zhang , Yixiong Liang , Hulin Kuang , Lihui Cen , Zhe Qu , Yigang Cen , Min Zeng , Shichao Kan This is my paper · ORCID
classification cs.LGcs.AIcs.CVcs.MM
keywords ContinualLearningLow-RankAdaptation(LoRA)ContrastiveRegularizationMultimodalBiomedicalImageIncrementalLargeVision-LanguageModelCatastrophicForgettingModality-SpecificLoRAOrthogonalityConstraint
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a single large vision-language model can keep learning new biomedical imaging modalities and tasks without forgetting, by freezing the base model and adding one low-rank LoRA branch per task. The new ingredient is a contrastive regularizer that pulls a new branch's parameters toward branches of the same modality and pushes them away from branches of other modalities, so knowledge is shared within a modality and interference between modalities is reduced. An orthogonality constraint on each branch's low-rank factors reinforces this separation. If the claims hold, a unified model can replace the usual practice of training a separate model per modality, cutting inference cost while actually improving overall accuracy. The headline evidence is an overall score of 990.25 versus 970.29 for separately fine-tuned models, a 1.88% gain.

What carries the argument

MSLoRA-CR: a set of task-specific LoRA branches attached to a frozen LVLM, each branch written as a low-rank pair $A_i, B_i$, merged at inference through a task-selection mask. The carrying mechanism is the contrastive regularization loss, which defines similarity between parameter matrices by $\mathrm{sim}(P,Q) = e^{-\mathrm{dis}(P,Q)}$ with Manhattan distance and applies a convergence loss to same-modality pairs and a divergence loss to cross-modality pairs; the orthogonality constraint $A^\top A = B B^\top = I$ decorrelates the factors. Proposition 1, the gradient-equivalence argument, is the theoretical load-bearing piece that connects branch expansion to plasticity and stability.

What would settle it

Re-run the six-task comparison under the orders listed in Table 6 (or a random sample of orders) for both MSLoRA-CR and separately fine-tuned per-task models, and compare the distribution of SUM scores; if the median or mean advantage is not positive across orders, the reported 1.88% improvement is an artifact of a favorable order. A single decisive signal: the radiology-first order B1,B2,B3,A1,A2,A3 already drops to 682.25, below the best order by 78.58.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that modality-specific LoRA expansion, MSLoRA, achieves a balance between plasticity and stability: Proposition 1 states that when learning task t, updating the new LoRA delta equals updating the shared pretrained weight W directly and is independent of previous deltas, so new tasks do not build on or distort old task parameters. The contrastive regularization loss then structures the growing set of LoRA branches so that same-modality branches converge and cross-modality branches diverge, and the orthogonality loss $A_i^\top A_i = B_i B_i^\top = I$ removes redundant capacity. Together these let the frozen biomedical LVLM outperform both separate

Load-bearing premise

The headline result assumes the specific training order used in the main experiments is representative; the paper's own order study shows totals swinging from 610.40 to 760.83 on a six-task subset, so if the gain over separate models disappears under a fair sample of orders, the main empirical claim fails.

Editorial extensions

If this is right

  • If correct, deployment of multi-modal biomedical assistants no longer requires one checkpoint per modality; a single frozen LVLM with mask-selected LoRA branches covers all tasks.
  • The method's gain rides on the foundation model's existing knowledge; weak zero-shot baselines improve only as far as the base LVLM can support, so the ceiling is tied to the base model.
  • Task order matters: the paper's Table 6 shows pathology-first, VQA-first curricula give the best totals, while radiology-first or report-generation-first orders drop sharply.
  • Orthogonality adds a few points (995.21 vs 990.25) and appears to help most on report generation (BLEU 14.12 vs 5.83), suggesting it mainly cleans up low-rank redundancy.
  • Because branches are merged by mask at inference, switching tasks is as cheap as changing the mask rather than loading a new model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Order sensitivity at the scale shown in Table 6 suggests the practical recipe must include curriculum choice; a natural extension would be to test whether the contrastive loss can be made order-robust, e.g., by asymmetric or temperature-scaled regularization.
  • The same 'pull same-domain, push cross-domain' logic applies to any grouping of continual tasks, not just biomedical modalities; natural-image domains such as sketches, clipart, and photos are an immediate test bed.
  • Proposition 1's equivalence is a gradient identity, not a guarantee about loss-landscape dynamics; whether the balance holds in practice depends on the optimizer, learning rate, and the scale of the CR and ortho terms, so the theoretical result is motivation rather than a full proof of no forgetting.
  • Because the regularization acts only on parameter matrices, it is agnostic to output format, so extending the same module to detection and segmentation tasks is plausible provided the LVLM can emit those outputs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces Multimodal Biomedical Image Incremental Learning (MBIIL), where a frozen large vision-language model is augmented with new LoRA branches for each incoming task, and an inference-time mask merges the task-specific branches. The proposed MSLoRA-CR adds a contrastive regularization loss that pulls the current task's LoRA parameters toward those of previous tasks in the same modality and pushes them away from those in different modalities, plus an orthogonality penalty on the LoRA factors. The authors report experiments on nine biomedical datasets spanning pathology, radiology, and dermatology, with VQA, classification, and report generation tasks. Their headline result is that MSLoRA-CR reaches an aggregate SUM of 990.25 (995.21 with the orthogonality term) versus 970.29 for a separate-model baseline, which they interpret as a 1.88% improvement. They also provide Proposition 1 claiming that fine-tuning the t-th LoRA delta is equivalent to fine-tuning the pretrained weight W and is independent of earlier LoRA deltas, which they use to argue for a balance between stability and plasticity.

Significance. If the empirical claim is robust, the paper addresses a useful and under-studied setting: incrementally adding biomedical imaging modalities/tasks to a single LVLM while avoiding separate models per task. The idea of using contrastive regularization over LoRA parameters based on modality membership is simple and plausible, and the public code release is a practical strength. However, the current evidence does not establish the claimed superiority. The main evaluation is a single run on one task order, the aggregate SUM metric adds heterogeneous scales without justification, and the paper's own task-order ablation shows very large performance swings. The theoretical stability-plasticity argument is also not rigorously proven as stated. These issues are fixable with additional experiments and a more careful analysis, so the underlying direction is worth pursuing.

major comments (5)
  1. [Section 4.3, Table 6] The central empirical claim (Table 2: MSLoRA-CR 990.25 vs. Data-Specific FT 970.29) is based on a single task order and no repeated runs or error bars. Table 6 shows that on a six-dataset subset, changing only the task order changes SUM from 610.40 to 760.83, a swing of over 150 points, and the best order is arguably the one used in the main evaluation. A different but equally reasonable order could erase or reverse the reported ~20-point advantage. Please report order-averaged results over multiple permutations, at least on the six-task subset, and include variance over at least three seeds.
  2. [Section 4.1, Evaluation Metrics] The aggregate 'SUM' adds accuracy, recall, F1, AUC, and BLEU scores directly. These are not commensurable metrics with comparable scales or distributions; summing them imposes an arbitrary equal weighting and makes the 1.88% improvement difficult to interpret. The paper should either justify the aggregation, use a normalized/rank-based aggregate, or present the per-dataset comparisons as the primary evidence. The per-task tables are useful, but the headline claim depends on the SUM definition.
  3. [Section 3.3, Proposition 1 and Eqs. (2)-(4)] The proof that fine-tuning ΔW_t is equivalent to fine-tuning W treats the layer input h as fixed. In a deep LVLM, h for a given layer depends on both W and all previous LoRA modules in earlier layers, so the gradient of the full loss w.r.t. W is not generally equal to the gradient w.r.t. ΔW_t. Moreover, ΔW_t = A_tB_t is low-rank while W is not, so even a local gradient equality does not imply equivalent updates. Thus the claimed 'theoretically proven balance between plasticity and stability' is not established by the argument given.
  4. [Section 3.4, Eq. (9)] There is a dimensional inconsistency in the orthogonality loss. If A_i ∈ R^{d_o×r} and B_i ∈ R^{r×d_i}, then A_i^T A_i is r×r and B_i^T B_i is d_i×d_i. The term ∥B_i^T B_i − I∥_F^2 cannot drive a rank-r matrix to a d_i-dimensional identity. The text correctly states the desired constraints as A_i^T A_i = I and B_iB_i^T = I, so Eq. (9) likely has a typo (B_i^T B_i should be B_iB_i^T). Since the ortho variant gives the best reported SUM, this needs correction and the experiments should be rerun or confirmed.
  5. [Section 4.3, hyperparameter selection] The hyperparameters α and β were selected by 'trained for several steps to observe the scale differences' and picking values that yielded the best performance on the same six tasks used later in the evaluation (Table 4). This is a form of test-set selection and makes the reported improvements optimistic. Please use a held-out validation split for hyperparameter selection or report results as a sensitivity study without claiming the selected configuration is unbiased.
minor comments (5)
  1. [Section 4.2, Table 2] The text calls the baseline 'LoRA-Each' but the table and caption use 'Data-Specific FT'; please reconcile the terminology and define what '⊤' marks.
  2. [Section 4.3, Table 6] The order labels (A1,A2,A3,B1,B2,B3 and A1,B1,A2,B2,A3,B3) are confusing: the first row uses all A's before all B's, while the last row interleaves them. A small schematic or legend would make the order effects easier to read.
  3. [Section 3.4, Eq. (5)] The similarity sim(P,Q) = exp(-dis(P,Q)) relies on a matrix Manhattan distance, but the exact definition of dis(P,Q) for two LoRA matrices is not stated. Please specify whether this is element-wise L1 averaged or summed, and how A and B are combined.
  4. [Section 4.4, Figure 3] The t-SNE visualizations are qualitative. If used as evidence for the CR loss's effect, a quantitative clustering metric (e.g., silhouette score) would be more convincing.
  5. [Abstract / Conclusion] The phrase 'unconstrained incremental learning methods' is vague; the abstract's 1.88% improvement seems to be relative to a particular baseline but the text does not clearly define the denominator. Please state the comparison explicitly.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the empirical claims are evaluated on unseen test data and the theoretical proposition is a direct consequence of the forward definition, not a fitted prediction.

full rationale

The paper's central empirical claim—MSLoRA-CR (SUM 990.25, 995.21 with ORTHO) outperforms separate-model fine-tuning (SUM 970.29)—rests on held-out test-split evaluations and does not fit any target constant to its own outputs. The CR loss and orthogonality penalty are explicit regularizers added to the base loss; their hyperparameters (α=0.1, β=0.01) are selected by observing loss scales during training, not by fitting the reported metrics. Proposition 1 is a direct consequence of the additive forward definition in Eq. (1): because e = Wh + Σ m_i ΔW_i h, the chain rule gives ∂L/∂ΔW_t = ∂L/∂W. This is a trivial but non-circular derivation; it does not by itself establish the claimed plasticity-stability trade-off, but that is an overclaim rather than a circular reduction. Section 4.3 (Table 6) explicitly acknowledges strong task-order sensitivity (SUM range 610.40–760.83 on a 6-task subset), which is a significant external-validity limitation for the headline order, but it is an evaluation-setup issue, not a circularity in the derivation. No load-bearing self-citations or imported uniqueness theorems appear. Overall circularity score: 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The method relies on fitted hyperparameters (rank, alpha, loss weights) and on several domain assumptions about parameter-space similarity, metric aggregation, and task-order representativeness. No new physical or model entities are postulated.

free parameters (3)
  • LoRA rank and alpha = 64, 64
    The main experiments use rank and alpha of 64; Table 3 shows the choice affects results, with 128 degrading performance.
  • CR loss weight (alpha) = 0.1
    Selected empirically after observing loss scales; Table 4 shows deviations from this value produce worse results.
  • Ortho loss weight (beta) = 0.01
    Selected empirically along with alpha; Table 4 shows sensitivity to this value.
assumptions (5)
  • ad hoc to paper Gradient equivalence in Proposition 1 implies a stability-plasticity balance
    The proposition shows dL/dDeltaW_t = dL/dW due to linearity of eq. (1), but the leap to a proven balance is not derived.
  • domain assumption Manhattan distance between LoRA parameter matrices is a meaningful measure of knowledge similarity
    Used in eq. (5) through (7) without empirical validation of the distance metric choice.
  • domain assumption The fixed task order in the main experiments is representative
    Table 6 shows order changes the SUM by more than 100 points on a six-task subset, so the chosen order is load-bearing.
  • domain assumption Summarizing performance by summing heterogeneous metrics is valid
    The overall claim relies on SUM of accuracy, F1, AUC, and BLEU scores, which are not comparable scales.
  • domain assumption The frozen LVLM (LLaVA-Med-v1.5) provides sufficient shared knowledge for cross-modality transfer
    The method assumes knowledge from previously learned modalities can help new modalities through the frozen backbone.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Contrastive Regularization over LoRA for Multimodal Biomedical Image Incremental Learning." pith.science (2026). https://pith.science/paper/BYU7DYZO

@misc{pith2026250811673,
  author       = {Pith},
  title        = {Pith review of: Contrastive Regularization over LoRA for Multimodal Biomedical Image Incremental Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BYU7DYZO}},
  note         = {Machine review of arXiv:2508.11673}
}
read the original abstract

Multimodal Biomedical Image Incremental Learning (MBIIL) is essential for handling diverse tasks and modalities in the biomedical domain, as training separate models for each modality or task significantly increases inference costs. Existing incremental learning methods focus on task expansion within a single modality, whereas MBIIL seeks to train a unified model incrementally across modalities. The MBIIL faces two challenges: I) How to preserve previously learned knowledge during incremental updates? II) How to effectively leverage knowledge acquired from existing modalities to support new modalities? To address these challenges, we propose MSLoRA-CR, a method that fine-tunes Modality-Specific LoRA modules while incorporating Contrastive Regularization to enhance intra-modality knowledge sharing and promote inter-modality knowledge differentiation. Our approach builds upon a large vision-language model (LVLM), keeping the pretrained model frozen while incrementally adapting new LoRA modules for each modality or task. Experiments on the incremental learning of biomedical images demonstrate that MSLoRA-CR outperforms both the state-of-the-art (SOTA) approach of training separate models for each modality and the general incremental learning method (incrementally fine-tuning LoRA). Specifically, MSLoRA-CR achieves a 1.88% improvement in overall performance compared to unconstrained incremental learning methods while maintaining computational efficiency. Our code is publicly available at https://github.com/VentusAislant/MSLoRA_CR.

Figures

Figures reproduced from arXiv: 2508.11673 by the authors.

Figure 1
Figure 1. Contrastive regularization aligns LoRA parameters [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The overview of modality-specific LoRA fine-tuning with contrastive regularization (MSLoRA-CR). [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. t-SNE visualization of the vision embeddings from three different models. Each subplot corresponds to a different [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

64 extracted references · 45 canonical work pages

  1. [1]

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al

  2. [2]

    Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. 2018. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European conference on computer vision (ECCV) . 139–154

    work page 2018

  3. [3]

    Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. 2017. Expert gate: Lifelong learning with a network of experts. InProceedings of the IEEE conference on computer vision and pattern recognition . 3366–3375

    work page 2017

  4. [4]

    Arthur Douillard, Alexandre Ramé, Guillaume Couairon, and Matthieu Cord

    work page

  5. [5]

    Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. 2024. Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision 132, 2 (2024), 581–595

    work page 2024

  6. [6]

    In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Dytox: Transformers for continual learning with dynamic token expan- sion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9285–9295

    work page

  7. [7]

    James A Hanley and Barbara J McNeil. 1982. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143, 1 (1982), 29–36

    work page 1982

  8. [8]

    Matthew Groh, Caleb Harris, Luis Soenksen, Felix Lau, Rachel Han, Aerin Kim, Arash Koochek, and Omar Badri. 2021. Evaluating deep neural networks trained on clinical images in dermatology with the fitzpatrick 17k dataset. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 1820– 1828

    work page 2021

Show all 64 references
  1. [9]

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In International conference on machine learning. PMLR, 2790–2799

    2019

  2. [10]

    Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. 2020. PathVQA: 30000+ Questions for Medical Visual Question Answering. arXiv preprint arXiv:2003.10286 (2020)

    2020 arXiv

  3. [11]

    Zhiyuan Hu, Yunsheng Li, Jiancheng Lyu, Dashan Gao, and Nuno Vasconcelos

  4. [12]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021)

    2021 arXiv

  5. [13]

    Qiushi Huang, Tom Ko, Zhan Zhuang, Lilian Tang, and Yu Zhang. 2025. HiRA: Parameter-Efficient Hadamard High-Rank Adaptation for Large Language Mod- els. In The Thirteenth International Conference on Learning Representations

    2025

  6. [14]

    David Isele and Akansel Cosgun. 2018. Selective experience replay for lifelong learning. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 32

    2018

  7. [15]

    Zi-Yuan Hu, Yanyang Li, Michael R Lyu, and Liwei Wang. 2023. Vl-pet: Vision- and-language parameter-efficient tuning via granularity control. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 3010–3020

    2023

  8. [16]

    Gangwei Jiang, Caigao Jiang, Zhaoyi Li, Siqiao Xue, Jun Zhou, Linqi Song, Defu Lian, and Yin Wei. 2025. Unlocking the Power of Function Vectors for Charac- terizing and Mitigating Catastrophic Forgetting in Continual Instruction Tuning. arXiv preprint arXiv:2502.11019 (2025)

    2025 arXiv

  9. [17]

    Daniel S Kermany, Michael Goldbaum, Wenjia Cai, Carolina CS Valentim, Huiy- ing Liang, Sally L Baxter, Alex McKeown, Ge Yang, Xiaokang Wu, Fangbing Yan, et al. 2018. Identifying medical diagnoses and treatable diseases by image-based deep learning. cell 172, 5 (2018), 1122–1131

    2018

  10. [18]

    Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. 2022. Visual prompt tuning. In Euro- pean Conference on Computer Vision . Springer, 709–727

    2022

  11. [19]

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of...

    2017

  12. [20]

    Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. 2018. A dataset of clinically generated visual questions and answers about radiology images. Scientific data 5, 1 (2018), 1–10

    2018

  13. [21]

    Sanghyeon Kim, Hyunmo Yang, Yunghyun Kim, Youngjoon Hong, and Eunbyung Park. 2024. Hydra: Multi-head low-rank adaptation for parameter efficient fine- tuning. Neural Networks 178 (2024), 106414

    2024

  14. [22]

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. 2024. Llava-med: Train- ing a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems 36 (2024)

    2024

  15. [23]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning . PMLR, 19730–19742

    2023

  16. [24]

    Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691 (2021)

    2021 arXiv

  17. [25]

    Yan-Shuo Liang and Wu-Jun Li. 2024. InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 23638–23647

    2024

  18. [26]

    Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. 2021. Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI). IEEE, 1650–1654

    2021

  19. [27]

    Youwei Liang, Han Guo, and Pengtao Xie. 2023. XrayChat: Towards Enabling ChatGPT-Like Capabilities on Chest X-ray Images. (2023)

    2023

  20. [28]

    Xialei Liu, Xusheng Cao, Haori Lu, Jia-wen Xiao, Andrew D Bagdanov, and Ming- Ming Cheng. 2023. Class incremental learning with pre-trained vision-language models. arXiv preprint arXiv:2310.20348 (2023)

    2023 arXiv

  21. [29]

    David Lopez-Paz and Marc’Aurelio Ranzato. 2017. Gradient episodic memory for continual learning. Advances in neural information processing systems 30 (2017)

    2017

  22. [30]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024. Visual in- struction tuning. Advances in neural information processing systems 36 (2024)

    2024

  23. [31]

    Arun Mallya and Svetlana Lazebnik. 2018. Packnet: Adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition . 7765–7773

    2018

  24. [32]

    Guy Oren and Lior Wolf. 2021. In defense of the learning without forgetting for task incremental learning. InProceedings of the IEEE/CVF International Conference on Computer Vision. 2209–2218

    2021

  25. [33]

    I Loshchilov. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)

    2017 arXiv

  26. [34]

    German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. 2019. Continual lifelong learning with neural networks: A review. Neural networks 113 (2019), 54–71

    2019

  27. [35]

    John Pavlopoulos, Vasiliki Kougia, and Ion Androutsopoulos. 2019. A survey on biomedical image captioning. In Proceedings of the second workshop on shortcom- ings in vision and language . 26–36

    2019

  28. [36]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics . 311–318

    2002

  29. [37]

    Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. 2017. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition . 2001–2010

    2017

  30. [38]

    Gobinda Saha, Isha Garg, and Kaushik Roy. 2021. Gradient Projection Memory for Continual Learning. In International Conference on Learning Representations . https://openreview.net/forum?id=3AOj0RCNC2

    2021

  31. [39]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sand- hini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al

  32. [40]

    Albin Soutif-Cormerais, Marc Masana, Joost Van de Weijer, and Bartlømiej Twardowski. 2021. On the importance of cross-task features for class-incremental learning. arXiv preprint arXiv:2106.11930 1 (2021)

    2021 arXiv

  33. [41]

    Yuxuan Sun, Chenglu Zhu, Sunyi Zheng, Kai Zhang, Zhongyi Shui, Xiaoxuan Yu, Yizhi Zhao, Honglin Li, Yunlong Zhang, Ruojia Zhao, et al. 2023. Pathasst: Redefining pathology through generative foundation ai assistant for pathology. arXiv preprint arXiv:2305.15072 (2023)

    2023 arXiv

  34. [42]

    Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. 2022. Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 5227–5237

    2022

  35. [43]

    Mehmet Saygin Seyfioglu, Wisdom O Ikezogwo, Fatemeh Ghezloo, Ranjay Kr- ishna, and Linda Shapiro. 2024. Quilt-llava: Visual instruction tuning by extract- ing localized narratives from open-source histopathology videos. In Proceedings of the IEEE/CVF Conference on Computer Vis...

    2024

  36. [44]

    Haojie Zhang, et al

    Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Charles Lau, Ryutaro Tanno, Ira Ktena, et al. Haojie Zhang, et al

  37. [45]

    Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling

  38. [46]

    Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. 2024. A comprehensive survey of continual learning: theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)

    2024

  39. [47]

    Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. 2018. The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific Data. 2018; 5: 180161. Search in 2 (2018)

    2018

  40. [48]

    Jiazuo Yu, Yunzhi Zhuge, Lu Zhang, Ping Hu, Dong Wang, Huchuan Lu, and You He. 2024. Boosting continual learning of vision-language models via mixture-of- experts adapters. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 23219–23230

    2024

  41. [49]

    Friedemann Zenke, Ben Poole, and Surya Ganguli. 2017. Continual learning through synaptic intelligence. In International conference on machine learning . PMLR, 3987–3995

    2017

  42. [50]

    Kai Zhang, Rong Zhou, Eashan Adhikarla, Zhiling Yan, Yixin Liu, Jun Yu, Zhengliang Liu, Xun Chen, Brian D Davison, Hui Ren, et al . 2024. A gener- alist vision–language foundation model for diverse biomedical tasks. Nature Medicine (2024), 1–13

    2024

  43. [51]

    Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. 2023. AdaLoRA: Adaptive budget allocation for parameter-efficient fine-tuning. arXiv preprint arXiv:2303.10512 (2023)

    2023 arXiv

  44. [52]

    Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. 2022. Tip-adapter: Training-free adaption of clip for few-shot classification. In European conference on computer vision . Springer, 493–510

    2022

  45. [53]

    Yichen Wu, Hongming Piao, Long-Kai Huang, Renzhen Wang, Wanhua Li, Hanspeter Pfister, Deyu Meng, Kede Ma, and Ying Wei. [n. d.]. SD-LoRA: Scal- able Decoupled Low-Rank Adaptation for Class Incremental Learning. In The Thirteenth International Conference on Learning Representations

  46. [54]

    Juexiao Zhou, Xiaonan He, Liyuan Sun, Jiannan Xu, Xiuying Chen, Yuetan Chu, Longxi Zhou, Xingyu Liao, Bin Zhang, and Xin Gao. 2023. SkinGPT-4: an interactive dermatology diagnostic system with visual large language model. (2023)

    2023

  47. [55]

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2022. Condi- tional prompt learning for vision-language models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition . 16816–16825

    2022

  48. [56]

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2022. Learning to prompt for vision-language models. International Journal of Computer Vision 130, 9 (2022), 2337–2348

    2022

  49. [57]

    Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, et al. 2022. Mixture-of-experts with expert choice routing. Advances in Neural Information Processing Systems 35 (2022), 7103–7114

    2022

  50. [58]

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large lan- guage models. arXiv preprint arXiv:2304.10592 (2023)

    2023 arXiv

  51. [59]

    Zizhao Zhang, Pingjun Chen, Mason McGough, Fuyong Xing, Chunbao Wang, Marilyn Bui, Yuanpu Xie, Manish Sapkota, Lei Cui, Jasreman Dhillon, et al. 2019. Pathologist-level interpretable whole-slide cancer diagnosis with deep learning. Nature Machine Intelligence 1, 5 (2019), 236

    2019

  52. [2018]

    In Medical Image Com- puting and Computer Assisted Intervention–MICCAI 2018: 21st International Con- ference, Granada, Spain, September 16-20, 2018, Proceedings, Part II 11

    Rotation equivariant CNNs for digital pathology. In Medical Image Com- puting and Computer Assisted Intervention–MICCAI 2018: 21st International Con- ference, Granada, Spain, September 16-20, 2018, Proceedings, Part II 11 . Springer, 210–218

    2018

  53. [2021]

    In International conference on machine learning

    Learning transferable visual models from natural language supervision. In International conference on machine learning . PMLR, 8748–8763

  54. [2022]

    Advances in neural information processing systems 35 (2022), 23716–23736

    Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems 35 (2022), 23716–23736

    2022

  55. [2023]

    In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Dense network expansion for class incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 11858– 11867

  56. [2024]

    NEJM AI 1, 3 (2024), AIoa2300138

    Towards generalist biomedical AI. NEJM AI 1, 3 (2024), AIoa2300138

    2024

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.