Pith. sign in

REVIEW 4 major objections 4 minor 59 references

Continually Evolved Multimodal Foundation Models for Cancer Prognosis

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A continually evolving multimodal Q-Former improves cancer prognosis by fusing text, image, and RNA data, and can add new modalities later without retraining.

desk verdict Sensible architecture idea, but the experimental presentation makes the central claims unverifiable; as submitted, this does not deserve peer review. read the letter →

arxiv 2501.18170 v2 pith:WAAERBHL submitted 2025-01-30 cs.LG

Jie Peng , Shuang Zhou , Longwei Yang , Yiran Song , Mohan Zhang , Kaixiong Zhou , Feng Xie , Mingquan Lin
show 2 more authors
Rui Zhang Tianlong Chen
This is my paper · ORCID
classification cs.LG
keywords cancerprognosismultimodalfoundationmodelcontinuallearningQ-Formerlow-rankadaptationsurvivalanalysisTCGAfusion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Cancer prognosis—predicting how long and how well a patient survives—is treated here as a multimodal fusion problem: clinical text, pathology images, and RNA expression each carry complementary signals. The paper claims that a single Q-Former architecture, augmented with per-modality low-rank adapters and a self-gating fusion step, can combine these modalities and also keep learning new ones as they arrive without catastrophic forgetting. If true, this would matter because hospital data arrive incrementally and from varying distributions, while most current fusion methods either concatenate features naively or rely on task-specific pipelines. The concrete evidence offered is concordance-index (c-index) improvement: 0.042 to 0.115 over fusion baselines, and up to 0.081 when a new modality is added in a continual-learning setup.

What carries the argument

The central objects are the Multimodal Q-Former, a shared Q-Former (the learnable cross-modal attention module from BLIP-2) that receives tokens from all modalities; MM-LoRA, which equips each modality with its own low-rank adapter (queries and LoRA projections) so that new modalities can be added without retraining the base Q-Former; and SMQF, which concatenates the supporting modalities' queries, projects them to the primary query's dimension, applies a sigmoid self-gate, and returns a fused token sequence that preserves the primary modality's dominance. The continual-learning side is carried by the Pseudo Target Generation Module (PTGM) and Instruction-based Knowledge Distillation (IKD), which the paper says preserve old-task knowledge and maintain generation capacity when new modalities arrive. What these components do is turn the fusion problem into a routing and compression problem: each modality gets its own lightweight path through a frozen backbone, and fusion happens by gated token compression rather than by concatenating raw feature vectors.

What would settle it

Measure the c-index on the original image-plus-text task after adding the genomic modality under the proposed continual-learning setup; if this old-task score drops by more than a small tolerance (e.g., a clinically meaningful margin) relative to a model that never learned the new modality, the claim that the method mitigates catastrophic forgetting would be refuted. A second test: train a from-scratch model on all modalities together; if the continual variant's final c-index does not beat it, the continual-learning gain may be an artifact of added training data.

Watch

Extended reading notes

Core claim

The paper's central claim is that a shared Multimodal Q-Former can serve as a foundation model for cancer prognosis across text, RNA, and pathology-image modalities, and that this same model can incorporate previously unseen modalities without forgetting older ones. Architecturally, the claim rests on Modality-specific Multi-Query Low-Rank Adaptation (MM-LoRA), which routes each modality to its own LoRA adapter so that adding a modality only adds a small parameter module, and on Self-gated Multimodal Query Fusion (SMQF), which compresses supporting-modality queries into a token budget that preserves the primary modality's dominance. The authors report average c-index gains of {0.042, 0.115, 0.072, 0.079, 0.036, 0.068, 0.081} over the LiMOE, MAGGate, MulT, TF, Cross-Attention Fusion, Early Fusion, and Late Fusion baselines, and improvements of up to {0.055 for LUAD, 0.081 for BLCA, 0.042 for MIMIC} when a new modality is added after initial training.

Load-bearing premise

The continual learning results depend on the unstated premise that performance on the originally learned modalities is preserved once a new modality is added, but the paper never measures old-task performance, so the reported gains on the new task cannot by themselves demonstrate that catastrophic forgetting was avoided.

Editorial extensions

If this is right

  • If the method holds, adding a new data type to an existing prognostic model (e.g., genomics for a cancer type that previously used only images and clinical text) can improve prediction without retraining the entire model.
  • The MM-LoRA design implies that scaling to many modalities adds only a small per-modality LoRA parameter set, keeping the architecture tractable as data sources multiply.
  • The reported c-index gains, if consistent in practice, would translate into better separation of high-risk from low-risk patients, which is the clinically relevant use of a prognostic model.
  • The paper's MIMIC result (0.042 improvement when a lab modality is added to codes and notes) suggests the same continual-fusion recipe transfers beyond TCGA pathology data to ICU-style clinical records.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The continual-learning evidence is incomplete as reported: because Section 4.3 measures only the new task's test c-index after adding a modality, the paper does not yet show that older modalities survive; a reader should treat the catastrophic-forgetting claim as unverified until old-task performance is reported.
  • The gains in Section 4.3 could in principle come from simply having more training data (the added modality), rather than from the continual-learning machinery; an ablation that trains from scratch on the combined set would separate the two effects.
  • The SMQF design, if validated, is a generic token-compression trick that could apply to other multimodal foundation models beyond prognosis, because it treats fusion as gated routing through a frozen backbone.
  • The paper does not compare against standard continual-learning baselines such as elastic weight consolidation or rehearsal; such comparisons would clarify whether PTGM and IKD add value over existing methods.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The manuscript proposes a "continually evolving multimodal foundation model" for cancer prognosis, combining pathology images, clinical text, and RNA-seq data through a Q-Former architecture with per-modality LoRA adapters (MM-LoRA) and a self-gated query fusion module (SMQF). The introduction also names two components, PTGM and IKD, as key to the approach. Experiments on TCGA and MIMIC report c-index improvements over fusion baselines and a continual-learning setting in which a modality is added after initial training. However, the main results table contains no row for the proposed method, the two named components are never described in the method or experiments, and the continual-learning evaluation does not measure forgetting.

Significance. The topic is relevant: integrating multiple modalities for cancer prognosis and enabling continual incorporation of new data would be practically valuable, and the paper evaluates on public datasets with a broad set of baselines. Some preprocessing details for the text, image, and RNA modalities are provided, which is a useful starting point. However, the central empirical claim is not verifiable from Table 1, the two named method components are absent from the method section, and the continual-learning experiment does not test the stated forgetting-avoidance objective. As presented, the significance cannot be assessed because the evidence for the headline claims is missing.

major comments (4)
  1. [Section 4.2, Table 1] Table 1 contains no row for the proposed method, so the claim in Section 4.2 of average c-index improvements of {0.042, 0.115, 0.072, 0.079, 0.036, 0.068, 0.081} over the listed baselines cannot be computed or checked. The problem is compounded by duplicated baseline rows with inconsistent values: "Cross Attn Fusion" appears twice with means 0.717 and 0.694, and "Early Fusion" appears twice with means 0.685 (Text, RNA, Image) and 0.746 (Text, Image). Because Table 1 is the only quantitative evidence for the headline multimodal fusion result, this is a load-bearing defect rather than a formatting issue.
  2. [Introduction and Section 3] The Introduction states that the model incorporates two key components, the Pseudo Target Generation Module (PTGM) and Instruction-based Knowledge Distillation (IKD), and attributes to PTGM the mitigation of catastrophic forgetting. However, neither PTGM nor IKD is defined, formulated, or evaluated anywhere in Section 3 or Section 4. Without a description of their operation, inputs, outputs, or training objectives, the method cannot be reproduced and the claimed continual-learning mechanism is unsupported.
  3. [Section 4.3, Figure 1] The continual-learning evaluation reports only testing c-index on the newly added modality and never reports performance on the originally learned modalities, such as image and text after adding genomics for LUAD and BLCA. The claim that the method "effectively learns new modalities" while mitigating catastrophic forgetting therefore has no supporting evidence, since the observed increases could result simply from training on more data. A standard forgetting metric on the old tasks is needed before the continual-learning contribution can be assessed.
  4. [Section 4.1 and Section 4.2] The experimental section omits essential training details: the survival prediction head, loss function, optimizer, learning rate, batch size, number of epochs, LoRA rank r, number of learnable queries per modality, and the patch-count threshold are not specified. No error bars, standard deviations, or statistical significance tests are reported for any c-index. These omissions make the quantitative claims non-reproducible and leave open the possibility that the reported improvements are within run-to-run noise.
minor comments (4)
  1. [General terminology] The paper repeatedly uses "foundation model" to describe a model trained only on the TCGA dataset without large-scale pretraining; this terminology should be justified or replaced with a more modest description.
  2. [Section 4.2, Table 1] LiMOE is listed as a baseline in Table 1 but is never described or cited in the related work or experiment sections, so the reader cannot tell what this baseline is or how it was configured.
  3. [Section 4.3] The MIMIC experiment is described only as starting with code and clinical notes and then adding a lab modality; the specific prediction task, outcome, cohort, and data split are not defined, which prevents reproducibility.
  4. [References and notation] Several bibliographic entries are incomplete or lack standard details, such as reference [22], which has no volume or page information, and reference [30] duplicates reference [16]; the notation around Equation (3), including the parameter theta and the definition of x_s, should also be made consistent.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the paper's claims are empirical benchmark results and architecture equations with no fitted-input-as-prediction or self-citation dependency.

full rationale

The derivation chain in this manuscript is not a derivation: the method is an architectural proposal (Multimodal Q-Former with MM-LoRA, SMQF, PTGM, IKD) and the central claims in Sections 4.2 and 4.3 are measured c-index values on TCGA and MIMIC. None of the paper's equations (Eqs. 1-4) define a target quantity in terms of itself; the claimed improvements are external comparisons against LiMOE, MAGGate, MulT, TF, and fusion baselines. The continual learning section reports final c-index after adding a modality, and it does not report old-task retention, so the 'mitigates catastrophic forgetting' claim is under-supported; however, missing evidence for retention is an evaluation weakness, not a circular step, because the reported numbers are not constructed to equal the model's inputs. The citation of CREMA is not self-citational by these authors, and no load-bearing argument reduces to the authors' own prior work. Reproducibility concerns (Table 1 lacking a row for 'Ours' and duplicated baseline rows with conflicting values) affect verifiability and correctness, not circularity. Accordingly, no circular step can be exhibited with a quote and a specific reduction, and the paper is treated as a benchmark-validated empirical contribution with score 0.

Assumptions & free parameters 6 free parameters · 4 assumptions · 2 invented entities

The central claim rests on several hand-picked hyperparameters, unstated assumptions about the evaluation protocol, and two named components that are never specified.

free parameters (6)
  • Patch entropy threshold = 5
    Hand-chosen threshold for discarding background patches in WSI preprocessing (Section 4.1).
  • Maximum patch count threshold = unspecified (value not reported)
    A 'predefined threshold' standardizes the number of patches per WSI; value not stated (Section 4.1).
  • LoRA rank r = unspecified (value not reported)
    Rank of modality-specific LoRA adapters in MM-LoRA (Eq. 2); no value or search reported.
  • Number of learnable queries per modality = unspecified (value not reported)
    Queries q^i_m in MM-LoRA (Eq. 1); not specified.
  • Choice of primary modality p = unspecified (value not reported)
    SMQF treats one modality as primary; which modality is primary is not specified (Eq. 3).
  • Text encoder choice = unspecified (feature dim 768)
    The raw text encoder used to produce 768-d embeddings is not named (Section 4.1).
assumptions (4)
  • domain assumption Frozen pre-trained feature extractors (Marugoto ViT, BulkRNABert, and an unnamed text encoder) produce reliable and equally suited representations for all modalities.
    All methods operate on these features, so the correctness of the comparison depends on these extractors being suitable across methods (Section 4.1).
  • domain assumption C-index on TCGA survival analysis is the appropriate and sufficient evaluation metric.
    No calibration, time-dependent AUC, or clinical utility metrics are reported (Section 4.2).
  • domain assumption Final task performance after continual learning is a valid measure of continual learning ability.
    The protocol in Section 4.3 assumes that improved final performance implies successful adaptation while implicitly ignoring forgetting of old modalities.
  • ad hoc to paper The model is a 'foundation model' despite training only on TCGA without large-scale pretraining.
    The term 'foundation model' is used in the title and abstract, but no large-scale pretraining is described; this is an unsubstantiated label.
invented entities (2)
  • Pseudo Target Generation Module (PTGM)
    purpose: Mitigate catastrophic forgetting when training on multiple tasks within the same modality
    Introduced in Section 1 but never described in Section 3 or evaluated in Section 4; impossible to verify.
  • Instruction-based Knowledge Distillation (IKD)
    purpose: Preserve generative capabilities when adapting to new modalities
    Same as PTGM: named but not defined or measured anywhere in the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Continually Evolved Multimodal Foundation Models for Cancer Prognosis." pith.science (2026). https://pith.science/paper/WAAERBHL

@misc{pith2026250118170,
  author       = {Pith},
  title        = {Pith review of: Continually Evolved Multimodal Foundation Models for Cancer Prognosis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WAAERBHL}},
  note         = {Machine review of arXiv:2501.18170}
}
read the original abstract

Cancer prognosis is a critical task that involves predicting patient outcomes and survival rates. To enhance prediction accuracy, previous studies have integrated diverse data modalities, such as clinical notes, medical images, and genomic data, leveraging their complementary information. However, existing approaches face two major limitations. First, they struggle to incorporate newly arrived data with varying distributions into training, such as patient records from different hospitals, thus rendering sub-optimal generalizability and limited utility in real-world applications. Second, most multimodal integration methods rely on simplistic concatenation or task-specific pipelines, which fail to capture the complex interdependencies across modalities. To address these, we propose a continually evolving multi-modal foundation model. Extensive experiments on the TCGA dataset demonstrate the effectiveness of our approach, highlighting its potential to advance cancer prognosis by enabling robust and adaptive multimodal integration.

Figures

Figures reproduced from arXiv: 2501.18170 by the authors.

Figure 1
Figure 1. Performance of Continual Learning. We validated our continual learning design us￾ing both TCGA and MIMIC datasets. Specifi￾cally, we focused on two cancer types from the TCGA dataset: LUAD and BLCA. The model was initially trained with image and clinical text modalities, then continued learning with genomics modalities. For the MIMIC dataset, the model started with code and clinical notes modalities and then incorpo… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

59 extracted references · 40 canonical work pages

  1. [1]

    The application of deep learning in cancer prognosis prediction

    Wan Zhu, Longxiang Xie, Jianye Han, and Xiangqian Guo. The application of deep learning in cancer prognosis prediction. Cancers, 12(3):603, 2020

    work page 2020

  2. [2]

    Global cancer statistics 2020: Globocan estimates of incidence and mortality worldwide for 36 cancers in 185 countries

    Hyuna Sung, Jacques Ferlay, Rebecca L Siegel, Mathieu Laversanne, Isabelle Soerjomataram, Ahmedin Jemal, and Freddie Bray. Global cancer statistics 2020: Globocan estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA: a cancer journal for clinicians, 71(3):209–249, 2021

    2020

  3. [3]

    Trends in cancer prognosis in a population-based cohort survey: can recent advances in cancer therapy affect the prognosis? Cancer Epidemiology, 39(1):97–103, 2015

    Eri Kawabata-Shoda, Hadrien Charvat, Ai Ikeda, Manami Inoue, Norie Sawada, Motoki Iwasaki, Shizuka Sasazuki, Taichi Shimazu, Taiki Yamaji, Hiromichi Kimura, et al. Trends in cancer prognosis in a population-based cohort survey: can recent advances in cancer therapy affect the prognosis? Cancer Epidemiology, 39(1):97–103, 2015

    work page 2015

  4. [4]

    Multimodal adversarial representation learning for breast cancer prognosis prediction

    Xiuquan Du and Yuefan Zhao. Multimodal adversarial representation learning for breast cancer prognosis prediction. Computers in Biology and Medicine, 157:106765, 2023

    work page 2023

  5. [5]

    Mbfusion: Multi- modal balanced fusion and multi-task learning for cancer diagnosis and prognosis

    Ziye Zhang, Wendong Yin, Shijin Wang, Xiaorou Zheng, and Shoubin Dong. Mbfusion: Multi- modal balanced fusion and multi-task learning for cancer diagnosis and prognosis. Computers in Biology and Medicine, 181:109042, 2024

    work page 2024

  6. [6]

    Machine learning applications in cancer prognosis and prediction

    Konstantina Kourou, Themis P Exarchos, Konstantinos P Exarchos, Michalis V Karamouzis, and Dimitrios I Fotiadis. Machine learning applications in cancer prognosis and prediction. Computational and structural biotechnology journal, 13:8–17, 2015

    2015

  7. [7]

    Pathology-and-genomics multimodal transformer for survival outcome prediction

    Kexin Ding, Mu Zhou, Dimitris N Metaxas, and Shaoting Zhang. Pathology-and-genomics multimodal transformer for survival outcome prediction. InInternational Conference on Medical Image Computing and Computer-Assisted Intervention, pages 622–631. Springer, 2023

    work page 2023

  8. [8]

    A pathology foundation model for cancer diagnosis and prognosis prediction

    Xiyue Wang, Junhan Zhao, Eliana Marostica, Wei Yuan, Jietian Jin, Jiayu Zhang, Ruijiang Li, Hongping Tang, Kanran Wang, Yu Li, et al. A pathology foundation model for cancer diagnosis and prognosis prediction. Nature, 634(8035):970–978, 2024

    2024

Show all 59 references
  1. [9]

    Llm-guided multi-modal multiple instance learning for 5-year overall survival prediction of lung cancer

    Kyungwon Kim, Yongmoon Lee, Doohyun Park, Taejoon Eo, Daemyung Youn, Hyesang Lee, and Dosik Hwang. Llm-guided multi-modal multiple instance learning for 5-year overall survival prediction of lung cancer. In International Conference on Medical Image Computing and Computer-Assis...

    2024

  2. [10]

    Multimodal whole slide foundation model for pathology

    Tong Ding, Sophia J Wagner, Andrew H Song, Richard J Chen, Ming Y Lu, Andrew Zhang, Anurag J Vaidya, Guillaume Jaume, Muhammad Shaban, Ahrong Kim, et al. Multimodal whole slide foundation model for pathology. arXiv preprint arXiv:2411.19666, 2024

    2024 arXiv

  3. [11]

    Clinical applications of continual learning machine learning

    Cecilia S Lee and Aaron Y Lee. Clinical applications of continual learning machine learning. The Lancet Digital Health, 2(6):e279–e281, 2020

    2020

  4. [12]

    Integrating multimodal information in large pretrained transformers

    Wasifur Rahman, Md Kamrul Hasan, Sangwu Lee, Amir Zadeh, Chengfeng Mao, Louis- Philippe Morency, and Ehsan Hoque. Integrating multimodal information in large pretrained transformers. In Proceedings of the conference. Association for Computational Linguistics. Meeting, volume 2...

    2020

  5. [13]

    Multimodal transformer for unaligned multimodal language sequences

    Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the conference. Association for computational linguistics. Meeting, volume 2...

    2019

  6. [14]

    3 4 5 Brat Daniel J

    Cancer Genome Atlas Research Network Tissue source sites: Duke University Medical School McLendon Roger 1 Friedman Allan 2 Bigner Darrell 1, Emory University Van Meir Erwin G. 3 4 5 Brat Daniel J. 5 6 M. Mastrogianakis Gena 3 Olson Jeffrey J. 3 4 5, Henry Ford Hospital Mikkels...

    2008

  7. [15]

    Attention is all you need

    Ashish Vaswani et al. Attention is all you need. In Advances in Neural Information Processing Systems, 2017

    2017

  8. [16]

    Learning transferable visual models from natural language supervision

    Alec Radford et al. Learning transferable visual models from natural language supervision. International Conference on Machine Learning, 2021

    2021

  9. [17]

    Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and- language tasks

    Jiasen Lu et al. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and- language tasks. In Neural Information Processing Systems, 2019

    2019

  10. [18]

    Lxmert: Learning cross-modality encoder representations from transformers

    Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. In Empirical Methods in Natural Language Processing, 2019

    2019

  11. [19]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac et al. Flamingo: a visual language model for few-shot learning. In Neural Information Processing Systems, 2022

    2022

  12. [20]

    Gpt-4v(ision) system card

    OpenAI. Gpt-4v(ision) system card. Technical report, OpenAI, 2023

    2023

  13. [21]

    Large language models for disease diagnosis: A scoping review

    Shuang Zhou, Zidu Xu, Mian Zhang, Chunpu Xu, Yawen Guo, Zaifu Zhan, Sirui Ding, Jiashuo Wang, Kaishuai Xu, Yi Fang, et al. Large language models for disease diagnosis: A scoping review. arXiv preprint arXiv:2409.00097, 2024

    2024 arXiv

  14. [22]

    Multi-modal medical image diagnosis

    Wei Xue et al. Multi-modal medical image diagnosis. Nature Machine Intelligence, 2022

    2022

  15. [23]

    Modality-aware integration with large language models for knowledge-based visual question answering

    Junnan Dong, Qinggang Zhang, Huachi Zhou, Daochen Zha, Pai Zheng, and Xiao Huang. Modality-aware integration with large language models for knowledge-based visual question answering. arXiv preprint arXiv:2402.12728, 2024

    2024 arXiv

  16. [24]

    Advances in multimodal human-computer interaction

    Dongwon Kim et al. Advances in multimodal human-computer interaction. In Association for Computational Linguistics, 2022

    2022

  17. [25]

    On the opportunities and risks of foundation models

    Rishi Bommasani et al. On the opportunities and risks of foundation models. arXiv preprint, 2021

    2021

  18. [26]

    Bert: Pre-training of deep bidirectional transformers

    Jacob Devlin et al. Bert: Pre-training of deep bidirectional transformers. In North American Chapter of the Association for Computational Linguistics, 2018

    2018

  19. [27]

    Language models are unsupervised multitask learners

    Alec Radford et al. Language models are unsupervised multitask learners. OpenAI Blog, 2019

    2019

  20. [28]

    Language models are few-shot learners

    Tom Brown et al. Language models are few-shot learners. Neural Information Processing Systems, 2020

    2020

  21. [29]

    Scaling laws for neural language models

    Jared Kaplan et al. Scaling laws for neural language models. arXiv preprint, 2020

    2020

  22. [30]

    Learning transferable visual models from natural language supervision

    Alec Radford et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021

    2021

  23. [31]

    Zero-shot text-to-image generation

    Aditya Ramesh et al. Zero-shot text-to-image generation. International Conference on Machine Learning, 2021

    2021

  24. [32]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach et al. High-resolution image synthesis with latent diffusion models. In Computer Vision and Pattern Recognition, 2022

    2022

  25. [33]

    Gpt-4 technical report

    OpenAI. Gpt-4 technical report. Technical report, arXiv, 2023

    2023

  26. [34]

    Highly accurate protein structure prediction with alphafold

    John Jumper et al. Highly accurate protein structure prediction with alphafold. Nature, 2021

    2021

  27. [35]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery et al. Palm: Scaling language modeling with pathways. arXiv preprint, 2022

    2022

  28. [36]

    Rt-1: Robotics transformer for real-world control

    Anthony Brohan et al. Rt-1: Robotics transformer for real-world control. In Conference on Robot Learning, 2022

    2022

  29. [37]

    On the dangers of stochastic parrots

    Emily Bender et al. On the dangers of stochastic parrots. Proceedings of the ACM Conference on Fairness, Accountability, and Transparency, 2021

    2021

  30. [38]

    Ethical and social risks of harm from language models

    Laura Weidinger et al. Ethical and social risks of harm from language models. arXiv preprint, 2022

    2022

  31. [39]

    Crema: Generalizable and efficient video- language reasoning via multimodal modular fusion

    Shoubin Yu, Jaehong Yoon, and Mohit Bansal. Crema: Generalizable and efficient video- language reasoning via multimodal modular fusion. ICLR, 2025

    2025

  32. [40]

    Pathformer: a biological pathway informed transformer for disease diagnosis and prognosis using multi-omics data

    Xiaofan Liu, Yuhuan Tao, Zilin Cai, Pengfei Bao, Hongli Ma, Kexing Li, Mengtao Li, Yunping Zhu, and Zhi John Lu. Pathformer: a biological pathway informed transformer for disease diagnosis and prognosis using multi-omics data. Bioinformatics, 40(5):btae316, 2024. 7

    2024

  33. [41]

    Multi- modal data integration for precision oncology: Challenges and future directions

    Huajun Zhou, Fengtao Zhou, Chenyu Zhao, Yingxue Xu, Luyang Luo, and Hao Chen. Multi- modal data integration for precision oncology: Challenges and future directions. arXiv preprint arXiv:2406.19611, 2024

    2024 arXiv

  34. [42]

    Multimodal prototyping for cancer survival prediction

    Andrew H Song, Richard J Chen, Guillaume Jaume, Anurag Jayant Vaidya, Alexander Baras, and Faisal Mahmood. Multimodal prototyping for cancer survival prediction. In Forty-first International Conference on Machine Learning

  35. [43]

    Samms: Multi-modality deep learning with the foundation model for the prediction of cancer patient survival

    Wen Zhu, Yiwen Chen, Shanling Nie, and Hai Yang. Samms: Multi-modality deep learning with the foundation model for the prediction of cancer patient survival. In 2023 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pages 3662–3668. IEEE, 2023

    2023

  36. [44]

    A multimodal knowledge-enhanced whole-slide pathology foundation model

    Yingxue Xu, Yihui Wang, Fengtao Zhou, Jiabo Ma, Shu Yang, Huangjing Lin, Xin Wang, Jiguang Wang, Li Liang, Anjia Han, et al. A multimodal knowledge-enhanced whole-slide pathology foundation model. arXiv preprint arXiv:2407.15362, 2024

    2024 arXiv

  37. [45]

    A continual learning survey: Defying forgetting in classification tasks

    Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence, 44(7):3366– 3385, 2021

    2021

  38. [46]

    Overcoming catastrophic forgetting in incremental few-shot learning by finding flat minima

    Guangyuan Shi, Jiaxin Chen, Wenlong Zhang, Li-Ming Zhan, and Xiao-Ming Wu. Overcoming catastrophic forgetting in incremental few-shot learning by finding flat minima. Advances in neural information processing systems, 34:6747–6761, 2021

    2021

  39. [47]

    A comprehensive survey of continual learning: theory, method and application

    Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  40. [48]

    Recent advances of foundation language models-based continual learning: A survey

    Yutao Yang, Jie Zhou, Xuanwen Ding, Tianyu Huai, Shunyu Liu, Qin Chen, Yuan Xie, and Liang He. Recent advances of foundation language models-based continual learning: A survey. ACM Computing Surveys, 2024

    2024

  41. [49]

    Continual learning of large language models: A comprehensive survey

    Haizhou Shi, Zihao Xu, Hengyi Wang, Weiyi Qin, Wenyuan Wang, Yibin Wang, Zifeng Wang, Sayna Ebrahimi, and Hao Wang. Continual learning of large language models: A comprehensive survey. arXiv preprint arXiv:2404.16789, 2024

    2024 arXiv

  42. [50]

    Continual learning for large language models: A survey

    Tongtong Wu, Linhao Luo, Yuan-Fang Li, Shirui Pan, Thuy-Trang Vu, and Gholamreza Haffari. Continual learning for large language models: A survey. arXiv preprint arXiv:2402.01364, 2024

    2024 arXiv

  43. [51]

    Efficient continual pre-training for building domain specific large language models

    Yong Xie, Karan Aggarwal, and Aitzaz Ahmad. Efficient continual pre-training for building domain specific large language models. arXiv preprint arXiv:2311.08545, 2023

    2023

  44. [52]

    Large- scale lifelong learning of in-context instructions and how to tackle it

    Jisoo Mok, Jaeyoung Do, Sungjin Lee, Tara Taghavi, Seunghak Yu, and Sungroh Yoon. Large- scale lifelong learning of in-context instructions and how to tackle it. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) ...

    2023

  45. [53]

    Copf: Continual learning human preference through optimal policy fitting

    Han Zhang, Lin Gui, Yuanzhao Zhai, Hui Wang, Yu Lei, and Ruifeng Xu. Copf: Continual learning human preference through optimal policy fitting. arXiv preprint arXiv:2310.15694, 2023

    2023 arXiv

  46. [54]

    Modality-inconsistent continual learning of multimodal large language models

    Weiguo Pian, Shijian Deng, Shentong Mo, Yunhui Guo, and Yapeng Tian. Modality-inconsistent continual learning of multimodal large language models. arXiv preprint arXiv:2412.13050, 2024

    2024 arXiv

  47. [55]

    Modalprompt: Dual- modality guided prompt for continual learning of large multimodal models

    Fanhu Zeng, Fei Zhu, Haiyang Guo, Xu-Yao Zhang, and Cheng-Lin Liu. Modalprompt: Dual- modality guided prompt for continual learning of large multimodal models. arXiv preprint arXiv:2410.05849, 2024

    2024 arXiv

  48. [56]

    Llm-assisted multi-teacher continual learning for visual question answering in robotic surgery

    Kexin Chen, Yuyang Du, Tao You, Mobarakol Islam, Ziyu Guo, Yueming Jin, Guangyong Chen, and Pheng-Ann Heng. Llm-assisted multi-teacher continual learning for visual question answering in robotic surgery. arXiv preprint arXiv:2402.16664, 2024

    2024 arXiv

  49. [57]

    BLIP-2: Bootstrapping language- image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, P...

    2023

  50. [58]

    marugoto: Machine learning for medical images, 2024

    Kather Lab. marugoto: Machine learning for medical images, 2024

    2024

  51. [59]

    Bulkrnabert: Cancer prognosis from bulk rna-seq based language models

    Maxence Gélard, Guillaume Richard, Thomas Pierrot, and Paul-Henry Cournède. Bulkrnabert: Cancer prognosis from bulk rna-seq based language models. bioRxiv, pages 2024–06, 2024. 9

    2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.