The reviewed record of science sign in
Pith

arxiv: 2606.25579 · v1 · pith:PENIK5EI · submitted 2026-06-24 · eess.IV · cs.CV

Cross-Attention Multimodal Learning for Predicting Response to Neoadjuvant Imatinib in Gastrointestinal Stromal Tumors: A Multicenter Retrospective Study

Fariba Tohidinezhad , Douwe J. Spaanderman , Natalia Oviedo Acosta , Kaouther Mouheb , Karthik Prathaban , David F. Hanff , Dirk J. Gr\"unhagen , Cornelis Verhoef
show 9 more authors
Joris M. van Sabben Evelyne Roets Jette J. Slettenhaar Hans Gelderblom Ingrid M.E. Desar Anna K.L. Reyners Neeltje Steeghs Stefan Klein Martijn P.A. Starmans
This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-06-25 19:37 UTCgrok-4.3pith:PENIK5EIrecord.jsonopen to challenge →

classification eess.IV cs.CV
keywords gastrointestinal stromal tumorsimatinibneoadjuvant therapycross-attentionmultimodal learningcomputed tomographytreatment response predictionexplainable AI
0
0 comments X

The pith

A cross-attention model fusing CT tumor images with clinical variables predicts neoadjuvant imatinib response in GIST, reaching high internal accuracy but lower external performance.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper builds a deep learning system that uses cross-attention to combine computed tomography scans centered on the tumor with patient clinical variables to forecast response to preoperative imatinib in gastrointestinal stromal tumors. Current markers cannot reliably separate responders from non-responders, so the multimodal approach aims to capture interactions across data types that single-modality methods miss. Internal cross-validation produced AUC values up to 0.99, while external testing on held-out centers yielded AUCs of 0.60-0.63, with clinical data contributing more to generalizability than imaging alone. Attention maps further show that features such as specific mutations, age, sex, and comorbidities carry different weight in responders versus non-responders.

Core claim

The cross-attention framework integrating clinical variables and tumor-centered CT imaging was developed to predict response to neoadjuvant imatinib; among 213 patients, it achieved the highest internal performance (AUC up to 0.99) but lower external performance (AUC 0.60-0.63), outperforming clinical-only (AUC 0.66) and imaging-only models, while ablation and attention analyses identified statistically significant differences in feature importance between responders and non-responders including CD117, BRAF, PDGFRA, age, sex, disease status, and comorbidities.

What carries the argument

Cross-attention mechanism that computes interactions between embeddings of clinical variables and features extracted from tumor-centered CT images, allowing each modality to attend to the other during prediction.

If this is right

  • Clinical variables alone achieve moderate prediction accuracy while adding imaging improves internal results but contributes less to external generalization.
  • Responders differ systematically from non-responders in tumor size, mitotic index, and mutation profiles, and these differences are captured by the attention weights.
  • Both self-supervised pretraining with low-rank adaptation and training from scratch can be used, with hyperparameters tuned via SMAC3 to reach the reported internal AUCs.
  • Explainability outputs can quantify modality contributions and highlight patient-level factors that drive treatment response predictions.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If additional data or adaptation techniques raise external AUC closer to internal levels, the model could support decisions on whether to proceed with neoadjuvant imatinib or alternative regimens.
  • The same cross-attention structure could be tested on other kinase inhibitors or tumor types where response to targeted therapy remains hard to predict from standard markers.
  • The identified differences in feature importance between groups suggest follow-up studies to test whether specific clinical variables or mutation subsets can be combined into simpler, non-deep-learning rules for initial screening.

Load-bearing premise

The retrospective data gathered from four centers between 2000 and 2023 is representative of future patients and imaging conditions so that the trained model generalizes beyond the observed internal-to-external performance drop.

What would settle it

A prospective multicenter validation collecting response outcomes on at least 100 new GIST patients where the cross-attention model's AUC falls below 0.55 would show that the claimed predictive improvement does not hold outside the training distribution.

Figures

Figures reproduced from arXiv: 2606.25579 by Anna K.L. Reyners, Cornelis Verhoef, David F. Hanff, Dirk J. Gr\"unhagen, Douwe J. Spaanderman, Evelyne Roets, Fariba Tohidinezhad, Hans Gelderblom, Ingrid M.E. Desar, Jette J. Slettenhaar, Joris M. van Sabben, Kaouther Mouheb, Karthik Prathaban, Martijn P.A. Starmans, Natalia Oviedo Acosta, Neeltje Steeghs, Stefan Klein.

Figure 3
Figure 3. Figure 3: Discrimination (area under the receiver operating characteristic curve, accuracy, sensitivity, and specificity) and calibration (expected calibration error, Brier score, and reliability diagrams) of multimodal and unimodal models in internal and external validation for predicting response to neoadjuvant imatinib in gastrointestinal stromal tumors [PITH_FULL_IMAGE:figures/full_fig_p016_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Explainability of multimodal predictions. A) Clinical feature importance analysis in the development cohort, comparing responders and non￾responders, with significance assessed using Mann–Whitney U tests and false discovery rate correction. B) Representative external true negative (non￾response, top) and true positive (response, bottom) cases with CT, tumor segmentation, and attention overlays highlighting… view at source ↗
read the original abstract

Background: Response to neoadjuvant imatinib in gastrointestinal stromal tumors (GISTs) is highly variable and cannot be reliably predicted using current clinical or molecular markers. This study developed and evaluated an explainable multimodal deep learning framework integrating computed tomography (CT) imaging and clinical variables to predict treatment response. Methods: Patients from four tertiary centers were retrospectively included between 2000-2023 in independent pretraining (n=935) and prediction (n=213) cohorts. A cross-attention framework integrating clinical variables and tumor-centered CT imaging was developed to predict response to neoadjuvant imatinib. Two training strategies were evaluated: (1) self-supervised pretraining with low-rank adaptation and (2) training from scratch. Hyperparameters were optimized using SMAC3. Performance was assessed through internal cross-validation and external testing. Ablation analyses and attention-based explanations were used to quantify modality contributions. Results: Among 213 patients (54.5% responders), responders had larger tumors (112 vs. 89 mm, P=0.026), higher mitotic index (3 vs. 0, P<0.001), and more frequent KIT mutations (69.0% vs. 56.7%, P=0.019). Cross-attention models achieved the highest internal performance (AUC up to 0.99) but lower external performance (AUC 0.60-0.63). Clinical-only performance was moderate (AUC 0.66), whereas imaging-only models showed limited generalizability (AUC 0.56-0.66). Explainability analyses identified significant differences in feature importance between responders and non-responders, including CD117, BRAF, PDGFRA, age, sex, disease status, and comorbidities (FDR-adjusted P<=0.036). Conclusion: The cross-attention framework shows potential for improving imatinib response prediction in GIST while providing interpretable insights into multimodal determinants of treatment response.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 1 minor

Summary. The manuscript presents a cross-attention multimodal deep learning framework that integrates tumor-centered CT imaging with clinical variables to predict response to neoadjuvant imatinib in GIST. Using retrospective multicenter data (pretraining n=935, prediction n=213 from 2000-2023), two training strategies are evaluated, with internal cross-validation yielding AUC up to 0.99 and external testing on independent centers yielding AUC 0.60-0.63. Clinical-only models achieve external AUC 0.66 while imaging-only range 0.56-0.66; ablation and attention-based explainability analyses are included. The authors conclude the framework shows potential for improving prediction and offers interpretable multimodal insights.

Significance. A generalizable multimodal model outperforming clinical baselines could aid treatment stratification in GIST where current markers are unreliable. The reported internal performance, ablation studies, and feature importance differences (e.g., CD117, BRAF) are technically interesting, but the external results limit translational significance.

major comments (3)
  1. [Abstract/Results] Abstract and Results (external testing paragraph): The multimodal cross-attention model reports external AUC 0.60-0.63, which is below the clinical-only baseline of AUC 0.66 on the same held-out multicenter data. This directly contradicts the claim of improvement over existing markers and indicates the model adds no predictive value externally.
  2. [Results/Discussion] Results (performance reporting) and Discussion: The internal-to-external drop (AUC 0.99 to 0.60-0.63) is load-bearing for the generalization claim, yet no quantitative analysis of center-specific or temporal shifts (CT protocols, mutation testing, or selection criteria across 2000-2023) is provided to explain or mitigate it.
  3. [Conclusion] Conclusion: The statement that the framework 'shows potential for improving imatinib response prediction' is not supported by the external metrics, where multimodal performance fails to exceed the clinical-only AUC of 0.66; the conclusion requires revision to align with the reported numbers.
minor comments (1)
  1. [Methods] Methods: The two training strategies (self-supervised pretraining with LoRA vs. from scratch) are described but results do not report separate external AUCs for each, limiting interpretation of which approach drives the reported figures.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive feedback. We agree that the external performance does not support claims of improvement over clinical baselines and have revised the abstract, results, and conclusion accordingly. We also acknowledge the lack of domain-shift analysis as a limitation and have updated the discussion.

read point-by-point responses

  1. Referee: [Abstract/Results] Abstract and Results (external testing paragraph): The multimodal cross-attention model reports external AUC 0.60-0.63, which is below the clinical-only baseline of AUC 0.66 on the same held-out multicenter data. This directly contradicts the claim of improvement over existing markers and indicates the model adds no predictive value externally.

    Authors: We agree that the reported external AUC of 0.60-0.63 for the multimodal model is below the clinical-only AUC of 0.66, indicating no added predictive value in external testing. We have revised the abstract and results sections to remove any implication of improvement over clinical markers and to clearly present the comparative performance metrics. revision: yes

  2. Referee: [Results/Discussion] Results (performance reporting) and Discussion: The internal-to-external drop (AUC 0.99 to 0.60-0.63) is load-bearing for the generalization claim, yet no quantitative analysis of center-specific or temporal shifts (CT protocols, mutation testing, or selection criteria across 2000-2023) is provided to explain or mitigate it.

    Authors: We acknowledge that no quantitative analysis of center-specific or temporal shifts was performed. Detailed metadata on CT protocols and selection criteria variations across the 2000-2023 period is not available in the retrospective dataset, precluding such analysis. We have expanded the discussion to explicitly list this as a limitation and have moderated claims regarding generalizability. revision: partial

  3. Referee: [Conclusion] Conclusion: The statement that the framework 'shows potential for improving imatinib response prediction' is not supported by the external metrics, where multimodal performance fails to exceed the clinical-only AUC of 0.66; the conclusion requires revision to align with the reported numbers.

    Authors: We agree that the original conclusion is not supported by the external results. We have revised the conclusion to state that the framework achieves high internal performance with interpretable insights but that external validation shows performance at or below clinical baselines, indicating the need for additional research to enhance generalizability. revision: yes

Circularity Check

0 steps flagged

No circularity: performance metrics are standard empirical evaluations on held-out splits, not reductions by construction

full rationale

The paper reports an empirical ML study with internal cross-validation and external testing on independent multicenter retrospective cohorts (2000-2023). AUC values (internal up to 0.99, external 0.60-0.63, clinical-only 0.66) are direct outputs of supervised training and evaluation on data splits, not self-definitional or fitted inputs renamed as predictions. No equations, ansatzes, or uniqueness theorems are invoked; the derivation chain consists of standard model training, ablation, and attention analysis without load-bearing self-citations or imported results that reduce to the target claim. The observed performance drop and lack of superiority over clinical baseline are issues of generalizability and correctness, not circularity in the reported metrics or framework. The abstract and described methods are self-contained against external benchmarks via the external test sets.

Axiom & Free-Parameter Ledger

1 free parameters · 2 axioms · 0 invented entities

The central claim rests on standard deep-learning assumptions about data representativeness and feature sufficiency; no new entities are introduced. Hyperparameter optimization introduces fitted values whose effect on external performance cannot be isolated from the abstract alone.

free parameters (1)
  • model hyperparameters
    Optimized via SMAC3 on the prediction cohort; values not reported.
axioms (2)
  • domain assumption Retrospective data from 2000-2023 across four centers is representative of future patients
    Invoked by the study design and external testing protocol.
  • domain assumption The selected clinical variables and tumor-centered CT features contain the relevant signals for response prediction
    Required for the cross-attention fusion to be meaningful.

pith-pipeline@v0.9.1-grok · 6003 in / 1480 out tokens · 43018 ms · 2026-06-25T19:37:55.823596+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

12 extracted references

  1. [1]

    (2021) Gastrointestinal stromal tumours

    (1) Blay, J.-Y ., Kang, Y .-K., Nishida, T., and von Mehren, M. (2021) Gastrointestinal stromal tumours. Nature Reviews. Disease Primers 7,

    2021

  2. [2]

    (2021) Contribution of Interstitial Cells of Cajal to Gastrointestinal Stromal Tumor Risk

    (2) Wang, Q., Huang, Z.-P ., Zhu, Y ., Fu, F ., and Tian, L. (2021) Contribution of Interstitial Cells of Cajal to Gastrointestinal Stromal Tumor Risk. Medical Science Monitor: International Medical Journal of Experimental and Clinical Research 27, e929575. (3) Corless, C. L., Fletcher, J. A., and Heinrich, M. C. (2004) Biology of gastrointestinal stromal...

    2021

  3. [3]

    (7) Chen, P ., Zong, L., Zhao, W., and Shi, L

    Annals of Surgical Oncology 19, 1074–1080. (7) Chen, P ., Zong, L., Zhao, W., and Shi, L. (2010) Efficacy evaluation of imatinib treatment in patients with gastrointestinal stromal tumors: A meta-analysis. World Journal of Gastroenterology : WJG 16, 4227–4232. (8) Lee, J.-H., Kim, Y ., Choi, J.-W., and Kim, Y .-S. (2013) Correlation of imatinib resistance...

    2010

  4. [4]

    (2023) Multi-modal cross-attention network for Alzheimer’s disease diagnosis with multi-modality data

    19 (12) Zhang, J., He, X., Liu, Y ., Cai, Q., Chen, H., and Qing, L. (2023) Multi-modal cross-attention network for Alzheimer’s disease diagnosis with multi-modality data. Computers in Biology and Medicine 162, 107050. (13) Zhao, W., Huang, Z., Tang, S., Li, W., Gao, Y ., Hu, Y ., Fan, W., Cheng, C., Yang, Y ., Zheng, H., Liang, D., and Hu, Z. (2024) MMCA...

    2023

  5. [5]

    arXiv.org

    nnInteractive: Redefining 3D Promptable Segmentation. arXiv.org. (18) de Vente, C., Venkadesh, K. V ., van Ginneken, B., and Sánchez, C. I. (2025, April

    2025

  6. [6]

    arXiv.org

    SlicerNNInteractive: A 3D Slicer extension for nnInteractive. arXiv.org. (19) Attention Is All You Need - arXiv.gg. (20) Golts, A., Raboh, M., Shoshan, Y ., Polaczek, S., Rabinovici-Cohen, S., and Hexter, E. (2023) FuseMedML: a framework for accelerated discovery in machine learning based biomedicine. Journal of Open Source Software 8,

    2023

  7. [7]

    (2022) Masked Autoencoders Are Scalable Vision Learners, in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 15979–15988

    (21) He, K., Chen, X., Xie, S., Li, Y ., Dollár, P ., and Girshick, R. (2022) Masked Autoencoders Are Scalable Vision Learners, in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 15979–15988. (22) Wasserthal, J., Breit, H.-C., Meyer, M. T., Pradella, M., Hinck, D., Sauter, A. W., Heye, T., Boll, D. T., Cyriac, J., Yang, S., ...

    2022

  8. [8]

    (2022) SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization

    (25) Lindauer, M., Eggensperger, K., Feurer, M., Biedenkapp, A., Deng, D., Benjamins, C., Ruhkopf, T., Sass, R., and Hutter, F . (2022) SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization. Journal of Machine Learning Research 23, 1–9. (26) Guo, C., Pleiss, G., Sun, Y ., and Weinberger, K. Q. (2017) On Calibration of Modern Neu...

    2022

  9. [9]

    (27) Wiegreffe, S., and Pinter, Y

    PMLR. (27) Wiegreffe, S., and Pinter, Y . (2019, August

    2019

  10. [10]

    arXiv.org

    Attention is not not Explanation. arXiv.org. (28) Wong, N. A. C. S., Garcia-Petit, C., Dangoor, A., and Andrew, N. (2022) A literature review and database of how the primary KIT/PDGFRA variant of a gastrointestinal stromal tumour predicts for sensitivity to imatinib. Cancer Genetics 268–269, 46–54. (29) Kirby, R. (2005) PDGFRA mutations and imatinib sensi...

    2022

  11. [11]

    C., Bordoni, A., Saletti, P ., Mazzucchelli, L., Pilotti, S., Pierotti, M

    (31) Miranda, C., Nucifora, M., Molinari, F ., Conca, E., Anania, M. C., Bordoni, A., Saletti, P ., Mazzucchelli, L., Pilotti, S., Pierotti, M. A., Tamborini, E., Greco, A., and Frattini, M. (2012) KRAS and BRAF Mutations Predict Primary Resistance to Imatinib in Gastrointestinal Stromal Tumors. Clinical Cancer Research 18, 1769–1776. (32) Cavnar, M. J., ...

    2012

  12. [12]

    Transfusion: Understanding Transfer Learning for Medical Imaging. arXiv. (38) Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., and Wichmann, F . A. (2020) Shortcut learning in deep neural networks. Nature Machine Intelligence 2, 665–673. 21 Table S1. Hyperparameter search space explored during SMAC optimization for the fine...

    2020