{"id":"8a731d48-5494-404b-9d7e-376d7737bc13","arxiv_id":"2606.25579","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"Multimodal cross-attention DL model for predicting imatinib response in GIST achieves high internal AUC but only 0.60-0.63 externally, with clinical-only baseline at 0.66.","lead":"This paper developed a cross-attention deep learning model combining CT images and clinical data to predict response to neoadjuvant imatinib in gastrointestinal stromal tumors. A smart generalist might read it to see the practical challenges of applying multimodal AI to rare cancers where existing markers are unreliable.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"External AUC 0.60-0.63 not superior to clinical-only AUC 0.66 undermines 'improving prediction' claim","rationale":"The reader's weakest assumption (representativeness for generalization) directly matches the load-bearing issue. The external numbers provide concrete evidence that the model does not improve prediction on new centers, so the verdict should move from UNVERDICTED to CONDITIONAL pending stronger external validation or explicit comparison showing superiority.","tokens_in":1870,"tokens_out":342,"duration_ms":14678,"concrete_test":"On the external cohort, recompute AUC for the best cross-attention model versus the clinical-only model and apply DeLong's test (or 1000-bootstrap CI); if multimodal AUC is not significantly higher (p>0.05 or CI overlap), the improvement claim fails.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim requires that the cross-attention multimodal model improves upon existing markers for imatinib response prediction. Internal CV reaches AUC 0.99, but external testing on independent centers yields only 0.60-0.63. The abstract reports clinical-only performance at AUC 0.66 (with imaging-only at 0.56-0.66), so the multimodal external result shows no added value. For the claim to hold, external performance must exceed the clinical baseline on held-out multicenter data; the observed numbers indicate the model captures center-specific patterns rather than generalizable multimodal signals. The 2000-2023 span across four centers makes unaddressed shifts in CT protocols, mutation testing, or patient selection the most plausible explanation for the drop.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript presents a cross-attention multimodal deep learning framework that integrates tumor-centered CT imaging with clinical variables to predict response to neoadjuvant imatinib in GIST. Using retrospective multicenter data (pretraining n=935, prediction n=213 from 2000-2023), two training strategies are evaluated, with internal cross-validation yielding AUC up to 0.99 and external testing on independent centers yielding AUC 0.60-0.63. Clinical-only models achieve external AUC 0.66 while imaging-only range 0.56-0.66; ablation and attention-based explainability analyses are included. The authors conclude the framework shows potential for improving prediction and offers interpretable multimodal insights.","tokens_in":2064,"tokens_out":545,"duration_ms":17663,"significance":"A generalizable multimodal model outperforming clinical baselines could aid treatment stratification in GIST where current markers are unreliable. The reported internal performance, ablation studies, and feature importance differences (e.g., CD117, BRAF) are technically interesting, but the external results limit translational significance.","major_comments":[{"comment":"Abstract and Results (external testing paragraph): The multimodal cross-attention model reports external AUC 0.60-0.63, which is below the clinical-only baseline of AUC 0.66 on the same held-out multicenter data. This directly contradicts the claim of improvement over existing markers and indicates the model adds no predictive value externally.","section":"Abstract/Results"},{"comment":"Results (performance reporting) and Discussion: The internal-to-external drop (AUC 0.99 to 0.60-0.63) is load-bearing for the generalization claim, yet no quantitative analysis of center-specific or temporal shifts (CT protocols, mutation testing, or selection criteria across 2000-2023) is provided to explain or mitigate it.","section":"Results/Discussion"},{"comment":"Conclusion: The statement that the framework 'shows potential for improving imatinib response prediction' is not supported by the external metrics, where multimodal performance fails to exceed the clinical-only AUC of 0.66; the conclusion requires revision to align with the reported numbers.","section":"Conclusion"}],"minor_comments":[{"comment":"Methods: The two training strategies (self-supervised pretraining with LoRA vs. from scratch) are described but results do not report separate external AUCs for each, limiting interpretation of which approach drives the reported figures.","section":"Methods"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback. We agree that the external performance does not support claims of improvement over clinical baselines and have revised the abstract, results, and conclusion accordingly. We also acknowledge the lack of domain-shift analysis as a limitation and have updated the discussion.","responses":[{"response":"We agree that the reported external AUC of 0.60-0.63 for the multimodal model is below the clinical-only AUC of 0.66, indicating no added predictive value in external testing. We have revised the abstract and results sections to remove any implication of improvement over clinical markers and to clearly present the comparative performance metrics.","revision_made":"yes","referee_comment":"[Abstract/Results] Abstract and Results (external testing paragraph): The multimodal cross-attention model reports external AUC 0.60-0.63, which is below the clinical-only baseline of AUC 0.66 on the same held-out multicenter data. This directly contradicts the claim of improvement over existing markers and indicates the model adds no predictive value externally."},{"response":"We acknowledge that no quantitative analysis of center-specific or temporal shifts was performed. Detailed metadata on CT protocols and selection criteria variations across the 2000-2023 period is not available in the retrospective dataset, precluding such analysis. We have expanded the discussion to explicitly list this as a limitation and have moderated claims regarding generalizability.","revision_made":"partial","referee_comment":"[Results/Discussion] Results (performance reporting) and Discussion: The internal-to-external drop (AUC 0.99 to 0.60-0.63) is load-bearing for the generalization claim, yet no quantitative analysis of center-specific or temporal shifts (CT protocols, mutation testing, or selection criteria across 2000-2023) is provided to explain or mitigate it."},{"response":"We agree that the original conclusion is not supported by the external results. We have revised the conclusion to state that the framework achieves high internal performance with interpretable insights but that external validation shows performance at or below clinical baselines, indicating the need for additional research to enhance generalizability.","revision_made":"yes","referee_comment":"[Conclusion] Conclusion: The statement that the framework 'shows potential for improving imatinib response prediction' is not supported by the external metrics, where multimodal performance fails to exceed the clinical-only AUC of 0.66; the conclusion requires revision to align with the reported numbers."}],"tokens_in":1623,"tokens_out":541,"duration_ms":21207,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The external numbers are the main thing here. The cross-attention model reaches AUC 0.60-0.63 on independent centers while the clinical-only baseline sits at 0.66, and imaging-only does no better. That gap means the work does not deliver evidence that the multimodal approach adds usable signal for this prediction task.\n\nWhat is new is the specific use of cross-attention on CT plus clinical variables to predict neoadjuvant imatinib response in GIST, with data pulled from four centers over 2000-2023. They split into a large pretraining set and a 213-patient prediction cohort, ran both self-supervised pretraining with LoRA and training from scratch, and included ablation checks plus attention-based explanations.\n\nThe paper handles the data collection reasonably for a rare tumor and reports external testing plus some interpretability work on features like KIT mutations, age, and comorbidities. Those steps are standard but executed on real multicenter retrospective records.\n\nThe clear soft spot is the internal-to-external drop from 0.99 down to 0.60-0.63 combined with the lack of gain over the clinical baseline externally. This pattern usually means the model is fitting center-specific artifacts in the CT scans or selection effects rather than learning generalizable multimodal relationships. The 23-year span across sites makes unaddressed changes in protocols or mutation testing the most likely driver, and nothing in the reported results closes that gap.\n\nThis is for readers who track AI applications in rare oncology and want to see how multimodal methods behave on actual external data. It illustrates the generalization problem more than it solves the clinical question.\n\nIt deserves peer review because the dataset and task are new enough that referees can usefully press on the external validation design and whether stronger domain adaptation or different fusion methods would change the outcome.","headline":"External results show the multimodal model at 0.60-0.63 AUC does not beat the clinical-only baseline of 0.66, so the improvement claim does not hold.","tokens_in":2637,"tokens_out":451,"would_cite":false,"duration_ms":12761,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"A cross-attention model fusing CT tumor images with clinical variables predicts neoadjuvant imatinib response in GIST, reaching high internal accuracy but lower external performance.","keywords":["gastrointestinal stromal tumors","imatinib","neoadjuvant therapy","cross-attention","multimodal learning","computed tomography","treatment response prediction","explainable AI"],"falsifier":"A prospective multicenter validation collecting response outcomes on at least 100 new GIST patients where the cross-attention model's AUC falls below 0.55 would show that the claimed predictive improvement does not hold outside the training distribution.","tokens_in":2798,"feed_emoji":"🧬","tokens_out":807,"duration_ms":27332,"temperature":0.7,"pith_summary":"The paper builds a deep learning system that uses cross-attention to combine computed tomography scans centered on the tumor with patient clinical variables to forecast response to preoperative imatinib in gastrointestinal stromal tumors. Current markers cannot reliably separate responders from non-responders, so the multimodal approach aims to capture interactions across data types that single-modality methods miss. Internal cross-validation produced AUC values up to 0.99, while external testing on held-out centers yielded AUCs of 0.60-0.63, with clinical data contributing more to generalizability than imaging alone. Attention maps further show that features such as specific mutations, age, sex, and comorbidities carry different weight in responders versus non-responders.","feed_headline":"Cross-attention model predicts imatinib response in GIST","feed_subtitle":"CT scans plus clinical data reach internal AUC 0.99 but drop to 0.60-0.63 on external centers, with attention maps showing key differing fea","key_machinery":"Cross-attention mechanism that computes interactions between embeddings of clinical variables and features extracted from tumor-centered CT images, allowing each modality to attend to the other during prediction.","core_discovery":"The cross-attention framework integrating clinical variables and tumor-centered CT imaging was developed to predict response to neoadjuvant imatinib; among 213 patients, it achieved the highest internal performance (AUC up to 0.99) but lower external performance (AUC 0.60-0.63), outperforming clinical-only (AUC 0.66) and imaging-only models, while ablation and attention analyses identified statistically significant differences in feature importance between responders and non-responders including CD117, BRAF, PDGFRA, age, sex, disease status, and comorbidities.","pith_inferences":["If additional data or adaptation techniques raise external AUC closer to internal levels, the model could support decisions on whether to proceed with neoadjuvant imatinib or alternative regimens.","The same cross-attention structure could be tested on other kinase inhibitors or tumor types where response to targeted therapy remains hard to predict from standard markers.","The identified differences in feature importance between groups suggest follow-up studies to test whether specific clinical variables or mutation subsets can be combined into simpler, non-deep-learning rules for initial screening."],"forward_implications":["Clinical variables alone achieve moderate prediction accuracy while adding imaging improves internal results but contributes less to external generalization.","Responders differ systematically from non-responders in tumor size, mitotic index, and mutation profiles, and these differences are captured by the attention weights.","Both self-supervised pretraining with low-rank adaptation and training from scratch can be used, with hyperparameters tuned via SMAC3 to reach the reported internal AUCs.","Explainability outputs can quantify modality contributions and highlight patient-level factors that drive treatment response predictions."],"fun_headline_variants":["Cross-attention achieves 0.99 AUC internally for GIST imatinib prediction","Internal AUC 0.99 but external 0.63 for GIST imatinib prediction model","Cross-attention model hits 0.99 AUC but generalizes poorly externally","Attention maps highlight differing features in GIST imatinib responders"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The retrospective data gathered from four centers between 2000 and 2023 is representative of future patients and imaging conditions so that the trained model generalizes beyond the observed internal-to-external performance drop.","fun_headline_variants_meta":{"raw":{"variants":["Cross-attention achieves 0.99 AUC internally for GIST imatinib prediction","Internal AUC 0.99 but external 0.63 for GIST imatinib prediction model","Cross-attention model hits 0.99 AUC but generalizes poorly externally","Attention maps highlight differing features in GIST imatinib responders"]},"model":"grok-4.3","cost_usd":0.014419,"raw_usage":{"total_tokens":6310,"prompt_tokens":870,"num_sources_used":0,"completion_tokens":82,"cost_in_usd_ticks":144187000,"prompt_tokens_details":{"text_tokens":870,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":5358,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":870,"tokens_out":82,"duration_ms":27997,"temperature":1.0,"reasoning_tokens":5358,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-25T19:37:55.823596+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A prospective multicenter validation collecting response outcomes on at least 100 new GIST patients where the cross-attention model's AUC falls below 0.55 would show that the claimed predictive improvement does not hold outside the training distribution.","supporting_citations":[],"review_version":1}