Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

MedTVT-R1: A Multimodal LLM Empowering Medical Reasoning and Diagnosis

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a 1-billion-parameter model trained to fuse ECG, chest X-ray, and laboratory data can reason through and diagnose seven diseases jointly, beating larger multimodal models on accuracy and language quality.

desk verdict A genuinely new three-modality medical QA dataset and a sensible integration pipeline, but the headline performance claim is not supported because every baseline is zero-shot while the proposed model is trained on the task. read the letter →

arxiv 2506.18512 v1 pith:RKC7WFNR submitted 2025-06-23 eess.IV cs.CLcs.CVq-bio.QM

Yuting Zhang , Kaishen Yuan , Hao Lu , Yutao Yue , Jintai Chen , Kaishun Wu This is my paper · ORCID
classification eess.IVcs.CLcs.CVq-bio.QM
keywords multimodallargelanguagemodelmulti-diseasediagnosischainofevidenceGRPOreinforcementfine-tuningJaccardrewardECGtimeserieschestX-rayimaginglaboratorytabulardata
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a 1-billion-parameter multimodal language model can combine three heterogeneous clinical data streams—electrocardiogram (ECG) time series, chest X-ray (CXR) images, and laboratory test tables—to reason through and jointly diagnose seven diseases, and that it beats larger general-purpose multimodal models on both language quality and clinical accuracy. The claim rests on three ingredients: a new instruction dataset, MedTVT-QA, whose answers are physiological explanations plus a GPT-4o-written 'Chain of Evidence' citing corroborating findings from each modality for every disease in the patient's set; a modality perception layer that cross-attends the three streams and adaptively weights each modality's contribution; and reinforcement fine-tuning in which a Jaccard reward scores the overlap between predicted and true disease sets. On a 375-case test split the model reports 0.519 F1 and 0.655 AUC, about 2.6 times the F1 of the best zero-shot baseline. If the numbers hold, interpretable comorbidity reasoning does not require a large model: a small one can be taught to triangulate evidence across time series, images, and tables.

What carries the argument

Three named components carry the argument. First, MedTVT-QA, an instruction dataset of 8,706 multimodal patient data combinations whose GPT-4o-written answers come in two layers: per-modality physiological explanations and a disease-level Chain of Evidence, a written justification that must cite corroborating findings from ECG, CXR, and lab data for each disease in a given set. Second, the Modality Perception Layer (MPL), a fusion block with two parts: Cyclic Multi-Head Attention, in which each modality's features cyclically take turns as query, key, and value so the three streams exchange information, and a Contribution-Aware Operator, a sigmoid gate that learns to weight each modality's contribution per diagnostic question (ECG more heavily for coronary artery disease, for instance). Third, the Jaccard Reward, the intersection-over-union of the predicted and ground-truth disease sets, which is used inside GRPO to push the model toward complete and precise sets of diseases rather than a single label.

What would settle it

Regenerate the disease-level training data with the identical prompts but without the {result_diseases} field, so that GPT-4o must infer each patient's diseases from the three modality reports on its own; then train the identical pipeline on that data and measure diagnostic F1 on the same 375-case test split. If the score collapses toward the zero-shot baselines, the Chain of Evidence was leaking the answer labels and the model learned post-hoc justification rather than evidence-based inference. A supplementary check: strip the <think> reasoning block from the model's outputs at inference; a large drop in answer-set accuracy would indicate the reasoning text itself is doing label-carrying work.

Watch

Extended reading notes

Core claim

MedTVT-R1 is presented as the first MLLM framework that integrates ECG, CXR, and laboratory data for interpretable multi-disease diagnosis rather than single-modality, single-disease classification. The authors build MedTVT-QA from MIMIC-IV records that keep clinical temporal order (ECG and lab tests from the first 24 hours of admission, chest X-rays from 24 to 72 hours), and they ask GPT-4o to expand physiological labels into long-form explanations and then to write a Chain of Evidence that justifies the patient's ICD-derived disease set using mutually corroborating findings from all three modalities. The model encodes each modality separately, fuses the streams through the modality perception layer, and is post-trained with GRPO, whose Jaccard reward measures intersection-over-union between predicted and ground-truth disease sets. In experiments against eight open multimodal models ranging from 1B to 8B parameters, MedTVT-R1 reports the highest scores on both language-quality metrics (BLEU, METEOR, ROUGE, BERTScore) and clinical metrics (precision, recall, F1, AUC), and it also leads on physiological-level description for each single modality. The ablation results are read by the authors as showing that physiological pre-training, the perception layer, all three modalities, and the reinforcement stage each contribute to the margin.

Load-bearing premise

The load-bearing premise is that a Chain of Evidence written by GPT-4o after being given the correct disease list is genuine diagnostic reasoning, so that training on it teaches the model to reason from the data rather than to restate labels it was told to justify.

Editorial extensions

If this is right

  • A 1B-parameter model with LoRA can produce structured, interpretable, multi-disease diagnostic text that beats much larger zero-shot multimodal models on this task, suggesting that model scale is not the binding constraint for this kind of clinical reasoning.
  • The Jaccard reward generalizes to any multi-label diagnosis setting: any task whose answer is a set of conditions can be scored by set overlap and optimized with GRPO.
  • The modality perception layer yields a per-case weighting of ECG, CXR, and lab contributions, so the model can in principle indicate which data source most strongly supports each diagnosis.
  • The paper's ablations imply that removing any single modality degrades performance, with ECG removal hurting most, so the three streams carry complementary rather than redundant evidence.
  • The three-stage recipe—physiological pre-training, disease-level supervised fine-tuning, then GRPO—transfers to other multimodal clinical inputs, such as alternative imaging modalities or additional time series.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the Chain of Evidence was written by GPT-4o after being handed the true disease list and told to prove exactly those diseases and no others, the dataset teaches retroductive justification rather than open-ended inference; a direct test is to regenerate the training data without revealing the disease labels and check whether the model's diagnostic F1 survives.
  • A clinically motivated variant of the Jaccard reward could weight severe conditions (for example, sepsis) more heavily than mild ones, changing the model's error profile toward triage priorities.
  • The Contribution-Aware Operator's learned weights could double as an explainability artifact—showing a clinician which modality drove a diagnosis—and a natural study is whether presenting those weights improves clinician trust or helps catch model errors.
  • The paper's limitation section flags missing modalities; medical history is the most natural addition, since the ICD-based ground-truth labels themselves encode information a clinician would have drawn from history the model never sees.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes MedTVT-R1, a multimodal large language model that integrates ECG time series, chest X-ray images, and blood-test tabular data for multi-disease diagnosis and interpretable reasoning. The authors construct MedTVT-QA, an instruction dataset derived from MIMIC-IV with GPT-4o-generated physiological-level reports and disease-level 'Chain of Evidence' reasoning, and train a LLaMA3.2-1B backbone with a modality perception layer (CMHA and CAO) and GRPO-based reinforcement fine-tuning with a Jaccard reward. The paper claims state-of-the-art performance against eight existing MLLMs in both disease-level diagnosis and physiological-level understanding, with ablation studies supporting the contributions of pre-training, RFT, MPL components, and each modality.

Significance. If the claims held, the work would be a useful step toward interpretable multimodal medical diagnosis: the dataset construction across three heterogeneous modalities is a genuine contribution, the MPL architecture is a plausible design, and the open release of dataset and code supports reproducibility. The ablations of CMHA, CAO, and modality missingness are informative. However, the two load-bearing claims—that MedTVT-R1 outperforms existing MLLMs and that its generated reasoning reflects diagnostic inference—are undermined by a circular training-target construction and by an evaluation that compares a fine-tuned model against zero-shot baselines. These issues affect the central contribution and would require substantial rework of the dataset and evaluation protocol to resolve.

major comments (4)
  1. [§3.1 Disease-QA Prompt] The Chain of Evidence target is generated by GPT-4o after being given the ground-truth disease set and explicit instruction to 'prove the reasons why the patient has the diseases I provided' and to 'not mention any diseases other than those I provided'. The resulting text is therefore a post-hoc justification conditioned on the answer, not an inference from the multimodal inputs. Training the model to reproduce this text and then evaluating its diagnostic agreement against the same labels measures how well the model can reproduce the training distribution, not the quality of medical reasoning. This is a load-bearing circularity for the paper's central claim of interpretable multi-disease reasoning.
  2. [§4.2, Table 1] The headline superiority claim is confounded: MedTVT-R1 is trained on MedTVT-QA through PT, SFT, and RFT, while all eight baseline MLLMs are evaluated zero-shot via SWIFT. No baseline receives the same fine-tuning intervention, the Chain-of-Evidence formatting, or the GRPO/Jaccard reward. The reported F1/AUC gains (0.5190/0.6554 vs. the best zero-shot baseline 0.1995/0.5053) therefore may reflect task-specific training rather than architectural or methodological superiority. A fair comparison requires at least the strongest baseline to be fine-tuned on the same training data; otherwise the claim 'outperforms these leading models' is not supported.
  3. [§4.2, Table 2; §4.1 Metrics] The physiological-level comparison has the same confound as Table 1: the test QA pairs for ECG, CXR, and LAB are drawn from MedTVT-QA, and MedTVT-R1 was pre-trained and SFT-tuned on this exact dataset, while the baselines are zero-shot. In addition, the test set is only 375 samples, and the paper provides no error bars, confidence intervals, or multiple-seed runs for the RL/SFT training. With high-variance training procedures and a small test set, the reported differences cannot be judged statistically reliable. The evaluation protocol should report variance and significance.
  4. [§4.3, Figure 4; Appendix E] The qualitative example in Figure 4 is presented as evidence of evidence-based reasoning, but the reasoning text closely follows the style and content of the GPT-4o-generated Chain-of-Evidence targets, which were produced from the answer labels. The limitation section acknowledges data scarcity and missing modalities but does not mention the circular construction of the reasoning targets or the zero-shot baseline comparison. Given that these are the two central limitations of the empirical study, they should be disclosed prominently rather than omitted.
minor comments (5)
  1. [§4.1 Training Details] The conversion of ECG signals to images and LAB tabular data to text for the baseline MLLMs is described only briefly; the exact input format, resolution, and prompt templates used for baselines should be specified to ensure comparability and reproducibility.
  2. [Figure 2(b)] The disease distribution bar chart lists numbers (e.g., 3893, 3081, 6456, 3298, etc.) that do not match the ICD-10 subtype counts reported in Table 5; please reconcile these statistics.
  3. [Appendix B, Figure 5] There is a typo: 'MIMIX-IV-CXR-report' should be 'MIMIC-IV-CXR-report'.
  4. [Appendix C, CXR-QA Example] The CXR-QA example appears to be generated from a text report rather than from the image; clarify whether the physiological-level CXR QA pairs are based on image features, report text, or both.
  5. [Introduction] The claim that MedTVT-QA is 'the first medical instruction dataset' with these three modalities is strong; please contrast with existing multimodal medical instruction datasets to justify this claim.

Circularity Check

1 steps flagged · score 6.0 of 10

The Chain-of-Evidence training target is generated from the ground-truth disease list, so the paper's interpretable-reasoning claim is partially circular; the disease-classification result itself is standard supervised learning.

  1. fitted input called prediction [Section 3.1 Disease-QA Prompt, Appendix A; used as SFT target (Eq. 5), NLG reference (Table 1), and Jaccard reward (Eq. 7)]
    "Please find definitive evidence from the ECG, CXR, and blood test results, leveraging the complementarity and mutual corroboration of these three modalities, to robustly prove the reasons why the patient has the diseases I provided. Your response must include every disease I provided, using the exact wording I provided, and you must not mention any diseases other than those I provided."

    The Disease-QA Prompt supplies GPT-4o with the exact disease set in its Task Description (Diseases:{result_diseases}) and instructs it to write a Chain of Evidence that includes exactly those diseases and excludes all others. That label-conditioned text becomes the supervised target for SFT (Eq. 5), the reference for the NLG metrics in Table 1, and the basis of the Jaccard Reward (Eq. 7) during RFT. The model is therefore trained and scored for reproducing rationalizations written from the final answer list, not for independently deriving diagnoses from the multimodal inputs. The claimed 'interpretable multi-disease reasoning' reduces by construction to matching a text that was generated after the ground-truth labels were revealed.

full rationale

MedTVT-R1's disease classification is a standard supervised split: the ICD-derived labels are external ground truth, and training on 8,331 samples and testing on 375 samples from the same MIMIC-derived distribution is a normal evaluation. That part is not circular, and the architecture contributions (MPL/CMHA/CAO) are tested by internal ablations rather than by self-citation or a load-bearing uniqueness theorem. However, the paper's central interpretability claim—the Chain of Evidence—is partially circular. Section 3.1's Disease-QA Prompt gives GPT-4o the exact disease set, tells it to pretend the modality reports are its own interpretation, and forbids mentioning any disease outside the provided list. The resulting CoE text is then the SFT target, the NLG reference, and the anchor of the Jaccard reward. Consequently, the model is trained and evaluated for generating label-conditioned rationalizations, not for demonstrating independent diagnostic reasoning from raw data. This is a fitted-input-called-prediction loop: the answer list is used to write the reasoning text, and the reasoning text is then used to score the model's reasoning. The zero-shot baseline comparison in Table 1 is an additional evaluation confound (task-tuned versus untuned models), but it is a fairness issue rather than a circularity. Overall score 6 reflects that one core 'reasoning' prediction reduces by construction, while the disease classification accuracy and the internal ablations still have independent content.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim rests on empirical assumptions about the quality of GPT-4o-generated labels and reasoning text, the sufficiency of the three modalities for the seven diseases, and the comparability of the evaluation. No mathematical free parameters are introduced; the model weights are trained on data.

assumptions (4)
  • domain assumption GPT-4o-generated physiological-level reports are clinically accurate after professional review.
    Section 3.1 states the content is manually reviewed, but the accuracy of the underlying GPT-4o synthesis is taken as ground truth for training.
  • ad hoc to paper The Chain of Evidence text generated from the true disease labels is a valid target for diagnostic reasoning.
    The Disease-QA prompt instructs GPT-4o to prove exactly the provided diseases and to mention no others, so the reasoning text is answer-conditioned by construction.
  • domain assumption MIMIC-IV ECG, CXR, and lab data are sufficient to diagnose the seven selected diseases.
    The authors filter diseases "for which evidence could not be found" in the three modalities, implying sufficiency rather than demonstrating it.
  • domain assumption CXR images contain the information present in the radiologist reports used to generate CXR-QA.
    CXR physiological QA is generated from the text report, while the model only sees the image, so the model must extract the same information from pixels.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MedTVT-R1: A Multimodal LLM Empowering Medical Reasoning and Diagnosis." pith.science (2026). https://pith.science/paper/RKC7WFNR

@misc{pith2026250618512,
  author       = {Pith},
  title        = {Pith review of: MedTVT-R1: A Multimodal LLM Empowering Medical Reasoning and Diagnosis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RKC7WFNR}},
  note         = {Machine review of arXiv:2506.18512}
}
read the original abstract

Accurate and interpretable multi-disease diagnosis remains a critical challenge in medical research, particularly when leveraging heterogeneous multimodal medical data. Current approaches often rely on single-modal data, limiting their ability to comprehensively understand complex diseases. To address this, we propose MedTVT-R1, a novel Multimodal Large Language Model (MLLM) framework designed to integrate clinical multimodal data for reasoning and diagnosing multiple diseases. We construct MedTVT-QA, a curated instruction dataset that provides question-answer pairs for physiological-level interpretations and disease-level diagnoses with a Chain of Evidence approach. MedTVT-R1 incorporates a modality perception layer to capture inter-modal dependencies and adaptively weight modality contributions. Additionally, we employ Group Relative Policy Optimization (GRPO)-based Reinforcement Fine-Tuning with a Jaccard Reward function to enhance diagnostic reasoning. Experimental results demonstrate MedTVT-R1's superiority in multimodal feature utilization and multi-disease diagnosis, offering significant potential for clinical applications such as diagnostic report generation and comorbidity reasoning. The dataset and code are available at https://github.com/keke-nice/MedTVT-R1.

Figures

Figures reproduced from arXiv: 2506.18512 by the authors.

Figure 1
Figure 1. Overview of MedTVT-R1: MedTVT-R1 seamlessly integrates Electrocardiogram (Time Series), Chest X-ray (Visual Image), and Blood Test (Tabular Data) to deliver comprehensive long-text Medical reasoning and diagnosis across various diseases. Abstract Accurate and interpretable multi-disease diagnosis remains a critical challenge in medical research, particularly when leveraging heterogeneous multimodal medical data. Cur… view at source ↗
Figure 2
Figure 2. MedTVT-QA dataset construction and disease distribution. (a) Pipeline of Dataset Construction: labels are refined to ensure consistency, prompts guide GPT-4o in generating verified physiological-level reports, which are combined with diagnostic labels to produce disease-level reports. Diagnostic labels are organized into seven primary categories with detailed subtypes. (b) Disease distribution of MedTVT-QA, with sub… view at source ↗
Figure 3
Figure 3. Pipeline of MedTVT-R1. Pretraining processes ECG, CXR, and LAB data through encoders and projectors, combined with prompts, to train projectors and LLM’s LoRA for enhanced physiological understanding. The SFT stage adds a Modality Perception Layer for interaction and integration, refining disease analysis. The RFT stage applies GRPO, using the SFT-trained model for policy and inference, optimizing KL divergence and … view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Inference example of MedTVT-R1: Blue highlights "evidence words," while green denotes [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: An CXR report example from MIMIX-IV-CXR-report dataset. [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: Performance of various MLLMs in handling medical reasoning and diagnosis. MedTVT-R1 [PITH_FULL_IMAGE:figures/full_fig_p020_6.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CoEmoGen: Towards Semantically-Coherent and Scalable Emotional Image Content Generation

    cs.CV 2025-08 unverdicted novelty 6.0 of 10

    CoEmoGen generates emotionally faithful images from emotion categories using MLLM-crafted captions and a hierarchical LoRA module, validated on a new EmoArt dataset.

Reference graph

Works this paper leans on

82 extracted references · 49 canonical work pages · cited by 1 Pith paper

  1. [1]

    Abdelaziz, T

    M. Abdelaziz, T. Wang, and A. Elazab. Alzheimer’s disease diagnosis framework from incom- plete multimodal data using convolutional neural networks.Journal of biomedical informatics, 121:103863, 2021

    2021

  2. [2]

    Achiam, S

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    J. M. L. Alcaraz and N. Strodthoff. Cardiolab: Laboratory values estimation and monitor- ing from electrocardiogram signals–a multimodal deep learning approach.arXiv preprint arXiv:2411.14886, 2024

    arXiv 2024

  4. [4]

    Ansari, O

    Y . Ansari, O. Mourad, K. Qaraqe, and E. Serpedin. Deep learning for ecg arrhythmia detection and classification: an overview of progress for period 2017–2023.Frontiers in Physiology, 14:1246746, 2023

    2017

  5. [5]

    S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  6. [6]

    Bisercic, M

    A. Bisercic, M. Nikolic, M. van der Schaar, B. Delibasic, P. Lio, and A. Petrovic. Interpretable medical diagnostics with structured data extraction by large language models.arXiv preprint arXiv:2306.05052, 2023

    arXiv 2023

  7. [7]

    Çallı, E

    E. Çallı, E. Sogancioglu, B. van Ginneken, K. G. van Leeuwen, and K. Murphy. Deep learning for chest x-ray analysis: A survey.Medical image analysis, 72:102125, 2021

    work page 2021

  8. [8]

    Cassar, D

    A. Cassar, D. R. Holmes Jr, C. S. Rihal, and B. J. Gersh. Chronic coronary artery disease: diagnosis and management. InMayo Clinic Proceedings, volume 84, pages 1130–1146. Elsevier, 2009

    work page 2009

Show all 82 references
  1. [9]

    X. Chen, Y . Niu, J. Fan, L. Lu, and H. Fan. Ecg-based intelligent model for coronary heart disease screening. InProceedings of the 2024 11th International Conference on Biomedical and Bioinformatics Engineering, pages 72–77, 2024

    2024

  2. [10]

    Dao and D

    A. Dao and D. B. Vu. Alphamaze: Enhancing large language models’ spatial intelligence via grpo.arXiv preprint arXiv:2502.14669, 2025

    2025 arXiv

  3. [11]

    Deitke, C

    M. Deitke, C. Clark, S. Lee, R. Tripathi, Y . Yang, J. S. Park, M. Salehi, N. Muennighoff, K. Lo, L. Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models.arXiv preprint arXiv:2409.17146, 2024

    2024 arXiv

  4. [12]

    Dosovitskiy, L

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020

    2010 arXiv

  5. [13]

    A. S. Elstein. On the origins and development of evidence-based medicine and medical decision making.Inflammation research, 53:S184–S189, 2004. 10

    2004

  6. [14]

    Gallifant, M

    J. Gallifant, M. Afshar, S. Ameen, Y . Aphinyanaphongs, S. Chen, G. Cacciamani, D. Demner- Fushman, D. Dligach, R. Daneshjou, C. Fernandes, et al. The tripod-llm reporting guideline for studies using large language models.Nature Medicine, pages 1–10, 2025

    2025

  7. [15]

    Ghaffar Nia, E

    N. Ghaffar Nia, E. Kaplanoglu, and A. Nasab. Evaluation of artificial intelligence techniques in disease diagnosis and prediction.Discover Artificial Intelligence, 3(1):5, 2023

    2023

  8. [16]

    B. Gow, T. Pollard, L. A. Nathanson, A. Johnson, B. Moody, C. Fernandes, N. Greenbaum, J. W. Waks, P. Eslami, T. Carbonati, et al. Mimic-iv-ecg: Diagnostic electrocardiogram matched subset.Type: dataset, 6:13–14, 2023

    2023

  9. [17]

    Grattafiori, A

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  10. [18]

    Gundapaneni, Z

    S. Gundapaneni, Z. Zhi, and M. Rodrigues. Deep learning-based noninvasive screening of type 2 diabetes with chest x-ray images and electronic health records.arXiv preprint arXiv:2412.10955, 2024

    2024 arXiv

  11. [19]

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  12. [20]

    Hernandez, G

    M. Hernandez, G. Epelde, A. Alberdi, R. Cilla, and D. Rankin. Synthetic data generation for tabular health records: A systematic review.Neurocomputing, 493:28–45, 2022

    2022

  13. [21]

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, W. Chen, et al. Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022

    2022

  14. [22]

    Huang, D

    J. Huang, D. M. Yang, R. Rong, K. Nezafati, C. Treager, Z. Chi, S. Wang, X. Cheng, Y . Guo, L. J. Klesse, et al. A critical assessment of using chatgpt for extracting structured data from clinical notes.npj Digital Medicine, 7(1):106, 2024

    2024

  15. [23]

    Irvin, P

    J. Irvin, P. Rajpurkar, M. Ko, Y . Yu, S. Ciurea-Ilcus, C. Chute, H. Marklund, B. Haghgoo, R. Ball, K. Shpanskaya, et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. InProceedings of the AAAI conference on artificial intelligence, ...

    2019

  16. [24]

    M. Jin, Q. Yu, C. Zhang, D. Shu, S. Zhu, M. Du, Y . Zhang, and Y . Meng. Health-llm: Personalized retrieval-augmented disease prediction model.arXiv preprint arXiv:2402.00746, 10, 2024

    2024 arXiv

  17. [25]

    Johnson, L

    A. Johnson, L. Bulgarelli, T. Pollard, S. Horng, L. A. Celi, and R. Mark. Mimic-iv.PhysioNet. Available online at: https://physionet. org/content/mimiciv/1.0/(accessed August 23, 2021), pages 49–55, 2020

    2021

  18. [26]

    A. E. Johnson, T. J. Pollard, N. R. Greenbaum, M. P. Lungren, C.-y. Deng, Y . Peng, Z. Lu, R. G. Mark, S. J. Berkowitz, and S. Horng. Mimic-cxr-jpg, a large publicly available database of labeled chest radiographs.arXiv preprint arXiv:1901.07042, 2019

    1901 arXiv

  19. [27]

    Kline, H

    A. Kline, H. Wang, Y . Li, S. Dennis, M. Hutch, Z. Xu, F. Wang, F. Cheng, and Y . Luo. Multimodal machine learning in precision health: A scoping review.npj Digital Medicine, 5(1):171, 2022

    2022

  20. [28]

    A. Kumar. Deep learning for multi-modal medical imaging fusion: Enhancing diagnostic accuracy in complex disease detection.Int J Eng Technol Res Manag, 6(11):183, 2022

    2022

  21. [29]

    Y . Lai, J. Zhong, M. Li, S. Zhao, and X. Yang. Med-r1: Reinforcement learning for generalizable medical reasoning in vision-language models.arXiv preprint arXiv:2503.13939, 2025

    2025

  22. [30]

    X. Lan, F. Wu, K. He, Q. Zhao, S. Hong, and M. Feng. Gem: Empowering mllm for grounded ecg understanding with time series and images.arXiv preprint arXiv:2503.06073, 2025

    2025

  23. [31]

    S. Lee, W. J. Kim, J. Chang, and J. C. Ye. Llm-cxr: instruction-finetuned llm for cxr image understanding and generation.arXiv preprint arXiv:2305.11490, 2023. 11

    2023 arXiv

  24. [32]

    S. Lee, J. Youn, H. Kim, M. Kim, and S. H. Yoon. Cxr-llava: a multimodal large language model for interpreting chest x-ray images.European Radiology, pages 1–13, 2025

    2025

  25. [33]

    B. Li, K. Zhang, H. Zhang, D. Guo, R. Zhang, F. Li, Y . Zhang, Z. Liu, and C. Li. Llava-next: Stronger llms supercharge multimodal capabilities in the wild, May 2024

    2024

  26. [34]

    B. Li, Y . Zhang, D. Guo, R. Zhang, F. Li, H. Zhang, K. Zhang, P. Zhang, Y . Li, Z. Liu, et al. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  27. [35]

    C. Li, C. Wong, S. Zhang, N. Usuyama, H. Liu, J. Yang, T. Naumann, H. Poon, and J. Gao. Llava-med: Training a large language-and-vision assistant for biomedicine in one day.Advances in Neural Information Processing Systems, 36:28541–28564, 2023

    2023

  28. [36]

    Li, K.-J

    C.-Y . Li, K.-J. Chang, C.-F. Yang, H.-Y . Wu, W. Chen, H. Bansal, L. Chen, Y .-P. Yang, Y .-C. Chen, S.-P. Chen, et al. Towards a holistic framework for multimodal llm in 3d brain ct radiology report generation.Nature Communications, 16(1):2258, 2025

    2025

  29. [37]

    Liévin, C

    V . Liévin, C. E. Hother, A. G. Motzfeldt, and O. Winther. Can large language models reason about medical questions?Patterns, 5(3), 2024

    2024

  30. [38]

    Lin, Y .-T

    C.-S. Lin, Y .-T. Lee, W.-H. Fang, Y .-S. Lou, F.-C. Kuo, C.-C. Lee, and C. Lin. Deep learning al- gorithm for management of diabetes mellitus via electrocardiogram-based glycated hemoglobin (ecg-hba1c): a retrospective cohort study.Journal of Personalized Medicine, 11(8):725, 2021

    2021

  31. [39]

    C. Liu, Y . Tian, W. Chen, Y . Song, and Y . Zhang. Bootstrapping large language models for radiology report generation. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18635–18643, 2024

    2024

  32. [40]

    H. Liu, C. Li, Y . Li, and Y . J. Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024

    2024

  33. [41]

    H. Liu, C. Li, Q. Wu, and Y . J. Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  34. [42]

    S. Liu, A. S. Hussain, Q. Wu, C. Sun, and Y . Shan. Mumu-llama: Multi-modal music un- derstanding and generation via large language models.arXiv preprint arXiv:2412.06660, 2024

    2024 arXiv

  35. [43]

    X. Liu, H. Wang, Z. Li, and L. Qin. Deep learning in ecg diagnosis: A review.Knowledge-Based Systems, 227:107187, 2021

    2021

  36. [44]

    Z. Liu, Z. Sun, Y . Zang, X. Dong, Y . Cao, H. Duan, D. Lin, and J. Wang. Visual-rft: Visual reinforcement fine-tuning.arXiv preprint arXiv:2503.01785, 2025

    2025 arXiv

  37. [45]

    H. Lu, W. Liu, B. Zhang, B. Wang, K. Dong, B. Liu, J. Sun, T. Ren, Z. Li, H. Yang, et al. Deepseek-vl: towards real-world vision-language understanding.arXiv preprint arXiv:2403.05525, 2024

    2024 arXiv

  38. [46]

    M. Y . Lu, B. Chen, D. F. Williamson, R. J. Chen, M. Zhao, A. K. Chow, K. Ikemura, A. Kim, D. Pouli, A. Patel, et al. A multimodal generative ai copilot for human pathology.Nature, 634(8033):466–473, 2024

    2024

  39. [47]

    J. Pan, C. Liu, J. Wu, F. Liu, J. Zhu, H. B. Li, C. Chen, C. Ouyang, and D. Rueckert. Medvlm-r1: Incentivizing medical reasoning capability of vision-language models (vlms) via reinforcement learning.arXiv preprint arXiv:2502.19634, 2025

    2025 arXiv

  40. [48]

    S. S. Ramesh, Y . Hu, I. Chaimalas, V . Mehta, P. G. Sessa, H. Bou Ammar, and I. Bogunovic. Group robust preference optimization in reward-free rlhf.Advances in Neural Information Processing Systems, 37:37100–37137, 2024

    2024

  41. [49]

    Saporta, A

    A. Saporta, A. M. Puli, M. Goldstein, and R. Ranganath. Contrasting with symile: Simple model- agnostic representation learning for unlimited modalities.Advances in Neural Information Processing Systems, 37:56919–56957, 2024. 12

    2024

  42. [50]

    Schulman, F

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  43. [51]

    Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y . Li, Y . Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  44. [52]

    Shentu and N

    J. Shentu and N. Al Moubayed. Cxr-irgen: an integrated vision and language model for the generation of clinically accurate chest x-ray image-report pairs. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5212–5221, 2024

    2024

  45. [53]

    Steyaert, M

    S. Steyaert, M. Pizurica, D. Nagaraj, P. Khandelwal, T. Hernandez-Boussard, A. J. Gentles, and O. Gevaert. Multimodal data fusion for cancer biomarker discovery with deep learning.Nature machine intelligence, 5(4):351–362, 2023

    2023

  46. [54]

    Strodthoff, J

    N. Strodthoff, J. M. Lopez Alcaraz, and W. Haverkamp. Mimic-iv-ecg-ext-icd: Diagnostic labels for mimic-iv-ecg (version 1.0.1).PhysioNet, 2024

    2024

  47. [55]

    M. S. I. Sumon, M. S. B. Islam, M. S. Rahman, M. S. A. Hossain, A. Khandakar, A. Hasan, M. Murugappan, and M. E. Chowdhury. Cardiotabnet: A novel hybrid transformer model for heart disease prediction using tabular medical data.arXiv preprint arXiv:2503.17664, 2025

    2025 arXiv

  48. [56]

    H. Tan, Y . Ji, X. Hao, M. Lin, P. Wang, Z. Wang, and S. Zhang. Reason-rft: Reinforcement fine-tuning for visual reasoning.arXiv preprint arXiv:2503.20752, 2025

    2025

  49. [57]

    Tanno, D

    R. Tanno, D. G. Barrett, A. Sellergren, S. Ghaisas, S. Dathathri, A. See, J. Welbl, C. Lau, T. Tu, S. Azizi, et al. Collaboration between clinicians and vision–language models in radiology report generation.Nature Medicine, 31(2):599–608, 2025

    2025

  50. [58]

    Y . Tian, Z. Li, Y . Jin, M. Wang, X. Wei, L. Zhao, Y . Liu, J. Liu, and C. Liu. Foundation model of ecg diagnosis: Diagnostics and explanations of any form and rhythm on ecg.Cell Reports Medicine, 5(12), 2024

    2024

  51. [59]

    Z. Tian, Y . Jin, Z. Liu, R. Yuan, X. Tan, Q. Chen, W. Xue, and Y . Guo. Audiox: Diffusion transformer for anything-to-audio generation.arXiv preprint arXiv:2503.10522, 2025

    2025 arXiv

  52. [60]

    Venugopalan, L

    J. Venugopalan, L. Tong, H. R. Hassanzadeh, and M. D. Wang. Multimodal deep learning models for early detection of alzheimer’s disease stage.Scientific reports, 11(1):3254, 2021

    2021

  53. [61]

    S. Wu, H. Fei, L. Qu, W. Ji, and T.-S. Chua. Next-gpt: Any-to-any multimodal llm. InForty-first International Conference on Machine Learning, 2024

    2024

  54. [62]

    K. Yang, M. Hong, J. Zhang, Y . Luo, S. Zhao, O. Zhang, X. Yu, J. Zhou, L. Yang, P. Zhang, et al. Ecg-lm: Understanding electrocardiogram with a large language model.Health Data Science, 5:0221, 2025

    2025

  55. [63]

    W. Yao, C. Liu, K. Yin, W. Cheung, and J. Qin. Addressing asynchronicity in clinical multimodal fusion via individualized chest x-ray generation.Advances in Neural Information Processing Systems, 37:29001–29028, 2024

    2024

  56. [64]

    C. Yu, A. Velu, E. Vinitsky, J. Gao, Y . Wang, A. Bayen, and Y . Wu. The surprising effectiveness of ppo in cooperative multi-agent games.Advances in neural information processing systems, 35:24611–24624, 2022

    2022

  57. [65]

    H. Yu, P. Guo, and A. Sano. Zero-shot ecg diagnosis with large language models and retrieval- augmented generation. InMachine learning for health (ML4H), pages 650–663. PMLR, 2023

    2023

  58. [66]

    D. Yuan, E. Rastogi, G. Naik, S. P. Rajagopal, S. Goyal, F. Zhao, B. Chintagunta, and J. Ward. A continued pretrained llm approach for automatic medical note generation.arXiv preprint arXiv:2403.09057, 2024

    2024 arXiv

  59. [67]

    Zhang, X

    H. Zhang, X. Li, and L. Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding.arXiv preprint arXiv:2306.02858, 2023. 13

    2023 arXiv

  60. [68]

    Y . Zhao, J. Huang, J. Hu, X. Wang, Y . Mao, D. Zhang, Z. Jiang, Z. Wu, B. Ai, A. Wang, et al. Swift: a scalable lightweight infrastructure for fine-tuning. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 29733–29735, 2025

    2025

  61. [69]

    Y . Zhao, T. Zhang, X. Wang, P. Han, T. Chen, L. Huang, Y . Jin, and J. Kang. Ecg-chat: A large ecg-language model for cardiac disease diagnosis.arXiv preprint arXiv:2408.08849, 2024

    2024 arXiv

  62. [70]

    Hematocrit

    J. Zhu, W. Wang, Z. Chen, Z. Liu, S. Ye, L. Gu, Y . Duan, H. Tian, W. Su, J. Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025. 14 A Details About the Prompts of MedTVT-QA’s Constru...

    2025 arXiv

  63. [71]

    Left Bundle Branch Block (LBBB): There is evidence of delayed or disrupted conduction through the left bundle of the heart’s electrical conduction system. This results in a characteristic pattern on the ECG, with a widened QRS complex (>120 ms) and specific morphologies in lea...

  64. [72]

    QRS Complexes: The QRS complexes are wide and have a dura�on of approximately 120ms , which is indica�ve of le� ventricular hypertrophy (LVH)

  65. [73]

    T Waves: The T waves are not significantly abnormal in any lead

  66. [74]

    Chest X-Ray Analysis:

    ST Segments: The ST segments are within normal limits. Chest X-Ray Analysis:

  67. [75]

    Cardiac Size: The cardiac silhoue�e appears enlarged, which could be due to LVH or other cardiomegaly causes

  68. [76]

    Lung Fields: The lungfields appear clear with no signs of consolida�on, effusion, or other abnormali�es

  69. [77]

    Heart Posi�on: The heart posi�on is normal, with no evidence of cardiomegaly or pericardial effusion. Blood Test Results: Crea�nine: 0.2264026403427124 mg/dL (normal range: 0.6–1.2 mg/dL) BUN: 0.2870349884033203 mg/dL (normal range: 7–20 mg/dL) Calcium: 0.6595468521118164 mg/dL...

  70. [78]

    Arrhythmias: These include abnormal heart rhythms such as bradycardia, tachycardia, or atrial fibrilla�on, indicated by the presence of irregular QRS complexes

  71. [79]

    Cardiomyopathy: Structural changes in the heart muscle, which might be visible on the ECG with abnormal heart rhythm pa�erns

  72. [80]

    Valvular Aor�c Stenosis: The semi-erect portable chest X-ray suggests possible pulmonary hypertension as the vascular structures seen suggest dila�on in pulmonary valve and possibly aor�c root shadow sugges�ng valve involvement

  73. [81]

    Coronary Artery Disease: Though less likely in this chest X-ray alone, subtle changes could indicate coronary artery disease

  74. [82]

    Electrolyte Imbalance: Such as hyperkalemia, which could be visible on an abnormal heart rhythm from changes in heart rhyt hmpa�erns. These are general possibili�es and specific diagnoses would need further clinical evalua�on and possibly addi�onal inves�gations such as Holter ...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.