V2T-CoT: From Vision to Text Chain-of-Thought for Medical Reasoning and Diagnosis

Bin Feng; Jian Wu; Jiaxiang Liu; Shujian Gao; Xiaotang Gai; Yuan Wang; Zhihang Tang; Zuozhu Liu

REVIEW 5 major objections 5 minor 36 references

V2T-CoT: From Vision to Text Chain-of-Thought for Medical Reasoning and Diagnosis

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read V2T-CoT claims state-of-the-art accuracy on four medical VQA benchmarks by chaining visual region localization with text rationale generation.

desk verdict Novel combination of visual grounding and rationale instruction tuning for Med-VQA, but the R-Med construction must document that test splits were excluded before the SOTA claims hold. read the letter →

arxiv 2506.19610 v2 pith:37BG4IIZ submitted 2025-06-24 cs.CE

Yuan Wang , Jiaxiang Liu , Shujian Gao , Bin Feng , Zhihang Tang , Xiaotang Gai , Jian Wu , Zuozhu Liu This is my paper

classification cs.CE

keywords MedicalVisualQuestionAnsweringChain-of-ThoughtVision-LanguageModelGroundingInstructionTuningInterpretabilityRadiologyPathology

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that medical visual question answering can be made both more accurate and more interpretable by chaining two distinct reasoning steps: first localizing disease-relevant image regions (Vision CoT), then generating a text rationale and answer conditioned on those regions (Text CoT). The authors report that a 2.7B-parameter model trained on their constructed R-Med 39K instruction-tuning dataset reaches the highest accuracy on VQA-RAD (84.86%) and SLAKE (87.61%) among the methods compared, with competitive results on VQA-2019 and PathVQA. A sympathetic reader would care because the method attacks a known limitation of prior Med-VQA systems: they answer accurately without giving a verifiable clinical reasoning path. If the result holds, small-parameter VLMs can deliver both diagnosis and explanation without relying on huge generalist models.

What carries the argument

The load-bearing mechanism is the V2T-CoT pipeline: a vision encoder and language encoder are fused with cross-modality multi-head attention, and a phrase-grounding alignment score $S_{\text{ground}} = V T^\top$ localizes disease-relevant regions described in the question. Regional pixel-level attention then mixes local and global features via $A_{\text{combined}} = \operatorname{SoftMax}(\alpha Q_{\text{regional}} K_{\text{regional}}^\top + (1-\alpha) Q_{\text{global}} K_{\text{global}}^\top)/\sqrt{d}$, with $\alpha$ balancing fine detail and context. The detected regions are fed into the language model, which is fine-tuned on the R-Med 39K dataset of quadruples $\langle Q, I, A, R\rangle$ (question, image, answer, rationale) generated by GPT-4 and Gemini and verified by LLM and expert checks. This two-stage design is what lets the model both look at the right pixels and say why.

What would settle it

Release R-Med 39K and check every quadruple against the test splits of VQA-RAD, SLAKE, VQA-2019, and PathVQA; if any test question or answer appears in the training rationales, retrain on a cleanly split version and compare, because the central accuracy claim would collapse if the difference vanishes.

Watch

Extended reading notes

Core claim

The central claim is that combining visual grounding with textual reasoning outperforms either component alone and beats existing Med-VQA baselines at comparable parameter scale. Concretely, V2T-CoT (Phi2) is reported to achieve 84.86% on VQA-RAD, 87.61% on SLAKE, 80.10% on VQA-2019, and 91.42% on PathVQA for closed-ended questions, with gains of +2.39% and +5.11% over LLaVA-Med on the first two benchmarks. The ablation table shows that Vision CoT and Text CoT are complementary: removing Vision CoT hurts SLAKE closed-ended accuracy, while removing Text CoT lowers open-ended recall on VQA-RAD by 2.27%. The paper also reports that rationale quality scores from LLM and human evaluation correlate with diagnostic accuracy, which it reads as evidence of interpretability.

Load-bearing premise

The R-Med 39K training rationales must have been built from training splits only; if any test question or answer seeped into them, the reported accuracy gains would reflect memorization rather than reasoning.

Editorial extensions

If this is right

Closed-ended accuracy on VQA-RAD and SLAKE would sit above 84% and 87% respectively for a 2.7B-parameter model, a level the paper claims beats larger baselines such as LLaVA-Med (7B).
Both reasoning chains are necessary: ablations show that dropping Vision CoT or Text CoT lowers accuracy or recall, so the reported gains come from their combination rather than from parameter scale.
On open-ended questions, V2T-CoT is reported to consistently outperform MedThink across BLEU and ROUGE-L on three datasets, indicating improved rationale generation as well as answer accuracy.
Improving region proposal quality (higher mAP) would directly translate into better VQA accuracy, since the paper reports a Pearson correlation of 0.92 between organ-level mAP and accuracy on SLAKE.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If R-Med 39K proves to exclude all test-set content, the same generation-and-verification recipe could be applied to other medical imaging domains, turning any VQA benchmark into an instruction-tuning corpus without manual rationale annotation.
The reported correlation between rationale quality scores and diagnostic accuracy suggests that rationale score could serve as a cheap proxy for diagnostic competence during model development, though this is an extension rather than a claim of the paper.
A testable extension would be swapping the frozen CLIP encoder for a medical-domain vision encoder; the framework's gains may be partly inherited from CLIP-ViT rather than from the CoT modules, which the current ablations do not isolate.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This paper is a straightforward and sensible entry in the growing Med-VQA reasoning line. The new piece is the combination: phrase-grounding (GLIP) to localize regions, a regional pixel-level attention term with a tunable alpha, and instruction tuning on R-Med 39K, a dataset of LLM-generated rationales. Each building block exists, but the integration is novel and the experiments are broader than most: four benchmarks, closed and open-ended questions, ablations, a detection mAP/VQA accuracy correlation, plus LLM and human rationale scoring. The ablations show both Vision CoT and Text CoT pull their weight; the correlation table (r=0.92) is a nice touch.

The soft spot is exactly where the stress-test points. Section 3.1 says the four benchmarks are used as training and testing datasets for V2T-CoT, and then says rationales were generated from them into R-Med. There is no sentence stating that test questions, images, or answers were excluded from R-Med. Without that, the +2.39% on VQA-RAD and +5.11% on SLAKE could be inflated by the model having seen the test answers during instruction tuning. This is not a manufactured flaw; it is a missing fact that the paper must supply. A second, smaller leak path: the GLIP detector is fine-tuned on SLAKE, and if that overlaps SLAKE's test split, the detection-side numbers inherit the same issue. The paper also reports no error bars or significance tests, and comparisons are to previously published numbers. The rationale evaluation is a closed loop (LLM generates, LLM judges), though the human validation step reduces that concern to a minor one.

None of this makes the method implausible. The gains are modest and the components are reasonable. But the SOTA claim is premature until the split provenance is documented and code/data are released to verify it. If the split exclusion is confirmed, this is a solid application paper. If not, the central claim collapses.

The right audience is Med-VQA researchers and anyone building explainable medical VLMs. It deserves serious peer review, not a desk reject, but the reviewers should push hard for a clear data-provenance statement and a data-size-matched baseline that isolates the rationale effect. I would read a revised version.

Referee Report

5 major / 5 minor

Summary. The paper proposes V2T-CoT, a Med-VQA method that combines a visual chain-of-thought (Vision CoT) and a textual chain-of-thought (Text CoT). Vision CoT uses a GLIP-based detector fine-tuned on SLAKE to localize disease-relevant regions and then applies regional pixel-level attention, while Text CoT fine-tunes StableLM (1.6B) and Phi2 (2.7B) on a newly constructed instruction-tuning dataset, R-Med 39K, consisting of <question, image, answer, rationale> quadruples generated by GPT-4 and Gemini. The authors report state-of-the-art closed-ended accuracy on VQA-RAD (84.86%) and SLAKE (87.61%), competitive results on VQA-2019 and PathVQA, and improved open-ended metrics, with ablations, detection mAP studies, and LLM/human rationale-quality evaluations.

Significance. If the reported results are valid, the paper makes a useful contribution: it integrates visual grounding with explicit textual rationales in a lightweight VLM and offers a practical pipeline for building reasoning-style instruction data from existing benchmarks. The method is clearly structured, the experimental design includes ablations for both components, and the paper explicitly aims at the clinically important goal of explainable diagnosis. However, the load-bearing empirical claims currently rest on unverifiable dataset provenance, LLM-generated rationales used both as training targets and as evaluation evidence, and comparisons against published numbers without uncertainty estimates; these issues must be resolved before the SOTA and interpretability claims can be accepted.

major comments (5)

[§2.3, §3.1] The R-Med 39K dataset is constructed from the same four benchmarks on which the model is evaluated, but the paper never states that the test splits of VQA-RAD, SLAKE, VQA-2019, or PathVQA were excluded when generating the <Q, I, A, R> quadruples. If any test question, image, or answer was included in R-Med, the instruction-tuning process exposes the test answers to the model, and the reported improvements over LLaVA-Med (Vicuna) in Table 1 (+2.39% on VQA-RAD, +5.11% on SLAKE) could reflect memorization rather than generalization. The GitHub repository is listed, but no dataset or construction script is described in the manuscript, so the split boundaries cannot be checked. Please report the exact train/test provenance of every example in R-Med, release the dataset or a verified construction script, and, ideally, validate on a held-out benchmark that did not contribute to the training rationales.
[§3.1] The Vision CoT detector is initialized with GLIP and fine-tuned on SLAKE, and SLAKE is also one of the four evaluation benchmarks. The paper does not state whether the SLAKE split used for detector fine-tuning overlaps the SLAKE test split used in Table 1. If the detector training included SLAKE test images, the reported SLAKE closed-ended accuracy of 87.61% would inherit detection-side leakage. Clarify the exact split used for detector fine-tuning and confirm that it is disjoint from both the SLAKE test images and the SLAKE VQA questions.
[Table 1, §3.2] All comparisons in Table 1 are against single numbers taken from the literature, with no standard deviations, confidence intervals, or significance tests. The claimed advantages are modest on VQA-2019 (+2.00 over MMBERT) and PathVQA (+1.25 over LLaVA-Med Phi2), and run-to-run variance could alter the ranking. Please report results over multiple random seeds with variance, and use a paired or appropriate statistical test where possible, or explicitly state if each reported number comes from a single run.
[§2.3, Fig. 4] The rationale-quality evaluation is a closed loop: GPT-4 and Gemini generate the R-Med rationales, the VLM is instruction-tuned on those same rationales, and then LLM-based scoring judges the quality of the generated rationales. The reported positive correlation between rationale quality and diagnostic accuracy is therefore not an independent validation of reasoning correctness. The paper mentions expert validation, but provides no number of experts, annotation protocol, or inter-annotator agreement. Please add an external validation protocol with quantitative human agreement, and note that the statement in §2.3 pointing to implementation details in 'Section 3.3' is inconsistent because Section 3.3 is the ablation analysis, not the dataset-construction details.
[§3.3, Table 2] The ablation narrative does not match the numbers in Table 2. The text states that removing Text CoT causes a 2.27% drop in VQA-RAD open-ended recall, but the table shows 58.72 with Vision CoT only versus 60.40 with both components, a 1.68-point difference, and 59.53 with Text CoT only. Similarly, the claimed 2.66% SLAKE closed-end error reduction from Vision CoT is not directly obtainable from the table as printed. Please specify exactly which rows are being differenced and recompute the claimed percentages.

minor comments (5)

[Abstract, §3.2] The abstract contains 'constructedR-Med 39K' with a missing space, and §3.2 says 'Rough-L' where 'ROUGE-L' is intended.
[Eq. (4)] The regional/global attention balance factor α is a free parameter, but the paper does not state its value or report sensitivity to it; please specify how α was chosen.
[Table 3] Table 3 uses 'Med-2019' while the rest of the paper uses 'VQA-2019'; please standardize the naming.
[Table 3] The abbreviations 'VCoT-GT(Tiny)' and 'VCoT-GL(Large)' are not defined in the text; please explain what GT and GL refer to.
[§3.1] The paper says rationales were generated by GPT-4 and Gemini, but no prompt templates or verification prompts are provided; including the exact prompts would significantly improve reproducibility.

Circularity Check

0 steps flagged · score 1.0 of 10

No exhibited circularity: the benchmark accuracy claims are externally grounded, though R-Med split provenance and LLM-based rationale validation are validity risks that need disclosure.

full rationale

The paper's central claims are empirical: V2T-CoT is trained on the constructed R-Med 39K dataset and evaluated on closed-end accuracy of four public Med-VQA benchmarks against their ground-truth answers. None of the reported accuracy numbers is an algebraic consequence of a fitted parameter or a definition: the model's output is compared with independent answer labels, and the ablations in Table 2 compare configurations on the same evaluation protocol. The vision phrase-grounding step (GLIP fine-tuned on SLAKE) and the text-rationale step (GPT-4/Gemini-generated rationales) are pipeline components, not results derived from their own premises. The paper is heavily self-cited (MedThink [7], MedCoT [27], etc.), but these citations are used as baselines and motivation, not as load-bearing proofs; there is no uniqueness theorem and no ansatz imported via self-citation. Two validity threats are present but are not demonstrated circularity. Section 3.1 does not explicitly state that R-Med was built only from the training splits of the four benchmarks, so test-set contamination is a correctness risk rather than an exhibited reduction. Section 3.3 relies in part on LLM scoring of LLM-generated rationales, but human validation is also claimed and the benchmark accuracy claims do not depend on that loop. Under the hard rule requiring a quoted reduction, no circular step is exhibited.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities. It combines existing detectors, VLMs, and LLMs, and constructs a dataset. The main unstated costs are the alpha hyperparameter and the assumption that LLM-generated rationales are clinically valid and that GLIP transfers across medical domains.

free parameters (1)

alpha (regional/global attention balance) = Not reported
Introduced in Eq. (4) to balance regional and global attention. No value or selection rule is given, so it may be hand-tuned on validation data, which would make part of the reported gains fitted.

assumptions (3)

domain assumption LLM-generated rationales are clinically valid after automated verification.
Section 2.3 says a separate language model checks logical consistency, but no clinical evidence is provided beyond that internal loop.
domain assumption GLIP detector fine-tuned on SLAKE generalizes to other medical modalities without manual annotation.
Section 3.1 states the detector is fine-tuned on SLAKE and deployed on other datasets. The transfer is only partially tested via mAP and is load-bearing for the Vision CoT results.
domain assumption Published baseline numbers in Table 1 are comparable because they use the same splits and metrics.
The paper mixes numbers from different papers with many dashes and does not re-run baselines, so comparability is assumed rather than demonstrated.

how reviews work

0 comments

Cite this review

Pith. "Pith review of V2T-CoT: From Vision to Text Chain-of-Thought for Medical Reasoning and Diagnosis." pith.science (2026). https://pith.science/paper/37BG4IIZ

@misc{pith2026250619610,
  author       = {Pith},
  title        = {Pith review of: V2T-CoT: From Vision to Text Chain-of-Thought for Medical Reasoning and Diagnosis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/37BG4IIZ}},
  note         = {Machine review of arXiv:2506.19610}
}

read the original abstract

Recent advances in multimodal techniques have led to significant progress in Medical Visual Question Answering (Med-VQA). However, most existing models focus on global image features rather than localizing disease-specific regions crucial for diagnosis. Additionally, current research tends to emphasize answer accuracy at the expense of the reasoning pathway, yet both are crucial for clinical decision-making. To address these challenges, we propose From Vision to Text Chain-of-Thought (V2T-CoT), a novel approach that automates the localization of preference areas within biomedical images and incorporates this localization into region-level pixel attention as knowledge for Vision CoT. By fine-tuning the vision language model on constructed R-Med 39K dataset, V2T-CoT provides definitive medical reasoning paths. V2T-CoT integrates visual grounding with textual rationale generation to establish precise and explainable diagnostic results. Experimental results across four Med-VQA benchmarks demonstrate state-of-the-art performance, achieving substantial improvements in both performance and interpretability.

Figures

Figures reproduced from arXiv: 2506.19610 by the authors.

**Figure 2.** Implementation of V2T-CoT for medical diagnosis. The model leverages [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** (a) A radar plot compares the performance of V2T-CoT and Med [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: LLM and Human evaluation for rationale from [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

36 extracted references · 19 canonical work pages

[1]

Bioengineering10(3), 380 (2023)

Bazi, Y., Rahhal, M.M.A., Bashmal, L., Zuair, M.: Vision–language model for visual question answering in medical imagery. Bioengineering10(3), 380 (2023)

work page 2023
[2]

Bellagente, M., Tow, J., Mahan, D., Phung, D., Zhuravinskyi, M., Adithyan, R., Baicoianu, J., Brooks, B., Cooper, N., Datta, A., Lee, M., Mostaque, E., Pieler, M., Pinnaparju, N., Rocha, P., Saini, H., Teufel, H., Zanichelli, N., Riquelme, C.: Stable lm 2 1.6b technical report (2024)

work page 2024
[3]

In: Proceedings of CLEF (Conference and Labs of the Evaluation Forum) 2019 Working Notes

Ben Abacha, A., Hasan, S.A., Datla, V.V., Demner-Fushman, D., Müller, H.: Vqa- med: Overview of the medical visual question answering task at imageclef 2019. In: Proceedings of CLEF (Conference and Labs of the Evaluation Forum) 2019 Working Notes. 9-12 September 2019 (2019)

2019
[4]

Chen, J., Cai, Z., Ji, K., Wang, X., Liu, W., Wang, R., Hou, J., Wang, B.: Huatuogpt-o1, towards medical complex reasoning with llms (2024)

work page 2024
[5]

In: Proceedings of the 30th ACM Interna- tional Conference on Multimedia

Chen, Z., Li, G., Wan, X.: Align, reason and learn: Enhancing medical vision-and- language pre-training with knowledge. In: Proceedings of the 30th ACM Interna- tional Conference on Multimedia. pp. 5152–5161 (2022)

work page 2022
[6]

Eslami, S., Meinel, C., De Melo, G.: Pubmedclip: How much does clip benefit visual question answering in the medical domain? In: Findings of the Association for Computational Linguistics: EACL 2023. pp. 1181–1193 (2023)

work page 2023
[7]

Gai, X., Zhou, C., Liu, J., Feng, Y., Wu, J., Liu, Z.: Medthink: Explaining medical visualquestionansweringviamultimodaldecision-makingrationale.arXivpreprint arXiv:2404.12372 (2024)

arXiv 2024
[8]

In: Proceedings of the 2021 International Conference on Multimedia Retrieval

Gong, H., Chen, G., Liu, S., Yu, Y., Li, G.: Cross-modal self-attention with multi- task pre-training for medical visual question answering. In: Proceedings of the 2021 International Conference on Multimedia Retrieval. pp. 456–460 (2021)

work page 2021

Show all 36 references

[9]

Gunasekar, S., Zhang, Y., Aneja, J., Mendes, C.C.T., Giorno, A.D., Gopi, S., Java- heripi, M., Kauffmann, P., de Rosa, G., Saarikivi, O., Salim, A., Shah, S., Behl, H.S., Wang, X., Bubeck, S., Eldan, R., Kalai, A.T., Lee, Y.T., Li, Y.: Textbooks are all you need (2023)

2023
[10]

arXiv preprint arXiv:2401.02797 (2024)

He, J., Li, P., Liu, G., Zhao, Z., Zhong, S.: Pefomed: Parameter efficient fine-tuning on multimodal large language models for medical visual question answering. arXiv preprint arXiv:2401.02797 (2024)

2024 arXiv
[11]

arXiv preprint arXiv:2003.10286 (2020)

He, X., Zhang, Y., Mou, L., Xing, E., Xie, P.: Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003.10286 (2020)

2020 arXiv
[12]

arXiv preprint arXiv:2506.12849 (2025)

Jiang, S., Wang, Y., Chen, R., Zhang, Y., Luo, R., Lei, B., Song, S., Feng, Y., Sun, J., Wu, J., et al.: Capo: Reinforcing consistent reasoning in medical decision- making. arXiv preprint arXiv:2506.12849 (2025)

2025 arXiv
[13]

Jiang, S., Wang, Y., Song, S., Zhang, Y., Meng, Z., Lei, B., Wu, J., Sun, J., Liu, Z.: Omniv-med: Scaling medical vision-language model for universal visual understanding (2025), https://arxiv.org/abs/2504.14692

2025 arXiv
[14]

arXiv preprint arXiv:2410.15334 (2024)

Jiang, S., Zhang, Y., Chen, R., Hu, T., Jin, Y., He, Q., Feng, Y., Wu, J., Liu, Z.: Modality-fair preference optimization for trustworthy mllm alignment. arXiv preprint arXiv:2410.15334 (2024)

2024 arXiv
[15]

arXiv preprint arXiv:2506.00805 (2025)

Jiang, S., Zhang, Y., Jin, Y., Tang, Z., Wu, Y., Feng, Y., Wu, J., Liu, Z.: Hscr: Hierarchical self-contrastive rewarding for aligning medical vision language models. arXiv preprint arXiv:2506.00805 (2025)

2025 arXiv
[16]

arXiv preprint arXiv:2404.04514 (2024)

Jiang, S., Zhang, Y., Zhou, C., Jin, Y., Feng, Y., Wu, J., Liu, Z.: Joint visual and text prompting for improved object-centric perception with multimodal large language models. arXiv preprint arXiv:2404.04514 (2024)

2024 arXiv
[17]

In: Find- ingsoftheAssociationforComputationalLinguistics:EMNLP2024.pp.3843–3860 (2024)

Jiang, S., Zheng, T., Zhang, Y., Jin, Y., Yuan, L., Liu, Z.: Med-moe: Mixture of domain-specific experts for lightweight medical vision-language models. In: Find- ingsoftheAssociationforComputationalLinguistics:EMNLP2024.pp.3843–3860 (2024)

2024
[18]

In: 2021 IEEE 18th International Symposium on Biomedical Imaging

Khare, Y., Bagal, V., Mathew, M., Devi, A., Priyakumar, U.D., Jawahar, C.: Mm- bert: multimodal bert pretraining for improved medical vqa. In: 2021 IEEE 18th International Symposium on Biomedical Imaging. pp. 1033–1036 (2021)

2021
[19]

Khare, Y., Bagal, V., Mathew, M., Devi, A., Priyakumar, U.D., Jawahar, C.: Mmbert: Multimodal bert pretraining for improved medical vqa (2021),https: //arxiv.org/abs/2104.01394

2021 arXiv
[20]

Scientific data 5(1), 1–10 (2018)

Lau, J.J., Gayen, S., Ben Abacha, A., Demner-Fushman, D.: A dataset of clinically generated visual questions and answers about radiology images. Scientific data 5(1), 1–10 (2018)

2018
[21]

Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang, J., Naumann, T., Poon, H., Gao, J.: Llava-med: Training a large language-and-vision assistant for biomedicine in one day (2023),https://arxiv.org/abs/2306.00890

2023 arXiv
[22]

Advances in Neural Information Processing Systems36 (2024)

Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang, J., Naumann, T., Poon, H., Gao, J.: Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems36 (2024)

2024
[23]

In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR)

Li, L.H., Zhang, P., Zhang, H., Yang, J., Li, C., Zhong, Y., Wang, L., Yuan, L., Zhang, L., Hwang, J.N., Chang, K.W., Gao, J.: Grounded language-image pre- training. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR). pp. 10955–10965 (2022)

2022
[24]

Liu, B., Zhan, L.M., Xu, L., Ma, L., Yang, Y., Wu, X.M.: Slake: A semantically- labeledknowledge-enhanceddatasetformedicalvisualquestionanswering.In:2021 IEEE 18th International Symposium on Biomedical Imaging. pp. 1650–1654 (2021)

2021
[25]

Advances in neural information processing systems36 (2024)

Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. Advances in neural information processing systems36 (2024)

2024
[26]

IEEE Transactions on Emerging Topics in Computational Intelligence (2023)

Liu, J., Hu, T., Zhang, Y., Feng, Y., Hao, J., Lv, J., Liu, Z.: Parameter-efficient transfer learning for medical visual question answering. IEEE Transactions on Emerging Topics in Computational Intelligence (2023)

2023
[27]

In: Proceedings of the 2024 Conference on Empirical Meth- ods in Natural Language Processing

Liu, J., Wang, Y., Du, J., Zhou, J.T., Liu, Z.: MedCoT: Medical chain of thought via hierarchical expert. In: Proceedings of the 2024 Conference on Empirical Meth- ods in Natural Language Processing. pp. 17371–17389 (2024)

2024
[28]

Nature Medicine pp

Liu, X., Liu, H., Yang, G., Jiang, Z., Cui, S., Zhang, Z., Wang, H., Tao, L., Sun, Y., Song, Z., et al.: A generalist medical language model for disease diagnosis assistance. Nature Medicine pp. 1–11 (2025)

2025
[29]

In: International Conference on Information Processing in Medical Imaging

Liu, Y., Wang, Z., Xu, D., Zhou, L.: Q2atransformer: Improving medical vqa via an answer querying decoder. In: International Conference on Information Processing in Medical Imaging. pp. 445–456. Springer (2023)

2023
[30]

arXiv preprint arXiv:2312.15011 (2023)

Qi, Z., Fang, Y., Zhang, M., Sun, Z., Wu, T., Liu, Z., Lin, D., Wang, J., Zhao, H.: Gemini vs gpt-4v: A preliminary comparison and combination of vision-language models through qualitative cases. arXiv preprint arXiv:2312.15011 (2023)

2023 arXiv
[31]

NPJ Digital Medicine7(1), 20 (2024)

Savage, T., Nayak, A., Gallo, R., Rangan, E., Chen, J.H.: Diagnostic reasoning prompts reveal the potential for large language model interpretability in medicine. NPJ Digital Medicine7(1), 20 (2024)

2024
[32]

In: International Conference on Medical Image Computing and Computer- Assisted Intervention

Van Sonsbeek, T., Derakhshani, M.M., Najdenkoska, I., Snoek, C.G., Worring, M.: Open-ended medical visual question answering through prefix tuning of language models. In: International Conference on Medical Image Computing and Computer- Assisted Intervention. pp. 726–736. Spri...

2023
[33]

arXiv preprint arXiv:2304.01097 (2023)

Xiong, H., Wang, S., Zhu, Y., Zhao, Z., Liu, Y., Huang, L., Wang, Q., Shen, D.: Doctorglm: Fine-tuning your chinese doctor is not a herculean task. arXiv preprint arXiv:2304.01097 (2023)

2023 arXiv
[34]

arXiv e-prints pp

Zhang,K.,Yu,J.,Adhikarla,E.,Zhou,R.,Yan,Z.,Liu,Y.,Liu,Z.,He,L.,Davison, B., Li, X., et al.: Biomedgpt: A unified and generalist biomedical generative pre- trained transformer for vision, language, and multimodal tasks. arXiv e-prints pp. arXiv–2305 (2023)

2023
[35]

arXiv preprint arXiv:2302.00923 (2023)

Zhang, Z., Zhang, A., Li, M., Zhao, H., Karypis, G., Smola, A.: Multimodal chain- of-thought reasoning in language models. arXiv preprint arXiv:2302.00923 (2023)

2023 arXiv
[36]

In: Thirty- seventh Conference on Neural Information Processing Systems (2023)

Zheng, G., Yang, B., Tang, J., Zhou, H.Y., Yang, S.: Ddcot: Duty-distinct chain- of-thought prompting for multimodal reasoning in language models. In: Thirty- seventh Conference on Neural Information Processing Systems (2023)

2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Bioengineering10(3), 380 (2023)

Bazi, Y., Rahhal, M.M.A., Bashmal, L., Zuair, M.: Vision–language model for visual question answering in medical imagery. Bioengineering10(3), 380 (2023)

work page 2023

[2] [2]

Bellagente, M., Tow, J., Mahan, D., Phung, D., Zhuravinskyi, M., Adithyan, R., Baicoianu, J., Brooks, B., Cooper, N., Datta, A., Lee, M., Mostaque, E., Pieler, M., Pinnaparju, N., Rocha, P., Saini, H., Teufel, H., Zanichelli, N., Riquelme, C.: Stable lm 2 1.6b technical report (2024)

work page 2024

[3] [3]

In: Proceedings of CLEF (Conference and Labs of the Evaluation Forum) 2019 Working Notes

Ben Abacha, A., Hasan, S.A., Datla, V.V., Demner-Fushman, D., Müller, H.: Vqa- med: Overview of the medical visual question answering task at imageclef 2019. In: Proceedings of CLEF (Conference and Labs of the Evaluation Forum) 2019 Working Notes. 9-12 September 2019 (2019)

2019

[4] [4]

Chen, J., Cai, Z., Ji, K., Wang, X., Liu, W., Wang, R., Hou, J., Wang, B.: Huatuogpt-o1, towards medical complex reasoning with llms (2024)

work page 2024

[5] [5]

In: Proceedings of the 30th ACM Interna- tional Conference on Multimedia

Chen, Z., Li, G., Wan, X.: Align, reason and learn: Enhancing medical vision-and- language pre-training with knowledge. In: Proceedings of the 30th ACM Interna- tional Conference on Multimedia. pp. 5152–5161 (2022)

work page 2022

[6] [6]

Eslami, S., Meinel, C., De Melo, G.: Pubmedclip: How much does clip benefit visual question answering in the medical domain? In: Findings of the Association for Computational Linguistics: EACL 2023. pp. 1181–1193 (2023)

work page 2023

[7] [7]

Gai, X., Zhou, C., Liu, J., Feng, Y., Wu, J., Liu, Z.: Medthink: Explaining medical visualquestionansweringviamultimodaldecision-makingrationale.arXivpreprint arXiv:2404.12372 (2024)

arXiv 2024

[8] [8]

In: Proceedings of the 2021 International Conference on Multimedia Retrieval

Gong, H., Chen, G., Liu, S., Yu, Y., Li, G.: Cross-modal self-attention with multi- task pre-training for medical visual question answering. In: Proceedings of the 2021 International Conference on Multimedia Retrieval. pp. 456–460 (2021)

work page 2021