Pith. sign in

REVIEW 5 major objections 9 minor 52 references

ClinKD: Cross-Modal Clinical Knowledge Distiller For Multi-Task Medical Images

T0 review · 5 major / 9 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ClinKD claims that a training recipe of MCG-RoPE and pseudo-label distillation turns a general-purpose MLLM into a state-of-the-art multi-task medical VQA system.

desk verdict ClinKD reports a large SOTA jump on Med-GRIT, but the biggest claimed gain comes from a pseudo-label distillation whose teacher is never identified; as written, the central attribution is unverifiable. read the letter →

arxiv 2502.05928 v4 pith:YYIFJ3MK submitted 2025-02-09 cs.CV

Hongyu Ge , Longkun Hao , Zihui Xu , Zhenxin Lin , Bin Li , Shoujun Zhou , Hongjin Zhao , Yihang Liu This is my paper · ORCID
classification cs.CV
keywords ClinKDMed-VQAmulti-taskmedicalimageanalysisrotarypositionembeddingknowledgedistillationpseudo-labelcurriculumvisualgroundingreferringand
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ClinKD aims to establish that a general-purpose multimodal large language model can be turned into a competitive multi-task medical VQA system by changing only the training recipe, not the architecture. The framework adds a position embedding that assigns different rotation intervals to image and text tokens, and a pseudo-label distillation stage that weights teacher predictions by confidence and margin under a curriculum threshold. On Med-GRIT-Test30k the paper reports an average score of 71.53%, against 56.66% for the prior BiRD model and 5.53% for LLaVA-Med. The authors attribute these gains to better image-text alignment and to supplying medical knowledge before supervised fine-tuning. If correct, the result makes medical VQA less a data-scarcity problem and more a training-recipe problem.

What carries the argument

The load-bearing object is Med-CLIP Guided Rotary Position Embedding (MCG-RoPE): it applies an affine transform $F(v^{(1)}_i,v^{(2)}_i)=(\alpha_1 v^{(1)}_i+\lambda_1,\alpha_2 v^{(2)}_i+\lambda_2)$ to image position indices so that inter-modal and intra-modal gaps are controlled separately, and then rotates the height and width halves of each query/key with frequency-scaled angles $\theta^{(h)}_{x_h,i}=x_h\omega^{(h)}_i$ and $\theta^{(w)}_{x_w,i}=x_w\omega^{(w)}_i$. The second mechanism is Adaptive Confidence-Margin Curriculum Pseudo-KD, which computes teacher confidence $C(x)=\max_i p_i(x)$ and margin $\Delta(x)=p_{(1)}(x)-p_{(2)}(x)$, keeps samples with $C(x)\ge\tau(t)$ under a linearly decaying threshold $\tau(t)=\tau_{\min}+(\tau_0-\tau_{\min})(1-t)$, weights them by $w(x)=C(x)^\gamma\Delta(x)^\beta$, and trains with $L_{\mathrm{Distill}}=(L_{\mathrm{CE}}+\alpha L_{\mathrm{KD}})/(1+\alpha)$. These are rounded out by Reflective Correction Training, which sends low-similarity predictions back to GPT-4o for rewriting, and Semantic-Aware Selective Generation, which selects the CLIP-best of multiple generated answers. The two main mechanisms are what carry the reported gains.

What would settle it

Identify the pseudo-label teacher used in Section 3.2 and compare its predictions on the Med-GRIT training set with the student's own fine-tuned predictions from the previous round; if the two are nearly identical or the teacher's training data is the same Med-GRIT data, the Pseudo-KD gain is self-distillation. A more direct test: rerun ClinKD with Pseudo-KD replaced by self-distillation from the student's own high-confidence predictions, and check whether the reported +12.69 average improvement survives.

Watch

Extended reading notes

Core claim

ClinKD's central claim is that two bottlenecks in medical VQA, inadequate image-text alignment and a medical-knowledge gap, can be addressed independently, and that fixing both gives state-of-the-art results on Med-GRIT. The paper reports 67.51% visual grounding, 82.35% referring object classification, 70.56% referring captioning, and 65.69% medical image analysis on Med-GRIT-Test30k, an average of 71.53% versus 56.66% for BiRD and 5.53% for LLaVA-Med; on LLaVA-Med-qa0.2k it reports an mBMR of 23.54 versus 21.06 for BiRD. Ablations attribute +7.78 average points to MCG-RoPE and +12.69 average points to Pseudo-KD, with the two combined outperforming either component alone. The paper also reports per-modality gains across CT, MR, X-ray, PET, endoscopy, dermoscopy, fundus, and ultrasound, together with few-shot gains on VQA-RAD, SLAKE, and Path-VQA.

Load-bearing premise

The load-bearing premise is that the unnamed pseudo-label teacher supplies medical knowledge the student does not already have; if that teacher was trained on the same Med-GRIT-style data and shares the student's biases, Pseudo-KD's reported +12.69 average gain would be self-distillation rather than new knowledge.

Editorial extensions

If this is right

  • On the paper's evidence, a changed position embedding plus a distillation schedule is enough to lift multi-task Med-VQA to the reported state-of-the-art level on Med-GRIT, without altering the base model family.
  • The MCG-RoPE gain is broadly distributed: all eight imaging modalities improve in at least some tasks, and MCG-RoPE beats RoPE-Mixed on all four task averages.
  • Pseudo-KD before supervised fine-tuning produces the largest single average improvement in the ablation (+12.69 points), supporting the paper's claim that medical knowledge can be injected through pseudo-labels rather than new data collection.
  • Combining MCG-RoPE and Pseudo-KD gives an average of 71.53%, above either component's individual average (64.44% and 69.35%), so the two mechanisms are complementary in the measured setting.
  • Semantic-Aware Selective Generation is not a reliable addition: it slightly improves VG and ROC but slightly lowers RC and MIA, so it should be used selectively per task.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the Pseudo-KD gain comes mostly from the curriculum weighting rather than from a genuinely distinct teacher, the same loss could be used as a self-distillation regularizer on unlabeled or weakly supervised medical images, which would make the recipe cheaper and easier to reproduce.
  • The affine parameters $\alpha_1,\alpha_2,\lambda_1,\lambda_2$ in MCG-RoPE are likely tunable per modality; sensitivity analysis on a validation split is a natural next experiment and could close the remaining per-modality gaps in Table 3.
  • Because the pseudo-label teacher is never identified in Section 3.2, the paper's own ablation cannot distinguish new medical knowledge from the student's own biases recycled through a confidence filter unless the teacher checkpoint and training data are released.
  • An outside check on a different base model would clarify whether the gains are specific to Qwen2-VL or transfer to other multimodal backbones.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 9 minor

Summary. The paper proposes ClinKD, a training and inference framework for multi-task medical VQA built on top of the BiRD/Qwen2-VL family. It introduces three components: (i) MCG-RoPE, a rotary position embedding that assigns distinct position intervals to image and text tokens to improve image-text alignment; (ii) Adaptive Confidence-Margin Curriculum Pseudo-KD, a knowledge-distillation loss with confidence- and margin-based sample weighting and a decaying threshold; and (iii) Reflective Correction Training, which uses GPT-4o to rewrite low-similarity predictions during fine-tuning, plus a Semantic-Aware Selective Generation strategy at inference that picks the highest-CLIP-scored answer among multiple samples. Experiments on Med-GRIT-Test30k report large gains over BiRD (67.51% VG, 82.35% ROC, 70.56% RC, 65.69% MIA, average 71.53% vs. 56.66%), with ablations attributing the largest component gain to Pseudo-KD (+12.69 average) and a substantial gain to MCG-RoPE (+7.78 average). Few-shot results on VQA-RAD, SLAKE, and Path-VQA are also reported, and code/checkpoints are promised via a GitHub link.

Significance. If the results are reproducible, ClinKD would be a strong empirical recipe for multi-task medical VQA, improving both visual grounding and medical reasoning without changing the base model family. The paper includes a broad set of experiments across eight modalities, reports standard deviations, and provides component-wise ablations, which are strengths. The central novelty, however, rests on two load-bearing mechanisms whose implementation details are missing: the identity and provenance of the pseudo-label teacher, and the specific values of the MCG-RoPE affine transform parameters. These omissions prevent verification and weaken the stated causal claim that Pseudo-KD injects external medical knowledge. The inference-time semantic selection also needs scrutiny as a potential source of evaluation bias, though the paper transparently reports its effect in the ablation. Overall, the direction is plausible and the empirical gains are large, but the manuscript is not currently reproducible.

major comments (5)
  1. [Section 3.2, Eqs. (9)-(14)] The teacher network ft(x) in the Pseudo-KD component is never specified: no architecture, no weights, no training data, and no relation to the student or to Med-GRIT-270k. Figure 2 labels the teacher as "ClinKD," which would make the procedure self-distillation. Since Table 2 shows Pseudo-KD contributes +12.69 average, the largest single component gain, the claim that this component supplies "prior medical knowledge" from an independent source is unsupported, and the method cannot be instantiated by a reader. The authors must either specify the teacher (including its training data and relationship to the student) or reframe the component as self-training/self-distillation and provide evidence that this still explains the gains.
  2. [Section 3.1, Eq. (2)] The affine transform F mapping image indices to new positions depends on α1, α2, λ1, λ2, but no values or derivation are given. The text claims this "enforces a pronounced interval difference" and "making inter-modal and intra-modal gaps equal," yet the formula does not by itself equalize gaps unless specific constraints are imposed. Without these values, the MCG-RoPE ablation gain of +7.78 average in Table 2 cannot be reproduced or tested, and the superiority over 2D-RoPE/RoPE-Mixed in Table 3 is not verifiable. Please provide the actual hyperparameter values, or a constraint that determines them, and a sensitivity study.
  3. [Section 3.3, Eq. (15)-(16)] Reflective Correction Training relies on GPT-4o to rewrite low-similarity answers, but the procedure is underspecified: how many samples are corrected per fine-tuning round, what prompt is used, whether corrected samples are added to the training set or simply reused once, and how the authors account for the dependence on a proprietary model. The weighted cosine similarity in Eq. (15) also uses µ and ν with only the qualitative statement "we typically choose µ > ν," and the threshold τ=0.8 is stated without sensitivity analysis. These details matter because the method is presented as part of the training recipe and could affect the reported gains.
  4. [Section 3, Eq. (2), (11), (12), (14)] A large set of free hyperparameters (α1, α2, λ1, λ2, γ, β, τ0, τmin, α, µ, ν) are introduced with no stated selection criterion. If these were tuned on the evaluation benchmark (Med-GRIT-Test30k) or on a test-derived validation split, the reported improvements would be inflated by selection bias. The manuscript should provide a table of hyperparameter values, specify the validation split used for tuning, and include a sensitivity analysis for at least the most influential ones (α in Eq. (14), τ0, τmin, γ, β).
  5. [Section 4.3.4, Table 4] The few-shot experiments on VQA-RAD, SLAKE, and Path-VQA do not report the number of shots, the random seed(s), or the sample-selection procedure. Without these details, it is unclear whether ClinKD is evaluated under the same few-shot protocol as LLaVA-Med, BiRD, and Med-Flamingo, so the claimed few-shot superiority cannot be assessed. Please specify the protocol and include variance over multiple runs.
minor comments (9)
  1. [Section 3.4] The text says "The pseudo code is shown in Algorithm," but no algorithm pseudocode appears in the manuscript; either include it or remove the reference.
  2. [Section 4.1] Typo: "We train ClinKD on the Med-GRIT-270k dataset, and evaluate the performance on Med-GRIT-30k and LLaVA-Med-qa0.2k dataset" — "alse" is misspelled and the sentence structure is awkward.
  3. [Section 4.3.4] Grammar: "It is can be seen that the BiRD" should be "It can be seen that BiRD."
  4. [Figure 2] In Figure 2, the teacher is labeled "ClinKD," which contradicts the idea that the teacher is an external model; this label should be corrected or clarified to avoid confusion.
  5. [Figure 4a] The caption and text refer to "BiRD [14]" in the case study, but the reference list entry for BiRD is [15]; please fix the citation.
  6. [Section 3.1] The name "Med-CLIP" suggests use of a specific medical CLIP model, but the methodology never mentions Med-CLIP beyond the name; if it is not used, consider renaming or clarifying the source of the name.
  7. [Section 3.1, Eq. (2)] It is unclear whether the affine transform F is a fixed preprocessing step, a learnable layer, or part of the position-embedding function; please specify the implementation.
  8. [Section 4.3.2] The row for Qwen2-VL in Table 2 shows "55.43 (+1.51)" for VG, but the baseline BiRD is 53.92; the formatting of the +1.51 is inconsistent with the other deltas, which are given as (+1.51) after the value; this is fine, but the table otherwise reports deltas in parentheses after the whole-model rows while the per-component rows list deltas directly; please standardize.
  9. [Abstract] The GitHub link is only in the abstract; the body does not mention code availability. Please add a "Code Availability" statement in the paper for consistency with the abstract.

Circularity Check

1 steps flagged · score 6.0 of 10

Pseudo-KD's teacher is never specified and is labeled as ClinKD itself, so the largest ablation gain (+12.69 avg) is presented as external 'prior medical knowledge' but reduces by construction to self-distillation unless an independent teacher is identified.

  1. self definitional [Section 3.2, Eq. (9)-(14); Figure 2 caption; Section 4.3.2, Table 2]
    "Given an input example x, the teacher network produces logits z_t = f_t(x). ... The ClinKD system undergoes distillation before supervised fine-tuning (SFT). ... Pseudo-KD provides prior medical knowledge by using pseudo-labels [21, 6], filling the gap of prior medical knowledge so that the model can better adapt to medical knowledge during the supervised fine-tuning."

    The paper never defines f_t: no architecture, weights, or training data are given. The only teacher depicted in the framework figures is 'ClinKD (Teacher)', i.e., the proposed system itself. If f_t is ClinKD or a frozen/EMA copy of it, then the pseudo-labels in Eqs. (9)-(13) are the student's own logits, and the 'prior medical knowledge' said to fill the student's gap is, by construction, already present in the student. Table 2's largest single component (Pseudo-KD, +12.69 average over BiRD) is therefore self-training/regularization, not an external medical-knowledge source. The headline 14.87% SOTA improvement and the abstract's 'medical knowledge transformation' mechanism rest on this self-referential loop unless an independent teacher is identified.

full rationale

There is no formal theorem derivation in the paper, so most of the work is empirical engineering rather than circular derivation. MCG-RoPE is a tunable positional-embedding modification, and Semantic-Aware Selective Generation is a test-time selection heuristic. The load-bearing circular step is the Pseudo-KD teacher: Section 3.2 defines the teacher only by anonymous logits f_t(x), the framework figures label the teacher as 'ClinKD', and the manuscript never identifies an external teacher checkpoint, training set, or any independence from the student. Because the Pseudo-KD ablation is the largest contributor to the reported gains and the paper credits those gains to 'prior medical knowledge', the central causal claim is either self-distillation or an uninstantiated external-knowledge claim. This warrants a partial-circularity score of 6. I do not score higher because the paper never explicitly writes 'teacher = student'; the reduction is conditional on the teacher identity, although the absence of any other teacher specification plus the figure labeling makes the self-referential reading the natural one. No self-citation chain or imported uniqueness theorem is involved.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The central claim rests on a set of hyperparameters whose values and fitting procedures are mostly undisclosed, plus domain assumptions about teacher pseudo-labels, CLIP ranking, and GPT-4o correction. The paper introduces no new physical entities or formal axioms, but the ad hoc position-interval design and the unidentified teacher are the main extra assumptions the reader must accept.

free parameters (7)
  • alpha1, alpha2, lambda1, lambda2 = not stated
    Affine transform of image position indices in Eq. 2; no fitting criterion or values given. If tuned on the test set, the MCG-RoPE gain is a fit.
  • gamma, beta = not stated
    Sensitivity exponents in the adaptive weighting function Eq. 12; no values reported.
  • tau0, tau_min = not stated
    Initial and minimum confidence thresholds in the curriculum schedule Eq. 11; no values reported.
  • alpha = 0.5
    Distillation loss weight in Eq. 14; the text reports that the best average is achieved at alpha=0.5.
  • T = not stated
    Distillation temperature in Eq. 13; no value reported.
  • mu, nu = not stated
    Weights in the semantic similarity Eq. 15; the text only says mu > nu.
  • tau (correction threshold) = 0.8
    Threshold in Eq. 16 for triggering GPT-4o correction; fixed without sensitivity analysis.
assumptions (4)
  • domain assumption Teacher pseudo-labels provide independent prior medical knowledge
    Sec 3.2 assumes distillation from a teacher fills a medical knowledge gap, but the teacher is never identified, so the independence of this knowledge is unverified.
  • ad hoc to paper Distinct inter-modal position intervals improve image-text alignment
    Sec 3.1 introduces affine position scaling and offsets without a derivation or an ablation isolating the interval choice from other hyperparameters.
  • domain assumption CLIP score ranks answer quality for medical images
    Sec 3.4 uses CLIP to select among candidate answers, although CLIP is not trained for medical images and the ablation shows it can hurt RC and MIA.
  • domain assumption GPT-4o corrections preserve ground-truth intent and improve label diversity
    Sec 3.3 assumes GPT-4o revisions keep the anchor box and align with ground truth; no manual verification or error analysis is given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ClinKD: Cross-Modal Clinical Knowledge Distiller For Multi-Task Medical Images." pith.science (2026). https://pith.science/paper/YYIFJ3MK

@misc{pith2026250205928,
  author       = {Pith},
  title        = {Pith review of: ClinKD: Cross-Modal Clinical Knowledge Distiller For Multi-Task Medical Images},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YYIFJ3MK}},
  note         = {Machine review of arXiv:2502.05928}
}
read the original abstract

Medical Visual Question Answering (Med-VQA) represents a critical and challenging subtask within the general VQA domain. Despite significant advancements in general VQA, multimodal large language models (MLLMs) still exhibit substantial limitations when handling multi-task VQA scenarios. These limitations manifest through erroneous spatial localization and misinterpretation of medical images, which primarily arise from two fundamental issues: inadequate image-text alignment and insufficient domain-specified knowledge for medical applications. To address these issues, we introduce the Cross-Modal Clinical Knowledge Distiller (ClinKD), an innovative framework designed to enhance image-text alignment and establish more effective medical knowledge transformation mechanisms, which enables MLLMs to perform better even when lacking prior medical knowledge. Our extensive experimental evaluations demonstrate that the ClinKD achieves state-of-the-art performance on several datasets which are challenging for Med-VQA task. The results indicate that our approach not only significantly improves image-text alignment but also effectively enables MLLMs to adapt to the medical knowledge. The source code for ClinKD is available at: https://github.com/overloadedHenry/ClinKD.

Figures

Figures reproduced from arXiv: 2502.05928 by the authors.

Figure 1
Figure 1. The left part shows that the BiRD [15] only utilizes traditional supervised fine-tuning (SFT) and 2D-RoPE [34] to obtain the capabilities of understanding and grounding medical images. The right part is our method which uses Med-CLIP Guided RoPE for enhancing image-text alignment and Adaptive Confidence-Margin Curriculum Pseudo-KD for filling the gap of medical knowledge in ClinKD. During the SFT procedure, we will … view at source ↗
Figure 2
Figure 2. Overview of our ClinKD’s framework: The framework begins with multi-task medical [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The visualized example of Med-CLIP Guided Rotary Position Embedding. The original [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Case Studies on Med-GRIT-Test30 [41, 7] and VQA-RAD [17]. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 23 canonical work pages

  1. [1]

    Hasan, Vivek Datla, Joey Liu, Dina Demner-Fushman, and Henning Müller

    Asma Ben Abacha, Sadid A. Hasan, Vivek Datla, Joey Liu, Dina Demner-Fushman, and Henning Müller. Vqa-med: Overview of the medical visual question answering task at imageclef 2019. InConference and Labs of the Evaluation Forum, 2019

    work page 2019

  2. [2]

    Spice: Semantic propositional image caption evaluation

    Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors,Computer Vision – ECCV 2016, pages 382–398, Cham, 2016. Springer International Publishing

    work page 2016

  3. [3]

    Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arxiv 2023.arXiv preprint arXiv:2308.12966, 1(8), 2023. 9

    arXiv 2023

  4. [4]

    The revolution of multimodal large language models: A survey

    Davide Caffagni, Federico Cocchi, Luca Barsellotti, Nicholas Moratelli, Sara Sarto, Lorenzo Baraldi, Lorenzo Baraldi, Marcella Cornia, and Rita Cucchiara. The revolution of multimodal large language models: A survey. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Findings of the Association for Computational Linguistics: ACL 2024, pages 13590–1...

    work page 2024

  5. [5]

    R-llava: Improving med-vqa understanding through visual region of interest.arXiv preprint arXiv:2410.20327, 2024

    Xupeng Chen, Zhixin Lai, Kangrui Ruan, Shichu Chen, Jiaxiang Liu, and Zuozhu Liu. R-llava: Improving med-vqa understanding through visual region of interest.arXiv preprint arXiv:2410.20327, 2024

    arXiv 2024

  6. [6]

    Mixed pseudo labels for semi-supervised object detection.arXiv preprint arXiv:2312.07006, 2023

    Zeming Chen, Wenwei Zhang, Xinjiang Wang, Kai Chen, and Zhi Wang. Mixed pseudo labels for semi-supervised object detection.arXiv preprint arXiv:2312.07006, 2023

    arXiv 2023

  7. [7]

    Sam-med2d.arXiv preprint arXiv:2308.16184, 2023

    Junlong Cheng, Jin Ye, Zhongying Deng, Jianpin Chen, Tianbin Li, Haoyu Wang, Yanzhou Su, Ziyan Huang, Jilong Chen, Lei Jiang, et al. Sam-med2d.arXiv preprint arXiv:2308.16184, 2023

    arXiv 2023

  8. [8]

    Kaiwen Cui, Yingchen Yu, Fangneng Zhan, Shengcai Liao, Shijian Lu, and Eric P. Xing. Kd-dlgan: Data limited image generation via knowledge distillation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3872–3882, June 2023

    work page 2023

Show all 52 references
  1. [9]

    Enhancing medical VQA with multimodal determination rationales

    Xiaotang Gai, Chenyi Zhou, Jiaxiang Liu, Y ANG FENG, Jian Wu, and Zuozhu Liu. Enhancing medical VQA with multimodal determination rationales. InGenAI for Health: Potential, Trust and Policy Compliance, 2024

    2024

  2. [10]

    Hasan, Yuan Ling, Oladimeji Farri, Joey Liu, Henning Müller, and Matthew P

    Sadid A. Hasan, Yuan Ling, Oladimeji Farri, Joey Liu, Henning Müller, and Matthew P. Lungren. Overview of imageclef 2018 medical domain visual question answering task. InConference and Labs of the Evaluation Forum, 2018

    2018

  3. [11]

    Parameter-efficient fine-tuning medical multimodal large language models for medical visual grounding.arXiv preprint arXiv:2410.23822, 2024

    Jinlong He, Pengfei Li, Gang Liu, and Shenjun Zhong. Parameter-efficient fine-tuning medical multimodal large language models for medical visual grounding.arXiv preprint arXiv:2410.23822, 2024

    2024 arXiv

  4. [12]

    Pathvqa: 30000+ questions for medical visual question answering.arXiv preprint arXiv:2003.10286, 2020

    Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. Pathvqa: 30000+ questions for medical visual question answering.arXiv preprint arXiv:2003.10286, 2020

    2003 arXiv

  5. [13]

    Rotary position embedding for vision transformer

    Byeongho Heo, Song Park, Dongyoon Han, and Sangdoo Yun. Rotary position embedding for vision transformer. In Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol, editors,Computer Vision – ECCV 2024, pages 289–305, Cham, 2025. Springer Nat...

    2024

  6. [14]

    Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531, 2015

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531, 2015

    2015 arXiv

  7. [15]

    A refer-and-ground multimodal large language model for biomedicine

    Xiaoshuang Huang, Haifeng Huang, Lingdong Shen, Yehui Yang, Fangxin Shang, Junwei Liu, and Jia Liu. A refer-and-ground multimodal large language model for biomedicine. In Marius George Linguraru, Qi Dou, Aasa Feragen, Stamatia Giannarou, Ben Glocker, Karim Lekadir, and Julia A...

    2024

  8. [16]

    Medvisionllama: Leveraging pre-trained large language model layers to enhance medical image segmentation.arXiv preprint arXiv:2410.02458, 2024

    Gurucharan Marthi Krishna Kumar, Aman Chadha, Janine Mendola, and Amir Shmuel. Medvisionllama: Leveraging pre-trained large language model layers to enhance medical image segmentation.arXiv preprint arXiv:2410.02458, 2024

    2024 arXiv

  9. [17]

    A dataset of clinically generated visual questions and answers about radiology images.Scientific data, 5(1):1–10, 2018

    Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images.Scientific data, 5(1):1–10, 2018

    2018

  10. [18]

    Vision-language instruction tuning: A review and analysis

    Chen Li, Yixiao Ge, Dian Li, and Ying Shan. Vision-language instruction tuning: A review and analysis. arXiv preprint arXiv:2311.08172, 2023

    2023 arXiv

  11. [19]

    Llava-med: Training a large language-and-vision assistant for biomedicine in one day.Advances in Neural Information Processing Systems, 36:28541–28564, 2023

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Nau- mann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-and-vision assistant for biomedicine in one day.Advances in Neural Information Processing Systems, 36:28541–2...

    2023

  12. [20]

    Towards visual-prompt temporal answer grounding in instructional video.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    Shutao Li, Bin Li, Bin Sun, and Yixuan Weng. Towards visual-prompt temporal answer grounding in instructional video.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  13. [21]

    Pseudo labels for unsupervised domain adaptation: A review

    Yundong Li, Longxia Guo, and Yizheng Ge. Pseudo labels for unsupervised domain adaptation: A review. Electronics, 12(15), 2023

    2023

  14. [22]

    A comprehensive survey and guide to multimodal large language models in vision-language tasks.arXiv preprint arXiv:2411.06284, 2024

    Chia Xin Liang, Pu Tian, Caitlyn Heqi Yin, Yao Yua, Wei An-Hou, Li Ming, Tianyang Wang, Ziqian Bi, and Ming Liu. A comprehensive survey and guide to multimodal large language models in vision-language tasks.arXiv preprint arXiv:2411.06284, 2024. 10

    2024

  15. [23]

    Medfilip: Medical fine-grained language-image pre-training.IEEE Journal of Biomedical and Health Informatics, pages 1–11, 2025

    Xinjie Liang, Xiangyu Li, Fanding Li, Jie Jiang, Qing Dong, Wei Wang, Kuanquan Wang, Suyu Dong, Gongning Luo, and Shuo Li. Medfilip: Medical fine-grained language-image pre-training.IEEE Journal of Biomedical and Health Informatics, pages 1–11, 2025

    2025

  16. [24]

    Healthgpt: A medical large vision-language model for unifying com- prehension and generation via heterogeneous knowledge adaptation.arXiv preprint arXiv:2502.09838, 2025

    Tianwei Lin, Wenqiao Zhang, Sijing Li, Yuqian Yuan, Binhe Yu, Haoyuan Li, Wanggui He, Hao Jiang, Mengze Li, Xiaohui Song, et al. Healthgpt: A medical large vision-language model for unifying com- prehension and generation via heterogeneous knowledge adaptation.arXiv preprint a...

    2025 arXiv

  17. [25]

    Lawrence Zitnick

    Tsung Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. InLecture Notes in Computer Science, Springer, volume 8693 LNCS ofLecture Notes in Computer Science (includin...

    2014

  18. [26]

    Medical visual question answering: A survey.Artificial Intelligence in Medicine, 143:102611, September 2023

    Zhihong Lin, Donghao Zhang, Qingyi Tao, Danli Shi, Gholamreza Haffari, Qi Wu, Mingguang He, and Zongyuan Ge. Medical visual question answering: A survey.Artificial Intelligence in Medicine, 143:102611, September 2023

    2023

  19. [27]

    Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering

    Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering. In2021 IEEE 18th international symposium on biomedical imaging (ISBI), pages 1650–1654. IEEE, 2021

    2021

  20. [28]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26296–26306, June 2024

    2024

  21. [29]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors,Advances in Neural Information Processing Systems, volume 36, pages 34892–34916. Curran Associates, Inc., 2023

    2023

  22. [30]

    Hc-llm: Historical-constrained large language models for radiology report generation.arXiv preprint arXiv:2412.11070, 2024

    Tengfei Liu, Jiapu Wang, Yongli Hu, Mingjie Li, Junfei Yi, Xiaojun Chang, Junbin Gao, and Baocai Yin. Hc-llm: Historical-constrained large language models for radiology report generation.arXiv preprint arXiv:2412.11070, 2024

    2024 arXiv

  23. [31]

    Vkd: Improving knowledge distillation using orthogonal projections

    Roy Miles, Ismail Elezi, and Jiankang Deng. Vkd: Improving knowledge distillation using orthogonal projections. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15720–15730, June 2024

    2024

  24. [32]

    Med-flamingo: a multimodal medical few-shot learner

    Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Yash Dalmia, Jure Leskovec, Cyril Zakka, Eduardo Pontes Reis, and Pranav Rajpurkar. Med-flamingo: a multimodal medical few-shot learner. In Machine Learning for Health (ML4H), pages 353–367. PMLR, 2023

    2023

  25. [33]

    Learning deep representations with probabilistic knowledge transfer

    Nikolaos Passalis and Anastasios Tefas. Learning deep representations with probabilistic knowledge transfer. InProceedings of the European Conference on Computer Vision (ECCV), September 2018

    2018

  26. [34]

    Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063, 2024

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063, 2024

    2024

  27. [35]

    Similarity-preserving knowledge distillation

    Frederick Tung and Greg Mori. Similarity-preserving knowledge distillation. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019

    2019

  28. [36]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  29. [37]

    ITA: Image-text alignments for multi-modal named entity recognition

    Xinyu Wang, Min Gui, Yong Jiang, Zixia Jia, Nguyen Bach, Tao Wang, Zhongqiang Huang, and Kewei Tu. ITA: Image-text alignments for multi-modal named entity recognition. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz, editors,Proceedings of the 2022 ...

    2022

  30. [38]

    Videorope: What makes for good video rotary position embedding?arXiv preprint arXiv:2502.05173, 2025

    Xilin Wei, Xiaoran Liu, Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Jian Tong, Haodong Duan, Qipeng Guo, Jiaqi Wang, et al. Videorope: What makes for good video rotary position embedding?arXiv preprint arXiv:2502.05173, 2025. 11

    2025 arXiv

  31. [39]

    Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities.arXiv preprint arXiv:2408.07666, 2024

    Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities.arXiv preprint arXiv:2408.07666, 2024

    2024 arXiv

  32. [40]

    Llm-medqa: Enhancing medical question answering through case studies in large language models

    Hang Yang, Hao Chen, Hui Guo, Yineng Chen, Ching-Sheng Lin, Shu Hu, Jinrong Hu, Xi Wu, and Xin Wang. Llm-medqa: Enhancing medical question answering through case studies in large language models. arXiv preprint arXiv:2501.05464, 2024

    2024 arXiv

  33. [41]

    Sa-med2d-20m dataset: Segment anything in 2d medical imaging with 20 million masks.arXiv preprint arXiv:2311.11969, 2023

    Jin Ye, Junlong Cheng, Jianpin Chen, Zhongying Deng, Tianbin Li, Haoyu Wang, Yanzhou Su, Ziyan Huang, Jilong Chen, Lei Jiang, et al. Sa-med2d-20m dataset: Segment anything in 2d medical imaging with 20 million masks.arXiv preprint arXiv:2311.11969, 2023

    2023 arXiv

  34. [42]

    Ferret: Refer and ground anything anywhere at any granularity.arXiv preprint arXiv:2310.07704, 2023

    Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity.arXiv preprint arXiv:2310.07704, 2023

    2023 arXiv

  35. [43]

    Visual-oriented fine-grained knowledge editing for multimodal large language models.arXiv preprint arXiv:2411.12790, 2024

    Zhen Zeng, Leijiang Gu, Xun Yang, Zhangling Duan, Zenglin Shi, and Meng Wang. Visual-oriented fine-grained knowledge editing for multimodal large language models.arXiv preprint arXiv:2411.12790, 2024

    2024 arXiv

  36. [44]

    Ferret-v2: An improved baseline for referring and grounding with large language models.arXiv preprint arXiv:2404.07973, 2024

    Haotian Zhang, Haoxuan You, Philipp Dufter, Bowen Zhang, Chen Chen, Hong-You Chen, Tsu-Jui Fu, William Yang Wang, Shih-Fu Chang, Zhe Gan, et al. Ferret-v2: An improved baseline for referring and grounding with large language models.arXiv preprint arXiv:2404.07973, 2024

    2024 arXiv

  37. [45]

    Davison, Hui Ren, Jing Huang, Chen Chen, Yuyin Zhou, Sunyang Fu, Wei Liu, Tianming Liu, Xiang Li, Yong Chen, Lifang He, James Zou, Quanzheng Li, Hongfang Liu, and Lichao Sun

    Kai Zhang, Rong Zhou, Eashan Adhikarla, Zhiling Yan, Yixin Liu, Jun Yu, Zhengliang Liu, Xun Chen, Brian D. Davison, Hui Ren, Jing Huang, Chen Chen, Yuyin Zhou, Sunyang Fu, Wei Liu, Tianming Liu, Xiang Li, Yong Chen, Lifang He, James Zou, Quanzheng Li, Hongfang Liu, and Lichao ...

    2024

  38. [46]

    Negative-aware attention framework for image-text matching

    Kun Zhang, Zhendong Mao, Quan Wang, and Yongdong Zhang. Negative-aware attention framework for image-text matching. In2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15640–15649, 2022

    2022

  39. [47]

    Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs.arXiv preprint arXiv:2303.00915, 2023

    Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, et al. Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs.arXiv preprint arXiv:2303.00915, 2023

    2023 arXiv

  40. [48]

    Instruction tuning for large language models: A survey.arXiv preprint arXiv:2308.10792, 2023

    Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. Instruction tuning for large language models: A survey.arXiv preprint arXiv:2308.10792, 2023

    2023

  41. [49]

    Consecutive knowledge meta-adaptation learning for unsupervised medical diagnosis.Knowledge-Based Systems, 291:111573, 2024

    Yumin Zhang, Hongliu Li, Yawen Hou, Xiuyi Chen, Hongyuan Yu, and Long Xia. Consecutive knowledge meta-adaptation learning for unsupervised medical diagnosis.Knowledge-Based Systems, 291:111573, 2024

    2024

  42. [50]

    Scene parsing through ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5122–5130, 2017

    2017

  43. [51]

    Semantic understanding of scenes through the ade20k dataset.International Journal of Computer Vision, 127:302–321, 2019

    Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset.International Journal of Computer Vision, 127:302–321, 2019. 12

    2019

  44. [2024]

    Springer Nature Switzerland

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.