Pith. sign in

REVIEW 3 major objections 5 minor 76 references

MedMax: Mixed-Modal Instruction Tuning for Training Biomedical Assistants

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MedMax, a 1.47-million-instance biomedical instruction-tuning dataset, lifts a 7B open mixed-modal model to 65.5% average accuracy on twelve biomedical VQA tasks, 18.3 percentage points above GPT-4o and 26 points above the base Chameleon.

desk verdict A genuinely useful biomedical instruction-tuning dataset, but the headline VQA gains are inflated by benchmark leakage and need a cleaner held-out evaluation before the numbers are taken at face value. read the letter →

arxiv 2412.12661 v2 pith:MFGEPO32 submitted 2024-12-17 cs.AI cs.CLcs.CV

Hritik Bansal , Daniel Israel , Siyan Zhao , Shufan Li , Tung Nguyen , Aditya Grover This is my paper · ORCID
classification cs.AIcs.CLcs.CV
keywords mixed-modalfoundationmodelinstructiontuningbiomedicalvisualquestionansweringmultimodalgenerationmedicalreportunderstandingassistantdatasetcurationevaluationsuite
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MedMax is a large-scale instruction-tuning dataset for native multimodal foundation models, containing 1.47 million instances that span visual question answering, image captioning and generation, visual chat, medical report understanding, and interleaved image-text generation across radiology and histopathology. The paper claims that instruction-tuning a 7B mixed-modal model (Anole, an instantiation of Chameleon) on this dataset raises its average accuracy on twelve biomedical VQA tasks to 65.5%, outperforming the base model by 26 percentage points and GPT-4o by 18.3 percentage points. The authors also introduce a unified evaluation suite covering these tasks, and report gains in image captioning, image generation, multimodal generation, and visual chat. If true, this shows that a carefully curated but largely existing data mix, plus a new GPT-generated interleaved-content component, can turn a generalist mixed-modal model into a competitive biomedical assistant without task-specific fine-tuning.

What carries the argument

The central object is the MedMax dataset itself, a mixture of curated sources (PMC-OA, Quilt-1M, LLaVA-Med, PubMedVision, VQA-RAD, SLAKE, PathVQA, PMC-VQA, OmniMedVQA, MIMIC-CXR) and a new MedMax-Instruct component of 88K interleaved image-text conversations produced with GPT-4o from caption-filtered pairs. The mechanism that transfers these data into skill is the instruction-tuning objective: given a sequence with text and VQGAN image tokens, the model maximizes likelihood of the response tokens conditioned on the instruction, with loss computed only on the response portion. Training uses LoRA on an autoregressive mixed-modal model (Anole-7B). The evaluation suite measures text tasks by exact match or GPT-4o-mini grading and generation tasks by BioMedCLIPScore.

What would settle it

Compute a near-duplicate overlap between the MedMax training set and the QuiltVQA and OmniMedVQA evaluation sets (for example by perceptual image hashing and embedding-based question similarity), and re-measure the VQA accuracy after removing any training instance within a similarity threshold; if the 18–26 point gaps collapse, the central claim of generalization is falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that the MedMax dataset is an effective instruction-tuning resource for mixed-modal foundation models in biomedicine. On the paper's own terms, fine-tuning Anole-7B (a Chameleon-7B instantiation) on MedMax yields an average accuracy of 65.5% across twelve biomedical VQA tasks, versus 39.4% for base Chameleon and 42.0% for GPT-4o; the model also outperforms LLaVA-Med and HuatuoGPT-Vision on average. The same fine-tuned model improves over the base by relative gains of 14–33% in image captioning, 14–100% in image generation, and 25.2%/31.5% on the text and image components of interleaved multimodal generation. The dataset includes 88K newly created interleaved image-text instruction instances (MedMax-Instruct) generated from filtered captions, alongside curated VQA, captioning, generation, visual chat, and report-understanding data. The authors frame this as a foundation for unified biomedical assistants that can both understand and generate multimodal content.

Load-bearing premise

The reported gains depend on the evaluation benchmarks being sufficiently disjoint from MedMax's training data that the model must generalize rather than recall near-duplicate images or questions, yet the paper's contamination check only rules out exact image-text pair matches.

Editorial extensions

If this is right

  • A single 7B instruction-tuned open model can outperform closed generalist models like GPT-4o on biomedical VQA, suggesting that domain-specific instruction-tuning data, not model scale, is the current bottleneck.
  • The same model checkpoint handles VQA, captioning, generation, visual chat, and interleaved report generation, so practitioners could replace several task-specific models with one unified assistant.
  • Performance on the twelve VQA tasks rises monotonically as MedMax is scaled from 25% to 100% of its data, indicating further dataset scaling should yield further gains.
  • Ablation results show removing VQA or visual-chat instances from the mix drops downstream performance by 23% and 17% respectively, so the task diversity in the mixture is what drives generalization.
  • The unified evaluation suite provides a reproducible protocol for comparing mixed-modal biomedical assistants, including new metrics for interleaved multimodal generation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the gains survive stricter contamination checks, the implication extends beyond biomedicine: the combination of large-scale caption curation and LLM-generated interleaved content may be a transferable recipe for other specialized multimodal domains where expert data are scarce.
  • The unusually high 99.5% accuracy on the hidden OmniMedVQA split suggests that when training and evaluation draw from the same distribution, the task becomes near-saturated; a more informative test of generalization would use benchmarks whose sources were not used in training at all.
  • One direct test of the paper's mechanism would be to replace the GPT-4o-generated MedMax-Instruct portion with randomly rewritten captions; if the interleaved generation gains vanish, then the structured multimodal response format itself is the active ingredient.
  • Because the paper uses only one backbone (Anole/Chameleon), it leaves open whether the gains transfer to other mixed-modal architectures such as Transfusion or Emu-3; showing that would separate the value of the data from the value of the model.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces MedMax, a 1.47M-instance multimodal instruction-tuning dataset for biomedical mixed-modal foundation models, spanning VQA, image captioning and generation, visual chat, report understanding, and interleaved image-text generation. The authors fine-tune Anole/Chameleon-7B with LoRA on MedMax and report an average accuracy of 65.5% across twelve biomedical VQA tasks, outperforming Chameleon-7B (39.4%) and GPT-4o (42.0%) by 26.1 and 18.3 percentage points respectively. They also evaluate image captioning/generation, visual chat, and multimodal generation, and present ablations on data scaling, task mixture, and visual encoder finetuning. The central claim is that MedMax is an effective instruction-tuning resource for building unified biomedical multimodal assistants.

Significance. If the reported gains are genuine generalization effects, the paper would make a strong contribution: it provides a large, publicly released instruction-tuning dataset covering a wider range of biomedical multimodal tasks than prior resources, including a newly created interleaved text-image generation subset, and it demonstrates the feasibility of instruction-tuning a native mixed-modal foundation model in the biomedical domain. The dataset curation pipeline is detailed and the authors release data, model, and code, which is valuable for reproducibility. However, the evaluation design has a serious flaw that directly affects the headline numbers: several evaluation benchmarks share source datasets (and, in some cases, images) with the MedMax training data. This prevents the paper from establishing the claimed generalization improvement as stated.

major comments (3)
  1. [§3.2 vs §4.2, Table 2]
  2. [§4.2]
  3. [§5.1, Table 2]
minor comments (5)
  1. [§3.1]
  2. [References]
  3. [§5.1 (Image captioning and generation)]
  4. [Appendix F]
  5. [§4.2]

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the reported VQA gains are empirical measurements on fixed benchmarks, not quantities derived from MedMax by construction.

full rationale

MedMax is an empirical dataset-and-finetuning paper. The central claim is the measured VQA accuracy (65.5% average) of a LoRA-tuned Anole/Chameleon model on a fixed set of 12 benchmarks. This is a benchmark measurement, not a quantity derived from the dataset's definition, so no self-definitional or fitted-input circularity is present. The only load-bearing external inputs are the benchmark test sets and the base model; neither is defined in terms of MedMax. The paper does not invoke a uniqueness theorem or rely on author self-citation to force its conclusions. Potential evaluation overlap (OmniMedVQA and QuiltVQA sharing sources with training data) is a data-contamination/external-validity concern, not a circular-derivation concern, and the paper explicitly acknowledges in-distribution OmniMedVQA exposure. Therefore no circular step is identified.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper contributes a dataset and an empirical model, so the ledger records curation and evaluation assumptions rather than fitted physical parameters. The central claim rests on the validity of the evaluation metrics and the absence of training and evaluation overlap, both of which are only partially established.

free parameters (3)
  • BioMedCLIP class thresholds = Youden's J optimal thresholds on 80 manually labeled images
    Used to filter LLaVA-Med-PMC images in Appendix A.1, affecting 37.8K captioning instances but not the headline VQA claim.
  • Caption quality filter by GPT-4o-mini = binary yes/no
    Removes 25% of captions in MedMax-Instruct Stage 2; the quality judgment is a model choice, not externally validated.
  • Subsampling ratios = 50K Quilt, 100K PubMedVision, 10% 'No findings' MIMIC-CXR
    Hand-chosen dataset sizes in Section 3.2 that influence skill balance and dataset composition.
assumptions (4)
  • domain assumption GPT-4o-generated MedMax-Instruct conversations are clinically accurate and instruction-following
    Data is generated by prompting GPT-4o with captions (Appendix Table 5); no expert verification of the 88K instances, only qualitative checks.
  • domain assumption BioMedCLIPScore is a valid automated metric for biomedical image-text alignment
    Used for captioning and generation evaluation in Table 1; the paper does not validate it against human judgments.
  • domain assumption GPT-4o-mini LLM judgments of open-ended VQA answers are reliable
    Open-ended VQA accuracy in Section 4.2 is computed by GPT-4o-mini comparing prediction to ground truth; judge bias could inflate or deflate scores.
  • domain assumption MIMIC-CXR impression summaries can serve as ground-truth captions
    Used for captioning and generation evaluation, following prior work [7]; these are clinical impressions, not exact image descriptions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MedMax: Mixed-Modal Instruction Tuning for Training Biomedical Assistants." pith.science (2026). https://pith.science/paper/MFGEPO32

@misc{pith2026241212661,
  author       = {Pith},
  title        = {Pith review of: MedMax: Mixed-Modal Instruction Tuning for Training Biomedical Assistants},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MFGEPO32}},
  note         = {Machine review of arXiv:2412.12661}
}
read the original abstract

Recent advancements in mixed-modal generative have opened new avenues for developing unified biomedical assistants capable of analyzing biomedical images, answering complex questions about them, and generating multimodal patient reports. However, existing datasets face challenges such as small sizes, limited coverage of biomedical tasks and domains, and a reliance on narrow sources. To address these gaps, we present MedMax, a large-scale multimodal biomedical instruction-tuning dataset for mixed-modal foundation models. With 1.47 million instances, MedMax encompasses a diverse range of tasks, including interleaved image-text generation, biomedical image captioning and generation, visual chat, and report understanding. These tasks span knowledge across diverse biomedical domains, including radiology and histopathology, grounded in medical papers and YouTube videos. Subsequently, we fine-tune a mixed-modal foundation model on the MedMax dataset, achieving significant performance improvements: a 26% gain over the Chameleon model and an 18.3% improvement over GPT-4o across 12 downstream biomedical visual question-answering tasks. Finally, we introduce a unified evaluation suite for biomedical tasks to guide the development of mixed-modal biomedical AI assistants. The data, model, and code is available at https://mint-medmax.github.io/.

Figures

Figures reproduced from arXiv: 2412.12661 by the authors.

Figure 2
Figure 2. Examples of diverse multimodal biomedical tasks covered in the MedMax dataset. The model inputs (yellow boxes) and corresponding outputs (red boxes) illustrate various task types: multimodal generation with interleaved text and images, medical report generation, text-to-image generation, visual question answering, medical image analysis through visual chat, and image captioning task. Note that report-conditioned ima… view at source ↗
Figure 3
Figure 3. A mixed-modal foundation model is capable of understanding text and image inputs and can [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 5
Figure 5. We source the data from biomedical sources that [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figures from the paper (6 more)
Figure 6
Figure 6. Figure 6: Performance on the multimodal generation task. Comparison between the performance of the MedMax and Chameleon mixed-modal model on the multimodal generation task. We find that MedMax finetuning improves the multimodal content generation capabilities for the biomedical …
Figure 7
Figure 7. Figure 7: Performance on the image captioning and image generation tasks. We find that MedMax model consistency outperforms the base Chameleon mixed-modal model across diverse biomedical domains. outperforms Chameleon by achieving a relative improvement of 25.2% on the quality o…
Figure 8
Figure 8. Figure 8: Performance on the visual chat task. We find that the chatting capabilities of our model is quite competitive, suggesting its ability to answer novel queries about biomedical images. 25 50 75 100 Percent of data 61 62 63 64 65 66 Accuracy (%) Avg. performance on VQA ta…
Figure 10
Figure 10. Figure 10: Results for the data ablation study. Finetuning the mixed-modal model with an ablated version of the MedMax data where the (a) VQA task instances and (b) visual chat instances are removed. The results highlight the usefulness of task-specific data in the mixture for d…
Figure 12
Figure 12. Figure 12: Positive qualitative examples. Sample generations from the MedMax model across diverse tasks: multimodal (interleaved) generation, visual chat, image captioning, and image generation. We verify with experts cited in acknowledgments that the provided medical informatio…
Figure 13
Figure 13. Figure 13: Negative qualitative examples. Sample generations from the MedMax model across diverse tasks: multimodal (interleaved) generation, visual chat, image captioning, and image generation. Pitfalls include poor image generation, confusion between segments within an image, …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

76 extracted references · 29 canonical work pages

  1. [1]

    Multimodal biomedical ai

    Julián N Acosta, Guido J Falcone, Pranav Rajpurkar, and Eric J Topol. Multimodal biomedical ai. Nature Medicine, 28(9):1773–1784, 2022

    work page 2022

  2. [2]

    The medical segmentation decathlon

    Michela Antonelli, Annika Reinke, Spyridon Bakas, Keyvan Farahani, Annette Kopp-Schneider, Bennett A Landman, Geert Litjens, Bjoern Menze, Olaf Ronneberger, Ronald M Summers, et al. The medical segmentation decathlon. Nature communications, 13(1):4128, 2022

    work page 2022

  3. [3]

    Openflamingo: An open-source framework for training large autoregressive vision-language models.arXiv preprint arXiv:2308.01390, 2023

    Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open-source framework for training large autoregressive vision-language models.arXiv preprint arXiv:2308.01390, 2023. 14

    arXiv 2023

  4. [4]

    Qwen-vl: A frontier large vision-language model with versatile abilities.arXiv preprint arXiv:2308.12966, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities.arXiv preprint arXiv:2308.12966, 2023

    arXiv 2023

  5. [5]

    One transformer fits all distributions in multi-modal diffusion at scale

    Fan Bao, Shen Nie, Kaiwen Xue, Chongxuan Li, Shi Pu, Yaole Wang, Gang Yue, Yue Cao, Hang Su, and Jun Zhu. One transformer fits all distributions in multi-modal diffusion at scale. InInternational Conference on Machine Learning, pages 1692–1717. PMLR, 2023

    2023

  6. [6]

    The revolution of multimodal large language models: A survey

    Davide Caffagni, Federico Cocchi, Luca Barsellotti, Nicholas Moratelli, Sara Sarto, Lorenzo Baraldi, Marcella Cornia, Rita Cucchiara, et al. The revolution of multimodal large language models: A survey. In Findings of the Association for Computational Linguistics: ACL 2024. 2024

    work page 2024

  7. [7]

    Roentgen: vision-language foundation model for chest x-ray generation.arXiv preprint arXiv:2211.12737, 2022

    Pierre Chambon, Christian Bluethgen, Jean-Benoit Delbrouck, Rogier Van der Sluijs, Małgorzata Połacin, Juan Manuel Zambrano Chaves, Tanishq Mathew Abraham, Shivanshu Purohit, Curtis P Langlotz, and Akshay Chaudhari. Roentgen: vision-language foundation model for chest x-ray generation.arXiv preprint arXiv:2211.12737, 2022

    arXiv 2022

  8. [8]

    Huatuogpt-vision, towards injecting medical visual knowledge into multimodal llms at scale, 2024

    Junying Chen, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Hardy Chen, Xidong Wang, Ruifei Zhang, Zhenyang Cai, Ke Ji, Guangjun Yu, Xiang Wan, and Benyou Wang. Huatuogpt-vision, towards injecting medical visual knowledge into multimodal llms at scale, 2024

    work page 2024

Show all 76 references
  1. [9]

    Anole: An open, autoregressive, native large multimodal models for interleaved image-text generation.arXiv preprint arXiv:2407.06135, 2024

    Ethan Chern, Jiadi Su, Yan Ma, and Pengfei Liu. Anole: An open, autoregressive, native large multimodal models for interleaved image-text generation.arXiv preprint arXiv:2407.06135, 2024

    2024 arXiv

  2. [10]

    Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023

    2023

  3. [11]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873–12883, 2021

    2021

  4. [12]

    Biomedjourney: Counterfactual biomedical image generation by instruction-learning from multimodal patient journeys.arXiv preprint arXiv:2310.10765, 2023

    Yu Gu, Jianwei Yang, Naoto Usuyama, Chunyuan Li, Sheng Zhang, Matthew P Lungren, Jianfeng Gao, and Hoifung Poon. Biomedjourney: Counterfactual biomedical image generation by instruction-learning from multimodal patient journeys.arXiv preprint arXiv:2310.10765, 2023

    arXiv

  5. [13]

    Pathvqa: 30000+ questions for medical visual question answering.arXiv preprint arXiv:2003.10286, 2020

    Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. Pathvqa: 30000+ questions for medical visual question answering.arXiv preprint arXiv:2003.10286, 2020

    2003 arXiv

  6. [14]

    Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  7. [15]

    Omnimedvqa: A new large-scale comprehensive evaluation benchmark for medical lvlm

    Yutao Hu, Tianbin Li, Quanfeng Lu, Wenqi Shao, Junjun He, Yu Qiao, and Ping Luo. Omnimedvqa: A new large-scale comprehensive evaluation benchmark for medical lvlm. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22170–22183, 2024

    2024

  8. [16]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  9. [17]

    Quilt-1m: One million image-text pairs for histopathology

    Wisdom Ikezogwo, Saygin Seyfioglu, Fatemeh Ghezloo, Dylan Geva, Fatwir Sheikh Mohammed, Pa- van Kumar Anand, Ranjay Krishna, and Linda Shapiro. Quilt-1m: One million image-text pairs for histopathology. Advances in neural information processing systems, 36, 2024. 15

    2024

  10. [18]

    Mimic-cxr-jpg, a large publicly available database of labeled chest radiographs.arXiv preprint arXiv:1901.07042, 2019

    Alistair EW Johnson, Tom J Pollard, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Yifan Peng, Zhiyong Lu, Roger G Mark, Seth J Berkowitz, and Steven Horng. Mimic-cxr-jpg, a large publicly available database of labeled chest radiographs.arXiv preprint arXiv:1901.07042, 2019

    1901 arXiv

  11. [19]

    Peir digital library: Online resources and authoring system

    Kristopher N Jones, Dwain E Woode, Kristina Panizzi, and Peter G Anderson. Peir digital library: Online resources and authoring system. InProceedings of the AMIA Symposium, page 1075. American Medical Informatics Association, 2001

    2001

  12. [20]

    Chaos challenge-combined (ct-mr) healthy abdominal organ segmentation.Medical Image Analysis, 69:101950, 2021

    A Emre Kavur, N Sinem Gezer, Mustafa Barış, Sinem Aslan, Pierre-Henri Conze, Vladimir Groza, Duc Duy Pham, Soumick Chatterjee, Philipp Ernst, Savaş Özkan, et al. Chaos challenge-combined (ct-mr) healthy abdominal organ segmentation.Medical Image Analysis, 69:101950, 2021

    2021

  13. [21]

    T\" ulu 3: Pushing frontiers in open language model post-training.arXiv preprint arXiv:2411.15124, 2024

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. T\" ulu 3: Pushing frontiers in open language model post-training.arXiv preprint arXiv:2411.15124, 2024

    2024 arXiv

  14. [22]

    A dataset of clinically generated visual questions and answers about radiology images.Scientific data, 5(1):1–10, 2018

    Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images.Scientific data, 5(1):1–10, 2018

    2018

  15. [23]

    Mimic-it: Multi-modal in-context instruction tuning, 2023

    Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning, 2023

    2023

  16. [24]

    Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  17. [25]

    Llava-med: Training a large language-and-vision assistant for biomedicine in one day.Advances in Neural Information Processing Systems, 36, 2024

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-and-vision assistant for biomedicine in one day.Advances in Neural Information Processing Systems, 36, 2024

    2024

  18. [26]

    Chatdoctor: A medical chat model fine-tuned on a large language model meta-ai (llama) using medical domain knowledge

    Yunxiang Li, Zihan Li, Kai Zhang, Ruilong Dan, Steve Jiang, and You Zhang. Chatdoctor: A medical chat model fine-tuned on a large language model meta-ai (llama) using medical domain knowledge. Cureus, 15(6), 2023

    2023

  19. [27]

    Pmc- clip: Contrastive language-image pre-training using biomedical documents

    Weixiong Lin, Ziheng Zhao, Xiaoman Zhang, Chaoyi Wu, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pmc- clip: Contrastive language-image pre-training using biomedical documents. InInternational Conference on Medical Image Computing and Computer-Assisted Intervention, pages 525–536. S...

    2023

  20. [28]

    Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering

    Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering. In2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pages 1650–1654. IEEE, 2021

    2021

  21. [29]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023

    2023 arXiv

  22. [30]

    Visual instruction tuning.arXiv preprint arXiv:2304.08485, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.arXiv preprint arXiv:2304.08485, 2023

    2023 arXiv

  23. [31]

    Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action.arXiv preprint arXiv:2312.17172, 2023

    Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action.arXiv preprint arXiv:2312.17172, 2023

    2023 arXiv

  24. [32]

    Macaw-llm: Multi-modal language modeling with image, audio, video, and text integration

    Chenyang Lyu, Minghao Wu, Longyue Wang, Xinting Huang, Bingshuai Liu, Zefeng Du, Shuming Shi, and Zhaopeng Tu. Macaw-llm: Multi-modal language modeling with image, audio, video, and text integration. arXiv preprint arXiv:2306.09093, 2023

    2023 arXiv

  25. [33]

    MedPix — medpix.nlm.nih.gov.https://medpix.nlm.nih.gov/home

    MedPix. MedPix — medpix.nlm.nih.gov.https://medpix.nlm.nih.gov/home. 16

  26. [34]

    Chameleon: Mixed-modal early-fusion foundation models

    Chameleon Meta. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818, 2024

    2024 arXiv

  27. [35]

    Med-flamingo: A multimodal medical few-shot learner

    Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Cyril Zakka, Yash Dalmia, Eduardo Pontes Reis, Pranav Rajpurkar, and Jure Leskovec. Med-flamingo: A multimodal medical few-shot learner. July

  28. [36]

    Can generalist foundation models outcompete special-purpose tuning? case study in medicine.arXiv preprint arXiv:2311.16452, 2023

    Harsha Nori, Yin Tat Lee, Sheng Zhang, Dean Carignan, Richard Edgar, Nicolo Fusi, Nicholas King, Jonathan Larson, Yuanzhi Li, Weishung Liu, et al. Can generalist foundation models outcompete special-purpose tuning? case study in medicine.arXiv preprint arXiv:2311.16452, 2023

    2023 arXiv

  29. [37]

    Gpt-4v(ision) system card

    OpenAI. Gpt-4v(ision) system card. 2023

    2023

  30. [38]

    Gpt-4v(ision) system card, 2023b

    OpenAI. Gpt-4v(ision) system card, 2023b. https://openai.com/research/gpt-4v-system-card, 2023

    2023

  31. [39]

    Gpt-4o-mini

    OpenAI. Gpt-4o-mini. 2024

    2024

  32. [40]

    Im2text: Describing images using 1 million captioned photographs

    Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using 1 million captioned photographs. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 24. Curran Associ...

    2011

  33. [41]

    A survey on biomedical image captioning

    John Pavlopoulos, Vasiliki Kougia, and Ion Androutsopoulos. A survey on biomedical image captioning. In Proceedings of the second workshop on shortcomings in vision and language, pages 26–36, 2019

    2019

  34. [42]

    PubMed Central (PMC) — pmc.ncbi.nlm.nih.gov.https://pmc.ncbi.nlm.nih.gov/

    PubMedCentral. PubMed Central (PMC) — pmc.ncbi.nlm.nih.gov.https://pmc.ncbi.nlm.nih.gov/

  35. [43]

    Laion-5b: An open large-scale dataset for training next generation image-text models.Advances in Neural Information Processing Systems, 35:25278–25294, 2022

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models.Advances in Neural Informat...

    2022

  36. [44]

    Neural machine translation of rare words with subword units

    Rico Sennrich. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909, 2015

    2015 arXiv

  37. [45]

    Quilt-llava: Visual instruction tuning by extracting localized narratives from open-source histopathology videos

    Mehmet Saygin Seyfioglu, Wisdom O Ikezogwo, Fatemeh Ghezloo, Ranjay Krishna, and Linda Shapiro. Quilt-llava: Visual instruction tuning by extracting localized narratives from open-source histopathology videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pa...

    2024

  38. [46]

    Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Iryna Gurevych and Yusuke Miyao, editors, Proceedings of the 56th Annual Meeting of the Association for Computati...

    2018

  39. [47]

    Pathmmu: A massive multimodal expert-level benchmark for understanding and reasoning in pathology

    Yuxuan Sun, Hao Wu, Chenglu Zhu, Sunyi Zheng, Qizi Chen, Kai Zhang, Yunlong Zhang, Dan Wan, Xiaoxiao Lan, Mengyue Zheng, et al. Pathmmu: A massive multimodal expert-level benchmark for understanding and reasoning in pathology. InEuropean Conference on Computer Vision, pages 56...

    2025

  40. [48]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model.https: //github.com/tatsu-lab/stanford_alpaca, 2023

    2023

  41. [49]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 17

    2023 arXiv

  42. [50]

    Image as a foreign language: Beit pretraining for all vision and vision-language tasks.arXiv preprint arXiv:2208.10442, 2022

    Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for all vision and vision-language tasks.arXiv preprint arXiv:2208.10442, 2022

    2022 arXiv

  43. [51]

    Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases

    Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. InProceedings of the IEEE conference on com...

    2017

  44. [52]

    Emu3: Next-token prediction is all you need.arXiv preprint arXiv:2409.18869, 2024

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need.arXiv preprint arXiv:2409.18869, 2024

    2024 arXiv

  45. [53]

    A perspective for adapting generalist ai to specialized medical ai applications and their challenges.arXiv preprint arXiv:2411.00024, 2024

    Zifeng Wang, Hanyin Wang, Benjamin Danek, Ying Li, Christina Mack, Hoifung Poon, Yajun Wang, Pranav Rajpurkar, and Jimeng Sun. A perspective for adapting generalist ai to specialized medical ai applications and their challenges.arXiv preprint arXiv:2411.00024, 2024

    2024 arXiv

  46. [54]

    Finetuned language models are zero-shot learners.arXiv preprint arXiv:2109.01652, 2021

    Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners.arXiv preprint arXiv:2109.01652, 2021

    2021 arXiv

  47. [55]

    Towards generalist foundation model for radiology, 2023

    Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Towards generalist foundation model for radiology, 2023

    2023

  48. [56]

    Medtrinity-25m: A large-scale multimodal dataset with multigranular annotations for medicine, 2024

    Yunfei Xie, Ce Zhou, Lang Gao, Juncheng Wu, Xianhang Li, Hong-Yu Zhou, Sheng Liu, Lei Xing, James Zou, Cihang Xie, and Yuyin Zhou. Medtrinity-25m: A large-scale multimodal dataset with multigranular annotations for medicine, 2024

    2024

  49. [57]

    Medicalgpt: Training medical gpt model.https://github.com/shibing624/MedicalGPT, 2023

    Ming Xu. Medicalgpt: Training medical gpt model.https://github.com/shibing624/MedicalGPT, 2023

    2023

  50. [58]

    Multiinstruct: Improving multi-modal zero-shot learning via instruction tuning

    Zhiyang Xu, Ying Shen, and Lifu Huang. Multiinstruct: Improving multi-modal zero-shot learning via instruction tuning. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11445–11465, 2023

    2023

  51. [59]

    Worse than random? an embarrassingly simple probing evaluation of large multimodal models in medical vqa.arXiv preprint arXiv:2405.20421, 2024

    Qianqi Yan, Xuehai He, Xiang Yue, and Xin Eric Wang. Worse than random? an embarrassingly simple probing evaluation of large multimodal models in medical vqa.arXiv preprint arXiv:2405.20421, 2024

    2024 arXiv

  52. [60]

    Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024

    2024 arXiv

  53. [61]

    Lamm: Language-assisted multi-modal instruction-tuning dataset, framework, and benchmark.arXiv preprint arXiv:2306.06687, 2023

    Zhenfei Yin, Jiong Wang, Jianjian Cao, Zhelun Shi, Dingning Liu, Mukai Li, Lu Sheng, Lei Bai, Xiaoshui Huang, Zhiyong Wang, et al. Lamm: Language-assisted multi-modal instruction-tuning dataset, framework, and benchmark.arXiv preprint arXiv:2306.06687, 2023

    2023 arXiv

  54. [62]

    Scaling autoregressive multi-modal models: Pretraining and instruction tuning.arXiv preprint arXiv:2309.02591, 2(3), 2023

    Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, et al. Scaling autoregressive multi-modal models: Pretraining and instruction tuning.arXiv preprint arXiv:2309.02591, 2(3), 2023

    2023 arXiv

  55. [63]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi.arXiv preprint arXiv:2311.16502, 2023

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi.arXiv preprint arXiv:2311.16502, 2023

    2023 arXiv

  56. [64]

    Huatuogpt, towards taming language model to be a doctor

    Hongbo Zhang, Junying Chen, Feng Jiang, Fei Yu, Zhihong Chen, Jianquan Li, Guiming Chen, Xiangbo Wu, Zhiyi Zhang, Qingying Xiao, et al. Huatuogpt, towards taming language model to be a doctor. arXiv preprint arXiv:2305.15075, 2023. 18

    2023 arXiv

  57. [65]

    Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs.arXiv preprint arXiv:2303.00915, 2023

    Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, et al. Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs.arXiv preprint arXiv:2303.00915, 2023

    2023 arXiv

  58. [66]

    Pmc- vqa: Visual instruction tuning for medical visual question answering.arXiv preprint arXiv:2305.10415, 2023

    Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pmc- vqa: Visual instruction tuning for medical visual question answering.arXiv preprint arXiv:2305.10415, 2023

    2023 arXiv

  59. [67]

    Llavar: Enhanced visual instruction tuning for text-rich image understanding.arXiv preprint arXiv:2306.17107, 2023

    Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. Llavar: Enhanced visual instruction tuning for text-rich image understanding.arXiv preprint arXiv:2306.17107, 2023

    2023 arXiv

  60. [68]

    Monoformer: One transformer for both diffusion and autoregression.arXiv preprint arXiv:2409.16280, 2024

    Chuyang Zhao, Yuxing Song, Wenhao Wang, Haocheng Feng, Errui Ding, Yifan Sun, Xinyan Xiao, and Jingdong Wang. Monoformer: One transformer for both diffusion and autoregression.arXiv preprint arXiv:2409.16280, 2024

    2024 arXiv

  61. [69]

    Transfusion: Predict the next token and diffuse images with one multi-modal model.arXiv preprint arXiv:2408.11039, 2024

    Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model.arXiv preprint arXiv:2408.11039, 2024

    2024 arXiv

  62. [70]

    Magnetic Resonance,

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023. 19 Table 3: Additional information about diverse biomedical dataset sources. We hi...

    2023 arXiv

  63. [71]

    Specificity: Does it contain precise details rather than vague descriptions?

  64. [72]

    Context: Does it provide relevant background or situational information?

  65. [73]

    Technical Details: Are any specific measurements, conditions, or technical terms included?

  66. [74]

    Purpose: Would the information be useful for professional analysis, decision-making, or documentation? For medical descriptions specifically, consider: - Anatomical details - Condition characteristics - Observable features - Diagnostic relevance Format your response as follows:

  67. [75]

    Analysis: Briefly explain why the description is or isn’t informative (2-3 sentences)

  68. [76]

    The answer is: Yes

    Conclusion: End with either "The answer is: Yes" or "The answer is: No" Example: Description: Juvenile polyp or retention polyp is present. Output: The description identifies a specific medical condition (juvenile/retention polyp) and confirms its presence, which is diagnostic...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.