Pith. sign in

REVIEW 3 major objections 4 minor 17 cited by

HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptation

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single autoregressive model can both comprehend and generate medical images, with H-LoRA preventing the two skills from interfering.

desk verdict Solid engineering in medical unified LVLMs, but the headline SOTA claims are unverifiable because the evaluation benchmarks appear in the training set with no held-out split. read the letter →

arxiv 2502.09838 v3 pith:RV4BK2TI submitted 2025-02-14 cs.CV cs.AI

Tianwei Lin , Wenqiao Zhang , Sijing Li , Yuqian Yuan , Binhe Yu , Haoyuan Li , Wanggui He , Hao Jiang
show 7 more authors
Mengze Li Xiaohui Song Siliang Tang Jun Xiao Hui Lin Yueting Zhuang Beng Chin Ooi
This is my paper
classification cs.CVcs.AI
keywords medicallargevision-languagemodelunifiedcomprehensionandgenerationheterogeneouslow-rankadaptationH-LoRAVQGANdiscretetokensvisualquestionansweringimagehierarchicalperception
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

HealthGPT is an attempt to build one medical large vision-language model that does both jobs usually split across two kinds of systems: reading medical images and answering questions about them, and generating medical images from text or other modalities. The paper's proposed mechanism is H-LoRA, a low-rank adaptation scheme that stores comprehension knowledge and generation knowledge in separate plugins, plus a hierarchical visual encoder that feeds abstract features to comprehension and concrete features to generation, trained in three stages. On its reported benchmarks, a 3.8B-parameter HealthGPT scores 61.3 average on seven medical comprehension tasks and beats dedicated generators on CT-to-MRI conversion and super-resolution, while a 14B version scores 66.4. The significance, if the results hold, is that a single generalist medical imaging assistant is feasible without the billions of training samples that general unified models require.

What carries the argument

The load-bearing object is H-LoRA, a mixture-of-experts variant of low-rank adaptation. For a task type T it selects a plugin of LoRA experts; the expert matrices are concatenated into merged low-rank matrices, and the router's weights are expanded by a replication operation before element-wise multiplication, so the added cost stays at O(6) regardless of the number of experts. In the paper's ablation, this takes only 67% of MoELoRA's training time with four experts while improving both comprehension and generation scores. It works together with hierarchical visual perception, which splits the CLIP ViT's hidden states into concrete-grained features from shallow layers for generation and abstract-grained features from deep layers for comprehension, and with a three-stage training strategy that keeps the base LLM frozen while tuning adapters, embedding/head, then H-LoRA plugins.

What would settle it

Hold out VQA-RAD, SLAKE, and PathVQA entirely from VL-Health training and evaluate HealthGPT on their official test sets; if the average score falls to roughly the level of the ordinary medical LVLMs in Table 1, the claimed state-of-the-art result is explained by training/evaluation overlap.

Watch

Extended reading notes

Core claim

The central claim is that the conflict between comprehension and generation in a unified autoregressive medical model can be resolved by decoupling the two task families instead of mixing their training data. HealthGPT represents text and images in one token vocabulary, with image generation supervised by VQGAN discrete indices, and uses H-LoRA to give comprehension and generation their own low-rank expert modules, a task-level hard router to select them, and hierarchical ViT features so generation sees shallow concrete detail while comprehension sees deep abstract semantics. A three-stage schedule first aligns each modality, then fuses the plugins by tuning only the embedding layer and output head, then instruction-tunes the plugins. The paper reports HealthGPT-M3 (3.8B) at 61.3 average comprehension, outperforming medical-specific models like HuatuoGPT-Vision (50.7) and general unified models like Unified-IO 2 (33.8), and HealthGPT-L14 at 66.4, while also achieving higher SSIM and PSNR than per-task specialists on four CT-MRI modality conversion tasks and on 4x MRI super-resolution.

Load-bearing premise

Section 5.1 lists VQA-RAD, SLAKE, and PathVQA as training sources for VL-Health, while Table 1 evaluates on those same benchmarks with no reported train/test split, so the reported comprehension gains may reflect memorization rather than generalization.

Editorial extensions

If this is right

  • A single model can handle both medical visual question answering and medical image synthesis, avoiding separate specialist systems.
  • The comprehension-generation conflict shown in the paper's mixed-training curves can be removed by task-level decoupling rather than by larger data mixtures.
  • H-LoRA's overhead is independent of the number of experts, so models could scale to many task-specific plugins without paying extra training cost.
  • Scaling the base LLM from 3.8B to 14B raises comprehension average from 61.3 to 66.4, suggesting the decoupled design still benefits from larger backbones.
  • The report-to-CXR results indicate that generating a medical image from a text description alone is feasible within the same autoregressive framework.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same plugin-decoupling recipe could transfer to other low-data domains where understanding and synthesis compete, such as pathology, ophthalmology, or remote sensing.
  • The claimed superiority should be re-tested on a held-out split: VQA-RAD, SLAKE, and PathVQA are listed in Section 5.1 as training sources yet appear in Table 1 as evaluation benchmarks with no reported train/test separation.
  • Because the base LLM stays frozen, H-LoRA plugins could plausibly be swapped at deployment time, allowing a hospital to update a CT-MRI plugin without retraining the full assistant.
  • A natural extension is to replace the hard task router with a learned router that decides from the instruction alone whether comprehension or generation is needed.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces HealthGPT, a unified medical large vision-language model that performs both visual comprehension (VQA-style tasks) and visual generation (modality conversion, super-resolution, reconstruction, report-to-CXR) in a single autoregressive framework. The method combines a heterogeneous low-rank adaptation (H-LoRA) module with a hierarchical visual perception scheme that feeds shallow ViT features to generation and deep ViT features to comprehension, and a three-stage training strategy intended to reduce comprehension/generation conflicts. The authors curate the VL-Health dataset and report state-of-the-art or competitive results on seven medical comprehension benchmarks and five generation tasks with 3.8B and 14B parameter variants.

Significance. If the empirical results are trustworthy, the paper would make a useful contribution: the unified autoregressive formulation of medical comprehension and generation is timely, the H-LoRA matrix-merging trick is a neat efficiency improvement over MoELoRA, and the VL-Health dataset is a potentially reusable resource. The internal ablation studies (Tables 4, 5 and Figure 5) are informative and appear self-consistent. However, the headline performance claims rest on an evaluation protocol that is not properly documented with respect to train/test separation, and this undermines the paper's main empirical contribution as written.

major comments (3)
  1. [Section 5.1 and Table 1; Appendix A.3, Table 8] The paper's central claim of state-of-the-art comprehension performance is not verifiable because the evaluation benchmarks VQA-RAD, SLAKE, and PathVQA are explicitly listed as components of the VL-Health Stage-3 training data (Section 5.1; Appendix A.3 and Table 8), and Table 1 reports results on these same datasets without any documented train/test split, image-level deduplication, or holdout procedure. If the authors used the official train splits of these datasets for training and the official test splits for evaluation, that must be stated precisely; otherwise the reported gains over prior models may reflect memorization of the test distribution rather than generalization. This concern also affects the human evaluation in Appendix C.4, which draws questions from the same three benchmarks.
  2. [Section 5.1 with Tables 2 and 3; Appendix A.3, Table 8] The generation claims are subject to the same contamination issue: IXI and SynthRAD2023 are listed as Stage-3 training sources (Section 5.1; Appendix Table 8), and the modality-conversion and super-resolution evaluations in Tables 2 and 3 are run on these datasets with no reported split. The paper does not state whether the same subjects or slices used in evaluation were also used in training, nor does it describe any data partitioning (e.g., patient-level or volume-level separation). Without this information, the reported SSIM/PSNR/MSE/LPIPS improvements over dedicated baselines cannot be interpreted.
  3. [Tables 1 and 8; Section 5.2] The evaluation protocol lacks several details that are load-bearing for the SOTA claim: Table 1 reports 'close' and 'all' columns but never defines what these mean (closed-ended vs. all questions, or something else); the exact number of test instances per benchmark is not given; and no statistical significance or variance information is reported for any of the headline comparisons. The authors should provide a precise evaluation protocol, including the prompt templates used for each benchmark, the exact test-set sizes, and how the reported averages are computed, so that the comparisons to previously published results are meaningful.
minor comments (4)
  1. [Section 5.2, paragraph on super-resolution] The text says 'excels in key metrics such as SSIM, PSNR, and ISE' and then refers to 'the lowest score of 12.34', but Table 3 lists LPIPS (not ISE) and the value 12.02; this is a typo or metric-name inconsistency that should be fixed.
  2. [Section 5.2, paragraph on modality conversion] The text says 'The experimental results, shown in Table 11', but the modality-conversion results are in Table 2; the in-text table references (Table 11) appear to be a leftover from an earlier draft and should be corrected throughout.
  3. [Figure 2 and Table 5] Figure 2 lacks axis labels and a description of the exact experiments that produced the shown degradation curves, making the 'conflict between comprehension and generation' motivation hard to verify; Table 5's column headers for the CT/MRI generation results are also ambiguous and should be clarified.
  4. [References] The references list LLaVA-Med twice as 'Li et al. 2024a' and 'Li et al. 2024b' for the same paper, which is confusing and should be consolidated.

Circularity Check

2 steps flagged · score 6.0 of 10

Table 1's SOTA numbers are not independent: VQA-RAD, SLAKE, and PathVQA are listed as VL-Health training sources with no reported holdout split, so the reported gains conflate memorization with generalization; generation metrics repeat the same issue for IXI and SynthRAD2023.

  1. fitted input called prediction [Section 5.1 Data and Experimental Setup; Section 5.2 Table 1; Appendix A.3 Table 8]
    "For medical visual comprehension, we leverage multiple medical-specific datasets, including PubMedVision (Chen et al. 2024a), LLaVA-Med (Li et al. 2024b), PathVQA (He et al. 2020), MIMIC-CXR-VQA (Bae et al. 2024), SLAKE (Liu et al. 2021), and VQA-RAD (Lau et al. 2018)."

    Appendix Table 8 lists the Stage-3 comprehension training data as 'LLaVA Med, MIMIC CXR VQA, PubMedVision-FT, LLaVA-665k, PathVQA, SLAKE, VQA-RAD', exactly the evaluation columns VQA-RAD, SLAKE, and PathVQA in Table 1. No train/test split, image-level de-duplication, or holdout procedure is reported anywhere. Because the model is trained on the same benchmark datasets it is then evaluated on, the 'prediction' on those benchmarks is partly a restatement of the training distribution, and the cross-model comparison in Table 1 cannot distinguish genuine medical visual understanding from memorization.

  2. fitted input called prediction [Section 5.1 Data and Experimental Setup; Section 5.2 Tables 2 and 3]
    "For generation tasks, we construct a reconstruction dataset based on LLaVA-558k (Liu et al. 2024b), and also explore two key tasks in personalized medical image enhancement—super-resolution and modality conversion—using the IXI (Davies et al. 2014) and SynthRAD2023 (Thummerer et al. 2023) datasets."

    Table 2 reports CT-to-MRI and MRI-to-CT conversion metrics on brain/pelvis pairs, and Table 3 reports 4x super-resolution metrics on the IXI dataset. Section 5.1 states that SynthRAD2023 was used for modality conversion and IXI for super-resolution training, and no split is specified. The generation-task numbers therefore are not held-out measurements and do not independently establish generalization compared with the listed baselines.

full rationale

The central empirical claim of the paper is that HealthGPT outperforms medical-specific and general unified LVLMs on seven medical comprehension tasks and five generation tasks. That claim rests on Table 1 (VQA-RAD, SLAKE, PathVQA, MMMU-Med, OMVQA) and Tables 2-3 (modality conversion, super-resolution). However, Section 5.1 explicitly names VQA-RAD, SLAKE, PathVQA, MIMIC-CXR-VQA, IXI, and SynthRAD2023 as sources of the VL-Health training data, and Appendix Table 8 confirms that PathVQA, SLAKE, and VQA-RAD are Stage-3 comprehension training data while IXI and SynthRAD2023 are generation training data. No train/test split, image-level de-duplication, or holdout procedure is reported. As written, the evaluation therefore cannot be distinguished from measuring performance on the training distribution, so the headline SOTA comparison is not an independent generalization measurement. The internal ablations comparing HealthGPT variants are less affected because all variants see the same data, but the paper's primary cross-model SOTA claim is statistically forced by the training/evaluation overlap. This is a benchmark-leakage form of fitted-input-called-prediction rather than a self-citation or equation-level circularity, so the score is 6 rather than higher.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claims rest on a small number of hyperparameters and domain assumptions. The most consequential is the assumed holdout of evaluation benchmarks from training, which is not documented.

free parameters (5)
  • H-LoRA rank = 16 (comp) / 64 (gen) for M3; 8 (comp) / 32 (gen) for L14
    Chosen for performance; sensitivity is shown in Figure 5 but no principled selection procedure is reported.
  • Number of LoRA experts = 4
    Used in main experiments; Table 10 explores 2, 4, 8, and 32, but the choice of 4 is not justified beyond performance.
  • ViT layer selection for concrete/abstract features = 2nd layer (concrete), penultimate layer (abstract)
    Based on an assumption about ViT hierarchy; no ablation over different layer choices is reported.
  • Learning rates = 1e-3 to 2e-5 depending on stage
    Reported in Table 7; the paper mentions loss spikes with larger learning rates, indicating tuning was performed.
  • Dataset proportions and sampling = 765,802 VQA samples and 783,045 generation samples; 47k mixed samples for stage-2
    Hand-crafted budget; no analysis of how these proportions affect performance or conflict mitigation.
assumptions (4)
  • domain assumption ViT features at shallow layers preserve concrete visual details, while deeper layers carry abstract semantics.
    Used to justify Hierarchical Visual Perception (Eq. 6, Section 4.2). If this hierarchy is not true for medical images, the routing is misguided.
  • domain assumption Discrete VQGAN tokens (f8-8192) provide sufficient fidelity for medical image generation.
    The generation pipeline relies on VQGAN encoder/decoder (Section 3). Medical images may require higher fidelity than natural images.
  • domain assumption Task type is known at inference time for routing.
    H-LoRA uses hard routing based on task type T (Eq. 6, Algorithm 1). In real deployment, the user must specify the task type, which may not always be known.
  • ad hoc to paper The evaluation benchmarks (VQA-RAD, SLAKE, PathVQA) are not in the VL-Health training data.
    Section 5.1 lists these datasets as training sources, and Table 1 evaluates on them. No holdout split is reported, making this an unverified assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptation." pith.science (2026). https://pith.science/paper/RV4BK2TI

@misc{pith2026250209838,
  author       = {Pith},
  title        = {Pith review of: HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RV4BK2TI}},
  note         = {Machine review of arXiv:2502.09838}
}
read the original abstract

We present HealthGPT, a powerful Medical Large Vision-Language Model (Med-LVLM) that integrates medical visual comprehension and generation capabilities within a unified autoregressive paradigm. Our bootstrapping philosophy is to progressively adapt heterogeneous comprehension and generation knowledge to pre-trained large language models (LLMs). This is achieved through a novel heterogeneous low-rank adaptation (H-LoRA) technique, which is complemented by a tailored hierarchical visual perception approach and a three-stage learning strategy. To effectively learn the HealthGPT, we devise a comprehensive medical domain-specific comprehension and generation dataset called VL-Health. Experimental results demonstrate exceptional performance and scalability of HealthGPT in medical visual unified tasks. Our project can be accessed at https://github.com/DCDmllm/HealthGPT.

Figures

Figures reproduced from arXiv: 2502.09838 by the authors.

Figure 1
Figure 1. HealthGPT enables medical multi-modal comprehension and generation, outperforming both state-of-the-art unified visual models and medical-specific models across various tasks. This highlights its superior capability in tackling com￾plex tasks in healthcare applications. Comp.Perf. and Gen.Perf. denote the results of comprehension and generation. Abstract We present HealthGPT, a powerful Medical Large Vision￾Language… view at source ↗
Figure 2
Figure 2. With a fixed amount of comprehension (genera [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The HealthGPT architecture integrates hierarchical visual perception and H-LoRA, employing a task-specific hard router to select visual features and H-LoRA plugins, ultimately generating outputs with an autoregressive manner. a design paradigm based on ViT, alignment adapters, and pre-trained LLMs(Liu et al. 2023, 2024b), enabling quick adaptation to downstream tasks. VQGAN. VQGAN (Esser, Rombach, and Ommer 2021) em… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Data statistics of VL-Health. 4.4 Training Pipeline 1st Stage: Multi-modal Alignment. In the first stage, we design separate visual adapters and H-LoRA submodules for medical unified tasks. For the medical comprehension task, we train abstract-grained visual adapters u…
Figure 6
Figure 6. Figure 6: The loss visualization (a) and performance com [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 5
Figure 5. Figure 5: Performance comparison of LoRA, MoELoRA, [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 7
Figure 7. Figure 7: Case study of report-to-CXR under different instructions. (a) shows a normal CXR image for comparison. (b) and (c) [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: VL-Health dataset collection distribution. [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Performance changes before and after the [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: (a) Proportion of model responses selected as the best in human evaluation. (b) Human Evaluation Dataset. [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: Case of modality transfer. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: Case of MRI image super-resolution. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 17 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DrVD-Bench: Do Vision-Language Models Reason Like Human Doctors in Medical Image Diagnosis?

    cs.CV 2025-05 conditional novelty 7.0 of 10

    A new five-level medical imaging benchmark, DrVD-Bench, shows that vision-language models lose accuracy sharply as reasoning complexity grows and often diagnose without grounding in lesion evidence.

  2. OralAgent: Integrating Reasoning, Tools, and Knowledge for Interactive Dental Image Analysis

    cs.CL 2026-04 accept novelty 6.5 of 10

    OralAgent, a ReAct-style dental agent with 22 vision tools and a 134.8M-token textbook RAG corpus, reaches SOTA on MMOral-Uni, MMOral-OPG, and OralQA-ZH.

  3. PathScale-R1: Cross-scale Reasoning for Pathological Image Analysis

    cs.CV 2026-07 conditional novelty 6.0 of 10

    PathScale-R1 and PathScale-VQA raise cross-scale pathology VQA accuracy to 83% by training on expert multi-magnification paths with anti-shortcut curation and scale-aware RL.

  4. Can Multimodal Large Language Models Understand OCT?

    cs.CV 2026-07 conditional novelty 6.0 of 10

    OCT-Bench, a 20-task benchmark across 10,076 questions, shows current MLLMs score up to 62% overall but only 43% on clinical reasoning over OCT images.

  5. Deep Expert Injection for Anchoring Retinal VLMs with Domain-Specific Knowledge

    cs.CV 2026-03 reject novelty 6.0 of 10

    EyExIn, a 7B retinal VLM with dual-stream expert encoding and adaptive deep-layer visual injection, reports state-of-the-art F1 on four fundus VQA benchmarks, exceeding larger proprietary models.

  6. Epistemic-aware Vision-Language Foundation Model for Fetal Ultrasound Interpretation

    cs.CV 2025-10 conditional novelty 6.0 of 10

    A multi-view vision-language model trained on 20,000 fetal ultrasound reports generates clinical text and diagnoses, reportedly outperforming general and medical baselines.

  7. Towards Better Dental AI: A Multimodal Benchmark and Instruction Dataset for Panoramic X-ray Analysis

    cs.CV 2025-09 reject novelty 6.0 of 10

    MMOral is a large new dental X-ray instruction dataset and benchmark, but the proposed model's 24.73% improvement is from fine-tuning and then testing on the same data pool.

  8. 4KAgent: Agentic Any Image to 4K Super-Resolution

    cs.CV 2025-07 reject novelty 6.0 of 10

    An agentic pipeline that plans and executes image restoration from a toolbox of pretrained models to upscale arbitrary images to 4K, reporting state-of-the-art results on many benchmarks.

  9. Graft: Integrating the Domain Knowledge via Efficient Parameter Synergy for MLLMs

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Graft merges two domain-specialized multimodal models by combining channel-wise gating, entropy-based global weighting, and an activation compatibility score to improve fusion without retraining.

  10. Clinical Cognition Alignment for Gastrointestinal Diagnosis with Multimodal LLMs

    cs.CV 2026-03 unverdicted novelty 5.5 of 10

    Hierarchical clinical-reasoning SFT plus counterfactual GRPO yields SoTA diagnostic accuracy for multimodal LLMs on gastrointestinal endoscopy benchmarks.

  11. Rad-JEPA 3D: Radiology Joint-Embedding Predictive Model for 3D Computed Tomography

    cs.CV 2026-07 conditional novelty 5.0 of 10

    A 19M-parameter JEPA-style 3D-CT encoder with a routed Mamba+GQA hybrid and orthogonal hidden-state regularization gives a 4B total model the best mean accuracy on M3D-VQA closed-ended questions and the best average o...

  12. M3CoTBench: Benchmark Chain-of-Thought of MLLMs in Medical Image Understanding

    eess.IV 2026-01 conditional novelty 5.0 of 10

    A medical-image benchmark that scores the step-by-step reasoning chains of multimodal LLMs shows current models explain poorly and chain-of-thought prompting frequently reduces diagnostic accuracy.

  13. CX-Mind: A Pioneering Multimodal Large Language Model for Interleaved Reasoning in Chest X-ray via Curriculum-Guided Reinforcement Learning

    cs.LG 2025-07 conditional novelty 5.0 of 10

    CX-Mind combines curriculum reinforcement learning and rule-based process rewards to train a chest X-ray vision-language model that produces interleaved think-answer reasoning and reports state-of-the-art results acro...

  14. From Bench to Bedside: A DeepSeek-Powered AI System for Automated Chest Radiograph Interpretation in Clinical Practice

    cs.HC 2025-05 conditional novelty 5.0 of 10

    A lightweight chest X-ray reporting AI, Janus-Pro-CXR, improved junior radiologists' report quality and cut reading time by 18.5% in a prospective three-hospital study.

  15. The Path to Self-Evolving Clinical Systems: Scaling Medical Agents from Assistance to Autonomy

    cs.AI 2026-07 conditional novelty 4.5 of 10

    Medical agents should be scaled mainly by richer clinical environments and self-evolution loops, not parameter growth alone, under a three-level autonomy taxonomy.

  16. ScatterPrism: convergence for generative simulation and inverse problems in particle and nuclear physics

    cs.LG 2026-04 unverdicted novelty 4.0 of 10

    Standard Conditional Flow Matching loss is a misleading early plateau; physics-informed metrics keep improving, so ScatterPrism and multi-metric diagnostics are needed for kinematic fidelity.

  17. Collaborative Editable Model

    cs.AI 2025-06 reject novelty 4.0 of 10

    CoEM scores user-contributed knowledge fragments using user ratings and LLM attribution, keeps the high scorers in a prompt-level knowledge pool, and reports 76% agreement with FinGPT on fragment value.

Reference graph

Works this paper leans on

60 extracted references · 18 canonical work pages · cited by 17 Pith papers

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    J.; Javaheripi, M.; Kauffmann, P.; et al

    Abdin, M.; Aneja, J.; Behl, H.; Bubeck, S.; Eldan, R.; Gunasekar, S.; Harrison, M.; Hewett, R. J.; Javaheripi, M.; Kauffmann, P.; et al. 2024. Phi-4 technical report. arXiv preprint arXiv:2412.08905

    arXiv 2024

  4. [4]

    Bae, S.; Kyung, D.; Ryu, J.; Cho, E.; Lee, G.; Kweon, S.; Oh, J.; JI, L.; Chang, E.; Kim, T.; et al. 2024. MIMIC-Ext-MIMIC-CXR-VQA: A Complex, Diverse, And Large-Scale Visual Question Answering Dataset for Chest X-ray Images

    work page 2024

  5. [5]

    H.; Wang, X.; Zhang, R.; Cai, Z.; Ji, K.; et al

    Chen, J.; Gui, C.; Ouyang, R.; Gao, A.; Chen, S.; Chen, G. H.; Wang, X.; Zhang, R.; Cai, Z.; Ji, K.; et al. 2024 a . Huatuogpt-vision, towards injecting medical visual knowledge into multimodal llms at scale. arXiv preprint arXiv:2406.19280

    arXiv 2024

  6. [6]

    Chen, Z.; Wang, W.; Tian, H.; Ye, S.; Gao, Z.; Cui, E.; Tong, W.; Hu, K.; Luo, J.; Ma, Z.; et al. 2024 b . How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821

    arXiv 2024

  7. [7]

    Chern, E.; Su, J.; Ma, Y.; and Liu, P. 2024. Anole: An open, autoregressive, native large multimodal models for interleaved image-text generation. arXiv preprint arXiv:2407.06135

    arXiv 2024

  8. [8]

    Dai, W.; Li, J.; Li, D.; Tiong, A. M. H.; Zhao, J.; Wang, W.; Li, B.; Fung, P.; and Hoi, S. 2023. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. arXiv:2305.06500

    arXiv 2023

Show all 60 references
  1. [9]

    L.; Royston, P

    Davies, R. L.; Royston, P. A.; Leung, M. S.; Haider, M. E. A. M. J.; Barkhof, S. G. A. L.; and B., P. E. T. M. 2014. The IXI Dataset. Accessed: 2025-01-30

    2014

  2. [10]

    Ding, N.; Qin, Y.; Yang, G.; Wei, F.; Yang, Z.; Su, Y.; Hu, S.; Chen, Y.; Chan, C.-M.; Chen, W.; et al. 2023. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3): 220--235

    2023

  3. [11]

    Dong, R.; Han, C.; Peng, Y.; Qi, Z.; Ge, Z.; Yang, J.; Zhao, L.; Sun, J.; Zhou, H.; Wei, H.; et al. 2023. Dreamllm: Synergistic multimodal comprehension and creation. arXiv preprint arXiv:2309.11499

    2023 arXiv

  4. [12]

    Dubey, A.; Jauhri, A.; Pandey, A.; Kadian, A.; Al-Dahle, A.; Letman, A.; Mathur, A.; Schelten, A.; Yang, A.; Fan, A.; et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  5. [13]

    Esser, P.; Rombach, R.; and Ommer, B. 2021. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 12873--12883

    2021

  6. [14]

    Ge, Y.; Ge, Y.; Zeng, Z.; Wang, X.; and Shan, Y. 2023. Planting a seed of vision in large language model. arXiv preprint arXiv:2307.08041

    2023 arXiv

  7. [15]

    Ge, Y.; Zhao, S.; Zhu, J.; Ge, Y.; Yi, K.; Song, L.; Li, C.; Ding, X.; and Shan, Y. 2024. Seed-x: Multimodal models with unified multi-granularity comprehension and generation. arXiv preprint arXiv:2404.14396

    2024 arXiv

  8. [16]

    He, X.; Zhang, Y.; Mou, L.; Xing, E.; and Xie, P. 2020. Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003.10286

    2020 arXiv

  9. [17]

    J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W

    Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685

    2021 arXiv

  10. [18]

    Hu, Y.; Li, T.; Lu, Q.; Shao, W.; He, J.; Qiao, Y.; and Luo, P. 2024. Omnimedvqa: A new large-scale comprehensive evaluation benchmark for medical lvlm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 22170--22183

    2024

  11. [19]

    Isola, P.; Zhu, J.-Y.; Zhou, T.; and Efros, A. A. 2017. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, 1125--1134

    2017

  12. [20]

    E.; Pollard, T

    Johnson, A. E.; Pollard, T. J.; Greenbaum, N. R.; Lungren, M. P.; Deng, C.-y.; Peng, Y.; Lu, Z.; Mark, R. G.; Berkowitz, S. J.; and Horng, S. 2019. MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs. arXiv preprint arXiv:1901.07042

    2019 arXiv

  13. [21]

    J.; Gayen, S.; Ben Abacha, A.; and Demner-Fushman, D

    Lau, J. J.; Gayen, S.; Ben Abacha, A.; and Demner-Fushman, D. 2018. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1): 1--10

    2018

  14. [22]

    Li, B.; Xue, K.; Liu, B.; and Lai, Y.-K. 2023 a . Bbdm: Image-to-image translation with brownian bridge diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern Recognition, 1952--1961

    2023

  15. [23]

    Li, C.; Wong, C.; Zhang, S.; Usuyama, N.; Liu, H.; Yang, J.; Naumann, T.; Poon, H.; and Gao, J. 2024 a . Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36

    2024

  16. [24]

    Li, C.; Wong, C.; Zhang, S.; Usuyama, N.; Liu, H.; Yang, J.; Naumann, T.; Poon, H.; and Gao, J. 2024 b . Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36

    2024

  17. [25]

    Li, J.; Li, D.; Savarese, S.; and Hoi, S. 2023 b . Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, 19730--19742. PMLR

    2023

  18. [26]

    Lin, T.; Liu, J.; Zhang, W.; Li, Z.; Dai, Y.; Li, H.; Yu, Z.; He, W.; Li, J.; Jiang, H.; et al. 2024. Teamlora: Boosting low-rank adaptation with expert collaboration and competition. arXiv preprint arXiv:2408.09856

    2024 arXiv

  19. [27]

    Liu, B.; Zhan, L.-M.; Xu, L.; Ma, L.; Yang, Y.; and Wu, X.-M. 2021. Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), 1650--1654. IEEE

    2021

  20. [28]

    Liu, D.; Zhao, S.; Zhuo, L.; Lin, W.; Qiao, Y.; Li, H.; and Gao, P. 2024 a . Lumina-mgpt: Illuminate flexible photorealistic text-to-image generation with multimodal generative pretraining. arXiv preprint arXiv:2408.02657

    2024 arXiv

  21. [29]

    Liu, H.; Li, C.; Li, Y.; and Lee, Y. J. 2024 b . Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 26296--26306

    2024

  22. [30]

    Liu, H.; Li, C.; Li, Y.; Li, B.; Zhang, Y.; Shen, S.; and Lee, Y. J. 2024 c . LLaVA-NeXT: Improved reasoning, OCR, and world knowledge. https://llava-vl.github.io/blog/2024-01-30-llava-next/

    2024

  23. [31]

    Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2023. Visual Instruction Tuning. In NeurIPS

    2023

  24. [32]

    Liu, Q.; Wu, X.; Zhao, X.; Zhu, Y.; Xu, D.; Tian, F.; and Zheng, Y. 2024 d . When moe meets llms: Parameter efficient fine-tuning for multi-task medical applications. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retri...

    2024

  25. [33]

    Liu, Y.; Tian, Y.; Zhao, Y.; Yu, H.; Xie, L.; Wang, Y.; Ye, Q.; and Liu, Y. 2024 e . VMamba: Visual State Space Model. arXiv preprint arXiv:2401.10166

    2024 arXiv

  26. [34]

    Lu, J.; Clark, C.; Lee, S.; Zhang, Z.; Khosla, S.; Marten, R.; Hoiem, D.; and Kembhavi, A. 2024. Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision Language Audio and Action. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, ...

    2024

  27. [35]

    Lu, J.; Clark, C.; Zellers, R.; Mottaghi, R.; and Kembhavi, A. 2022. Unified-io: A unified model for vision, language, and multi-modal tasks. In The Eleventh International Conference on Learning Representations

    2022

  28. [36]

    Luo, T.; Lei, J.; Lei, F.; Liu, W.; He, S.; Zhao, J.; and Liu, K. 2024 a . Moelora: Contrastive learning guided mixture of experts on parameter-efficient fine-tuning for large language models. arXiv preprint arXiv:2402.12851

    2024 arXiv

  29. [37]

    Luo, Y.; Zhang, J.; Fan, S.; Yang, K.; Hong, M.; Wu, Y.; Qiao, M.; and Nie, Z. 2024 b . Biomedgpt: An open multimodal large language model for biomedicine. IEEE Journal of Biomedical and Health Informatics

    2024

  30. [38]

    Masoudnia, S.; and Ebrahimpour, R. 2014. Mixture of experts: a literature survey. Artificial Intelligence Review, 42: 275--293

    2014

  31. [39]

    P.; and Rajpurkar, P

    Moor, M.; Huang, Q.; Wu, S.; Yasunaga, M.; Dalmia, Y.; Leskovec, J.; Zakka, C.; Reis, E. P.; and Rajpurkar, P. 2023. Med-flamingo: a multimodal medical few-shot learner. In Machine Learning for Health (ML4H), 353--367. PMLR

    2023

  32. [40]

    M.; Tang, Y.; et al

    Nath, V.; Li, W.; Yang, D.; Myronenko, A.; Zheng, M.; Lu, Y.; Liu, Z.; Yin, H.; Law, Y. M.; Tang, Y.; et al. 2024. Vila-m3: Enhancing vision-language models with medical expert knowledge. arXiv preprint arXiv:2411.12915

    2024 arXiv

  33. [41]

    OpenAI. 2023. GPT-4V(ision) System Card. https://cdn.openai.com/papers/GPTV_System_Card.pdf

    2023

  34. [42]

    Pan, K.; Tang, S.; Li, J.; Fan, Z.; Chow, W.; Yan, S.; Chua, T.-S.; Zhuang, Y.; and Zhang, H. 2024. Auto-Encoding Morph-Tokens for Multimodal LLM. arXiv preprint arXiv:2405.01926

    2024 arXiv

  35. [43]

    W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al

    Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, 8748--8763. PMLR

    2021

  36. [44]

    S.; Wei, J.; Chung, H

    Singhal, K.; Azizi, S.; Tu, T.; Mahdavi, S. S.; Wei, J.; Chung, H. W.; Scales, N.; Tanwani, A.; Cole-Lewis, H.; Pfohl, S.; et al. 2023. Large language models encode clinical knowledge. Nature, 620(7972): 172--180

    2023

  37. [45]

    Team, C. 2024. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818

    2024 arXiv

  38. [46]

    S.; Cholakkal, H.; Anwer, R

    Thawkar, O.; Shaker, A.; Mullappilly, S. S.; Cholakkal, H.; Anwer, R. M.; Khan, S.; Laaksonen, J.; and Khan, F. S. 2023. Xraygpt: Chest radiographs summarization using medical vision-language models. arXiv preprint arXiv:2306.07971

    2023 arXiv

  39. [47]

    J.; Langendijk, J

    Thummerer, A.; van der Bijl, E.; Galapon Jr, A.; Verhoeff, J. J.; Langendijk, J. A.; Both, S.; van den Berg, C. N. A.; and Maspero, M. 2023. SynthRAD2023 Grand Challenge dataset: Generating synthetic CT for radiotherapy. Medical physics, 50(7): 4664--4674

    2023

  40. [48]

    Tian, D.; Jiang, S.; Zhang, L.; Lu, X.; and Xu, Y. 2023. The role of large language models in medical image processing: a narrative review. Quantitative Imaging in Medicine and Surgery, 14(1): 1108

    2023

  41. [49]

    Tong, S.; Fan, D.; Zhu, J.; Xiong, Y.; Chen, X.; Sinha, K.; Rabbat, M.; LeCun, Y.; Xie, S.; and Liu, Z. 2024. MetaMorph: Multimodal Understanding and Generation via Instruction Tuning. arXiv preprint arXiv:2412.14164

    2024 arXiv

  42. [50]

    Tu, T.; Azizi, S.; Driess, D.; Schaekermann, M.; Amin, M.; Chang, P.-C.; Carroll, A.; Lau, C.; Tanno, R.; Ktena, I.; et al. 2024. Towards generalist biomedical AI. NEJM AI, 1(3): AIoa2300138

    2024

  43. [51]

    Vig, J. 2019. A multiscale visualization of attention in the transformer model. arXiv preprint arXiv:1906.05714

    2019 arXiv

  44. [52]

    Wang, X.; Zhang, X.; Luo, Z.; Sun, Q.; Cui, Y.; Wang, J.; Zhang, F.; Wang, Y.; Li, Z.; Yu, Q.; et al. 2024 a . Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869

    2024 arXiv

  45. [53]

    Wang, Z.; Wu, Z.; Agarwal, D.; and Sun, J. 2022. Medclip: Contrastive learning from unpaired medical images and text. arXiv preprint arXiv:2210.10163

    2022 arXiv

  46. [54]

    Wang, Z.; Zhang, L.; Wang, L.; and Zhang, Z. 2024 b . Soft Masked Mamba Diffusion Model for CT to MRI Conversion. arXiv preprint arXiv:2406.15910

    2024 arXiv

  47. [55]

    Wu, C.; Chen, X.; Wu, Z.; Ma, Y.; Liu, X.; Pan, Z.; Liu, W.; Xie, Z.; Yu, X.; Ruan, C.; and Luo, P. 2024. Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation. arXiv:2410.13848

    2024 arXiv

  48. [56]

    Wu, S.; Fei, H.; Qu, L.; Ji, W.; and Chua, T.-S. 2023. Next-gpt: Any-to-any multimodal llm. arXiv preprint arXiv:2309.05519

    2023 arXiv

  49. [57]

    J.; Wang, W.; Lin, K

    Xie, J.; Mao, W.; Bai, Z.; Zhang, D. J.; Wang, W.; Lin, K. Q.; Gu, Y.; Chen, Z.; Yang, Z.; and Shou, M. Z. 2024. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408.12528

    2024 arXiv

  50. [58]

    Young, A.; Chen, B.; Li, C.; Huang, C.; Zhang, G.; Zhang, G.; Li, H.; Zhu, J.; Chen, J.; Chang, J.; et al. 2024. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04652

    2024 arXiv

  51. [59]

    S.; Zhou, P.; Liu, J.; et al

    Zhou, H.; Liu, F.; Gu, B.; Zou, X.; Huang, J.; Wu, J.; Li, Y.; Chen, S. S.; Zhou, P.; Liu, J.; et al. 2023. A survey of large language models in medicine: Progress, application, and challenge. arXiv preprint arXiv:2311.05112

    2023 arXiv

  52. [60]

    Zhu, J.-Y.; Park, T.; Isola, P.; and Efros, A. A. 2017. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, 2223--2232

    2017

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.