Pith. sign in

REVIEW 2 major objections 5 minor 105 references

The paper claims that pretraining a 3D medical image encoder with 2D slice images and text descriptions generated by a multimodal large language model improves downstream 3D segmentation and classification beyond image-only self-supervised

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 19:44 UTC pith:IOGMM4CY

load-bearing objection Solid method with a real data-overlap problem on MSD-Liver; worth refereeing, but the headline claim needs reframing. the 2 major comments →

arxiv 2509.09064 v1 pith:IOGMM4CY submitted 2025-09-11 cs.CV

Enhancing 3D Medical Image Understanding with Pretraining Aided by 2D Multimodal Large Language Models

Qiuhui Chen , Xuancheng Yao , Huping Ye , Yi Hong This is my paper
classification cs.CV
keywords 3D medical image understandingvision-language pretrainingself-supervised learningmultimodal large language modelsoptimal transport alignmentmedical image segmentationmedical image classificationplane-slice-aware transformer
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Med3DInsight adds language supervision to 3D medical image pretraining without human annotations. For each 3D volume, it samples one 2D slice, has a multimodal large language model write a text description of that slice, and trains the volume encoder to reconstruct the volume while aligning its features with both the 2D slice and the generated text. The paper reports consistent gains over image-only self-supervised baselines on segmentation datasets spanning CT and MRI, and on two disease classification datasets. The central idea is that language descriptions carry clinical semantics that pixel-level reconstruction and contrastive learning alone miss.

Core claim

The paper's central claim is that a 3D medical image encoder learns more transferable representations when its features are aligned with 2D slice features and with text descriptions of those slices produced by a 2D multimodal large language model. A learnable query transformer, informed by the slice's plane and position within the volume, projects 3D features into the 2D image-text space. The matching loss is a mini-batch partial optimal transport that tolerates noise in generated text. The authors report consistent improvements over image-only self-supervised baselines on eight or more segmentation datasets and on classification of Alzheimer's and Parkinson's disease, interpreting the gains

What carries the argument

The Plane-Slice-Aware Transformer (PSAT) is the bridge between the 3D volume and the 2D vision-language space. It uses learnable query tokens with a plane-slice position embedding that encodes which orientation and slice index the 2D image came from; the queries attend to the 3D volume features and produce projected features comparable to 2D image and text embeddings. The second piece is mini-batch Partial Optimal Transport (POT): instead of contrastive push-apart, the loss compares an identity coupling to a transport plan between volume and slice/text features, allowing some features to remain unmatched and thereby absorbing noise in LLM-generated descriptions. A reconstruction loss on the

Load-bearing premise

The whole generalization story rests on the pretraining corpora being disjoint from downstream evaluation, specifically the claim that MSD-Liver is unseen during pretraining; if the LiTS liver data inside 3DSeg-8 overlaps with MSD-Liver, that transfer claim collapses.

What would settle it

Compare the actual image volumes used for pretraining with the MSD-Liver evaluation cases, e.g., by matching image hash values or patient IDs. If overlap exists, rerun the MSD-Liver evaluation with a provably disjoint pretraining corpus; if the reported Dice advantage disappears or shrinks, the cross-domain transfer claim for liver segmentation is unsupported.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If the central claim is correct, medical image pretraining no longer needs curated image-text pairs: generated 2D descriptions from a general multimodal model can serve as supervision for 3D encoders.
  • The alignment recipe is encoder-agnostic: any 3D encoder and any pre-aligned 2D vision-language encoder pair could be plugged in, so gains should extend beyond the specific backbone tested.
  • Because one slice per volume suffices, pretraining cost stays modest while still bringing the semantic advantage of language.
  • Disease classification improves along with segmentation, suggesting the learned features capture pathology-related cues, not just anatomy.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's claim that MSD-Liver is unseen during pretraining is load-bearing; if the LiTS liver data inside 3DSeg-8 overlaps with MSD-Liver volumes, the MSD-Liver result is a domain-familiarity test rather than a cross-domain transfer test.
  • The plane-slice position embedding implicitly teaches the model where a slice lives in a volume; this could be turned into an explicit pretraining signal for slice localization or 3D landmark detection.
  • Partial optimal transport's tolerance to noisy text suggests the method could work with lower-quality text sources such as automated radiology reports or weaker multimodal models, possibly with a different performance trade-off.
  • One-slice-per-volume sampling leaves most anatomical context unused; a multi-slice or volume-level text aggregation might improve performance for pathologies only visible in particular planes.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. Med3DInsight proposes a pretraining framework for 3D medical image encoders. For each 3D volume in 3DSeg-8 and M3D, the method samples one 2D slice, generates a text description with GPT-4V, and fine-tunes CLIP on the resulting slice-text pairs. A 3D encoder-decoder (nnFormer) is then trained with two objectives: a plane-slice-aware transformer (PSAT) aligns 3D volume features with CLIP image/text features using a mini-batch partial optimal transport (mPOT) loss, and an L2 reconstruction loss preserves low-level detail. The pre-trained encoder is evaluated by fine-tuning on ten segmentation datasets (Table IV) and two classification datasets (Table V), reporting improvements over six self-supervised baselines and several supervised baselines. The central claimed contribution is that language-guided semantic supervision from 2D MLLMs yields transferable 3D representations that outperform vision-only SSL.

Significance. If the results hold, Med3DInsight is a significant practical step: it provides a scalable, annotation-free way to inject semantic and clinical knowledge into 3D medical encoders, and the PSAT module plus mPOT alignment are plausible mechanisms for bridging 3D volume features with 2D vision-language spaces. The paper includes useful ablations (PSAT components, loss terms, alignment strategy, MLLM choice, slice sampling, preprocessing, data efficiency) and reports consistent gains across CT/MRI, segmentation, and classification. However, the headline 'unseen transfer' claim depends on the pretraining corpora being disjoint from the downstream evaluation datasets; this is questionable for MSD-Liver and, if confirmed, would inflate a portion of the reported average improvement. The mPOT formulation is also underspecified in ways that affect reproducibility. With those two issues resolved, the contribution would be solid.

major comments (2)
  1. [Section IV.B, Table IV(j), Table I] The paper states that MSD-Heart, MSD-Liver, and MSD-Colon are 'unseen during pretraining' and explains that overlapping MSD tasks (Hippocampus, Prostate, Pancreas, Vessel, Spleen) were excluded from evaluation. However, 3DSeg-8 is the Med3D collection [1], whose liver subset is widely understood to be derived from LiTS; Table III lists both MSD-Liver and LiTs with 131 CT scans, and Table I counts 6,684 liver samples in the pretraining set. If LiTS-derived volumes appear in 3DSeg-8, then MSD-Liver is not unseen, and the 88.41 vs. 87.25 Dice gain in Table IV(j) is at least partly a domain-familiarity effect rather than cross-domain transfer. The paper's own enumeration of overlaps omits liver, which is internally inconsistent with the 'unseen' claim. This is load-bearing for the sentence in Section IV-C that Med3DInsight 'consistently outperforms SOTA SSL methods over eight datasets.' The
  2. [Section III-D.1, Eq. (2), Algorithm 1] The mPOT loss is not well-defined as written. The empirical plan \hat\pi_b is a b×b matrix with \hat\pi_ii = 1/b and zero off-diagonal, so it has total mass 1, while \pi^{X,S}_{s,\epsilon,c} is supposed to transport only mass s. A KL divergence between a probability measure and a measure of mass s is undefined unless s=1 or the two arguments are normalized. Algorithm 1 never states how s is set or how the constraint \Pi_s is enforced; the call 'Bregman(s, C, \epsilon)' is not a standard interface for partial OT. The Mahalanobis matrix M is updated with 'Proj' but the feasible set (e.g., PSD, fixed trace, bounded entries) is not given. These details are necessary to reproduce Table VI(c), where mPOT is claimed to outperform contrastive learning.
minor comments (5)
  1. [Section IV-C and Table IV] The text says 'eight datasets' and Figure 3(a) says 'eight different 3D medical image segmentation datasets,' but Table IV reports ten downstream segmentation panels (MM-WHS, CHAOS, OASIS1, MSD-Heart, ADNI, VS, AbdomenCT-1K, LiTs, MSD-Liver, MSD-Colon), and Table IV(a) is titled 'Average over 10 datasets.' The count should be made consistent.
  2. [Section IV-C] The sentence 'All SSL methods employed identical fine-tuning procedures, including pretraining hyperparameters (learning rate, batch size, training epochs) pretrain datasets, and evaluation metrics' is grammatically ambiguous. It should state exactly which pretraining datasets and hyperparameters are shared across baselines and which are method-specific; otherwise the claim of fair comparison is hard to verify.
  3. [Eq. (1)] The double sum is written as \sum_{i=1}^{nX} \sum_{j=1}^{nX}, but the second index should presumably range over the target sample count (nY or nS/nT). This is a notation error that makes the transport plan dimensions unclear.
  4. [Table VI(f)] The row for '1' slice per volume appears to have a formatting issue (the value 92.07 is merged with adjacent columns). Please reformat for readability.
  5. [Figure 3(d)] The modality-gap numbers (e.g., 82.48 vs. 76.07, 15.77 vs. 12.98) are reported without confidence intervals or an explanation of how many random samples/restarts were used; since t-SNE and UMAP projections are stochastic, a small repeated-run analysis would strengthen this visualization claim.

Circularity Check

2 steps flagged

MSD-Liver is claimed 'unseen' despite liver volumes in pretraining Table I, and the same LiTS data is evaluated twice as MSD-Liver and LiTs, inflating headline gains.

specific steps
  1. fitted input called prediction [Section IV.B (Downstream Tasks and Corresponding Datasets); Table I]
    "The 3DSeg-8 dataset [1], used for pretraining, includes five tasks that overlap with the MSD benchmark (Hippocampus, Prostate, Pancreas, Vessel, and Spleen). To eliminate domain familiarity and prevent potential data leakage, we exclude these overlapping tasks during evaluation. Instead, we select the Heart, Liver, and Colon tasks from MSD, which are unseen during pretraining, as the downstream evaluation target. This design ensures that performance reflects true cross-domain generalization rather than memorization effects from the pre-training data."

    The pretraining corpus 3DSeg-8 is the Med3D collection, which includes LiTS-derived liver volumes; the paper's own Table I lists Liver (6,684) among pretraining internal organs. MSD-Liver is the same LiTS data (131 CT scans). Therefore the Section IV.B assertion that MSD-Liver is 'unseen during pretraining' is false. The MSD-Liver row of Table IV is an in-distribution test: the reported 88.41 vs. 87.25 Dice advantage over vox2vec can be attributed to memorization/domain familiarity rather than cross-domain transfer. The claimed generalization result for this dataset reduces, by construction, to having included the evaluation volumes in the training distribution.

  2. renaming known result [Table III; Table IV rows (i) and (j)]
    "MSD-Liver [62] CT 3 131 Liver, Kidneys, Spleen ... LiTs [68] CT 2 131 Liver, Liver Tumor"

    MSD-Liver is a re-packaging of the LiTS dataset, and both Table III rows report the identical scan count (131). The paper evaluates 'MSD-Liver' and 'LiTs' as separate datasets in Table IV and then uses both when computing the average Dice/HD95 gain 'over eight datasets'. This double-counts the same 131 volumes, so the headline improvement of over 1% average Dice is not measured over eight independent datasets; it is measured over at most seven, with LiTS contributing twice. The 'eight datasets' claim is thus inflated by renaming the same known dataset and treating it as a distinct result.

full rationale

The paper is an empirical systems paper; its pretraining objective Ltotal = LmPOT + Lres is self-contained and no formal derivation becomes circular by construction. The circularity lies in the evaluation design. Section IV.B asserts that MSD-Liver is 'unseen during pretraining,' but Table I lists Liver among the pretraining internal organs with 6,684 triplets from 3DSeg-8 (Med3D), and MSD-Liver is the LiTS dataset (131 volumes) already present in that corpus. Consequently, the MSD-Liver row in Table IV is not a cross-domain transfer prediction but an in-distribution familiarity/memorization test, so that portion of the headline 'unseen generalization' gain reduces to a training-input overlap. Additionally, Table III lists MSD-Liver and LiTs with the same 131 scans, and Table IV evaluates both as separate datasets, double-counting LiTS in the reported 'over eight datasets' average. These issues are concrete and quoteable; they undermine the central quantitative claim. No load-bearing self-citation, uniqueness import, or ansatz smuggling was found; the methodology itself is not definitionally circular. Overall, partial circularity via fitted/contaminated evaluation gives score 6.

Axiom & Free-Parameter Ledger

4 free parameters · 3 axioms · 0 invented entities

The ledger is dominated by evaluation-setup assumptions and unreported hyperparameters rather than mathematical axioms. The most consequential entry is the disjointness assumption, which is contradicted by known dataset composition. No new physical or ontological entities are postulated; PSAT and position embeddings are architectural components, not invented entities.

free parameters (4)
  • s (transported mass fraction) = not reported
    In Eq. 1 and Algorithm 1, partial OT requires the mass s to transport. The paper never gives a value, and the KL loss in Eq. 2 compares a full-mass diagonal plan with a partial plan, so the meaning of the loss depends on this unreported parameter.
  • epsilon (entropic regularization coefficient) = not reported
    The entropy regularizer in Eq. 1 controls plan smoothness and must be chosen in advance. No value is reported and no sensitivity analysis is provided.
  • Q (number of learnable queries in PSAT) = 300
    Chosen as 'a common setting' in Section IV-A; no ablation over query count is reported, so it is a hand-picked hyperparameter.
  • M (Mahalanobis interaction matrix) = learned during pretraining
    The ground metric matrix M is updated by gradient descent (Algorithm 1), allowing the alignment cost to be fitted to the training data rather than fixed a priori.
axioms (3)
  • domain assumption GPT-4V captions of 2D slices provide semantically valid supervision for 3D volumes.
    Introduced in Section III.A; the paper relies on MLLM output quality, and its own Table II shows inaccurate captions. POT mitigates but does not remove dependence on caption quality.
  • ad hoc to paper Pretraining datasets (3DSeg-8 and M3D) are disjoint from downstream evaluation datasets, specifically MSD-Liver.
    Stated in Section IV.B as 'MSD-Heart, MSD-Liver, and MSD-Colon are unseen during pretraining.' This appears false because 3DSeg-8 is the Med3D set that includes LiTS liver data, the same source used by MSD-Liver.
  • domain assumption CLIP, after fine-tuning on generated pairs, provides a stable 2D image-text space into which 3D features should be projected.
    Used throughout Section III.C; if the fine-tuned CLIP space does not capture medical semantics, the alignment loss trains toward a noisy target, and the text supervision adds little.

pith-pipeline@v1.3.0-alltime-deepseek · 25569 in / 16700 out tokens · 161130 ms · 2026-08-04T19:44:03.553456+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Enhancing 3D Medical Image Understanding with Pretraining Aided by 2D Multimodal Large Language Models." pith.science (2026). https://pith.science/paper/IOGMM4CY

@misc{pith2026250909064,
  author       = {Pith},
  title        = {Pith review of: Enhancing 3D Medical Image Understanding with Pretraining Aided by 2D Multimodal Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IOGMM4CY}},
  note         = {Machine review of arXiv:2509.09064}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Understanding 3D medical image volumes is critical in the medical field, yet existing 3D medical convolution and transformer-based self-supervised learning (SSL) methods often lack deep semantic comprehension. Recent advancements in multimodal large language models (MLLMs) provide a promising approach to enhance image understanding through text descriptions. To leverage these 2D MLLMs for improved 3D medical image understanding, we propose Med3DInsight, a novel pretraining framework that integrates 3D image encoders with 2D MLLMs via a specially designed plane-slice-aware transformer module. Additionally, our model employs a partial optimal transport based alignment, demonstrating greater tolerance to noise introduced by potential noises in LLM-generated content. Med3DInsight introduces a new paradigm for scalable multimodal 3D medical representation learning without requiring human annotations. Extensive experiments demonstrate our state-of-the-art performance on two downstream tasks, i.e., segmentation and classification, across various public datasets with CT and MRI modalities, outperforming current SSL methods. Med3DInsight can be seamlessly integrated into existing 3D medical image understanding networks, potentially enhancing their performance. Our source code, generated datasets, and pre-trained models will be available at https://github.com/Qybc/Med3DInsight.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

105 extracted references · 17 linked inside Pith

  1. [1]

    A classical technique in SSL involves image restoration to learn visual features by reconstructing corrupted images back to their original state [6], [8], [26], [27]

    Reconstruction-based Methods:Self-supervised learning (SSL) has significantly advanced computer vision by utilizing pretraining tasks that do not require annotations. A classical technique in SSL involves image restoration to learn visual features by reconstructing corrupted images back to their original state [6], [8], [26], [27]. Some of these approache...

  2. [2]

    Describe the image in fewer than 100 words

    Alignment-based Methods:Recently, contrastive learning has emerged as the standard method in SSL [11], [12]. To address the limitations of traditional contrastive learning, such as the requirement for large batch sizes and strong augmentations [29], BYOL [11] and BarlowTwins [12] em- ploy a dual-branch structure to align the embeddings of two augmented im...

  3. [3]

    Alignment Loss:Contrastive learning, as used in CLIP [24], offers an efficient self-learning technique by bring- ing the representations of paired data closer together in the 6 IEEE TRANSACTIONS AND JOURNALS TEMPLA TE embedding space while pushing apart the representations of non-paired data. However, the use of large language models inevitably leads to s...

  4. [4]

    Reconstruction Loss:To enhance visual feature learning with low-level visual semantics, we introduce an auxiliary pretext task, image restoration, which aims to reconstruct the 3D medical volume using latent visual encoders. The training objective for this task is formulated using the L2-norm below: Lres =E (X b) bX i=1 ||xi −ˆxi||2.(3) Input:Initialized ...

  5. [5]

    Total Loss:With a synergy of triplet alignment learning and image reconstruction, the overall pretraining objective of our Med3DInsight framework is represented as follows: Ltotal =L mPOT +L res (4) After pretraining, the resulting 3D image encoder serves as a visual backbone for downstream task-specific fine-tuning. IV. EXPERIMENTS A. Pretraining Setup a...

  6. [6]

    All pretraining experiments are implemented on four NVIDIA GeForce RTX 3090 GPUs

    The training is set for 300 epochs to ensure convergence. All pretraining experiments are implemented on four NVIDIA GeForce RTX 3090 GPUs. B. Downstream Tasks and Corresponding Datasets We evaluate the effectiveness of our method across a diverse set of 3D medical imaging tasks, varying in size, objectives, and modalities. Specifically, we evaluate the r...

  7. [7]

    Multi-Organs or Tumors Segmentation:For the 3D seg- mentation task, we implement this task that includes car- diac structure segmentation (MM-WHS [61] and MSD- Heart [62]), abdominal organ segmentation (MSD-Liver [62], CHAOS [63], and AbdomenCT-1K [64]), brain segmentation (OASIS1 [65] and ADNI [66]), and tumor segmentation (MSD-Colon [62], VS [67], LiTs ...

  8. [8]

    Each MRI scan’s image size is224×224×224, and the voxel spacing is 1.75 mm

    Disease Classification:For Alzheimer’s disease classi- fication, we use the OASIS2 dataset [69], consisting of 312 T1-weighted structural MRI scans collected from 135 subjects, including both AD subjects and healthy volunteers. Each MRI scan’s image size is224×224×224, and the voxel spacing is 1.75 mm. For Parkinson’s disease classification, the PPMI data...

  9. [9]

    Our Med3DInsight con- sistently outperforms SOTA SSL methods over eight datasets, surpassing the best baseline by over 1% in average Dice score and 0.7mm in average HD95

    Experimental Results for 3D Segmentation:Table IV showcases the experimental results of multi-organ and sub- structure segmentation across the various datasets, which in- clude 19 different organs or tumors. Our Med3DInsight con- sistently outperforms SOTA SSL methods over eight datasets, surpassing the best baseline by over 1% in average Dice score and 0...

  10. [10]

    The OASIS2 dataset presents the challenge of discerning Alzheimer’s disease, and the PPMI dataset presents the challenge of discerning Parkinson’s disease

    Experimental Results for 3D Classification:Table V re- ports the 3D classification results on the OASIS2 dataset and PPMI dataset. The OASIS2 dataset presents the challenge of discerning Alzheimer’s disease, and the PPMI dataset presents the challenge of discerning Parkinson’s disease. Nevertheless, our approach consistently excels in recognizing patholog...

  11. [11]

    In Table VI(a),Ex1represents the model without any query transformer or place-slice position embedding

    Effectiveness of the PSAT Model:To demonstrate the effectiveness of each component in PSAT, we conduct an ablation study on the OASIS1 3D segmentation task. In Table VI(a),Ex1represents the model without any query transformer or place-slice position embedding. In this scenario, the 3D image feature is directly fed into a projection layer to align the dime...

  12. [12]

    As depicted in Table VI(b), excluding eitherL mPOT orL res leads to reduced performance

    Loss Ablation:We further investigate the influence of different losses during the pre-training stage. As depicted in Table VI(b), excluding eitherL mPOT orL res leads to reduced performance. However, when combining all components as in our method, we achieve the highest performance. This underscores the importance of both losses for enhancing the effectiv...

  13. [13]

    We report dice scores for contrastive loss and the mini-batch partial optimal transport loss in Ta- ble VI(c)

    Alignment Strategy:We conduct experiments on the OASIS1 test set to examine the transferability of the learned joint embedding space. We report dice scores for contrastive loss and the mini-batch partial optimal transport loss in Ta- ble VI(c). The mPOT loss acquires the highest dice score for the segmentation task, which means the proposed mPOT loss achi...

  14. [14]

    Vision/Text Encoder:To assess the versatility of our method, we employ two distinct 2D encoders: the CLIP [24] and the BioMedCLIP [76]. As shown in Table VI(d), the performance differences between these encoders are marginal, underscoring that our method is agnostic to these two choices of backbones for vision and text encoders

  15. [15]

    Ablation of different image backbones:As seen in Fig- ure 3(b), we employ two distinct image backbones UNETR [4] and nnFormer [5]. Med3DInsight consistently improves differ- ent backbones across three datasets, indicating it is a compre- hensive representation learning framework that enriches visual representation for medical image segmentation

  16. [16]

    in which the GPT-4V we use have more accurate descriptions

    Ablation of different multimodal large language models: We conduct a qualitative comparison of the generation quality across three MLLMs, as shown in Table VII, including GPT- 4Vision and Claude, which are general-purpose models, and HuatuoGPT-Vision, which is specifically designed for the medical domain. in which the GPT-4V we use have more accurate desc...

    arXiv 1994

  17. [17]

    Different Slice-per-volume Strategies:To validate our design choice, we conduct ablation experiments comparing different slice-per-volume sampling strategies during pretrain- ing. Specifically, we test a setting using only the M3D subset (8,000 volumes) with three slices per volume, resulting in the same total number of slices (24,000) as our original app...

  18. [18]

    As shown in Table VI(g), Sliding window achieves slightly higher accuracy for targets with rich edge details, but resize provided faster inference with comparable Dice scores

    Different Preprocess Types:To evaluate the impact of different image preprocessing strategies, we conduct addi- tional experiments comparing two approaches: (1) Isotropic resampling (i.e., resizing) of the entire volume to a fixed 1283 resolution, as we did; (2) Sliding-window-based ROI cropping using MONAI [79], which extracts128 3 patches from the origi...

  19. [19]

    To validate the data efficiency of our Med3DInsight framework, we check the average dice changes with varying amounts of fine-tuning samples

    Data Efficiency:Model pretraining can potentially re- duce the demand for labeled data in downstream tasks. To validate the data efficiency of our Med3DInsight framework, we check the average dice changes with varying amounts of fine-tuning samples. Our method consistently outperforms baseline methods across all training data percentages. As shown in Figu...

  20. [20]

    multiple hypodense lesions

    Latent Representation of Pretrained Model:We utilize the pretrained visual encoder to extract latent representations from 3D medical volume images of the brain, abdomen, and chest. We then apply the t-SNE algorithm [80] and UMAP [81] for dimensionality reduction and visualization. Figure 3(d) shows our resulting 2D representations, comparing the SOTA SSL ...

  21. [21]

    Med3d: Transfer learning for 3d medical image analysis,

    S. Chen, K. Ma, and Y . Zheng, “Med3d: Transfer learning for 3d medical image analysis,”arXiv preprint arXiv:1904.00625, 2019

    Pith/arXiv arXiv 1904

  22. [22]

    M3t: three-dimensional medical image classifier using multi-plane and multi-slice transformer,

    J. Jang and D. Hwang, “M3t: three-dimensional medical image classifier using multi-plane and multi-slice transformer,” inProceedings of the IEEE/CVF conference on CVPR, 2022, pp. 20 718–20 729

    2022

  23. [23]

    Universeg: Universal medical image segmentation,

    V . I. Butoi, J. J. G. Ortiz, T. Ma, M. R. Sabuncu, J. Guttag, and A. V . Dalca, “Universeg: Universal medical image segmentation,” in Proceedings of the IEEE/CVF ICCV, 2023, pp. 21 438–21 451

    2023

  24. [24]

    Unetr: Transformers for 3d medical image segmentation,

    A. Hatamizadeh, Y . Tang, V . Nath, D. Yang, A. Myronenko, B. Land- man, H. R. Roth, and D. Xu, “Unetr: Transformers for 3d medical image segmentation,” inIEEE/CVF WACV, 2022, pp. 574–584

    2022

  25. [25]

    nn- former: V olumetric medical image segmentation via a 3d transformer,

    H.-Y . Zhou, J. Guo, Y . Zhang, X. Han, L. Yu, L. Wang, and Y . Yu, “nn- former: V olumetric medical image segmentation via a 3d transformer,” IEEE Transactions on Image Processing, 2023

    2023

  26. [26]

    Simmim: A simple framework for masked image modeling,

    Z. Xie, Z. Zhang, Y . Cao, Y . Lin, J. Bao, Z. Yao, Q. Dai, and H. Hu, “Simmim: A simple framework for masked image modeling,” in Proceedings of the IEEE/CVF CVPR, 2022, pp. 9653–9663

    2022

  27. [27]

    Models genesis: Generic autodidactic models for 3d medical image analysis,

    Z. Zhou, V . Sodha, M. M. Rahman Siddiquee, R. Feng, N. Tajbakhsh, M. B. Gotway, and J. Liang, “Models genesis: Generic autodidactic models for 3d medical image analysis,” inMICCAI 2019, Part IV 22. Springer, 2019, pp. 384–393

    2019

  28. [28]

    Masked au- toencoders are scalable vision learners,

    K. He, X. Chen, S. Xie, Y . Li, P. Doll ´ar, and R. Girshick, “Masked au- toencoders are scalable vision learners,” inProceedings of the IEEE/CVF Conference on CVPR, 2022, pp. 16 000–16 009

    2022

  29. [29]

    Masked image modeling advances 3d medical image anal- ysis,

    Z. Chen, D. Agarwal, K. Aggarwal, W. Safta, M. M. Balan, and K. Brown, “Masked image modeling advances 3d medical image anal- ysis,” inProceedings of the IEEE/CVF WACV, 2023, pp. 1970–1980

    2023

  30. [30]

    Self pre- training with masked autoencoders for medical image classification and segmentation,

    L. Zhou, H. Liu, J. Bae, J. He, D. Samaras, and P. Prasanna, “Self pre- training with masked autoencoders for medical image classification and segmentation,” inISBI. IEEE, 2023, pp. 1–6

    2023

  31. [31]

    Bootstrap your own latent-a new approach to self-supervised learning,

    J.-B. Grill, F. Strub, F. Altch ´e, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pireset al., “Bootstrap your own latent-a new approach to self-supervised learning,”Advances in neural information processing systems, vol. 33, pp. 21 271–21 284, 2020

    2020

  32. [32]

    Barlow twins: Self-supervised learning via redundancy reduction,

    J. Zbontar, L. Jing, I. Misra, Y . LeCun, and S. Deny, “Barlow twins: Self-supervised learning via redundancy reduction,” inInternational conference on machine learning. PMLR, 2021, pp. 12 310–12 320

    2021

  33. [33]

    Exploring simple siamese representation learning,

    X. Chen and K. He, “Exploring simple siamese representation learning,” inIEEE/CVF CVPR, 2021, pp. 15 750–15 758

    2021

  34. [34]

    A unified visual information preservation framework for self-supervised pre-training in medical image analysis,

    H.-Y . Zhou, C. Lu, C. Chen, S. Yang, and Y . Yu, “A unified visual information preservation framework for self-supervised pre-training in medical image analysis,”IEEE TPAMI, 2023

    2023

  35. [35]

    vox2vec: A framework for self-supervised contrastive learning of voxel- level representations in medical images,

    M. Goncharov, V . Soboleva, A. Kurmukov, M. Pisov, and M. Belyaev, “vox2vec: A framework for self-supervised contrastive learning of voxel- level representations in medical images,” inMICCAI, 2023, pp. 605–614

    2023

  36. [36]

    Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,”arXiv preprint arXiv:2301.12597, 2023

    Pith/arXiv arXiv 2023

  37. [37]

    Flamingo: a visual language model for few-shot learning,

    J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y . Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynoldset al., “Flamingo: a visual language model for few-shot learning,”Advances in Neural Information Processing Systems, vol. 35, pp. 23 716–23 736, 2022

    2022

  38. [38]

    Gpt-4v(ision) system card,

    OpenAI, “Gpt-4v(ision) system card,” 2023

    2023

  39. [39]

    Medblip: Bootstrapping language-image pre-training from 3d medical images and texts,

    Q. Chen, X. Hu, Z. Wang, and Y . Hong, “Medblip: Bootstrapping language-image pre-training from 3d medical images and texts,”arXiv preprint arXiv:2305.10799, 2023

    Pith/arXiv arXiv 2023

  40. [40]

    Gen- erative text-guided 3d vision-language pretraining for unified medical image segmentation,

    Y . Chen, C. Liu, W. Huang, S. Cheng, R. Arcucci, and Z. Xiong, “Gen- erative text-guided 3d vision-language pretraining for unified medical image segmentation,”arXiv preprint arXiv:2306.04811, 2023

    Pith/arXiv arXiv 2023

  41. [41]

    Performance of multimodal gpt- 4v on usmle with image: Potential for imaging diagnostic support with explanations,

    Z. Yang, Z. Yao, M. Tasmin, P. Vashisht, W. S. Jang, F. Ouyang, B. Wang, D. Berlowitz, and H. Yu, “Performance of multimodal gpt- 4v on usmle with image: Potential for imaging diagnostic support with explanations,”medRxiv, pp. 2023–10, 2023

    2023

  42. [42]

    Can gpt-4v (ision) serve medical applications? case studies on gpt-4v for multimodal medical diagnosis,

    C. Wu, J. Lei, Q. Zheng, W. Zhao, W. Lin, X. Zhanget al., “Can gpt-4v (ision) serve medical applications? case studies on gpt-4v for multimodal medical diagnosis,”arXiv preprint arXiv:2310.09909, 2023

    Pith/arXiv arXiv 2023

  43. [43]

    End-to-end object detection with transformers,

    N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in ECCV. Springer, 2020, pp. 213–229

    2020

  44. [44]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askellet al., “Learning transferable visual models from natural language supervision,” inICML, 2021, pp. 8748–8763

    2021

  45. [45]

    Ulip: Learning a unified representation of language, images, and point clouds for 3d understanding,

    L. Xue, M. Gao, C. Xing, R. Mart ´ın-Mart´ın, J. Wu, C. Xiong, R. Xu, J. C. Niebles, and S. Savarese, “Ulip: Learning a unified representation of language, images, and point clouds for 3d understanding,” inPro- ceedings of the IEEE/CVF Conference on CVPR, 2023, pp. 1179–1189

    2023

  46. [46]

    Masked feature prediction for self-supervised visual pre-training,

    C. Wei, H. Fan, S. Xie, C.-Y . Wu, A. Yuille, and C. Feichtenhofer, “Masked feature prediction for self-supervised visual pre-training,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 14 668–14 678

    2022

  47. [47]

    Unsupervised repre- sentation learning by predicting image rotations,

    S. Gidaris, P. Singh, and N. Komodakis, “Unsupervised repre- sentation learning by predicting image rotations,”arXiv preprint arXiv:1803.07728, 2018

    Pith/arXiv arXiv 2018

  48. [48]

    Self-supervised learning for medical image analysis using image context restoration,

    L. Chen, P. Bentley, K. Mori, K. Misawa, M. Fujiwara, and D. Rueck- ert, “Self-supervised learning for medical image analysis using image context restoration,”Medical image analysis, vol. 58, p. 101539, 2019

    2019

  49. [49]

    Momentum contrast for unsupervised visual representation learning,

    K. He, H. Fan, Y . Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” inProceedings of the IEEE/CVF Conference on CVPR, 2020, pp. 9729–9738

    2020

  50. [50]

    Contrastive learning of medical visual representations from paired images and text,

    Y . Zhang, H. Jiang, Y . Miura, C. D. Manning, and C. P. Langlotz, “Contrastive learning of medical visual representations from paired images and text,” inMachine Learning for Healthcare Conference. PMLR, 2022, pp. 2–25

    2022

  51. [51]

    Medclip: Contrastive learning from unpaired medical images and text,

    Z. Wang, Z. Wu, D. Agarwal, and J. Sun, “Medclip: Contrastive learning from unpaired medical images and text,”arXiv preprint arXiv:2210.10163, 2022

    Pith/arXiv arXiv 2022

  52. [52]

    Gloria: A multimodal global-local representation learning framework for label- efficient medical image recognition,

    S.-C. Huang, L. Shen, M. P. Lungren, and S. Yeung, “Gloria: A multimodal global-local representation learning framework for label- efficient medical image recognition,” inIEEE/CVF ICCV, 2021, pp. 3942–3951

    2021

  53. [53]

    Uniseg: A prompt- driven universal segmentation model as well as a strong representation learner,

    Y . Ye, Y . Xie, J. Zhang, Z. Chen, and Y . Xia, “Uniseg: A prompt- driven universal segmentation model as well as a strong representation learner,” inInternational Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2023, pp. 508–518

    2023

  54. [54]

    Towards generalist foundation model for radiology,

    C. Wu, X. Zhang, Y . Zhang, Y . Wang, and W. Xie, “Towards generalist foundation model for radiology,”arXiv:2308.02463, 2023

    Pith/arXiv arXiv 2023

  55. [55]

    M3d: Advancing 3d medical image analysis with multi-modal large language models,

    F. Bai, Y . Du, T. Huang, M. Q.-H. Meng, and B. Zhao, “M3d: Advancing 3d medical image analysis with multi-modal large language models,” arXiv preprint arXiv:2404.00578, 2024

    Pith/arXiv arXiv 2024

  56. [56]

    Utilizing synthetic data for medical vision-language pre-training: Bypassing the need for real images,

    C. Liu, A. Shah, W. Bai, and R. Arcucci, “Utilizing synthetic data for medical vision-language pre-training: Bypassing the need for real images,”arXiv preprint arXiv:2310.07027, 2023

    Pith/arXiv arXiv 2023

  57. [57]

    Roentgen: Vision-language foundation model for chest x-ray generation,

    P. Chambon, C. Bluethgen, J.-B. Delbrouck, R. Van der Sluijs, M. Połacin, J. M. Z. Chaves, T. M. Abraham, S. Purohit, C. P. Langlotz, and A. Chaudhari, “Roentgen: Vision-language foundation model for chest x-ray generation,”arXiv preprint arXiv:2211.12737, 2022

    Pith/arXiv arXiv 2022

  58. [58]

    Medimp: 3d medical images with 14 IEEE TRANSACTIONS AND JOURNALS TEMPLA TE clinical prompts from limited tabular data for renal transplantation,

    L. Milecki, V . Kalogeiton, S. Bodard, D. Anglicheau, J.-M. Correas, M.-O. Timsit, and M. Vakalopoulou, “Medimp: 3d medical images with 14 IEEE TRANSACTIONS AND JOURNALS TEMPLA TE clinical prompts from limited tabular data for renal transplantation,” in Medical Imaging with Deep Learning, 2023

    2023

  59. [59]

    Joint distribution optimal transportation for domain adaptation,

    N. Courty, R. Flamary, A. Habrard, and A. Rakotomamonjy, “Joint distribution optimal transportation for domain adaptation,”Advances in neural information processing systems, vol. 30, 2017

    2017

  60. [60]

    Disentangle then calibrate with gradient guidance: A unified framework for common and rare disease diagnosis,

    Y . Chen, X. Guo, Y . Xia, and Y . Yuan, “Disentangle then calibrate with gradient guidance: A unified framework for common and rare disease diagnosis,”IEEE Transactions on Medical Imaging, 2024

    2024

  61. [61]

    Spatial and modal optimal transport for fast cross-modal mri reconstruction,

    Q. Wang, Z. Wen, J. Shi, Q. Wang, D. Shen, and S. Ying, “Spatial and modal optimal transport for fast cross-modal mri reconstruction,”IEEE Transactions on Medical Imaging, 2024

    2024

  62. [62]

    Uniter: Universal image-text representation learning,

    Y .-C. Chen, L. Li, L. Yu, A. El Kholy, F. Ahmed, Z. Gan, Y . Cheng, and J. Liu, “Uniter: Universal image-text representation learning,” inECCV. Springer, 2020, pp. 104–120

    2020

  63. [63]

    Weakly-supervised temporal ac- tion alignment driven by unbalanced spectral fused gromov-wasserstein distance,

    D. Luo, Y . Wang, A. Yue, and H. Xu, “Weakly-supervised temporal ac- tion alignment driven by unbalanced spectral fused gromov-wasserstein distance,” in30th ACM International Conference on Multimedia, 2022, pp. 728–739

    2022

  64. [64]

    Self-supervised video summarization guided by semantic inverse optimal transport,

    Y . Wang, H. Xu, and D. Luo, “Self-supervised video summarization guided by semantic inverse optimal transport,” inProceedings of 31st ACM International Conference on Multimedia, 2023, pp. 6611–6622

    2023

  65. [65]

    Partial unbalanced feature transport for cross-modality cardiac image segmentation,

    S. Dong, Z. Pan, Y . Fu, D. Xu, K. Shi, Q. Yang, Y . Shi, and C. Zhuo, “Partial unbalanced feature transport for cross-modality cardiac image segmentation,”IEEE Transactions on Medical Imaging, 2023

    2023

  66. [66]

    Prototype correlation matching and class-relation reasoning for few-shot medical image segmentation,

    Y . Zhang, H. Li, Y . Gao, H. Duan, Y . Huang, and Y . Zheng, “Prototype correlation matching and class-relation reasoning for few-shot medical image segmentation,”IEEE Transactions on Medical Imaging, 2024

    2024

  67. [67]

    Learning to match via inverse optimal transport,

    R. Li, X. Ye, H. Zhou, and H. Zha, “Learning to match via inverse optimal transport,”Journal of machine learning research, vol. 20, no. 80, pp. 1–37, 2019

    2019

  68. [68]

    Inverse optimal transport,

    A. M. Stuart and M.-T. Wolfram, “Inverse optimal transport,”SIAM Journal on Applied Mathematics, vol. 80, no. 1, pp. 599–619, 2020

    2020

  69. [69]

    Data efficient language-supervised zero-shot recognition with optimal transport distillation,

    B. Wu, R. Cheng, P. Zhang, T. Gao, P. Vajda, and J. E. Gonzalez, “Data efficient language-supervised zero-shot recognition with optimal transport distillation,”arXiv preprint arXiv:2112.09445, 2021

    Pith/arXiv arXiv 2021

  70. [70]

    A computational fluid mechanics solution to the monge-kantorovich mass transfer problem,

    J.-D. Benamou and Y . Brenier, “A computational fluid mechanics solution to the monge-kantorovich mass transfer problem,”Numerische Mathematik, vol. 84, no. 3, pp. 375–393, 2000

    2000

  71. [71]

    Sinkhorn distances: Lightspeed computation of optimal transport,

    M. Cuturi, “Sinkhorn distances: Lightspeed computation of optimal transport,”Advances in neural information processing systems, vol. 26, 2013

    2013

  72. [72]

    Revisiting deep audio-text retrieval through the lens of transportation,

    M. Luong, K. Nguyen, N. Ho, R. Haf, D. Phung, and L. Qu, “Revisiting deep audio-text retrieval through the lens of transportation,”arXiv preprint arXiv:2405.10084, 2024

    Pith/arXiv arXiv 2024

  73. [73]

    The ma- halanobis distance,

    R. De Maesschalck, D. Jouan-Rimbaud, and D. L. Massart, “The ma- halanobis distance,”Chemometrics and intelligent laboratory systems, vol. 50, no. 1, pp. 1–18, 2000

    2000

  74. [74]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,”Advances in neural information processing systems, vol. 30, 2017

    2017

  75. [75]

    Computational optimal transport: With applications to data science,

    G. Peyr ´e, M. Cuturiet al., “Computational optimal transport: With applications to data science,”Foundations and Trends® in Machine Learning, vol. 11, no. 5-6, pp. 355–607, 2019

    2019

  76. [76]

    Scaling algorithms for unbalanced optimal transport problems,

    L. Chizat, G. Peyr ´e, B. Schmitzer, and F.-X. Vialard, “Scaling algorithms for unbalanced optimal transport problems,”Mathematics of Computa- tion, vol. 87, no. 314, pp. 2563–2609, 2018

    2018

  77. [77]

    Iterative bregman projections for regularized transportation problems,

    J.-D. Benamou, G. Carlier, M. Cuturi, L. Nenna, and G. Peyr ´e, “Iterative bregman projections for regularized transportation problems,”SIAM Journal on Scientific Computing, vol. 37, no. 2, pp. A1111–A1138, 2015

    2015

  78. [78]

    Ground metric learning,

    M. Cuturi and D. Avis, “Ground metric learning,”The Journal of Machine Learning Research, vol. 15, no. 1, pp. 533–564, 2014

    2014

  79. [79]

    Euclidean distance mapping,

    P.-E. Danielsson, “Euclidean distance mapping,”Computer Graphics and image processing, vol. 14, no. 3, pp. 227–248, 1980

    1980

  80. [80]

    A swiss army knife for minimax optimal transport,

    S. Dhouib, I. Redko, T. Kerdoncuff, R. Emonet, and M. Sebban, “A swiss army knife for minimax optimal transport,” inInternational Conference on Machine Learning. PMLR, 2020, pp. 2504–2513

    2020

Showing first 80 references.