{"id":"bfcb9728-3397-4291-a335-0b8682661ef7","arxiv_id":"2508.12108","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"VELVET-Med combines a sentence-aware TriBERT text encoder, hierarchical contrastive learning, and uni-modal self-supervision to pre-train 3D CT vision-language encoders on 38,875 scan-report pairs.","lead":"This paper trains a vision-language model on 3D CT scans and their radiology reports using only about 39,000 pairs, adding sentence-level text modeling and multi-level contrastive learning to make up for the small dataset. The authors report gains over earlier baselines on segmentation, retrieval, question answering, and report generation, but the results depend on a manually filtered version of the M3D-CAP dataset.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline gains may owe more to dataset curation than to VELVET-Med: all method comparisons use the manually filtered M3D-CAP-filtered subset (Sec. 4.1), and the only filter-effect comparison shows a large retrieval jump that precedes any architectural change.","rationale":"The reader's weakest assumption identifies the dataset filter as the load-bearing premise, and I agree. The paper has a coherent framework and several ablations that support individual components, but the central causal claim that the method, rather than data selection, produces the reported SOTA results is not established. The cleanest evidence for the confound is already inside the paper: Table 3 shows that switching from unfiltered to filtered data with the same architecture changes R@1 from 19.10 to 26.29. Because all proposed-method comparisons then use filtered data only, the method's marginal contribution is measured against a selected subset, not against the original distribution. The internal retrieval results further weaken the claim that the full combination is necessary, since the full model is not the best row in Table 3. I do not recommend changing the CONDITIONAL verdict: no internal contradiction or fabrication is apparent, and the issue could be resolved by releasing the filter and re-running the key comparisons on an unfiltered or independently validated dataset. The requested 2x2 interaction test would settle whether curation or architecture/objectives drive the gains.","tokens_in":19959,"tokens_out":6971,"duration_ms":77793,"concrete_test":"Run a 2x2 pre-training comparison: CLIP-3D (BERT-B/ViT-B) and VELVET-Med, each trained with identical hyperparameters on M3D-CAP-filtered and on M3D-CAP-clean (the 69,686-pair set with only the subjective disorder filter, excluding the 48-slice cutoff). Evaluate SRR/RSR on the same held-out test set. If VELVET-Med's retrieval advantage over CLIP-3D largely shrinks or disappears on M3D-CAP-clean, then the 48-slice curation step is a major confound and the architecture/objective claims need to be re-benchmarked on an unfiltered or externally validated dataset.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim attributes the reported downstream gains to TriBERT, hierarchical contrastive learning, and uni-modal self-supervision, but the evidence cannot separate the method from the data filter. In Section 4.1, the pre-training set is reduced from 120,092 to 38,875 pairs by manually excluding scans judged to have \"disordered\" z-axis slice order and scans with fewer than 48 slices; Appendix A states that scans are stored without metadata and gives no automated criterion for disorder. The paper's own 'Effect of data quality' experiment in Table 3 shows that simply changing the dataset while holding the architecture fixed (BERT-B/ViT-B) raises SRR R@1 from 19.10 to 26.29, so curation alone is a large effect. Every subsequent comparison of the proposed architecture and objectives is performed only on M3D-CAP-filtered, with no external validation of the filter on independent clinical data. Thus the margin over prior M3D-based models could be substantially driven by selecting an easier or cleaner subset rather than by the proposed method. The internal ablations also do not fully support the claim: in Table 3, the full VELVET-Med row (34.86 R@1 SRR) is worse than Lcm+Llan+Lvis (36.18 R@1 SRR), so the complete combination is not even the best configuration on retrieval within the paper's own results. The promised release of the filtered dataset and filter code is absent, making the curation step non-reproducible.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes VELVET-Med, a vision-and-language pre-training framework for 3D CT volumes paired with radiology reports. The main contributions are a TriBERT language encoder that models report-, sentence-, and word-level semantics through special input embeddings and an attention mask; a hierarchical contrastive learning objective that aligns top/middle/bottom visual features with report/sentence/word text embeddings; and the inclusion of uni-modal vision and language self-supervised losses alongside cross-modal and multi-modal objectives. The model is pre-trained on M3D-CAP-filtered, a manually curated subset of M3D-CAP containing 38,875 scan-report pairs, and then evaluated on 3D segmentation (AbdomenCT-1K, CT-ORG), cross-modal retrieval, generative and classification-based VQA, and medical report generation. The paper claims state-of-the-art transferability across these tasks.","tokens_in":20204,"tokens_out":6892,"duration_ms":68137,"significance":"If the results hold, the paper makes a useful contribution to data-efficient 3D medical vision-language pre-training. The TriBERT sentence-aware encoding and the hierarchical contrastive alignment are reasonable architectural ideas, and the combination of uni-modal and cross-modal objectives is worth studying in the low-data volumetric setting. The paper also promises to release a curated dataset, which would aid reproducibility. However, the significance is currently conditional: the evaluation does not isolate the proposed method from the dataset curation step, the main generative-task baseline is a re-implementation, all results appear to be single-run point estimates, and the full proposed objective is not the best configuration in the paper's own retrieval ablation. These issues must be addressed before the SOTA claims can be accepted.","major_comments":[{"comment":"The central claim that the reported gains come from TriBERT, hierarchical contrastive learning, and uni-modal supervision is not separable from the dataset curation step. Pre-training is performed exclusively on M3D-CAP-filtered (38,875 pairs), obtained by manually excluding scans judged to have disordered z-axis slice order and scans with fewer than 48 slices. Appendix A states that scans are stored without metadata and gives no automated criterion for 'disordered,' and the promised release of the filter and filtered dataset is absent. The paper's own data-quality experiment (Table 3) shows that holding the architecture fixed (BERT-B/ViT-B) and moving from the unfiltered M3D retrieval model to CLIP-3D on M3D-CAP-filtered raises SRR R@1 from 19.10 to 26.29, a larger jump than most subsequent architectural or objective changes. Because the filter is validated on the same benchmark used for the SOTA claims and is not externally validated on independent clinical data, the reported margins over prior M3D-based models could be substantially due to selection of easier or cleaner volumes. To support the attribution, the authors should train the M3D retrieval baseline and CLIP-3D on the same filtered data, train the proposed method on unfiltered or intermediate-quality data, and provide an automated, reproducible filter criterion or external validation of the curation decisions.","section":"Sec. 4.1; Appendix A; Table 3"},{"comment":"The full VELVET-Med objective combination is not the best configuration in the paper's own retrieval results. In Table 3, VELVET-Med (Lcm,Lmm,Llan,Lvis) obtains SRR R@1 34.86 and RSR R@1 33.12, while Lcm+Llan+Lvis reaches 36.18 and 36.60, and Lcm+Lvis reaches 36.85 and 35.37. The text says VELVET-Med 'achieves competitive retrieval results,' but the ablation does not support the claim that the complete proposed model is the best or that adding multi-modal supervision helps retrieval. The authors need to explain this degradation, report which configuration is recommended for each downstream task, and state whether the full model was selected on a held-out validation criterion or by another principle.","section":"Table 3"},{"comment":"All experiments appear to be single runs without error bars or significance testing, and several reported differences are small (e.g., Table 2 Dice 94.03 vs 93.96; Table 4 classification-based VQA AUC 84.27 vs 84.01). The SOTA claim therefore rests on point estimates that may be within run-to-run noise. The authors should provide results over multiple seeds with means and standard deviations and, where appropriate, significance tests. In addition, the generative VQA and report-generation results are fine-tuned on a 10% subsample disclosed only in Appendix B.3; this caveat should be stated in the main tables and taken into account when claiming that VELVET-Med achieves the best performance on those tasks.","section":"Tables 1–4; Appendix B.3"},{"comment":"The generative VQA and report-generation comparisons are not on equal footing with the official M3D-LaMed baseline. The baseline M3D-LaMed$ is the authors' own re-implementation using the pre-trained vision encoder from CLIP-3D, and both the proposed and baseline generative models are fine-tuned on a 10% subsample of M3D-VQA/M3D-CAP-filtered. The 'state-of-the-art' claim on these tasks therefore depends on a non-official baseline and a small subsample. Please report official M3D-LaMed numbers, evaluate on the full fine-tuning set if feasible, or clearly qualify the claim and state the subsample size in the main text.","section":"Table 4; Appendix B.3"}],"minor_comments":[{"comment":"The row label 'Lcm,Lvis,Llan,Lvis(VELVET-Med)' in Table 3 duplicates Lvis and does not match the objective set in Eq. (4); it should read Lcm,Lmm,Llan,Lvis. The same inconsistency appears in Table 4, and Tables C/D refer to 'De-MedViL' instead of VELVET-Med.","section":"Table 3; Table 4; Tables C/D"},{"comment":"There is an unresolved placeholder '2D interpolation [cite]' in Appendix A; this citation needs to be completed before publication.","section":"Appendix A"},{"comment":"The notation for visual features is inconsistent: the text writes 'f^k_v in R^{N x c^k_v x h^k x w^k x d^i}' with mixed superscripts and a dangling 'di'. This should be corrected to a single coherent form such as R^{N x c^k_v x h^k x w^k x d^k}.","section":"Sec. 3.5"},{"comment":"The total loss is written as an unweighted sum Lvis + Llan + Lcm + Lmm; please state whether all weights are fixed to 1 and whether any sensitivity analysis or tuning of these weights was performed.","section":"Eq. (5)"},{"comment":"The cross-modal retrieval evaluation uses a 'M3D-CAP-filtered test set,' but the construction and size of this 2000-pair test set are not fully described in the main text. In addition, Table 1 reports scan sizes 256,256,32 alongside 96,96,96; the meaning of these size tuples should be clarified.","section":"Tables 1 and 3"},{"comment":"There are several typos and grammatical issues (e.g., 'adatabple' in the introduction, 'parings' in Section 1) that should be corrected in a careful proofread.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the dataset-curation confound: the paper's own Table 3 shows that changing only the data gives a large retrieval jump, and all architecture/objective comparisons are on the filtered subset. A controlled comparison with the same architectures on filtered vs. unfiltered data, plus official or carefully matched baselines for generative tasks, is essential before the SOTA claims can be supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Honest take: this is a real attempt at a data-efficient 3D medical VLP, and the TriBERT sentence-level encoder plus hierarchical contrastive learning are sensible ideas. The ablation tables mostly support the value of hierarchical alignment over a plain CLIP-style objective, and the breadth of downstream tasks (segmentation, retrieval, VQA, report generation) is respectable. The dataset curation story—filtering out disordered and short scans—is also a legitimate practical contribution if the filtered set is actually released.\n\nBut the paper does not support its headline claim. First, the full VELVET-Med model is not the best configuration in the paper's own retrieval table: Lcm+Lvis gives 36.85 SRR R@1, while VELVET-Med gets 34.86. The full model only tops the generative tables, so 'state of the art across a wide range of tasks' is at best half-true. Second, the data filter is a serious confound. The paper's own 'effect of data quality' experiment shows CLIP-3D on the filtered set jumps from 19.10 to 26.29 SRR R@1 compared to the M3D model on unfiltered data—a gain larger than any single architectural change. Since every method comparison runs on the filtered set, we cannot separate the method's contribution from curation. The filter itself uses a manual 'disordered' criterion that is not automated or externally validated, so the whole pipeline is non-reproducible as described.\n\nThen there are smaller but real issues: no error bars, single-run results; the generative VQA and report generation fine-tune on a 10% subsample disclosed only in Appendix B.3; the main baseline is the authors' own re-implementation of M3D-LaMed; and the model is called De-MedViL in the appendix tables with no explanation, which is odd and needs clarification before I'd trust the novelty claim. Code and data are promised but not linked.\n\nIf the authors release the filtered dataset and filter criteria, run multiple seeds, and add an experiment that holds the data fixed while varying the method (or vice versa), the central claim could become testable. As it stands, this is a solid engineering contribution with an overclaimed narrative.\n\nWho should read it: people building 3D medical VLMs will find the TriBERT design and the ablations useful. I would send it to referees, but with the expectation of heavy revision. My advice: engage, but explicitly ask for the missing artifacts and a cleaner separation of data quality from method.","headline":"The framework is plausible and the ablation story is mostly coherent, but the SOTA claim is undercut by the data-curation confound and an unexplained name change.","tokens_in":20806,"tokens_out":3233,"would_cite":false,"duration_ms":31524,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Aligning coarse-to-fine visual features with report-, sentence-, and word-level text lets a 3D medical vision-language model beat larger models while pre-training on only 38,875 CT-report pairs.","keywords":["vision-language pre-training","volumetric medical imaging","3D CT scans","hierarchical contrastive learning","self-supervised learning","radiology report generation","cross-modal retrieval","visual question answering"],"falsifier":"Train VELVET-Med with identical hyperparameters on the unfiltered M3D-CAP set (120,092 pairs) and on M3D-CAP-filtered (38,875 pairs), then compare retrieval and segmentation; if the filtered model does not clearly beat the unfiltered one, the gains are curation rather than the architecture or objectives. A second check is to have independent radiologists re-score the excluded scans: if many excluded volumes have clinically valid anatomy, the filter is introducing selection bias.","tokens_in":19659,"feed_emoji":"🩻","tokens_out":7850,"duration_ms":78748,"temperature":0.7,"pith_summary":"Volumetric medical imaging lacks the large scan-report datasets that power 2D vision-language models, and the paper sets out to show that clever pre-training can substitute for data volume. VELVET-Med predicts that if a text encoder is taught to represent reports, sentences, and words separately, and those levels are aligned one-by-one with coarse-to-fine visual features of a 3D CT scan, then strong transferable encoders can be learned from only 38,875 pairs. The resulting encoders are claimed to reach state-of-the-art performance on 3D semantic segmentation, cross-modal retrieval, visual question answering, and report generation. The paper also argues that a smaller, manually cleaned dataset of anatomically ordered volumes beats a larger noisy one, making data quality a central part of the result.","feed_headline":"38,875 CT-report pairs beat much larger 3D medical models","feed_subtitle":"Aligning report, sentence, and word semantics to 3D scans transfers scarce CT data to four clinical tasks.","key_machinery":"The load-bearing mechanism is TriBERT, a BERT-style text encoder augmented with a learnable [SENT] token per sentence and sentence-type embeddings, plus a tri-level attention mask that lets sentence tokens attend only to their own sentence and the global [CLS] token, preventing inter-sentence leakage. Its three text levels (report, sentence, word) are matched by a hierarchical contrastive learning objective to three visual levels (top, middle, bottom feature maps), replacing a single global CLIP-style loss. Uni-modal self-supervised losses on both modalities complete the training signal. This three-layer alignment is what the paper credits for capturing both coarse concepts and fine-grained anatomical detail, and it is the component that carries the reported transferability.","core_discovery":"The central discovery is that cross-modal alignment for volumetric medical data should be hierarchical rather than whole-pair-only. VELVET-Med aligns the three deepest visual feature maps of a 3D Swin-based vision encoder to report-level, sentence-level, and word-level embeddings of a custom text encoder called TriBERT, while simultaneously training each modality alone with self-supervision (masked volume inpainting, rotation prediction, contrastive coding for images; masked language modeling for text). With this combination, the paper reports that the pre-trained encoders transfer to four downstream families of tasks and beat the prior M3D-based models even though pre-training uses only 38,875 scan-report pairs. The paper further claims that filtering the M3D-CAP dataset to exclude scans with disordered z-axis slice order and fewer than 48 slices substantially improves retrieval, indicating that dataset curation is itself a source of the gains.","pith_inferences":["Editorial inference: the dataset filter is not automated, so the clean-versus-noisy comparison conflates data curation with the model; a same-architecture ablation on unfiltered data would isolate the curation effect.","Editorial inference: the paper's own table shows that multi-modal matching can hurt retrieval when added to cross-modal contrast, suggesting the optimal weighting of the four losses is task-dependent and that a tuned schedule could push results further.","Editorial inference: TriBERT's sentence-level masking is a general idea that could extend to any multi-sentence document paired with images, such as pathology reports or longitudinal imaging, and to sentence-level grounding of findings in regions.","Editorial inference: scaling these objectives to seven-digit pair counts, with curriculum learning or distillation, is a natural next test; the current 38,875-pair results do not bound what the same objectives could do at larger scale."],"forward_implications":["Pre-training 3D medical vision-language models no longer requires millions of paired scans; a few tens of thousands of clean pairs can be enough if the learning objectives exploit volumetric structure.","Text encoders for radiology should represent sentence structure explicitly, since whole-report BERT-style encoding misses the sentence-level dependencies that hierarchical contrastive learning uses.","Adding uni-modal self-supervision to cross-modal pre-training is expected to help downstream segmentation and retrieval more than cross-modal alignment alone.","A modest curated dataset can outperform a larger automatically collected one, so filtering protocols such as z-axis order and slice count deserve to be a standard part of volumetric medical vision-language pre-training pipelines.","If the framework is modality-agnostic as claimed, the same recipe should work for MRI or ultrasound volumes paired with reports."],"supporting_citations":[{"why":"Supplies the M3D-CAP dataset, the M3D retrieval model, the M3D-VQA benchmark, and the instruction set used for baselines and evaluation.","marker":"[15]"},{"why":"Provides the CLIP-style top-level contrastive objective that the paper extends into hierarchical multi-level alignment.","marker":"[5]"},{"why":"Supplies the align-before-fuse multi-modal encoder design and the hard-negative image-text matching objective used in multi-modal supervision.","marker":"[16]"},{"why":"Supplies the masked language modeling framework and attention masking conventions that TriBERT modifies with sentence-level tokens and masks.","marker":"[17]"},{"why":"Supplies the Swin Transformer vision backbone whose hierarchical feature maps feed the three-level contrastive alignment.","marker":"[45]"},{"why":"Supplies SwinUNETR, the segmentation model whose vision encoder is initialized with VELVET-Med pre-trained weights.","marker":"[54]"},{"why":"Supplies the 7-billion-parameter medical language model used as the decoder for generative visual question answering and report generation.","marker":"[62]"},{"why":"Supplies a 2D biomedical vision-language baseline for cross-modal retrieval comparisons.","marker":"[67]"}],"fun_headline_variants":["VELVET-Med: Hierarchical alignment makes 38K CT pairs beat bigger models","Hierarchical contrastive pretraining: 38K CT-report pairs top larger models","38K scans, four tasks: VELVET-Med's hierarchical pretraining wins","Self-supervision plus hierarchy: VELVET-Med needs only 38K pairs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire reported advantage rests on the assumption that the manually filtered M3D-CAP-filtered set of 38,875 scan-report pairs is a fair representation of real medical CT data, rather than an easy subset created by excluding scans the authors judged atypical.","fun_headline_variants_meta":{"raw":{"variants":["VELVET-Med: Hierarchical alignment makes 38K CT pairs beat bigger models","Hierarchical contrastive pretraining: 38K CT-report pairs top larger models","38K scans, four tasks: VELVET-Med's hierarchical pretraining wins","Self-supervision plus hierarchy: VELVET-Med needs only 38K pairs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00062,"raw_usage":{"total_tokens":2910,"prompt_tokens":1013,"completion_tokens":1897,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":629,"completion_tokens_details":{"reasoning_tokens":1806}},"tokens_in":629,"tokens_out":1897,"duration_ms":12276,"temperature":1.0,"reasoning_tokens":1806,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:24:30.175364+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train VELVET-Med with identical hyperparameters on the unfiltered M3D-CAP set (120,092 pairs) and on M3D-CAP-filtered (38,875 pairs), then compare retrieval and segmentation; if the filtered model does not clearly beat the unfiltered one, the gains are curation rather than the architecture or objectives. A second check is to have independent radiologists re-score the excluded scans: if many excluded volumes have clinically valid anatomy, the filter is introducing selection bias.","supporting_citations":[{"cited_title":"https://huggingface.co/ContactDoctor/Bio- Medical-Llama-3-8B, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the 7-billion-parameter medical language model used as the decoder for generative visual question answering and report generation."},{"cited_title":"Pmc- clip: Contrastive language-image pre-training using biomedical documents","cited_arxiv_id":null,"evidence_quote":"Supplies a 2D biomedical vision-language baseline for cross-modal retrieval comparisons."}],"review_version":1}