{"id":"918578a1-463e-48c7-8849-ce34b42ba080","arxiv_id":"2506.14136","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"BiomedCLIP zero-shot over-predicts rare chest X-ray labels on IU-Xray, while linear probing nearly matches full fine-tuning and preserves the model's spatial attention fidelity.","lead":"This paper evaluates BiomedCLIP, a large biomedical vision-language model, on a highly imbalanced chest X-ray dataset it was not trained on. It compares zero-shot inference, full fine-tuning, and linear probing, finding that zero-shot over-predicts rare diseases while linear probing nearly matches fine-tuning at lower cost.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Zero-shot 'over-predicts all labels' is threshold-dependent, yet the paper never states how similarity scores are converted to binary predictions; the headline finding may be an artifact of the unspecified decision rule.","rationale":"The reader's weakest assumption was CheXbert label noise, which is a genuine concern: noisy ground truth can bias per-class F1, especially for rare classes with 20-49 samples, and may systematically disadvantage zero-shot because supervised methods can overfit to the labeler's errors. However, the more load-bearing issue for the paper's central claim is the missing decision rule for zero-shot predictions. The abstract and Section 5.1 claim that zero-shot 'over-predicts all labels,' but without knowing the threshold used to convert cosine similarities or logits into binary predictions, this finding cannot be reproduced or even semantically evaluated. The same applies to fine-tuning and linear probing, whose BCE outputs need a threshold to produce the reported F1, exact-match, and precision values. This is a protocol gap, not a disagreement with consensus; it is directly addressable by obtaining the code or the threshold. If the threshold turns out to be arbitrary or miscalibrated, the quantitative comparison collapses. If the authors can supply a principled threshold and the results hold, the central claim survives. The Grad-CAM analysis is also selective, but it is qualitative and less central than the quantitative headline. I therefore keep the reader's CONDITIONAL verdict: the paper is potentially sound but must clarify the prediction protocol before its central claim can be trusted.","tokens_in":11853,"tokens_out":4188,"duration_ms":42815,"concrete_test":"Obtain the GitHub code or ask the authors for the exact zero-shot prediction script and threshold value; re-run zero-shot using the same embedding similarities but set the positive threshold on the validation split (e.g., maximize validation macro-F1, or use per-label thresholds). If macro-F1 and precision remain materially unchanged and predictions are not all-positive, the concern is resolved. If macro-F1 shifts substantially or the all-positive behavior disappears, then 'over-predicts all labels' is an artifact of the unspecified default threshold, and Tables 1-3 need to be re-reported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing gap is that the paper never specifies the decision rule that turns BiomedCLIP's image-text similarity scores (or logits) into the binary multi-label predictions used in Tables 1-3. Section 4.2 describes only that zero-shot yields image and text embeddings; Section 4.4 lists metrics but gives no threshold. For zero-shot, 'over-predicts all labels' is exactly a statement about where the positive/negative boundary was placed. If predictions were made by thresholding cosine similarities at, say, 0, then any threshold below the minimum similarity would mechanically predict all 14 labels for every image, guaranteeing macro-F1 near 0.105 and LRAP near 0.250 regardless of model quality. Fine-tuning and linear probing use BCE-trained heads whose output thresholds are also unspecified. Thus the central quantitative contrast (macro-F1 0.105 vs 0.235 vs 0.183, LRAP 0.250 vs 0.779 vs 0.741) and the headline qualitative claim about zero-shot over-prediction are uninterpretable without the exact decision rule. The CheXbert-label noise flagged by the reader is real, but it does not independently determine the zero-shot over-prediction result; the threshold does.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates BiomedCLIP on the IU-Xray chest radiograph dataset, which is treated as a highly imbalanced, out-of-distribution, multi-label benchmark. Labels are obtained automatically from radiology reports using CheXbert. The authors compare three adaptation regimes: zero-shot inference, full fine-tuning, and linear probing, reporting embedding-space separability, multi-label classification metrics (macro-F1, exact-match, LRAP, coverage error), and per-label/domain F1. They also present Grad-CAM visualizations compared against 15 radiologist-annotated radiographs. The central claims are that zero-shot BiomedCLIP 'over-predicts all labels' with poor precision and separability, full fine-tuning improves classification of distinct diseases, and linear probing recovers much of the fine-tuning gain at lower cost while preserving spatial fidelity.","tokens_in":12089,"tokens_out":2818,"duration_ms":29116,"significance":"If the quantitative results are taken at face value, the paper provides a useful practical message for users of biomedical VLMs: zero-shot inference on imbalanced OOD radiology data is unreliable, and linear probing is a cost-effective alternative to full fine-tuning. The authors release code and include 15 radiologist-annotated samples for qualitative comparison, which are valuable assets. The embedding-space analysis and the comparison of three adaptation regimes on a realistic imbalanced dataset address a question of genuine interest to the medical imaging community. However, the significance is currently conditional: the headline zero-shot result is threshold-dependent and the manuscript does not specify the decision rule, so the quantitative contrast could be an artifact. The CheXbert labeling and the single train/val/test split further weaken the reliability of per-class comparisons.","major_comments":[{"comment":"The paper never specifies the decision rule that converts BiomedCLIP's similarity scores or logits into the binary predictions used in Tables 1 and 2. Section 4.2 (1) states only that zero-shot yields image and text embeddings, and Section 4.3 gives implementation details without any threshold. The claim in the abstract and Section 5.1 that zero-shot 'over-predicts all labels' is precisely a statement about where the positive/negative boundary was placed; a sufficiently low threshold would mechanically predict all 14 labels for every image, producing macro-F1 near 0.105 and LRAP near 0.250 regardless of model quality. The same issue applies to the BCE-trained heads in fine-tuning and linear probing. The authors must specify the threshold for each setting, state how it was chosen (e.g., fixed at 0.5, tuned on validation, or based on a percentile), and ideally supplement the threshold-dependent metrics with threshold-independent ones such as average precision or precision-recall curves. Without this, the central quantitative contrast in Table 1 is uninterpretable.","section":"Section 4.2 and Section 5.1"},{"comment":"There is a direct numerical inconsistency in the reported exact-match accuracy for linear probing. Table 1 lists exact-match as 0.143 (14.3%), while Section 5.1 and the Conclusion state 'exact-match to 4.3%' and 'exact-match = 4.3%', respectively. Since exact-match is one of the headline metrics used to compare the three settings, the authors must correct this inconsistency and verify that all numbers in the text, tables, and abstract are mutually consistent.","section":"Table 1 vs. Section 5.1"},{"comment":"The quantitative comparison rests on a single train/val/test split (70/10/20) with no repeated runs or confidence intervals, and the ground-truth labels are produced by CheXbert rather than manual review, as the authors acknowledge in the limitations. This is particularly concerning for rare classes such as Pneumothorax (n=20) and Consolidation (n=31), where a handful of mislabeled reports could change per-class F1 values substantially. Claims such as 'linear probing generalizes better on mid-frequency diseases' (Section 5.2) are based on small per-class differences that could be within label-noise or split-induced variation. The authors should report results across multiple seeds or cross-validation folds, provide error bars, and discuss the sensitivity of the reported per-class gaps to CheXbert label noise; a patient-level split should also be used if patients contribute multiple images.","section":"Section 4.3 and Section 6"}],"minor_comments":[{"comment":"The phrase 'over-predicts all labels' is used as a definitive finding, but without the decision-threshold details it is better framed as a conditional observation; please qualify it until the threshold is specified.","section":"Abstract and Section 5.1"},{"comment":"There are several typos: 'Morever' should be 'Moreover'; 'Standford' should be 'Stanford'; 'onewarm-up' should be 'one warm-up'; 'multiperceptron' should be 'multilayer perceptron'.","section":"Introduction"},{"comment":"References [4] and [5] appear to refer to the same Boecking et al. ECCV 2022 paper and are duplicated; please merge or disambiguate.","section":"References"},{"comment":"The Grad-CAM discussion would benefit from clearer quantification of 'aligns very closely' and 'co-localized'; consider reporting a quantitative overlap metric (e.g., Dice or IoU) against the radiologist annotations rather than only qualitative descriptions.","section":"Section 5.3"},{"comment":"The dataset description should mention that IU-Xray contains frontal and lateral views and clarify whether the model is evaluated on both views or only one, since this affects the interpretation of the OOD claim.","section":"Section 3"},{"comment":"The zero-shot F1 scores in Table 2 are all positive even for rare classes; this is surprising given the claim of 'over-predicting all labels' and suggests the threshold may not be extremely low. Clarifying this would help the reader reconcile Table 2 with the headline narrative.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the venue and addresses an important practical question. The main blocker is the missing decision-threshold specification for all three settings, which makes the headline zero-shot claim uninterpretable. This is fixable by adding the threshold details, threshold-independent metrics, and possibly a small ablation over thresholds. The single-split and CheXbert-label issues are also significant but can be addressed with additional experiments and a more careful discussion. I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Diego—quick take on arXiv:2506.14136.\n\nThis is a solid, if rough, empirical paper: it evaluates BiomedCLIP on IU-Xray under zero-shot, fine-tuning, and linear probing, and reports per-domain F1, embedding distances, and layer-depth Grad-CAM. The most useful observation is that linear probing nearly closes the gap to full fine-tuning (macro-F1 0.183 vs 0.235, LRAP 0.741 vs 0.779) at less than half the training time, and sometimes beats it on mid-frequency diseases like Atelectasis (0.410 vs 0.091) and Pneumonia (0.286 vs 0.267). That's a concrete, actionable result for practitioners who can't afford fine-tuning. The embedding-distance analysis is a nice addition, and the training time comparison is refreshingly practical.\n\nThe soft spots are real. The biggest one: the paper never states how zero-shot predictions are made. BiomedCLIP gives you image and text embeddings, but to compute precision, F1, and exact-match you need a threshold on the similarity score. No threshold is given. So the headline claim that zero-shot 'over-predicts all labels' is uninterpretable—it could be an artifact of a poorly chosen cutoff. The ranking metrics (LRAP, coverage error) don't have this problem, and they also show zero-shot is much worse, so the broad conclusion probably holds. But the specific over-prediction statement needs the decision rule reported, plus a sensitivity analysis over thresholds.\n\nOther issues: labels come from CheXbert, not manual review; the authors admit this, but it's a real limitation for the rarest classes (Pneumothorax n=20). There's a single 70/10/20 split with no error bars, and no patient-level split, which matters because IU-Xray has multiple images per patient. The Grad-CAM analysis is based on 15 radiologist-annotated images and selected examples—fine for illustration, not strong evidence. And the GitHub link promised in the abstract is missing from the text.\n\nThat said, the paper is honest about its limitations, the tables are clear, and the direction is right. It deserves a serious referee, but the reviewers should ask for the exact decision rule and a patient-level split before acceptance. I'd use it as a data point on linear probing for medical VLMs, though I'd cite it with the threshold caveat.","headline":"A useful but under-specified empirical comparison: the zero-shot 'over-predicts all labels' headline is not interpretable because the decision threshold is never stated.","tokens_in":12606,"tokens_out":3121,"would_cite":true,"duration_ms":30441,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Zero-shot BiomedCLIP over-predicts every label on imbalanced chest X-rays, while linear probing recovers most fine-tuning gains at a fraction of the training cost.","keywords":["BiomedCLIP","vision-language models","zero-shot classification","linear probing","full fine-tuning","class imbalance","out-of-distribution","Grad-CAM"],"falsifier":"Recompute the per-class F1 scores and LRAP on a test set where a radiologist manually labels every image, especially Pneumothorax and Consolidation; if fine-tuning then matches or beats linear probing on mid-frequency classes, the paper's claim that linear probing generalizes better on scarce classes is refuted.","tokens_in":11655,"feed_emoji":"🩻","tokens_out":5121,"duration_ms":46944,"temperature":0.7,"pith_summary":"This paper asks whether a large pretrained biomedical vision-language model can classify a highly imbalanced, out-of-distribution radiology dataset, and it answers with a qualified no for zero-shot use. On IU-Xray, zero-shot BiomedCLIP fires on nearly every label, producing low precision, zero exact-match accuracy, and poor inter-class separation. Full fine-tuning reshapes the embedding space and roughly doubles macro-F1, while linear probing recovers most of that gain at less than half the training time and preserves the original spatial fidelity of the model's attention. The paper argues that adaptation strategy, not just model choice, determines whether these models are usable in real clinical settings.","feed_headline":"Zero-shot BiomedCLIP over-predicts every chest X-ray label","feed_subtitle":"On imbalanced chest X-rays, linear probing recovers most fine-tuning gains at a fraction of the training time.","key_machinery":"The argument is carried by the contrastive embedding space of BiomedCLIP and by three measurement tools applied to it. The inter-class versus intra-class Euclidean distance ratio quantifies separability, multi-label metrics (macro-F1, exact-match, LRAP, coverage error) quantify ranking and detection quality, and Grad-CAM heatmaps compare model attention against radiologist-annotated regions. Together they let the paper attribute differences in behavior to the adaptation regime rather than to the dataset or the pretrained weights alone.","core_discovery":"The paper's central claim is that BiomedCLIP's zero-shot behavior is systematically over-predictive on imbalanced out-of-distribution radiographs: it assigns labels too liberally, so macro-F1 is 0.105, exact-match accuracy is 0.0%, LRAP is 0.250, and coverage error is 7.70, and the inter/intra-class distance ratio stays at 1.506. Full fine-tuning increases the separation ratio to 1.784 and raises macro-F1 to 0.235, exact-match to 13.4%, and LRAP to 0.779, while linear probing on the frozen encoder reaches 0.183 macro-F1, 4.3% exact-match, and 0.741 LRAP at 6.10 minutes of training versus 15.47 minutes. Per-class, fine-tuning wins on abundant classes and fails on several rare ones, whereas linear probing does better on mid-frequency diseases like Pneumonia, Atelectasis, and Fracture. Grad-CAM comparisons against 15 radiologist annotations show zero-shot heatmaps tightly localized to pathology, fine-tuning heatmaps broad and non-specific, and linear probing retaining near-zero-shot spatial fidelity while adding accuracy.","pith_inferences":["The same evaluation could be run on a manually labeled out-of-distribution set to test whether the rare-class advantage of linear probing is real or an artifact of label noise.","A natural extension is to test prompt-based and adapter-based adaptations, which the paper lists as future work, against the same three heads to see whether the fine-tuning/linear-probing trade-off is specific to full weight updates.","The shallow-layer localization result suggests that clinicians may get more useful saliency maps by reading earlier blocks of the encoder, even when the final layer is used for classification.","If the zero-shot over-prediction pattern holds on other imbalanced medical datasets, it would argue for defaulting to lightweight probing before deploying any pretrained biomedical VLM in a low-data clinic."],"forward_implications":["Zero-shot inference with a biomedical VLM should not be trusted as a clinical classifier on imbalanced out-of-distribution data; its high recall comes at the cost of overwhelming false positives.","Linear probing is a viable low-cost adaptation: it recovers most of full fine-tuning's ranking performance and takes less than half the training time.","Full fine-tuning is the best choice when abundant classes matter most, but it can erase performance on scarce classes and degrade spatial interpretability.","Grad-CAM fidelity and classification accuracy do not move together; zero-shot can localize well while ranking poorly, so interpretability evaluations must be separate from accuracy evaluations."],"supporting_citations":[{"why":"Supplies the BiomedCLIP model, its pretrained image and text encoders, and the PMC-15M pretraining corpus that the paper evaluates.","marker":"[28]"},{"why":"Supplies the IU-Xray dataset, the highly imbalanced, out-of-distribution radiograph and report collection at the center of the evaluation.","marker":"[7]"},{"why":"Supplies CheXbert, the automated labeler used to generate the 14 disease-class ground-truth labels from IU-Xray reports.","marker":"[21]"},{"why":"Supplies the Grad-CAM method used to produce the saliency heatmaps compared against radiologist annotations.","marker":"[18]"},{"why":"Provides the multi-label evaluation metrics (macro-F1, LRAP, coverage error, ranking loss) that score the three adaptation settings.","marker":"[27]"},{"why":"Motivates the framing of zero-shot, fine-tuning, and linear probing as a continuum of cost and performance trade-offs.","marker":"[9]"},{"why":"Supports the expectation that linear probing can recover much of the fine-tuning performance at a fraction of the computational cost.","marker":"[17]"}],"fun_headline_variants":["Zero-shot BiomedCLIP over-predicts every chest X-ray label","BiomedCLIP zero-shot fails imbalanced X-rays; linear probing is cheap","Fine-tuning beats zero-shot on X-rays, but linear probing saves time","Linear probing recovers most fine-tuning gains on rare X-ray labels","Zero-shot BiomedCLIP over-predicts; tuning needed for imbalanced X-rays"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes the CheXbert-generated labels on IU-Xray reports are accurate enough ground truth for all 14 disease classes, even though CheXbert has roughly 80% validated accuracy and the rarest classes have only 20 to 31 samples.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot BiomedCLIP over-predicts every chest X-ray label","BiomedCLIP zero-shot fails imbalanced X-rays; linear probing is cheap","Fine-tuning beats zero-shot on X-rays, but linear probing saves time","Linear probing recovers most fine-tuning gains on rare X-ray labels","Zero-shot BiomedCLIP over-predicts; tuning needed for imbalanced X-rays"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000536,"raw_usage":{"total_tokens":2596,"prompt_tokens":990,"completion_tokens":1606,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":606,"completion_tokens_details":{"reasoning_tokens":1506}},"tokens_in":606,"tokens_out":1606,"duration_ms":11534,"temperature":1.0,"reasoning_tokens":1506,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:53:31.210198+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the per-class F1 scores and LRAP on a test set where a radiologist manually labels every image, especially Pneumothorax and Consolidation; if fine-tuning then matches or beats linear probing on mid-frequency classes, the paper's claim that linear probing generalizes better on scarce classes is refuted.","supporting_citations":[{"cited_title":"P., Nau- mann, T., W ang, S., and Poon, H.A multimodal biomedical foundation model trained from fifteen million image–text pairs.NEJM AI 2, 1 (2025), AIoa2400640","cited_arxiv_id":null,"evidence_quote":"Supplies the BiomedCLIP model, its pretrained image and text encoders, and the PMC-15M pretraining corpus that the paper evaluates."},{"cited_title":"D., Rosenman, M","cited_arxiv_id":null,"evidence_quote":"Supplies the IU-Xray dataset, the highly imbalanced, out-of-distribution radiograph and report collection at the center of the evaluation."},{"cited_title":"Y., and Lungren, M","cited_arxiv_id":null,"evidence_quote":"Supplies CheXbert, the automated labeler used to generate the 14 disease-class ground-truth labels from IU-Xray reports."},{"cited_title":"R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D.Grad-cam: Visual explanations from deep networks via gradient-based local- ization","cited_arxiv_id":null,"evidence_quote":"Supplies the Grad-CAM method used to produce the saliency heatmaps compared against radiologist annotations."},{"cited_title":"IEEE Transactions on Knowledge and Data Engineering 26, 8 (2014), 1819–1837","cited_arxiv_id":null,"evidence_quote":"Provides the multi-label evaluation metrics (macro-F1, LRAP, coverage error, ranking loss) that score the three adaptation settings."},{"cited_title":"InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2023), pp","cited_arxiv_id":null,"evidence_quote":"Motivates the framing of zero-shot, fine-tuning, and linear probing as a continuum of cost and performance trade-offs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports the expectation that linear probing can recover much of the fine-tuning performance at a fraction of the computational cost."}],"review_version":1}