REVIEW 5 major objections 8 minor 16 references
MedBLIP: Fine-tuning BLIP for Medical Image Captioning
T0 review · 5 major / 8 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Fine-tuning the BLIP model on radiology captions lifts measured scores and attention, yet every example still fails clinical review.
desk verdict A routine fine-tuning study whose own clinical evaluation table undercuts the abstract's qualitative-improvement claim; the honest negative result is the only part worth keeping. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing component is BLIP itself, a vision-language architecture with a ViT-B/16 image encoder and a BERT-style text decoder. The paper fine-tunes this pair on radiology image-caption pairs with cross-entropy loss and teacher forcing, then reads decoder cross-attention from the final layer—averaged over heads and overlaid on the image—as a window into visual grounding. The ablation (encoder-only, decoder-only, full) uses the same machinery to attribute the gains to language-side versus vision-side adaptation.
What would settle it
A reader could take the four cases in the paper's clinical-evaluation table, expand them to 100 randomly selected held-out images, and have two blinded radiologists score each fine-tuned caption for whether it correctly states modality, laterality, primary finding, and anatomical location. If fine-tuned captions do not receive significantly more 'correct' verdicts than the zero-shot baseline, the qualitative-improvement claim is falsified; the table already shows this is a live possibility.
Extended reading notes
Core claim
The central claim, stated the way a sympathetic reader would take it, is that domain-specific full fine-tuning of BLIP on ROCO is the most effective of the tested routes to medical captioning: it achieves the best measured CIDEr, SPICE, and BERTScore while making decoder attention more localized, and decoder-only fine-tuning offers a competitive economy. The paper's own clinical-correctness table, however, marks every fine-tuned output as Incorrect, Misleading, Poor, or Incomplete, so the authors conclude that improved metrics and attention do not guarantee clinically accurate captions. Thus the discovery is double: adaptation helps alignment, and alignment is not enough.
Load-bearing premise
The load-bearing assumption is that ROCO's figure captions and the four reported scores measure what matters for medical captions; the paper's own clinical review shows that all fine-tuned outputs still carry incorrect, misleading, poor, or incomplete medical statements.
Editorial extensions
If this is right
- If the paper's central claim holds, full fine-tuning of BLIP on ROCO gives the best measured captioning scores in this comparison, with CIDEr rising from 0.0294 to 0.0917 and SPICE from 0.0171 to 0.0409.
- Decoder-only fine-tuning is a competitive cheaper baseline, offering most of the measured benefit while updating fewer parameters and, as the paper reports, a roughly 5% training-time saving in at least one configuration.
- Encoder-only fine-tuning scores below the other strategies, indicating that adapting the decoder's language model matters more than adapting the visual encoder alone.
- Standard overlap and embedding metrics can improve while every generated example remains clinically unacceptable, so medical deployment needs clinical verification rather than metric-based acceptance.
Reading between the lines
- A blinded clinician preference test on a larger held-out set would directly test whether the metric gains correspond to usable captions; the paper's clinical table suggests fine-tuned captions would often still be rejected.
- The paper's 5% training-time claim is attributed to decoder-only fine-tuning in the abstract but to encoder-only training in the results; reconciling this would clarify which configuration actually saves time.
- Attention localization could be repurposed as a human-in-the-loop verification aid: the paper shows focused attention on the correct region can accompany a hallucinated finding, so attention maps should flag, not certify, clinical statements.
- One could build a composite failure score from the paper's four clinical dimensions—modality, laterality, finding, anatomical specificity—to penalize hallucinations and missed findings in a way CIDEr and BERTScore do not.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents MedBLIP, a project that fine-tunes the BLIP image-captioning model on the ROCO radiology dataset and compares it against zero-shot BLIP, BLIP-2, BLIP-2 Instruct, Gemini 1.5 Flash, and ViT-GPT2. The authors report quantitative gains for fine-tuned BLIP on CIDEr, SPICE, BERTScore, and cosine similarity, show cross-attention visualizations, and conduct an ablation of full, encoder-only, and decoder-only fine-tuning. The paper claims that domain-specific fine-tuning improves both quantitative and qualitative performance, while also acknowledging in the text that fine-tuned captions can hallucinate findings or miss key diagnoses.
Significance. If the quantitative results were accompanied by proper statistical support, the paper would provide a modest confirmation that fine-tuning a general vision-language model on radiology captions improves lexical and semantic similarity to reference captions. The attention visualizations and the explicit clinical failure analysis are useful as cautionary evidence that standard captioning metrics do not imply clinical correctness. The paper does not ship code, specify the held-out evaluation size, or report any variance across runs, and its central positive claim is contradicted by its own clinical table: all four fine-tuned examples are graded clinically incorrect, misleading, or incomplete. The negative result is valuable, but the paper as written overstates the benefits of fine-tuning.
major comments (5)
- [Abstract and §4.3, Table 2] The abstract's claim that fine-tuning significantly improves performance across both quantitative and qualitative evaluation metrics is contradicted by the paper's own clinical evaluation. Table 2 grades the four fine-tuned captions as Incorrect (chest X-ray), Misleading (brain MRI), Misleading (knee X-ray), and Incomplete (abdominal US), and none is clinically correct. The paper itself states that "improved metrics do not guarantee clinical correctness." The qualitative claim should be narrowed to lexical and semantic similarity to ROCO references, with the persistent clinical failures explicitly acknowledged in the abstract and conclusion.
- [Abstract and §4.5, Table 4] The efficiency claim is misattributed. The abstract says "decoder-only fine-tuning (encoder-frozen) offers a strong performance baseline with 5% lower training time than full fine-tuning," but §4.5 reports encoder-only training at 4h59m versus 5h15m for full fine-tuning, which is the 5% saving. No decoder-only training time is reported. Moreover, Table 4 shows decoder-only fine-tuning has the lowest CIDEr (0.0664), SPICE (0.0275), and BERTScore F1 (0.7228) among the three configurations, so calling it a strong performance baseline is unsupported. The abstract should refer to encoder-only fine-tuning, or the authors should report decoder-only timing and justify the claim.
- [§4.1 and Table 1] The quantitative evaluation lacks essential experimental details and statistical rigor. The size and composition of the "held-out validation subset" are not specified, metrics are reported from a single run without standard deviations, and no significance tests are provided. Given that the absolute scores are low (CIDEr 0.0917, SPICE 0.0409) and the gaps to some baselines are small, the claim that fine-tuned BLIP "consistently outperformed" all baselines cannot be assessed as robust. The authors should report the number of test images, the distribution of modalities, and at least mean±std over multiple seeds.
- [§3.5 and Table 2] The clinical evaluation is based on only four hand-picked examples with no selection criteria, no inter-rater reliability, and no systematic scoring rubric. This is anecdotal evidence, not a qualitative evaluation metric. It cannot support the abstract's claim of improved qualitative performance. A larger, randomly sampled or independently reviewed clinical assessment would be needed to make any statement about clinical correctness; in its absence, the authors should explicitly state that clinical correctness remains an open failure.
- [§3.1] The use of ROCO figure captions as ground truth is a strong assumption that is not discussed. These captions are extracted from open-access publications and may be descriptive figure captions rather than verified clinical reports. While this does not create circularity because all models are scored against the same references, it limits the clinical validity of the reported quantitative improvements and should be stated as a caveat in the evaluation section.
minor comments (8)
- [Abstract and §5] The GitHub link "github.com/Med Img Captioning" contains a space and is not a usable URL; it should be either corrected or removed.
- [§3.3] The training setup says "1–3 epochs" with early stopping, but the actual epoch count is never reported; please specify the epoch at which training stopped for each configuration.
- [§3.5] There are typographical issues in the metric names, including "BER TScore" and "Hugging Face.For"; these should be fixed.
- [§4.1] The decoding parameters (beam size = 4, max length = 128) are given but not justified, and it is unclear how many reference captions are used per image in the ROCO subset; please clarify.
- [References] Several references are incomplete or suspicious: [1] is listed as "Journal of Important Results," [9] uses implausibly round page numbers, and [7] lacks venue information; these need correction.
- [Figure 1] The figure caption says "Each method generates different interpretations" but does not specify which images are shown or how the attention maps are aggregated; please add this information.
- [Title and §2.3] The model name "MedBLIP" is identical to the title of reference [2], an existing medical vision-language model; this naming collision is confusing and should be clarified or renamed.
- [Table 1] Gemini 1.5 Flash is reported with CIDEr 0.0 and SPICE 0.0; this is suspicious and likely due to output format mismatch, and it should be explained or excluded from the comparison.
Circularity Check
No significant circularity: the evaluation uses external ROCO references and off-the-shelf metrics, and no fitted parameter is fed back into the conclusion.
full rationale
The paper's central empirical move is fine-tuning BLIP on ROCO and comparing it against zero-shot baselines on a held-out ROCO validation subset using CIDEr, SPICE, BERTScore, and cosine similarity. These metrics compare generated captions to external ROCO reference captions; no metric value is defined in terms of the model's own training objective or fitted parameters. The fine-tuned model is not scored by a quantity it was optimized to maximize in a way that forces the reported ranking. The ablation study compares full fine-tuning, decoder-only, and encoder-only; each is a standard training configuration, and the outcome is empirical. There is no uniqueness theorem imported from prior work, no ansatz smuggled via citation, and no renamed empirical pattern presented as a derivation. The main defect in the paper is the tension between the abstract's claim of qualitative improvement and Table 2's clinical scoring, in which all four fine-tuned captions are graded Incorrect, Misleading, Misleading, or Incomplete; the paper itself concedes that 'improved metrics do not guarantee clinical correctness.' That is a validity or correctness concern about the strength of the claim, not a circularity concern. The derivation chain is self-contained against external benchmarks, so the circularity score is 0.
Assumptions & free parameters
free parameters (3)
- learning rate =
5e-5
- number of epochs =
1 to 3 with early stopping
- beam size =
4
assumptions (2)
- domain assumption ROCO figure captions are treated as correct clinical ground truth for scoring.
- domain assumption The hand-picked validation subset is representative of radiology captioning difficulty.
Cite this review
Pith. "Pith review of MedBLIP: Fine-tuning BLIP for Medical Image Captioning." pith.science (2026). https://pith.science/paper/ZQNMXKTG
@misc{pith2026250514726,
author = {Pith},
title = {Pith review of: MedBLIP: Fine-tuning BLIP for Medical Image Captioning},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZQNMXKTG}},
note = {Machine review of arXiv:2505.14726}
}
read the original abstract
Medical image captioning is a challenging task that requires generating clinically accurate and semantically meaningful descriptions of radiology images. While recent vision-language models (VLMs) such as BLIP, BLIP2, Gemini and ViT-GPT2 show strong performance on natural image datasets, they often produce generic or imprecise captions when applied to specialized medical domains. In this project, we explore the effectiveness of fine-tuning the BLIP model on the ROCO dataset for improved radiology captioning. We compare the fine-tuned BLIP against its zero-shot version, BLIP-2 base, BLIP-2 Instruct and a ViT-GPT2 transformer baseline. Our results demonstrate that domain-specific fine-tuning on BLIP significantly improves performance across both quantitative and qualitative evaluation metrics. We also visualize decoder cross-attention maps to assess interpretability and conduct an ablation study to evaluate the contributions of encoder-only and decoder-only fine-tuning. Our findings highlight the importance of targeted adaptation for medical applications and suggest that decoder-only fine-tuning (encoder-frozen) offers a strong performance baseline with 5% lower training time than full fine-tuning, while full model fine-tuning still yields the best results overall.
Figures
Reference graph
Works this paper leans on
-
[1]
ImageCLEF 2024. Imageclef medical caption. Jour- nal of Important Results , 2024
work page 2024
-
[2]
Medblip: Bootstrapping language-image pre-training from 3d medical images and texts
Qiuhui Chen et al. Medblip: Bootstrapping language-image pre-training from 3d medical images and texts. arXiv preprint arXiv:2305.10799 , 2023
arXiv 2023
-
[3]
Rocov2: Radiology objects in context version 2, an updated multimodal image dataset
R¨ uckert et.al. Rocov2: Radiology objects in context version 2, an updated multimodal image dataset. arXiv preprint arXiv:2405.10004 , 2024
arXiv 2024
-
[4]
Vuk B. et.al. Radiology objects in context (roco): A multimodal image dataset. https://github.com/ razorx89/roco-dataset, 2018. Accessed: 2025-05- 06. 7
work page 2018
-
[5]
Visual cluster grounding for im- age captioning
Wenhui Jiang et al. Visual cluster grounding for im- age captioning. IEEE Transactions on Image Pro- cessing, 31:1234–1245, 2022
work page 2022
-
[6]
Junnan Li et al. Blip: Bootstrapping language-image pre-training for unified vision-language understand- ing and generation. In Proceedings of the 39th In- ternational Conference on Machine Learning , pages 12888–12900. PMLR, 2022
work page 2022
-
[7]
Junnan Li et al. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models, 2023
work page 2023
-
[8]
A systematic review of deep learning-based research on radiology report gener- ation
Chang Liu et al. A systematic review of deep learning-based research on radiology report gener- ation. arXiv preprint arXiv:2311.14199 , 2023
arXiv 2023
Show all 16 references
-
[9]
Image caption generation using vision transformer and gpt architecture
Swapneel Mishra et al. Image caption generation using vision transformer and gpt architecture. IEEE Access, 11:123456–123467, 2023
2023
-
[10]
Uit-darkcow team at image- clefmedical caption 2024: Diagnostic captioning for radiology images efficiency with transformer models, 2024
Quan Van Nguyen et al. Uit-darkcow team at image- clefmedical caption 2024: Diagnostic captioning for radiology images efficiency with transformer models, 2024
2024
-
[11]
Learning transferable visual models from natural language supervision
Alec Radford et al. Learning transferable visual models from natural language supervision. In Pro- ceedings of the 38th International Conference on Ma- chine Learning, pages 8748–8763. PMLR, 2021
2021
-
[12]
Overview of Image- CLEFmedical 2024 – Caption Prediction and Con- cept Detection
Johannes R¨ uckert et al. Overview of Image- CLEFmedical 2024 – Caption Prediction and Con- cept Detection. In CLEF2024 Working Notes , CEUR Workshop Proceedings, Grenoble, France,
2024
-
[13]
Medical image captioning with blip2 and opt-6.7b
Shakkak. Medical image captioning with blip2 and opt-6.7b. https://github.com/Shakkak/ Medical-Image-Captioning-blip2-opt-6.7b . Ac- cessed: 2025-05-06
2025
-
[14]
Iu chest x-ray collection
Indiana University. Iu chest x-ray collection. https: //openi.nlm.nih.gov/faq#collection. Accessed: 2025-05-06
2025
-
[15]
Show and tell: A neural image caption generator
Oriol Vinyals et al. Show and tell: A neural image caption generator. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition (CVPR), pages 3156–3164. IEEE, 2015
2015
-
[16]
Show, attend and tell: Neural image caption generation with visual attention
Kelvin Xu et al. Show, attend and tell: Neural image caption generation with visual attention. In Proceed- ings of the 32nd International Conference on Ma- chine Learning, pages 2048–2057. PMLR, 2015. 8
2015
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.