{"id":"a644d0cc-b3a9-4315-84a0-e8613927a29e","arxiv_id":"2505.05804","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"MedDAM, a prompt-tuned DAM applied to medical images, reports 63.6% on the authors' MedDLC attribute-verification benchmark versus 50.2% for GPT-4o.","lead":"This paper adapts the Describe Anything Model (DAM) to medical imaging, adding expert-designed prompts for chest X-ray, lung CT, and skin lesions, and introduces a reference-free benchmark, MedDLC-score, that checks captions with binary attribute questions. MedDAM outscores GPT-4o, Claude 3.7 Sonnet, LLaMA-3.2 Vision, Qwen2.5-VL, GPT-4Rol, and OMG-LLaVA on that benchmark across the three datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MedDLC-score may reward prompt-vocabulary matching rather than clinical factuality, so the reported superiority of MedDAM may be a prompting artifact.","rationale":"The reader's weakest assumption already identifies the author-constructed MedDLC-score as the load-bearing pillar, including the possibility of bias toward MedDAM's clinical phrasing. My stress-test sharpens this into a concrete, testable confound: the QA templates are keyed to specific attribute vocabulary, and MedDAM's prompt explicitly coaches that vocabulary, while baselines receive an unspecified, likely generic prompt. This makes the headline result vulnerable to a construct-validity failure: the metric may reward term-frequency matching, not clinical factuality. This is a genuine soft spot because the paper's central contribution is the evaluation protocol, and the claimed advantage of MedDAM over GPT-4o and Claude 3.7 Sonnet is the main evidence for its value. A simple control experiment—giving baselines the same MedDAM prompt—would resolve the ambiguity. I do not think this concern warrants changing the reader's conditional verdict: the paper is preliminary work, and the requested condition (controlled prompt comparison and metric validation) is exactly what would restore confidence. Therefore the verdict remains unchanged, with the condition stated more precisely.","tokens_in":9685,"tokens_out":5695,"duration_ms":55708,"concrete_test":"Re-run the evaluation with GPT-4o and Claude 3.7 Sonnet using the exact MedDAM-prompt template (Fig. 3) instead of a generic region-caption prompt, keeping all region crops, sampling, and evaluation settings identical. If the MedDLC-score gap between MedDAM and these baselines narrows by more than half (e.g., from roughly 13 points to below 5 points), the original advantage is largely attributable to prompt design rather than medical reasoning capability.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—MedDAM's 13.4-point MedDLC-score advantage in Table 4 demonstrates clinically grounded localized captioning—rests entirely on the author-constructed MedDLC-score. The most load-bearing gap is that this metric may measure prompt-vocabulary alignment instead of clinical factuality. The positive QA templates (Sec. 3.3, Table 3) ask whether the generated description 'mentions' a specific attribute term (e.g., 'spiculated nodule margin'), and the negative templates ask whether a specific false term is 'incorrectly mentioned'. The LLM judge checks lexical mention, not semantic equivalence. Meanwhile, MedDAM-prompt (Sec. 3.5, Fig. 3) explicitly instructs the model to use anatomically precise terminology and follow professional report style, and its output format likely enumerates the same attribute dimensions used in the QA templates. Baselines are described as 'prompted to generate region-specific descriptions' (Sec. 3.4), but the paper does not state that they receive the same specialized vocabulary prompt. A clinically correct baseline caption phrased in natural language (e.g., 'irregular border') can fail a question keyed to 'spiculated margin', while MedDAM passes by lexical matching. The use of GPT-4o as judge (Sec. 3.4) may compound this bias, and Table 4 reports only dataset-averaged scores without per-dataset breakdown or error bars, leaving open the possibility that the advantage is driven by one domain. If confirmed, the reported superiority is a prompting artifact, not evidence of stronger medical understanding.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MedDAM, a framework that adapts the Describe Anything Model (DAM) to medical images by combining expert-designed, modality-specific prompts with a region-of-interest preprocessing pipeline, and proposes MedDLC-score, a reference-free evaluation protocol based on attribute-level binary QA tasks. The method is evaluated on VinDr-CXR, LIDC-IDRI, and SkinCon against several general-purpose and region-specific VLMs. The authors report that MedDAM achieves the highest MedDLC-score (63.6%) versus GPT-4o (50.2%) and Claude 3.7 Sonnet (47.5%), and interpret this as evidence that region-specific prompting and DAM adaptation yield clinically grounded localized captioning.","tokens_in":10014,"tokens_out":4164,"duration_ms":41162,"significance":"If the evaluation protocol is valid, the paper would constitute a useful first step toward region-specific medical captioning without ground-truth region-text pairs, and its benchmark could benefit future work on medical vision-language models. The three datasets cover diverse modalities, and the use of attribute-level verification to bypass caption-level annotation is an interesting and timely idea. However, the central claim of superiority rests entirely on the author-constructed MedDLC-score, which is not validated against human expert judgment and is likely confounded with prompt vocabulary. Because the main evidence is a single table with no per-dataset breakdown or statistical testing, the current support for the paper's headline claim is insufficient.","major_comments":[{"comment":"The MedDLC-score may reward lexical mention rather than clinical semantic equivalence. The positive QA examples ask whether the description 'mentions' a specific attribute term (e.g., 'spiculated nodule margin'), and the negative examples ask whether a specific false term is 'incorrectly mentioned.' Meanwhile, the MedDAM-prompt (Section 3.5) explicitly instructs the model to use anatomically precise terminology and a professional clinical report style, while the baselines are described only as being 'prompted to generate region-specific descriptions' (Section 3.4). The paper does not state whether the baselines receive the same specialized vocabulary prompt. A clinically correct baseline caption phrased in natural language (e.g., 'irregular border with radiating strands') could fail a question keyed to 'spiculated margin,' while MedDAM passes by lexical overlap. To support the claim that MedDAM is more clinically factual, the authors must either provide the exact baseline prompts, demonstrate that the LLM judge performs semantic equivalence matching rather than keyword matching (e.g., by validating the judge against human experts on rephrased answers), or control for vocabulary by giving all models the same terminology instructions. As it stands, the 13.4-point advantage in Table 4 may be a prompting artifact.","section":"Section 3.3 (SkinCon)"},{"comment":"The SkinCon evaluation lacks an independent ground-truth anchor. The paper states that 'since the original dataset lacks region-level annotations, we generate bounding boxes ourselves using a lightweight lesion detection method' and that 'based on these detected regions, we construct verification questions.' It is not explained how the correct positive/negative answers for these detected boxes are obtained, nor whether the image-level SkinCon concepts are propagated to the detected regions. Without a documented source of ground-truth attribute labels for these regions, the SkinCon MedDLC-score does not measure factuality against the dataset; it only measures agreement with whatever labels the authors assigned during question construction. This is a circular step in the evaluation pipeline and needs to be resolved and reported transparently.","section":"Section 3.4, Table 4"},{"comment":"Table 4 reports only dataset-averaged scores with no per-dataset breakdown, no number of evaluated regions or questions per dataset, no error bars or confidence intervals, and no significance tests. Given that the three datasets have different modalities, different region sampling procedures (10% margin for VinDr-CXR, no margin for LIDC-IDRI, 15% padding for SkinCon), and different question construction methods, it is possible that MedDAM's advantage is concentrated in one domain. The paper should report per-dataset results for LLM-score, MedDLC-score, positive QA, and negative QA, along with uncertainty estimates and a paired significance test across regions where appropriate.","section":"Section 3.6"},{"comment":"The use of GPT-4o as the LLM judge for MedDLC-score is problematic because GPT-4o is also one of the baseline models. The paper acknowledges this bias for the LLM-score but does not address it for MedDLC-score. Since the question templates are author-designed and the judge is also a baseline model, the evaluation instrument may systematically favor outputs that resemble the vocabulary used in the questions. At minimum, the authors should report inter-judge agreement with an alternative judge, compare MedDLC-score against clinician ratings on a held-out sample, or replace the GPT-4o judge with an independent judge or a committee.","section":"Section 3.6"}],"minor_comments":[{"comment":"Table 1 lists the SkinCon sample size as 3,000 images, while Section 3.1 states 3,230 images; the discrepancy should be reconciled.","section":"Table 1 vs. Section 3.1"},{"comment":"There are typographical inconsistencies in model names: 'OMG-LLaV A' in the abstract and Table 4, and 'GPT-4Rol' without a proper citation; these should be unified and referenced consistently.","section":"Throughout"},{"comment":"Figure 2(d) refers to 'ground-truth semantic attributes,' but the evaluation is explicitly reference-free; rewording to 'annotated attributes' or 'expected attributes' would avoid confusion.","section":"Figure 2"},{"comment":"The paper states that MedDAM-prompt was designed by medical experts, but no details are provided about the experts, the design process, or any validation of the prompt; a brief description would strengthen the claim.","section":"Section 3.5"},{"comment":"The paper does not specify how many regions and questions are evaluated per dataset, or how the QA templates are instantiated per sample; these details are needed for reproducibility.","section":"Section 3.3 / Section 3.4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is preliminary work, and the central contribution is essentially a prompt-engineering and evaluation protocol built on the existing DAM model. The main weakness is that the evaluation metric is author-constructed, not independently validated, and appears confounded with the specialized vocabulary injected by the MedDAM prompt. This is correctable if the authors supply controlled baseline prompts, human validation of the judge, per-dataset statistics, and a resolution of the SkinCon ground-truth issue. I recommend major revision rather than rejection because the underlying idea is plausible and the benchmark could become a useful community resource after these fixes."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The paper is a genuinely useful first application of the Describe Anything Model to medical region captioning, and the MedDLC-score protocol is a reasonable starting point for reference-free evaluation in a domain where region captions don't exist. The question grounding comes from external dataset annotations, so it's not circular in the worst sense. What's not established is the headline result: MedDAM's 63.6% vs GPT-4o's 50.2%. The metric may be rewarding prompt-vocabulary matching rather than clinical factuality. MedDAM-prompt explicitly tells the model to use anatomically precise terminology and follow professional report style; the positive QA templates ask whether the description 'mentions' terms like 'spiculated nodule margin'; the baselines get a generic region-description prompt. A clinically correct baseline that says 'irregular border' could fail a question keyed to 'spiculated margin.' That's a plausible artifact, and the paper doesn't rule it out. Table 4 is also just dataset-averaged scores with no error bars, per-dataset breakdown, or significance tests, and the judge family (GPT-4o) overlaps with a baseline. These are fixable: validate MedDLC-score against human expert judgments, run baselines with the same specialized vocabulary prompt, and report per-dataset numbers. The authors are honest about the LLM-score self-judge bias and label the work preliminary, which helps. In short: the application is real, the evaluation idea is worth discussing, but the superiority claim doesn't hold up as stated. It deserves a serious referee—this is exactly the kind of paper where open review can push the authors to tighten the evaluation. I'd bring it to reading group and would cite the benchmark protocol, but I wouldn't cite the 13-point gap as evidence.","headline":"A useful first application of DAM to medical region captioning, but the headline superiority claim rests on a metric that may reward prompt-vocabulary matching; worth refereeing, not trusting as-is.","tokens_in":10519,"tokens_out":3186,"would_cite":true,"duration_ms":31216,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a prompt-adapted localized captioning model, MedDAM, beats GPT-4o on clinically factual region-specific medical captions, as measured by a new attribute-level benchmark.","keywords":["MedDAM","region-specific captioning","medical image understanding","vision-language models","Describe Anything Model","attribute-level verification","zero-shot evaluation","clinical factuality"],"falsifier":"Select a random sample of regions from the three datasets, ask a panel of radiologists and dermatologists to judge, from the images alone, which categorical attributes are present, then compare MedDAM's and GPT-4o's captions against that human ground truth. If GPT-4o equals or beats MedDAM on human-adjudicated attribute accuracy, the MedDLC-score ranking does not reflect true clinical factuality. A simpler check is to recompute MedDLC-score with a different judge model, such as Claude 3.7 Sonnet instead of GPT-4o, and see whether MedDAM's 13-point lead persists.","tokens_in":9507,"feed_emoji":"🩻","tokens_out":7182,"duration_ms":64423,"temperature":0.7,"pith_summary":"The paper claims that a general localized-captioning model designed for natural images, the Describe Anything Model (DAM), can be adapted to medical imaging by pairing it with expert-written clinical prompts and a region-of-interest extraction pipeline, yielding a system the authors call MedDAM. It then proposes a reference-free evaluation benchmark, MedDLC-score, that uses positive and negative yes/no questions about clinically relevant attributes, with GPT-4o as judge, to measure factual accuracy without ground-truth region captions. On three datasets spanning chest X-rays, lung CTs, and skin photographs, MedDAM scores 63.6% on this benchmark, ahead of GPT-4o at 50.2% and Claude 3.7 Sonnet at 47.5%. A sympathetic reader would take the paper's claim to be that region-level semantic alignment, obtained by prompt-adapting a localized captioning model, is the practical route to clinically grounded region description in medical images.","feed_headline":"Medical region captions: new model tops GPT-4o on clinical facts","feed_subtitle":"With expert prompts and region crops, MedDAM reaches 63.6% attribute accuracy versus GPT-4o's 50.2%.","key_machinery":"The argument rests on three coupled components. The first is DAM's focal prompting mechanism, which combines a high-resolution crop of the target region with the full image via gated cross-attention, letting the model attend to local detail while keeping global context. The second is the MedDAM-prompt, a structured template that enforces anatomical focus, clinical terminology, report style, and grounding, applied identically to all baselines for fair comparison. The third is the MedDLC-score evaluation: for each dataset, a QA template library of positive and negative attribute questions is built from the dataset's own annotations, a GPT-4o judge reads the generated caption and answers each question, and the model's score is the average accuracy across the three datasets. This attribute-verification design is what allows the paper to measure clinical factuality without any ground-truth region captions.","core_discovery":"MedDAM is a framework that takes the Describe Anything Model — a vision-language model designed for localized captioning of natural images, which uses focal prompting with a high-resolution region crop alongside the global image — and adapts it to medical imaging. The adaptation consists of expert-designed text prompts that instruct the model to focus only on the marked region, use precise anatomical terminology, and write in a clinical report style, plus a region-of-interest pipeline that obtains bounding boxes from existing annotations or from a segmentation-based detector when annotations are missing. For evaluation, the paper introduces MedDLC-score, a reference-free protocol that builds a library of binary yes/no questions about clinically relevant attributes per dataset (radiological findings, nodule morphology, dermatological lesion characteristics), asks a judge model whether the generated caption supports each attribute, and reports accuracy on positive and negative questions. On this benchmark, MedDAM scores 63.6% overall, ahead of GPT-4o at 50.2%, Claude 3.7 Sonnet at 47.5%, and other general and region-specific vision-language models, and it leads on both positive questions (65.1%) and negative questions (62.0%), which the paper interprets as evidence that region-level semantic alignment improves medical image understanding and reduces hallucinated findings. The paper's claim is therefore that region-specific captioning of medical images is achievable zero-shot, and that the MedDAM prompt-plus-benchmark recipe is what makes the difference.","pith_inferences":["The same recipe of focal prompting plus attribute-level QA should transfer to other modalities the paper lists as future work, such as pathology and ophthalmology, where region-level findings are similarly decisive.","If the benchmark is trusted, the 13-point gap implies that region-level semantic alignment, not just global image understanding, is the main bottleneck for clinical vision-language models, and that general models could narrow it with prompting rather than new pretraining.","The MedDLC-score design has a hidden calibration requirement: without human-expert validation of the QA templates, its absolute numbers are hard to interpret, so an expert-judgment study would determine whether 63.6% versus 50.2% is clinically meaningful."],"forward_implications":["Region-specific captioning for medical images can be performed zero-shot with an off-the-shelf natural-image model, provided the region can be boxed and prompts are medically phrased.","The attribute-level QA protocol supplies a way to measure and audit medical captioning where ground-truth region captions are absent, which is the normal situation in clinical data.","General large vision-language models score markedly lower on negative questions, so the benchmark exposes a systematic tendency of such models to fabricate localized findings.","Because MedDAM trails GPT-4o on LLM-score (78.9 vs 81.5), linguistic fluency and clinical factuality are separable; the paper argues the gap partly reflects GPT-4o judging itself."],"supporting_citations":[{"why":"Supplies DAM, the localized captioning backbone, and DLC-Bench, the evaluation idea that MedDAM adapts.","marker":"Lian et al., 2025"},{"why":"Provides VinDr-CXR bounding boxes for thoracic abnormalities used to prompt and evaluate regions.","marker":"Nguyen et al., 2022"},{"why":"Supplies LIDC-IDRI nodule masks and semantic attributes that ground the positive and negative QA questions.","marker":"Armato III et al., 2011"},{"why":"Provides SkinCon dense dermatology concept annotations used to construct attribute questions and regions.","marker":"Daneshjou et al., 2022"},{"why":"GPT-4o is both a leading baseline and the LLM judge in the LLM-score and MedDLC-score protocols.","marker":"OpenAI, 2024"},{"why":"Offers the lesion-detection ROI approach adapted to generate bounding boxes for SkinCon where none exist.","marker":"Wang et al., 2024a"}],"fun_headline_variants":["MedDAM: region captions that beat GPT-4o on clinical facts","New model pinpoints medical findings with region-specific captions","Region-focused captions: MedDAM tops GPT-4o in clinical accuracy","Describe anything in medicine? MedDAM nails region details","MedDAM: zero-shot region captions outperform GPT-4o"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire superiority claim rests on the author-built MedDLC-score — the specific yes/no attribute questions, the GPT-4o judge, and the averaging across three datasets — being a valid measure of clinical factuality; if that benchmark favors MedDAM's phrasing or does not reflect what clinicians actually need, the ranking of MedDAM over GPT-4o is not established.","fun_headline_variants_meta":{"raw":{"variants":["MedDAM: region captions that beat GPT-4o on clinical facts","New model pinpoints medical findings with region-specific captions","Region-focused captions: MedDAM tops GPT-4o in clinical accuracy","Describe anything in medicine? MedDAM nails region details","MedDAM: zero-shot region captions outperform GPT-4o"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000725,"raw_usage":{"total_tokens":3320,"prompt_tokens":1088,"completion_tokens":2232,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":704,"completion_tokens_details":{"reasoning_tokens":2138}},"tokens_in":704,"tokens_out":2232,"duration_ms":15665,"temperature":1.0,"reasoning_tokens":2138,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:55:17.748537+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Select a random sample of regions from the three datasets, ask a panel of radiologists and dermatologists to judge, from the images alone, which categorical attributes are present, then compare MedDAM's and GPT-4o's captions against that human ground truth. If GPT-4o equals or beats MedDAM on human-adjudicated attribute accuracy, the MedDLC-score ranking does not reflect true clinical factuality. A simpler check is to recompute MedDLC-score with a different judge model, such as Claude 3.7 Sonnet instead of GPT-4o, and see whether MedDAM's 13-point lead persists.","supporting_citations":[],"review_version":1}