Pith. sign in

REVIEW 4 major objections 6 minor 70 references

Seeing the Trees for the Forest: Rethinking Weakly-Supervised Medical Visual Grounding

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that two measurable defects in medical vision-language models—high background token norms and a global image token that ignores local disease patches—are what break weakly-supervised visual grounding, and that reweighting…

desk verdict Two sharp diagnostics about medical VLMs, a simple and plausible fix, but the evaluation leaks MS-CXR test information and the CNR numbers do not reconcile; worth referee time with required revision. read the letter →

arxiv 2505.15123 v2 pith:SHBFCN7V submitted 2025-05-21 cs.CV cs.AI

Ta Duc Huy , Duy Anh Huynh , Yutong Xie , Yuankai Qi , Qi Chen , Phi Le Nguyen , Sen Kim Tran , Son Lam Phung
show 5 more authors
Anton van den Hengel Zhibin Liao Minh-Son To Johan W. Verjans Vu Minh Hieu Phan
This is my paper · ORCID
classification cs.CVcs.AI
keywords weakly-supervisedvisualgroundingmedicalvision-languagemodeldisease-awarepromptingexplainabilitymapchestX-raycontrastivelearningtokennormanalysisglobal-localalignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that weakly-supervised visual grounding on medical images fails for two diagnosable reasons inside current vision-language models (VLMs): local image tokens in healthy background regions carry high L2 norms and so dominate attention, and the global image [IMG] token that bridges image and text is aligned with background patches rather than with the small disease patches. To fix both, the authors introduce Disease-Aware Prompting (DAP), which uses the VLM's own explainability map for a text query to reweight visual token features during training, amplifying disease regions and suppressing background. The reweighted features feed a global contrastive loss that pulls the text token toward a disease-aware global image token, a local contrastive loss that pulls foreground disease patches toward the text and pushes background patches away, and a segmentation decoder trained with the explainability map as pseudo-label. Across three chest X-ray datasets (MS-CXR, RSNA Pneumonia, COVID Rural), DAP reports state-of-the-art weakly-supervised results without pixel annotations, including a 20.74% relative gain in contrast-to-noise ratio over the g++ baseline on RSNA. If the claim is right, cheap image-text supervision is enough to learn fine-grained disease localization, which is what makes the paper matter.

What carries the argument

The central object is the Disease-Aware Prompt $\Phi$, a per-patch importance map produced by a bi-modal explainability method applied to the frozen VLM, used to reweight visual token features and as the segmentation pseudo-label. It carries the argument because it converts the VLM's own text-conditioned attention into a training signal that directly targets the two diagnosed defects: multiplying $\Phi$ into the visual features suppresses high-norm background tokens and amplifies disease tokens, making the global [IMG] token more representative of disease regions. The three losses then redistribute the feature geometry so that disease patches, the global [IMG] token, and the text [CLS] token end up aligned, while the segmentation decoder is trained directly on the same map as pseudo-label.

What would settle it

Run DAP on a chest X-ray dataset in which the frozen VLM's explainability map has mean Dice with ground truth at or below 0.3; the paper's noise experiments predict DAP will match or fall below the g++ baseline there. A second check targets the mechanism: measure patch-token L2 norms after DAP training; if grounding improves while background norms remain high, the norm-dominance diagnosis is not what drives the gain.

Watch

Extended reading notes

Core claim

The central claim is that the poor visual grounding of medical VLMs is caused by two measurable representational defects, and that both can be corrected by prompting in feature space. The first defect is that the L2 norms of patch tokens are highest in non-disease background regions, so attention and similarity-based grounding are pulled toward healthy tissue. The second is that the global [IMG] token, trained only against the text [CLS] token by a global contrastive objective, is closer to background patch tokens than to disease patch tokens, making it an unreliable proxy for linking text to fine-grained disease regions. DAP computes a disease-aware prompt $\Phi$ from the VLM's own explainability map for the query text and multiplies it into the visual token features ($\hat{V}^l = \Phi \cdot V^l$), then trains with a disease-aware global contrastive loss, a foreground/background local contrastive loss, and Dice loss with $\Phi$ as pseudo-label. The paper reports that this restores intra-modal alignment—disease patches move closer to [IMG] and to the text—and produces crisp localization on MS-CXR, RSNA, and COVID Rural without dense annotations, improving CNR by 20.74% relative to g++ on RSNA.

Load-bearing premise

The method's gains rest on the frozen VLM's explainability map being a trustworthy indicator of where the disease is: that map is used both to reweight visual features and as the segmentation pseudo-label, and the paper's own noise experiments show DAP drops below g++ when the map's Dice with ground truth is at or below about 0.3 under heavy noise.

Editorial extensions

If this is right

  • Because DAP needs no pixel-level annotations, it can be trained on the large image-text corpora already used to pretrain medical VLMs, such as MIMIC-CXR's frontal-view studies (146.7k train samples).
  • Because the explainability map is used only during training, a trained DAP model can ground on the original image and text at inference without recomputing the map.
  • Measuring background token norms and global-to-local token similarity before training would reveal whether a candidate VLM is ready for weakly-supervised disease grounding.
  • On RSNA, DAP's weakly supervised results are comparable to the 20-shot fully supervised fine-tuning results of the baselines, meaning the prompted features serve as a strong starting point for minimal additional supervision.
  • The three-dataset benchmark gives future weakly supervised medical grounding methods a common evaluation across MS-CXR, RSNA, and COVID Rural.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The headline 20.74% figure is a relative gain on the contrast-to-noise ratio metric, a foreground-background separation score; the absolute Dice and IoU increases over baselines are smaller, so the practical impact depends on which quality measure matches the clinical use.
  • If the background-dominance and global-token misalignment are generic properties of contrastively trained VLMs, DAP's feature-space reweighting should transfer to other 'stuff-dominated' medical images such as CT slices, pathology whole slides, or ultrasound, where target findings occupy a small fraction of the field.
  • The paper's self-enhancement plots, where DAP improves on the explainability map's own Dice, suggest an iterative loop: a trained DAP model could generate sharper maps for another round of prompting, though the paper stops at a single pass.
  • The paper's noise experiment implies a practical pre-screening rule: on datasets where the frozen VLM's explainability map has Dice with ground truth at or below about 0.3, DAP should be expected to fall below a strong baseline, so map quality should be checked before adoption.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper identifies two obstacles to weakly-supervised medical visual grounding in current VLMs: background tokens carry high norms that divert attention, and the global [IMG] token is poorly aligned with local disease tokens. It proposes Disease-Aware Prompting (DAP), which uses the VLM's own explainability map to reweight visual features, trains with global and local contrastive losses plus a segmentation decoder supervised by the same map as a pseudo-label, and evaluates on MS-CXR, RSNA, and COVID Rural. Experiments report consistent gains over existing weakly-supervised methods in CNR, PG, Dice, and IoU, with ablation studies and robustness analyses in the appendix.

Significance. If the empirical claims hold, DAP is a simple and practical contribution: it avoids pixel-level annotations, repurposes the VLM's own explainability map as a training signal, and is accompanied by a useful diagnostic analysis of why medical VLMs underperform in visual grounding. The paper also provides a comparative benchmark across three chest X-ray datasets and includes a noise-robustness study of the critical map Phi, which is a genuine strength. However, the evaluation is currently compromised by a test-set tuning protocol, an unexplained numerical inconsistency in the headline metric, and a contradiction in the reported learning rate, so the empirical claims are not yet fully supported.

major comments (4)
  1. [Sec. 4 (Datasets) and Appendix B] Hyperparameters were tuned on a 20% subset of the MS-CXR evaluation set, which leaks test information. The paper states in Sec. 4 that 'Evaluation was done using MS-CXR' and Appendix B says 'We conduct the hyperparameters tuning on 20% of the MS-CXR dataset.' Since MS-CXR is one of the three evaluation datasets, the reported MS-CXR numbers in Table 1 are not obtained from a fully held-out test set. This undermines the fairness of the MS-CXR comparison, including the claimed 15.89% CNR improvement over Self-EQ. The authors should retrain with hyperparameters chosen without using MS-CXR, or clearly separate a validation split and report results on the untouched remainder.
  2. [Appendix B vs. Table 1] The appendix's best tuning result on the 20% MS-CXR subset is CNR 1.042, PG 0.449, Dice 0.350 (Tables 5-7), while Table 1 reports DAP on MS-CXR as CNR 1.254, PG 0.457, Dice 0.352. Dice and PG differ by only 0.002-0.008, but CNR differs by 0.212 (about 20% relative), and the paper offers no explanation. This inconsistency suggests that CNR may have been computed differently in the tuning and final evaluations, or that the final model uses additional unstated factors. The authors must reconcile these numbers and report the exact metric definitions used in both places.
  3. [Sec. 4 (Implementation Details) and Appendix B.1] There is a direct contradiction in the learning rate. The main text states 'We set up an Adam optimizer with a learning rate of 0.008 and a batch size of 512 across all experiments,' while Appendix B.1 reports that the optimal learning rate is 1e-3 and says 'We set batch size to 512 and learning rate to 1e−3 for other experiments.' If the main text is correct, the tuning description is wrong; if the appendix is correct, the main text misstates a key training hyperparameter. This must be fixed, and the reported results should be clearly tied to the actual configuration used.
  4. [Appendix E and Sec. 5] The method's dependence on the quality of the explainability map Phi is acknowledged only in the appendix: under 50% noise, where Dice(Phi, GT) is at or below 0.3, DAP falls below g++ (Fig. 15). Because Phi is used both as a feature reweighting term and as the pseudo-label for the segmentation decoder, this failure regime is directly relevant to the paper's central claim of superior performance. The main text should state this limitation explicitly and discuss under what conditions the method is expected to be useful, rather than presenting unconditional state-of-the-art results.
minor comments (6)
  1. [Abstract] The phrase 'improves visual grounding accuracy by 20.74%' is misleading: the 20.74% figure is a relative CNR improvement over g++ on the RSNA dataset, not a general accuracy improvement across all datasets. Please state the comparison explicitly.
  2. [Eq. (5)] The notation 'V L' and '[CLS]L' is undefined; the superscript L is used in Eq. (2) as a layer index, but the notation should be made consistent (e.g., V^L and [CLS]^L) and the layer at which the decoder operates should be specified.
  3. [Table 3] Table 3 compares prompt-tuning methods such as CoOp, CoCoOp, Visual prompt, and Multi-modal prompt, but the text does not explain how these methods are adapted for weakly-supervised visual grounding. A brief description of the adaptation is needed for the comparison to be meaningful.
  4. [Fig. 15] The caption refers to 'g++[44]' but the correct reference for g++ is [47]; please check all citation numbers in the appendix.
  5. [Sec. 6] The final sentence 'This paper constructs a comprehensive weakly-supervised medical VG on three datasets' is grammatically incomplete; it should be 'a comprehensive weakly-supervised medical VG benchmark on three datasets.'
  6. [Sec. 1] The paper calls the proposed method 'Disease-Aware Prompting' but the abstract uses 'Disease-Aware Prompting (DAP) process'; please use the acronym consistently throughout.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DAP is a pseudo-label bootstrap whose outputs are scored against external ground-truth masks, and the central gain does not reduce by construction to its own inputs.

full rationale

The derivation is self-contained. The only self-referential element is that the VLM's own explainability map Phi (Chefer et al.) is used both to reweight visual features (Eq. 3: V-hat^l = Phi * V^l) and as the pseudo-label for the segmentation loss, which is a standard bootstrapping design rather than a circular reduction. The final output is not defined to equal Phi: the visual grounding map is produced by a decoder, y-hat = D(CrossAttn(V^L, [CLS]^L)) (Eq. 5), and all headline evaluations in Tables 1-2 are against external ground-truth annotations (MS-CXR bounding boxes, RSNA pneumonia masks, COVID Rural masks). The paper explicitly reports that Phi alone achieves only 0.34/0.42/0.33 Dice on MS-CXR/RSNA/COVID, and Fig. 14 shows DAP improves on Phi for most samples, so the reported results are not Phi by construction. Appendix E further corrupts Phi and shows that DAP degrades monotonically with map quality, falling below g++ when Dice(Phi, GT) <= 0.3 at 50% noise, which is an empirically falsifiable dependence rather than a tautology. No load-bearing self-citations are present: the authors' prior MA-VL [44] appears only as related work and is not used to justify DAP. The evaluation-fairness concern that hyperparameters were tuned on 20% of the MS-CXR test set (Appendix B) could inflate Table 1's MS-CXR numbers, but this is a methodological issue, not a circularity reduction, because the selected hyperparameters do not by themselves define the predicted maps; per the hard rules it does not raise the circularity score.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the quality of the explainability map (a domain assumption), the transfer of prompted training to unprompted inference (another domain assumption), and several tuned hyperparameters. No new physical entities are introduced. The free parameters are numerous and some are not reported, which weakens reproducibility.

free parameters (5)
  • FG/BG token selection threshold tau = not reported
    Section 3.2 states 'We threshold the disease-aware prompting mask Phi to select FG tokens' but the threshold value is never given; it is presumably tuned on a validation subset.
  • Interpretability layer index l = last layer (Table 9)
    The layer at which the Chefer et al. explainability map is extracted is chosen empirically (Table 9); the paper does not specify how the map is applied across layers.
  • Loss weights for L_glb and L_lcl = 1 and 0.1 (Table 7)
    Loss weights are tuned on 20% of MS-CXR (Appendix B.2).
  • Batch size and learning rate = 512, 1e-3
    Tuned on 20% of MS-CXR (Appendix B.1).
  • Number of text prompts per disease = 50
    Tuned; Table 10 shows increasing prompts helps; 50 chosen. Tuning was on a subset of MS-CXR.
assumptions (4)
  • domain assumption The L2 norm of a patch token indicates its activation strength and its importance for attention or grounding.
    Section 3.1 uses norm maps as evidence of background dominance and as a causal mechanism for mis-grounding; this is an interpretive assumption about ViT token norms.
  • domain assumption The explainability map Phi from Chefer et al. is a reliable proxy for disease foreground locations.
    Section 3.2, Eq. (2)-(3), Phi is used to reweight features and to pseudo-label the segmentation decoder; Appendix E shows performance degrades when Phi is noisy.
  • domain assumption Training with disease-prompted features transfers disease-awareness to the unprompted encoder at inference.
    Section 3.2 inference paragraph states the map is only needed at training; the weight-shared encoder must generalize the prompted alignment to unprompted inputs.
  • standard math Standard properties of CLIP-style contrastive losses (gradients, negatives) hold as assumed.
    Equations (6)-(7) use standard cross-modal and local contrastive losses.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Seeing the Trees for the Forest: Rethinking Weakly-Supervised Medical Visual Grounding." pith.science (2026). https://pith.science/paper/SHBFCN7V

@misc{pith2026250515123,
  author       = {Pith},
  title        = {Pith review of: Seeing the Trees for the Forest: Rethinking Weakly-Supervised Medical Visual Grounding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SHBFCN7V}},
  note         = {Machine review of arXiv:2505.15123}
}
read the original abstract

Visual grounding (VG) is the capability to identify the specific regions in an image associated with a particular text description. In medical imaging, VG enhances interpretability by highlighting relevant pathological features corresponding to textual descriptions, improving model transparency and trustworthiness for wider adoption of deep learning models in clinical practice. Current models struggle to associate textual descriptions with disease regions due to inefficient attention mechanisms and a lack of fine-grained token representations. In this paper, we empirically demonstrate two key observations. First, current VLMs assign high norms to background tokens, diverting the model's attention from regions of disease. Second, the global tokens used for cross-modal learning are not representative of local disease tokens. This hampers identifying correlations between the text and disease tokens. To address this, we introduce simple, yet effective Disease-Aware Prompting (DAP) process, which uses the explainability map of a VLM to identify the appropriate image features. This simple strategy amplifies disease-relevant regions while suppressing background interference. Without any additional pixel-level annotations, DAP improves visual grounding accuracy by 20.74% compared to state-of-the-art methods across three major chest X-ray datasets.

Figures

Figures reproduced from arXiv: 2505.15123 by the authors.

Figure 1
Figure 1. Left: Average cosine similarity between disease patch to￾kens and the corresponding disease texts of CheXzero [53] on MS￾CXR [6] dataset. Without disease-aware prompting, the current VLM shows undifferentiated alignment scores between disease and non-disease patches with text. Right: Local feature norms, and visual grounding of CheXzero [53], Self-EQ [23], and our disease-aware prompted model. gions enhances interpr… view at source ↗
Figure 2
Figure 2. In current VLMs, text [CLS] (•) are pulled close to a global image token [IMG] (•) through contrastive loss, and thus they use the global token as a proxy to link with disease patch to￾kens (•). Left: w/o prompt, the global image token and the disease patch of the original input on the left show weak alignment, while our proposed disease-aware prompting enhances contrast for dis￾ease regions, aligning disease patche… view at source ↗
Figure 3
Figure 3. Performance analysis of BioVIL [ [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (13 more)
Figure 5
Figure 5. Figure 5: Dice score (%) between (i) Norm map and the back [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 4
Figure 4. Figure 4: Comparative analysis of token norm distribution (Norm) [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 7
Figure 7. Figure 7: Average cosine similarity between the global im [PITH_FULL_IMAGE:figures/full_fig_p004_7.png]
Figure 8
Figure 8. Figure 8: Feature space visualization for six common chest conditions without (Top) and with our disease-focused prompting(Bottom). [PITH_FULL_IMAGE:figures/full_fig_p005_8.png]
Figure 9
Figure 9. Figure 9: Model architecture. We use the disease-aware prompt to align the global image token [IMG] and [CLS] for the Disease-aware global contrastive loss, and to select FG/BG patch tokens for the Local Contrastive Loss. During inference, only the original image patch token and…
Figure 10
Figure 10. Figure 10: Local contrastive loss. The FG and BG tokens are [PITH_FULL_IMAGE:figures/full_fig_p006_10.png]
Figure 11
Figure 11. Figure 11: Qualitative results of recent VLMs, and weakly-supervised VG methods on MS-CXR[ [PITH_FULL_IMAGE:figures/full_fig_p007_11.png]
Figure 12
Figure 12. Figure 12: Token norm and VG analysis of prompt tuning meth [PITH_FULL_IMAGE:figures/full_fig_p008_12.png]
Figure 13
Figure 13. Figure 13: Results of interpretability map, our DAP, and others. [PITH_FULL_IMAGE:figures/full_fig_p012_13.png]
Figure 14
Figure 14. Figure 14: (Left) plots the Dice score of our model against Φ on RSNA. Most points lie above the red line, showing our model imprives Dice upon the interpretability map, indicat￾ing self-enhancement [PITH_FULL_IMAGE:figures/full_fig_p012_14.png]
Figure 15
Figure 15. Figure 15: Dice score (left) and CNR (right) of DAP and Φ against the ground truth of RSNA dataset under different noise levels, compared to the strong baseline g++[44]. 20-shot fine-tuning using ground truth dense labels. As shown in Table. 11, our proposed DAP achieves results…
Figure 16
Figure 16. Figure 16: Qualitative results of DAP on MS-CXR [6] dataset [PITH_FULL_IMAGE:figures/full_fig_p015_16.png]
Figure 17
Figure 17. Figure 17: More qualitative results of DAP on MS-CXR [ [PITH_FULL_IMAGE:figures/full_fig_p016_17.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

70 extracted references · 55 canonical work pages

  1. [1]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736,

  2. [2]

    Detector-free weakly supervised grounding by separation

    Assaf Arbelle, Sivan Doveh, Amit Alfassy, Joseph Shtok, Guy Lev, Eli Schwartz, Hilde Kuehne, Hila Barak Levi, Prasanna Sattigeri, Rameswar Panda, et al. Detector-free weakly supervised grounding by separation. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion (ICCV), pages 1801–1812, 2021. 2, 3

    work page 2021

  3. [3]

    Exploring visual prompts for adapting large- scale models

    Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large- scale models. arXiv preprint arXiv:2203.17274, 2022. 3

    arXiv 2022

  4. [4]

    Learning to exploit temporal structure for biomedical vision-language processing

    Shruthi Bannur, Stephanie Hyland, Qianchu Liu, Fernando Perez-Garcia, Maximilian Ilse, Daniel C Castro, Benedikt Boecking, Harshita Sharma, Kenza Bouzid, Anja Thieme, et al. Learning to exploit temporal structure for biomedical vision-language processing. In CVPR, pages 15016–15027,

    work page

  5. [5]

    Im- proving pneumonia localization via cross-attention on med- ical images and reports

    Riddhish Bhalodia, Ali Hatamizadeh, Leo Tam, Ziyue Xu, Xiaosong Wang, Evrim Turkbey, and Daguang Xu. Im- proving pneumonia localization via cross-attention on med- ical images and reports. In Medical Image Computing and Computer Assisted Intervention (MICCAI) , pages 571–581. Springer, 2021. 1

    work page 2021

  6. [6]

    Making the most of text semantics to improve biomedical vision–language processing

    Benedikt Boecking, Naoto Usuyama, Shruthi Bannur, Daniel C Castro, Anton Schwaighofer, Stephanie Hyland, Maria Wetscherek, Tristan Naumann, Aditya Nori, Javier Alvarez-Valle, et al. Making the most of text semantics to improve biomedical vision–language processing. In ECCV, pages 1–21. Springer, 2022. 1, 2, 3, 4, 6, 7, 8, 12, 15, 16

    work page 2022

  7. [7]

    Coco- stuff: Thing and stuff classes in context

    Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco- stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pages 1209–1218, 2018. 2

    work page 2018

  8. [8]

    Generic attention- model explainability for interpreting bi-modal and encoder- decoder transformers

    Hila Chefer, Shir Gur, and Lior Wolf. Generic attention- model explainability for interpreting bi-modal and encoder- decoder transformers. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision, pages 397–406,

Show all 70 references
  1. [9]

    Knowledge aided consistency for weakly supervised phrase grounding

    Kan Chen, Jiyang Gao, and Ram Nevatia. Knowledge aided consistency for weakly supervised phrase grounding. InPro- ceedings of the IEEE conference on computer vision and pat- tern recognition, pages 4042–4050, 2018. 1

    2018

  2. [10]

    Querymatch: A query-based contrastive learning framework for weakly supervised visual grounding

    Shengxin Chen, Gen Luo, Yiyi Zhou, Xiaoshuai Sun, Guan- nan Jiang, and Rongrong Ji. Querymatch: A query-based contrastive learning framework for weakly supervised visual grounding. In Proceedings of the ACM International Con- ference on Multimedia, pages 4177–4186, 2024. 2

    2024

  3. [11]

    Medical phrase ground- ing with region-phrase context contrastive alignment

    Zhihao Chen, Yang Zhou, Anh Tran, Junting Zhao, Liang Wan, Gideon Su Kai Ooi, Lionel Tim-Ee Cheng, Choon Hua Thng, Xinxing Xu, Yong Liu, et al. Medical phrase ground- ing with region-phrase context contrastive alignment. In In- ternational Conference on Medical Image Computing...

  4. [12]

    Chexagent: Towards a foun- dation model for chest x-ray interpretation

    Zhihong Chen, Maya Varma, Jean-Benoit Delbrouck, Mag- dalini Paschali, Louis Blankemeier, Dave Van Veen, Jeya Maria Jose Valanarasu, Alaa Youssef, Joseph Paul Cohen, Eduardo Pontes Reis, et al. Chexagent: Towards a foun- dation model for chest x-ray interpretation. arXiv prepr...

    2024 arXiv

  5. [13]

    Randaugment: Practical automated data augmen- tation with a reduced search space

    Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmen- tation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 702–703, 2020. 6

    2020

  6. [14]

    Align2ground: Weakly supervised phrase grounding guided by image-caption align- ment

    Samyak Datta, Karan Sikka, Anirban Roy, Karuna Ahuja, Devi Parikh, and Ajay Divakaran. Align2ground: Weakly supervised phrase grounding guided by image-caption align- ment. In Proceedings of the IEEE/CVF international confer- ence on computer vision, pages 2601–2610, 2019. 2

    2019

  7. [15]

    Transvg: End-to-end visual ground- ing with transformers

    Jiajun Deng, Zhengyuan Yang, Tianlang Chen, Wengang Zhou, and Houqiang Li. Transvg: End-to-end visual ground- ing with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1769– 1779, 2021. 1

    2021

  8. [16]

    Chest imaging representing a covid-19 positive rural us population

    Shivang Desai, Ahmad Baghal, Thidathip Wongsurawat, Piroon Jenjaroenpun, Thomas Powell, Shaymaa Al-Shukri, Kim Gates, Phillip Farmer, Michael Rutherford, Geri Blake, et al. Chest imaging representing a covid-19 positive rural us population. Scientific Data, 7(1):414, 2020. 6, 7, 8

    2020

  9. [17]

    Adam: A method for stochastic opti- mization

    P Kingma Diederik. Adam: A method for stochastic opti- mization. (No Title), 2014. 6

    2014

  10. [18]

    Neural se- quential phrase grounding (seqground)

    Pelin Dogan, Leonid Sigal, and Markus Gross. Neural se- quential phrase grounding (seqground). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4175–4184, 2019. 1

    2019

  11. [19]

    Learning to prompt for open-vocabulary ob- ject detection with vision-language model

    Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open-vocabulary ob- ject detection with vision-language model. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14084–14093, 2022. 3

    2022

  12. [20]

    Medrax: Medical reasoning agent for chest x-ray, 2025

    Adibvafa Fallahpour, Jun Ma, Alif Munim, Hongwei Lyu, and Bo Wang. Medrax: Medical reasoning agent for chest x-ray, 2025. 1

    2025

  13. [21]

    Contrastive learning for weakly supervised phrase grounding

    Tanmay Gupta, Arash Vahdat, Gal Chechik, Xiaodong Yang, Jan Kautz, and Derek Hoiem. Contrastive learning for weakly supervised phrase grounding. In European Confer- ence on Computer Vision , pages 752–768. Springer, 2020. 2

    2020

  14. [22]

    Parameter-efficient fine-tuning medical multimodal large language models for medical visual grounding

    Jinlong He, Pengfei Li, Gang Liu, and Shenjun Zhong. Parameter-efficient fine-tuning medical multimodal large language models for medical visual grounding. arXiv preprint arXiv:2410.23822, 2024. 1

    2024 arXiv

  15. [23]

    Improved visual ground- ing through self-consistent explanations

    Ruozhen He, Paola Cascante-Bonilla, Ziyan Yang, Alexan- der C Berg, and Vicente Ordonez. Improved visual ground- ing through self-consistent explanations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13095–13105, 2024. 1, 3, 4, 6, 7, 8

    2024

  16. [24]

    Multi-grained attention with object-level ground- ing for visual question answering

    Pingping Huang, Jianhui Huang, Yuqing Guo, Min Qiao, and Yong Zhu. Multi-grained attention with object-level ground- ing for visual question answering. InProceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics, pages 3595–3600, 2019. 1

    2019

  17. [25]

    Verjans, Minh-Son To, and Vu Minh Hieu Phan

    Ta Duc Huy, Sen Kim Tran, Phan Nguyen, Nguyen Hoang Tran, Tran Bao Sam, Anton van den Hengel, Zhibin Liao, Johan W. Verjans, Minh-Son To, and Vu Minh Hieu Phan. Interactive medical image analysis with concept-based sim- ilarity reasoning. In Proceedings of the IEEE/CVF Confer-...

    2025

  18. [26]

    Visual grounding of whole radiology reports for 3d ct images

    Akimichi Ichinose, Taro Hatsutani, Keigo Nakamura, Yoshiro Kitamura, Satoshi Iizuka, Edgar Simo-Serra, Shoji Kido, and Noriyuki Tomiyama. Visual grounding of whole radiology reports for 3d ct images. In International Confer- ence on Medical Image Computing and Computer-Assiste...

    2023

  19. [27]

    Rad- graph: Extracting clinical entities and relations from radiol- ogy reports

    Saahil Jain, Ashwin Agrawal, Adriel Saporta, Steven QH Truong, Du Nguyen Duong, Tan Bui, Pierre Chambon, Yuhao Zhang, Matthew P Lungren, Andrew Y Ng, et al. Rad- graph: Extracting clinical entities and relations from radiol- ogy reports. In NeurIPS, 2021. 3

    2021

  20. [28]

    Scaling up visual and vision-language representa- tion learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representa- tion learning with noisy text supervision. In International Conference on Machine Learning, pages 4904–4916. PMLR,

  21. [29]

    Vi- sual prompt tuning

    Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Vi- sual prompt tuning. In European Conference on Computer Vision, pages 709–727. Springer, 2022. 3, 8

    2022

  22. [30]

    Pseudo-q: Generating pseudo language queries for visual grounding

    Haojun Jiang, Yuanze Lin, Dongchen Han, Shiji Song, and Gao Huang. Pseudo-q: Generating pseudo language queries for visual grounding. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 15513–15523, 2022. 1, 6

    2022

  23. [31]

    Refclip: A universal teacher for weakly supervised referring expression comprehension

    Lei Jin, Gen Luo, Yiyi Zhou, Xiaoshuai Sun, Guannan Jiang, Annan Shu, and Rongrong Ji. Refclip: A universal teacher for weakly supervised referring expression comprehension. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 2681–2690...

    2023

  24. [32]

    Mimic-cxr, a de- identified publicly available database of chest radiographs with free-text reports

    Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Roger G Mark, and Steven Horng. Mimic-cxr, a de- identified publicly available database of chest radiographs with free-text reports. Scientific Data, 6(1):317, 2019. 6

    2019

  25. [33]

    Deep visual-semantic align- ments for generating image descriptions

    Andrej Karpathy and Li Fei-Fei. Deep visual-semantic align- ments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pages 3128–3137, 2015. 1

    2015

  26. [34]

    Maple: Multi-modal prompt learning

    Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19113–19122, 2023. 8

    2023

  27. [35]

    Covlm: Compos- ing visual entities and relationships in large language models via communicative decoding

    Junyan Li, Delin Chen, Yining Hong, Zhenfang Chen, Pei- hao Chen, Yikang Shen, and Chuang Gan. Covlm: Compos- ing visual entities and relationships in large language models via communicative decoding. In ICLR, 2024. 1

    2024

  28. [36]

    Visual prompt tuning for weakly supervised phrase grounding

    Pengyue Lin, Zhihan Yu, Mingcong Lu, Fangxiang Feng, Ruifan Li, and Xiaojie Wang. Visual prompt tuning for weakly supervised phrase grounding. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7895–7899. IEEE, 2024. 2,...

    2024

  29. [37]

    Adaptive recon- struction network for weakly supervised referring expression grounding

    Xuejing Liu, Liang Li, Shuhui Wang, Zheng-Jun Zha, Dechao Meng, and Qingming Huang. Adaptive recon- struction network for weakly supervised referring expression grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2611–2620, 2019. 1

    2019

  30. [38]

    Knowledge-guided pairwise recon- struction network for weakly supervised referring expression grounding

    Xuejing Liu, Liang Li, Shuhui Wang, Zheng-Jun Zha, Li Su, and Qingming Huang. Knowledge-guided pairwise recon- struction network for weakly supervised referring expression grounding. In Proceedings of the 27th ACM International Conference on Multimedia, pages 539–547, 2019

    2019

  31. [39]

    Relation- aware instance refinement for weakly supervised visual grounding

    Yongfei Liu, Bo Wan, Lin Ma, and Xuming He. Relation- aware instance refinement for weakly supervised visual grounding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5612–5621,

  32. [40]

    Confidence-aware pseudo-label learning for weakly super- vised visual grounding

    Yang Liu, Jiahua Zhang, Qingchao Chen, and Yuxin Peng. Confidence-aware pseudo-label learning for weakly super- vised visual grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2828– 2838, 2023. 1, 2

    2023

  33. [41]

    Vividmed: Vision language model with versatile visual grounding for medicine

    Lingxiao Luo, Bingda Tang, Xuanzhong Chen, Rong Han, and Ting Chen. Vividmed: Vision language model with versatile visual grounding for medicine. arXiv preprint arXiv:2410.12694, 2024. 1

    2024 arXiv

  34. [42]

    Localizing be- fore answering: A hallucination evaluation benchmark for grounded medical multimodal llms

    Dung Nguyen, Minh Khoi Ho, Huy Ta, Thanh Tam Nguyen, Qi Chen, Kumar Rav, Quy Duong Dang, Satwik Ramchan- dre, Son Lam Phung, Zhibin Liao, et al. Localizing be- fore answering: A hallucination evaluation benchmark for grounded medical multimodal llms. arXiv e-prints , pages arX...

    2025

  35. [43]

    Gpt-4 technical report, 2023

    OpenAI. Gpt-4 technical report, 2023. 6, 13

    2023

  36. [44]

    Decomposing disease de- scriptions for enhanced pathology detection: A multi-aspect vision-language pre-training framework

    Vu Minh Hieu Phan, Yutong Xie, Yuankai Qi, Lingqiao Liu, Liyang Liu, Bowen Zhang, Zhibin Liao, Qi Wu, Minh- Son To, and Johan W Verjans. Decomposing disease de- scriptions for enhanced pathology detection: A multi-aspect vision-language pre-training framework. In Proceedings o...

    2024

  37. [45]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In International Conference on Machine Learning...

    2021

  38. [46]

    Grounding of textual phrases in images by reconstruction

    Anna Rohrbach, Marcus Rohrbach, Ronghang Hu, Trevor Darrell, and Bernt Schiele. Grounding of textual phrases in images by reconstruction. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part I 14 , pages 817–

    2016

  39. [47]

    Similarity maps for self- training weakly-supervised phrase grounding

    Tal Shaharabany and Lior Wolf. Similarity maps for self- training weakly-supervised phrase grounding. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6925–6934, 2023. 2, 3, 6, 7, 8, 14

    2023

  40. [48]

    What is where by looking: Weakly-supervised open-world phrase- grounding without text inputs

    Tal Shaharabany, Yoad Tewel, and Lior Wolf. What is where by looking: Weakly-supervised open-world phrase- grounding without text inputs. Advances in Neural Informa- tion Processing Systems, 35:28222–28237, 2022. 2, 3, 6, 7, 8, 14

    2022

  41. [49]

    Augmenting the national institutes of health chest radiograph dataset with expert annotations of possi- ble pneumonia

    George Shih, Carol C Wu, Safwan S Halabi, Marc D Kohli, Luciano M Prevedello, Tessa S Cook, Arjun Sharma, Judith K Amorosa, Veronica Arteaga, Maya Galperin- Aizenberg, et al. Augmenting the national institutes of health chest radiograph dataset with expert annotations of possi...

    2019

  42. [50]

    Interactive visual grounding of referring expressions for human-robot interaction

    Mohit Shridhar and David Hsu. Interactive visual grounding of referring expressions for human-robot interaction. arXiv preprint arXiv:1806.03831, 2018. 1

    2018 arXiv

  43. [51]

    What does clip know about a red circle? vi- sual prompt engineering for vlms

    Aleksandar Shtedritski, Christian Rupprecht, and Andrea Vedaldi. What does clip know about a red circle? vi- sual prompt engineering for vlms. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 11987–11997, 2023. 8

    2023

  44. [52]

    Discriminative triad matching and reconstruc- tion for weakly referring expression grounding

    Mingjie Sun, Jimin Xiao, Eng Gee Lim, Si Liu, and John Y Goulermas. Discriminative triad matching and reconstruc- tion for weakly referring expression grounding. IEEE trans- actions on pattern analysis and machine intelligence, 43(11): 4189–4195, 2021. 1

    2021

  45. [53]

    Expert-level detection of pathologies from unannotated chest x-ray images via self- supervised learning

    Ekin Tiu, Ellie Talius, Pujan Patel, Curtis P Langlotz, An- drew Y Ng, and Pranav Rajpurkar. Expert-level detection of pathologies from unannotated chest x-ray images via self- supervised learning. Nature Biomedical Engineering, 6(12): 1399–1406, 2022. 1, 2, 3, 4, 6, 7, 8

    2022

  46. [54]

    Few-shot visual grounding for natural human-robot interaction

    Giorgos Tziafas and Hamidreza Kasaei. Few-shot visual grounding for natural human-robot interaction. In2021 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC), pages 50–55. IEEE, 2021. 1

    2021

  47. [55]

    Found a reason for me? weakly-supervised grounded visual question answering us- ing capsules

    Aisha Urooj, Hilde Kuehne, Kevin Duarte, Chuang Gan, Niels Lobo, and Mubarak Shah. Found a reason for me? weakly-supervised grounded visual question answering us- ing capsules. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8465– 8...

    2021

  48. [56]

    A framework for interpretability in machine learn- ing for medical imaging

    Alan Q Wang, Batuhan K Karaman, Heejong Kim, Ja- cob Rosenthal, Rachit Saluja, Sean I Young, and Mert R Sabuncu. A framework for interpretability in machine learn- ing for medical imaging. IEEE Access, 2024. 1

    2024

  49. [57]

    Phrase localization without paired training examples

    Josiah Wang and Lucia Specia. Phrase localization without paired training examples. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4663– 4672, 2019. 2

    2019

  50. [58]

    Improving weakly supervised visual ground- ing by contrastive knowledge distillation

    Liwei Wang, Jing Huang, Yin Li, Kun Xu, Zhengyuan Yang, and Dong Yu. Improving weakly supervised visual ground- ing by contrastive knowledge distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14090–14100, 2021. 2

    2021

  51. [59]

    Learning visual grounding from generative vi- sion and language model

    Shijie Wang, Dahun Kim, Ali Taalimi, Chen Sun, and We- icheng Kuo. Learning visual grounding from generative vi- sion and language model. arXiv preprint arXiv:2407.14563,

    arXiv

  52. [60]

    Medclip: Contrastive learning from unpaired medical images and text

    Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun. Medclip: Contrastive learning from unpaired medical images and text. 2022. 3

    2022

  53. [61]

    MedKLIP: Medical knowledge enhanced language-image pre-training

    Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. MedKLIP: Medical knowledge enhanced language-image pre-training. In ICCV, 2023. 3

    2023

  54. [62]

    Rethinking masked image modelling for medical image representation

    Yutong Xie, Lin Gu, Tatsuya Harada, Jianpeng Zhang, Yong Xia, and Qi Wu. Rethinking masked image modelling for medical image representation. Medical Image Analysis, 98: 103304, 2024. 3

    2024

  55. [63]

    A simple baseline for zero- shot semantic segmentation with pre-trained vision-language model

    Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, Xiang Bai, et al. A simple baseline for zero- shot semantic segmentation with pre-trained vision-language model. arXiv preprint arXiv:2112.14757, 3:2, 2021. 3, 6, 8

    2021 arXiv

  56. [64]

    Florence: A new foundation model for computer vision

    Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432, 2021. 3

    2021 arXiv

  57. [65]

    Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs

    Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, et al. Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs. arXiv preprint arXiv:2303.00915,

    arXiv

  58. [66]

    Conditional prompt learning for vision-language mod- els

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language mod- els. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition , pages 16816–16825,

  59. [67]

    Learning to prompt for vision-language models

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. In- ternational Journal of Computer Vision, 130(9):2337–2348,

  60. [68]

    Scanreason: Empowering 3d visual grounding with reasoning capabilities

    Chenming Zhu, Tai Wang, Wenwei Zhang, Kai Chen, and Xihui Liu. Scanreason: Empowering 3d visual grounding with reasoning capabilities. In European Conference on Computer Vision, pages 151–168. Springer, 2024. 1

    2024

  61. [69]

    Read before grounding: Scene knowledge visual ground- ing via multi-step parsing

    HaiXiang Zhu, Lixian Su, ShuangMing Mao, and Jing Ye. Read before grounding: Scene knowledge visual ground- ing via multi-step parsing. In Proceedings of the 31st In- ternational Conference on Computational Linguistics, pages 1136–1149, 2025. 1 A. Self-enhancement with DAP The...

    2025

  62. [2021]

    2, 3, 5, 6, 7, 8, 12, 13

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.